llama_index_sentence_splitter issues about ragatouille

[bclavie]

Hey, could you provide more of your code? It looks like the issue here is that the documents that makes it to the preprocessor is None, so would be helpful to figure out how that happened!

[manisnesan]

@bclavie - I faced a similar issue when I ran the notebook 06-index_free_use.ipynb from examples.

Tried to create a reproducer in the code using Colab and faced

ValidationError: 1 validation error for Document
text
  none is not an allowed value (type=type_error.none.not_allowed)

The root cause is due to one of the page being empty and hence ragatouille is throwing the error as "ValidationError" which is the right behavior.

The user need to ensure passing only valid docs before passing to Corpus_Processor.process_corpus method. This issue is not a bug and can be closed.