jina-ai / jina-commons Goto Github PK

Currently, if we set needs_attr="text" on get_docs_batch_generator the function will yield docs with doc.text=="".
It might be expected that these docs get filtered because the empty string is also the default argument for the text attribute on the Document object.

This means

docs = DocumentArray([Document(blob=...)])
for batch in get_docs_batch_generator(docs, needs_attr='text')):
    print(len(batch))
>>> 1

will yield the doc.

Currently, it is difficult to tell whether there is something hung up or just due to a large dumping task.

We can add it to https://github.com/jina-ai/jina-commons/blob/main/jina_commons/indexers/dump.py#L69

Refactor the functionality for document batching. This is currently used in the following executors and can be implemented once and imported from jina_commons instead.

It roughly looks like this:

def _batch_generator(data: List[Any], batch_size: int):
    for i in range(0, len(data), batch_size):
        yield data[i: i + batch_size]

def _get_docs_batch_generator(self, docs: DocumentArray, parameters: Dict):
    traversal_path = parameters.get('traversal_path', self.default_traversal_path)
    batch_size = parameters.get('batch_size', self.default_batch_size)

    flat_docs = docs.traverse_flat(traversal_path)

    filtered_docs = [doc for doc in flat_docs if doc is not None and doc.blob is not None]

    return _batch_generator(filtered_docs, batch_size)

Name | Repo URL | PR

• Executor Transformer Torch Encoder | https://github.com/jina-ai/executor-transformer-torch-encoder | jina-ai/executor-transformer-torch-encoder#5
• TFIDF Encoder | https://github.com/jina-ai/executor-text-tfidfencoder | jina-ai/executor-text-tfidfencoder#5
• Paddle | https://github.com/jina-ai/executor-text-paddle | jina-ai/executor-text-paddle#8
• Image Custom Torch Encoder | https://github.com/jina-ai/executor-image-custom-torch-encoder | jina-ai/executor-image-custom-torch-encoder#4
• Text Clip | https://github.com/jina-ai/executor-text-clip-text-encoder | jina-ai/executor-text-clip-text-encoder#4
• Big Transfer | https://github.com/jina-ai/executor-big-transfer-encoder | jina-ai/executor-big-transfer-encoder#5
• Video Encoder | https://github.com/jina-ai/executor-video-torch-encoder | jina-ai/executor-video-torch-encoder#8
• Sentence Transformer | https://github.com/jina-ai/executor-sentence-transformer | jina-ai/executor-sentence-transformer#3
• Object Segmenter | https://github.com/jina-ai/executor-image-torch-object-detection-segmenter | jina-ai/executor-image-torch-object-detection-segmenter#4
• Flair Text | https://github.com/jina-ai/executor-text-flair-encoder | jina-ai/executor-text-flair-encoder#5
• Laser encoder | https://github.com/jina-ai/executor-text-laser-encoder | jina-ai/executor-text-laser-encoder#5
• Image paddle | https://github.com/jina-ai/executor-image-paddle-encoder | jina-ai/executor-image-paddle-encoder#5
• Image tf | https://github.com/jina-ai/executor-image-tf-encoder | jina-ai/executor-image-tf-encoder#4
• clip image | https://github.com/jina-ai/executor-clip-image | jina-ai/executor-clip-image#24

When the embedding is missing, this will break the whole program

https://github.com/jina-ai/jina-commons/blob/main/jina_commons/indexers/dump.py#L102