Missing documents on Vespa Cloud about pyvespa

[thigm85]

You need to specify id, e.g.:

batch_feed = [
    {
        "id": idx,
        "fields": sentence
    }
    for idx, sentence in enumerate(sentence_data)
]

Maybe you have duplicate id`s?

[neo-anderson]

That's it! Generated ids were not unique and resulted in documents getting overwritten in the index. Thanks for the pointer.

[thigm85]

There are likely schema issues with the documents that are failing. You can inspect the results and see the error messages, for ex.:

results = app.feed_batch()
failed_docs = [x for x in results if x.status_code != 200]
failed_docs[0].json # inspect the error message

[neo-anderson]

I tried on a more powerful machine.

In [9]: failed_docs = [x for x in response if x.status_code != 200]
In [10]: len(failed_docs)
Out[10]: 0

No failed feed this time.
The number of documents in the data store is still 735.1k instead of 1M+.

For context,

# command used to ingest data
response = app.feed_batch(schema="coso", batch=data, batch_size=1000, total_timeout=200, asynchronous=True, connections=100)
len(data)
#1093460

I tried ingesting data into prod and see the same number of documents in the prod store as well - 735.1k.

What else can I do to check what went wrong?