duplicated entities in response about cosmos

[cambro]

[davenquinn]

Also found on this page, along with figures apparently grouped with the wrong caption.

[iross]

This isn't a bug in retrieval or the API. It's a user error in ingesting the data. The bottom line is that I accidentally put a small set of PDFs into the queue twice. We don't do duplicate checking on ingest, so I ended up doubling some of the results. Part of the issue is that I was doing separate runs on small piles of documents instead of one big run on the full pile (as the committed code would do it).

The foreign keys don't cascade at the moment and making that change + yanking out the documents is too big a change for the live set at the moment.

We could also add pdf name (or pdf checksum) checks to make sure we don't ingest the same document more than once.

[cambro]

@iross the DOI/GDD id should definitely be carried along all the way though every single thing that any of our applications do!

I know that this is being done, but just a reminder!