[BUG] `KeyError: 'reference_database_name'` when running summarize about hamronization

[fmaguire]

Updated in bioconda now: bioconda/bioconda-recipes#37140

[fmaguire]

Great! Thanks for your patience!

It should already be automatically updating on pypi, dockerhub, and (pending the attentiveness of bioconda bot) updated on bioconda at some point today.

[fmaguire]

(although it does seem the badges on the README aren't updating for some reason...)

[fmaguire]

Ah damn, sorry, I thought I'd solved that issue/covered it with tests. I thought the concatenation should be adding those fields but I'll try initialising the empty combined dataframe earlier.

Does the same error occur if you run hamronize summarize on just WAL001-megahit.mapping.potential.ARG.deeparg.json or only with the big list of jsons?

If the former could you send me just that output file (to [email protected]) and if the latter the big ole zip?

[jfy133]

Yes, only WAL001-megahit, but strangely it seems to happen in all cases, e.g. VLC009-metaspades.mapping.potential.ARG.deeparg.json which does have hits.

I'll send you the zip and you can test everything. Happy to also test any dev versions!

[fmaguire]

This seems to work now but please test in your workflow. Instead of trying to add additional columns if needed post-concatenation, I now just initialise an empty dataframe with all the headers in summarize before concatenating.

One question to make sure I haven't failed to fix another issue: These input jsons to summarize weren't cached and not regenerated with the hamronization v1.1.0 right? hamronization should now be generating valid empty jsons (i.e., just files containing []) when parsing empty tool reports but I see these files still have the ] malformation.

[jfy133]

1. OK! I will test this :)
2. Ah yes correct sorry, the JSONs in the ZIP were still from 1.0.3 - It took a few days for the pipeline to run, so didn't want to run the whole thing again with 1.1.0 to find the same/different summarize issue 😅 . I can try to take a few and re-generate them with 1.1.0 to double check now though

[jfy133]

@fmaguire I can confirm 761fe77 fixes the bug, and that re-running e.g. harmonizate deeparg on an 'empty' outputfile produces the correct empty JSON of [].

Once this version is released on bioconda (I sped that up for 1.1.0 this morning btw 😬 ), I will update our nf-core nextflow module and re-run the full pipeline again and let you know how well it performs.

This issue can be closed now!

[jfy133]

oh and thanks for the quick turnaround :D