The datasets `conditioned/gsim_XXX` are using too much disk space about oq-engine

[micheles]

Actually the only solution is to reduce the number of sites, since the memory/disk space occupation is quadratic with the number of sites.

[raoanirudh]

Why doesn't storing them in the .tmp.hdf5 work? This is data needed only during the calculation and doesn't need to be stored in the final calc.hdf5

[micheles]

Because you will soon run out of disk space, this is how Cata discovered the issue. Also, once you start storing 100+GB then reading the data will kill your calculation (out of memory or so slow to be impossible to run). No matter how big is your machine, a quadratic calculation will run out of resources pretty soon. You would need an algorithm not quadratic with the number of sites.

[raoanirudh]

Opening this issue again as it still persists.

The issue is not related to having too many sites in the calculation. It was that the conditioned/mean_covs data that is now stored in the calc_xxx.hdf5 file is useful only while the calculation is running, and can safely be deleted from the datastore once the calculation is completed. Or the other option might be to store it in the calc_xxx.tmp.hdf5 file instead, which gets deleted at the end of the calculation, since this interim data is not useful to the user after the calculation is over. If the conditioned/mean_covs data is deleted from the datastore, the hdf5 file sizes in oqdata should go back to the regular sizes for scenario calculations that do not involve conditioning.

[micheles]

You are partially right @raoanirudh , but my point still stand that calculations with too many points will be impossible. The only solution I see for Aristotle calculations is to use a large enough region_grid_spacing so that calculations can run. Then, to avoid wasting too much disk space we can store the temporary data in _tmp.hdf5 or even better only keep it in memory as it was originally, before #9094 (retrospectively, it was a bad idea, trading a decent but not impressive speedup for too much disk space).