Reasonable range of MBs for a .xyz file to train? about quip

[gabor1]

In MB I don't know but we regularly train databases with several hundred thousand atoms. Note that with MPI training you can use lots of memory on multiple nodes (or are you already doing that and that's what 1-2 TB refers to?)

[MES-physics]

Just one node , not MPI.

[gabor1]

So if you switch to multiple nodes you can significantly increase database sizes and reduce training times.

[MES-physics]

Thanks. How do I find how many atoms are in my xyz files?

[jameskermode]

You could load it into ASE and sum the lengths of all the Atoms objects, with something like (not tested)

from ase.io import read
dataset = read("file.xyz", index=":")
total_atoms = sum([len(atoms) for atoms in dataset])

or if you have only a single species eg Si then grep -Ec ‘^Si' file.xyz would suffice (this counts the number of lines starting with Si).

[gabor1]

If you have many elements, you can also grep -c 'Lattice' file.xyz which will count the number of structures, and subtract twice this number from the result of wc file.xyz to get the overall number of atoms.

[Sideboard]

You can also sum up the atom numbers at the start of each frame/configuration:
grep '^[0-9]\+$' data.xyz | awk '{s+=$1} END {print s}'

Number of frames accordingly:
grep '^[0-9]\+$' data.xyz | wc -l