optimize `Tree.from_list()` about dvc-data

[skshetry]

@dberenbaum, Tree.load() is used to .dir file from cache.
Tree.from_list is a bit generic in that it can also load from files section in .dvc files.

Tree loading creates 2 objects: Meta, HashInfo, and this above profiling is for CelebA where we build 500k of those instances. You can see above that it takes 1.5s just to load that in memory.

Of course, we are not doing anything bad here. It's just Python being slow, we have already optimized this in terms of memory and performance. I opened this issue for exploring other ways to improve this.

This affects a lot of commands. Eg: data status has to load all .dir files in memory times 2 (from Git HEAD and from the hash in .dvc file). So a lot of time is lost just loading this in memory.

[skshetry]

I played with Cython (which required 0 code changes), and got 20-30% improvement. But it will require a lot of infrastructure changes. It has to be 5x-10x improvement to be worth it here.

[dberenbaum]

@skshetry Do you have a more "high-level" use case where you can show how much of an impact it has?