A Question About the Map Function about datasets

[Modexus]

All data is saved in the arrow format on disk.
If you return a tensor it gets converted to arrow before saving to disk when using map.

To get a tensor when you access data elements you can use dataset.set_format("pt").
Note that this just changes how the data is loaded, not how it is stored.

[YangsongLan]

All data is saved in the arrow format on disk. If you return a tensor it gets converted to arrow before saving to disk when using map.

To get a tensor when you access data elements you can use dataset.set_format("pt"). Note that this just changes how the data is loaded, not how it is stored.

Thank you very much for your explanation, I understand what you mean now. So you're saying that when streaming=True, there's no need to convert it to the arrow format and save it to disk. But if we directly load all formats and then convert them into the arrow format after passing through the map function, it will convert torch.Tensor into a List. I see.