Comments (1)
When torch creates a parallel dataloader (num_workers > 1) it will create some new R processes using callr and then copy the dataset you passed on into each one of those processes. It will then run .getitem()
in each of theses precesses.
Problems can arise when copying dataset into those processes, for example:
- if the dataset contains torch_tensors as attributes. torch tensors are not serializable using saveRDS() thus it's hard to reliably move them between process. The alternative in this case is to not have any dataset attribute that is a tensor.
- the dataset has very large attributes. if your dataset has very large attributes, theywill be copied into each process potentially using a lot of memory.
- other kinds of objects that are not copiable using saveRDS(). eg connections, XML objects, anything that is a pointer.
Here's a small example running the mnist dataset in parallel:
library(torch)
library(torchvision)
dir <- "~/Downloads/mnist2"
train_ds <- mnist_dataset(
dir,
download = TRUE,
transform = transform_to_tensor
)
train_dl <- dataloader(train_ds, batch_size = 128, shuffle = TRUE, num_workers = 4)
d <- coro::collect(train_dl)
from torch.
Related Issues (20)
- Cuda 11.6 deprecated HOT 2
- torch fails on new Mac M3 architecture HOT 13
- Release torch 0.13.0
- `torch_triu_indices` and `torch_tril_indices` return 0-based index
- first and second order derivative of gamma samples HOT 1
- Unable to load pre-trained alexnet weights HOT 2
- Can't download lantern from https://torch-cdn.mlverse.org when install torch 0.13.0 in win11 HOT 4
- Number of dimensions of a tensor increases when indexing a dimension of length 1 with a one-dimension logical vector HOT 5
- Adding Regularization Term in NLS optimization
- `set_num_threads` on Linux does not seem to work
- Question about installation from pre-built binaries HOT 4
- Indexing a tensor by an R matrix should behave as if the matrix is a tensor
- Cannot install lantern dependencies on aarm64 architecture HOT 6
- GPU runtime + memory leakage HOT 1
- Backward pass fails on torch_max with the "inplace operation" error HOT 1
- GLIBCXX version error after installing 0.13.0 with CUDA support
- dataloader issure when use parallel HOT 3
- training in epoch ,error in sizes of tensors must match except in dimension1.expected size 1 but got size 28 for tensor number 2 in the list
- `torch_iinfo()` is failing for dtypes `torch_uint8()` and `torch_int8()`
- R session Aborted HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from torch.