I'm getting an OOM error that allegedly says not enough memory can be found for 1.19 GB. I'm running SLURM jobs with ~80GB.
How can I investigate the cause? Is it possible that previous layers' activations are consuming memory? If so, is there some flag or some mechanism to free that memory?
Traceback (most recent call last):
File "scripts/compute_eigenspectra_and_fit_encoding_model.py", line 63, in <module>
activations_extractor=model,
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regression_dimensionality/custom_model_tools/eigenspectrum.py", line 44, in fit
image_transform_name=transform_name)
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regdim_venv/lib/python3.7/site-packages/result_caching/__init__.py", line 223, in wrapper
result = function(**reduced_call_args)
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regression_dimensionality/custom_model_tools/eigenspectrum.py", line 141, in _fit
activations = self._extractor(image_paths, layers=[layer])
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regdim_venv/lib/python3.7/site-packages/model_tools/activations/pytorch.py", line 41, in __call__
return self._extractor(*args, **kwargs)
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regdim_venv/lib/python3.7/site-packages/model_tools/activations/core.py", line 43, in __call__
return self.from_paths(stimuli_paths=stimuli, layers=layers, stimuli_identifier=stimuli_identifier)
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regdim_venv/lib/python3.7/site-packages/model_tools/activations/core.py", line 73, in from_paths
activations = fnc(layers=layers, stimuli_paths=reduced_paths)
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regdim_venv/lib/python3.7/site-packages/model_tools/activations/core.py", line 85, in _from_paths
layer_activations = self._get_activations_batched(stimuli_paths, layers=layers, batch_size=self._batch_size)
File "/home/gridsan/rschaeffer/FieteLab-Reg-Eff-Dim/regdim_venv/lib/python3.7/site-packages/model_tools/activations/core.py", line 141, in _get_activations_batched
layer_activations[layer_name] = np.concatenate((layer_activations[layer_name], layer_output))
File "<__array_function__ internals>", line 6, in concatenate
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.19 GiB for an array with shape (1216, 256, 32, 32) and data type float32