Installation failed on Linux HPC nodes via Slurm and pip about deepwave

[ar4]

Hi Muhammad, Do you know which version of PyTorch is installed on the cluster? I ask because it looks like the error might be due to PyTorch also using "DIM" as a symbol (Deepwave uses it to separate the code for 1D, 2D, and 3D propagation). I haven't had that problem before, so perhaps this is a change in the latest version of PyTorch (a new version was released a few days ago, but I haven't tested it yet). The best solution in that case would probably be for Deepwave to use a different symbol. You say that you tried to install it via Slurm. Do I correctly interpret that you mean your Slurm job installs Deepwave on each node? If so, that may not be necessary - if you can access a shared directory from each node (such as perhaps your home directory), then I think it should probably be sufficient to install Deepwave once in that shared directory from the head node and then access that installed version on each node. You might need to set the PYTHONPATH environment variable in your Slurm job to include the path to that installed version so that each node can find it.

[izzatum]

Hi Alan,

Thanks for the swift response. I set up a new environment with miniconda and install the latest version of PyTorch as on their website, version 1.10 with cudatoolkit 11.3.

For Slurm, I already did as you suggested so it is okay now. Just the deepwave still can't be installed.

[ar4]

Hi Muhammad, It is probably as I suspected that this is a change in the latest version of PyTorch. I will update my local version to debug it. Hopefully I just need to change the symbol name from DIM to something else.

[izzatum]

Alright. Hopefully, it will work. Thanks a lot, Alan!

…

On Mon, Oct 25, 2021 at 11:09 PM Alan Richardson ***@***.***> wrote: Hi Muhammad, It is probably as I suspected that this is a change in the latest version of PyTorch. I will update my local version to debug it. Hopefully I just need to change the symbol name from DIM to something else. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/ar4/deepwave/issues/37*issuecomment-951268685__;Iw!!Nmw4Hv0!goqL_XudfRM9wuzOxfGbs8nbnBBQJ57wCQ_HDlZhB2GFLCDG1b40sI92c_qZd1osjejB6de09SKx$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALH7GBDC4UU6X3AWVWXEWGLUIW2O3ANCNFSM5GWBO5OQ__;!!Nmw4Hv0!goqL_XudfRM9wuzOxfGbs8nbnBBQJ57wCQ_HDlZhB2GFLCDG1b40sI92c_qZd1osjejB6e0FGSnr$> . Triage notifications on the go with GitHub Mobile for iOS <https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!Nmw4Hv0!goqL_XudfRM9wuzOxfGbs8nbnBBQJ57wCQ_HDlZhB2GFLCDG1b40sI92c_qZd1osjejB6fRqofzx$> or Android <https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!Nmw4Hv0!goqL_XudfRM9wuzOxfGbs8nbnBBQJ57wCQ_HDlZhB2GFLCDG1b40sI92c_qZd1osjejB6cH-V3r9$>.

-- *Muhammad Izzatullah* King Abdullah University of Science and Technology (KAUST) Division of Physical Sciences and Engineering Bldg. 1, Desk #0202-WS023 Thuwal 23955-6900, Kingdom of Saudi Arabia *Tel*.: [phone] *Email*: ***@***.*** ***@***.***> *Seismic Modeling & Inversion (SMI) Research Group* https://smi.kaust.edu.sa/

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[ar4]

Changing the symbol name was sufficient for it to compile successfully, but it is now failing some of the tests. I will need to investigate this further tomorrow. In the meantime, if you wish to workaround the problem, I suggest installing the previous version of PyTorch.

[izzatum]

Thanks a lot, Alan! Since deepwave runs smoothly on Google Colab, I will try using the same version of PyTorch as Google Colab has for the moment.

…

On Mon, Oct 25, 2021 at 11:50 PM Alan Richardson ***@***.***> wrote: Changing the symbol name was sufficient for it to compile successfully, but it is now failing some of the tests. I will need to investigate this further tomorrow. In the meantime, if you wish to workaround the problem, I suggest installing the previous version of PyTorch. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/ar4/deepwave/issues/37*issuecomment-951319369__;Iw!!Nmw4Hv0!l0Uypvxc80qiVPkXKgpJKhC1WEil-WPo66iGHB0_WYwl2mvX99EOrVUTJxiZV4uDvXXEHsy4fe1p$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALH7GBECQSSSLCKQDHMOP23UIW7JJANCNFSM5GWBO5OQ__;!!Nmw4Hv0!l0Uypvxc80qiVPkXKgpJKhC1WEil-WPo66iGHB0_WYwl2mvX99EOrVUTJxiZV4uDvXXEHnB2jNas$> . Triage notifications on the go with GitHub Mobile for iOS <https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!Nmw4Hv0!l0Uypvxc80qiVPkXKgpJKhC1WEil-WPo66iGHB0_WYwl2mvX99EOrVUTJxiZV4uDvXXEHtknUJZ0$> or Android <https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!Nmw4Hv0!l0Uypvxc80qiVPkXKgpJKhC1WEil-WPo66iGHB0_WYwl2mvX99EOrVUTJxiZV4uDvXXEHl5RB1E4$>.

-- *Muhammad Izzatullah* King Abdullah University of Science and Technology (KAUST) Division of Physical Sciences and Engineering Bldg. 1, Desk #0202-WS023 Thuwal 23955-6900, Kingdom of Saudi Arabia *Tel*.: [phone] *Email*: ***@***.*** ***@***.***> *Seismic Modeling & Inversion (SMI) Research Group* https://smi.kaust.edu.sa/

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[ar4]

It should hopefully work now with the new v0.0.9 release of Deepwave. Thank you for reporting the problem!

[izzatum]

Hi Alan, Thanks a lot!

…

On Tue, Oct 26, 2021 at 1:13 PM Alan Richardson ***@***.***> wrote: It should hopefully work now with the new v0.0.9 release of Deepwave. Thank you for reporting the problem! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/ar4/deepwave/issues/37*issuecomment-951789161__;Iw!!Nmw4Hv0!k5pb72lGZ2xrViEFXwrnJIewN3hG4BC35gXRnnGN5bdt0eoKErJ2DxHMzmJkj1ZENAn1bJJMl_Ak$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALH7GBEXFGRDRFVEMBLNUMTUIZ5NDANCNFSM5GWBO5OQ__;!!Nmw4Hv0!k5pb72lGZ2xrViEFXwrnJIewN3hG4BC35gXRnnGN5bdt0eoKErJ2DxHMzmJkj1ZENAn1bM8oH5IO$> . Triage notifications on the go with GitHub Mobile for iOS <https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!Nmw4Hv0!k5pb72lGZ2xrViEFXwrnJIewN3hG4BC35gXRnnGN5bdt0eoKErJ2DxHMzmJkj1ZENAn1bNiJdRb5$> or Android <https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!Nmw4Hv0!k5pb72lGZ2xrViEFXwrnJIewN3hG4BC35gXRnnGN5bdt0eoKErJ2DxHMzmJkj1ZENAn1bD5l62Mg$>.

-- *Muhammad Izzatullah* King Abdullah University of Science and Technology (KAUST) Division of Physical Sciences and Engineering Bldg. 1, Desk #0202-WS023 Thuwal 23955-6900, Kingdom of Saudi Arabia *Tel*.: [phone] *Email*: ***@***.*** ***@***.***> *Seismic Modeling & Inversion (SMI) Research Group* https://smi.kaust.edu.sa/

-- This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

[pplotn]

Hello everyone,
Installation of gpu version of deepwave on ibex is possible. There is my experience on link below.
https://github.com/pplotn/deepwave/blob/master/installation_gpu_cluster

[ar4]

Hello Pavel,

Thank you for sharing your experience. I am glad that the process looks pretty clean and easy.

I am currently working on the next release of Deepwave. It is a complete re-write, but I hope that your installation script will still work.

[pplotn]

As for me the problem was that deepwave does not see some libraries like nvcc in conda provided cudatoolkit. So thats why using properly installed cuda with module load helps the deepwave installer to find necessary destinations.

[ar4]

The new 0.0.10 version of Deepwave that I mentioned has been released. I hope that installation might cause problems for slightly fewer users, but we shall have to wait and see. I'll close this issue for now, but please feel free to reopen it or create a new one if you encounter problems.