conda is slow about ramp-ua

[cheginit]

The main reason that I use conda is the need for some hard-to-install-from-pip packages such as rasterio. Many of the geospatial packages that I use depend on gdal which its installation is nightmare when it comes to cross-platform development. All packages on conda-forge are pre-compiled so installing them is straight forward from a UX perspective. I agree that poetry provides automation and more control over dependency management and might be a better choice for cases with dependencies that are easy to install on all platforms.

BTW, There's another package that I use for generating meta.yml files called grayskull. You can use the run section of the generated meta.yml as a starting point for creating environment.yml.

[nickmalleson]

Welcome @dabreegster ! Yes I think that would be welcome, you're not alone in having trouble setting up the environment initially with conda. I'm also keen to test whether it will work with docker, but haven't gotten round to it (my main concern is whether the opencl parts of the code will be able to access the hardware properly).

[dabreegster]

dabreegster@7e075a7 is working enough to run the initialization step, but this still needs more work to fully replace Conda.

This Poetry change is also targeting the EcoTwins branch, so maybe one of the first things I need to sort out is where to focus my efforts. There are lots of branches in this repo and seemingly a few different groups working independently. Is there any intention to get everybody working off one upstream branch, or has everyone's work diverged a fair bit by this point?

I'll also mention that I've been experimenting with rewriting the initialization half of RAMP in Rust: https://github.com/dabreegster/rampfs/. https://github.com/dabreegster/rampfs/blob/main/writeup.md has some initial results (incredible performance gains) and an argument for why static type systems could be helpful for a project of this complexity.

my main concern is whether the opencl parts of the code will be able to access the hardware properly

This is a good concern. I spotted python-poetry/poetry#105 (comment), which seems to imply there may be some issue with Poetry and packages that have system dependencies -- although I haven't had any issues with bindings to glfw, imgui, geopandas, etc yet.

[dabreegster]

my main concern is whether the opencl parts of the code will be able to access the hardware properly

Good news, no problems at all! I just successfully ran the OpenCL model with my Intel GPU using poetry run python main_model.py -p model_parameters/default.yml.

I'll continue with cutting over Ecotwins to poetry.

[cheginit]

I just stumbled upon this issue and thought I share my 2 cents! I've had great success with using mamba. It's a drop-in replacement for conda and is very fast! I use Mambaforge which install packages from conda-forge by default so no more -c conda-forge --strict-channel-priority!

[dabreegster]

I've had great success with using mamba

Thanks for the pointer! It certainly seems like a strictly better option than conda, but unless it produces lockfiles by default, there's still the issue of dependencies changing for different people based on when they first create the environment. Any reason to prefer conda/mamba over poetry, which doesn't appear to have any performance problems at all and creates lockfiles?

Another reason to be suspicious of the conda setup -- in this repo, we're importing requests and sklearn, but those dependencies aren't declared in environment.yml. They just happen to be transitive dependencies of some other package, so it works. This doesn't work in poetry -- I had to explicitly declare the dependency, which seems desirable.

[dabreegster]

Cleaning up this old issue. For future reference, we got poetry working just fine for this project: https://github.com/dabreegster/uatk-aspics/blob/main/pyproject.toml