nim-lang / atlas

The Atlas Package cloner. It manages an isolated workspace that contains projects and dependencies.

atlas

The Atlas Package cloner. It manages an isolated workspace that contains projects and dependencies.

Installation

Upcoming Nim version 2.0 will ship with atlas. Building from source:

git clone https://github.com/nim-lang/atlas.git
cd atlas
nim c src/atlas.nim
# copy src/atlas[.exe] somewhere in your PATH

Documentation

Read the full documentation or go through the following tutorial.

Tutorial

Create a new workspace. A workspace contains everything we need and can safely be deleted after this tutorial:

mkdir workspace
cd workspace
atlas init

Create a new project inside the workspace:

mkdir myproject
cd myproject

Tell Atlas we want to use the "malebolgia" library:

atlas use malebolgia

Now import malebolgia in your Nim code and run the compiler as usual:

echo "import malebolgia" >myproject.nim
nim c myproject.nim

Using URLs and local folders

atlas use https://github.com/zedeus/nitter
atlas use file://../../existingDepdency/

Debugging

Sometimes it's helpful to understand what Atlas is doing. You can run commands with: atlas --verbosity:<trace|debug> to get more information.

Installing Nim with Atlas

atlas env 2.0.0
source $WORKSPACE/nim-2.0.0/activate.sh

Vendoring with Atlas

Atlas also supports vendoring using an "inverted workspace". The project layout is where the workspace is a top-level subfolder like vendor/ or deps/ in your project. Like this:

someProject/vendor/atlas.workspace
someProject/vendor/dep1/
...

This is especially helpful for working with projects that have dependencies pinned as git submodules, which was common in the pre-Atlas era.

atlas's Issues

Installing Atlas #head with Nimble is broken

nimble install https://github.com/nim-lang/atlas@\#head

/tmp/nimble_1894/githubcom_nimlangatlas_#head/src/lockfiles.nim(11, 12) Warning: imported and not used: 'sha1' [UnusedImport]
  Success:  atlas installed successfully.
       Tip: 7 messages have been suppressed, use --verbose to show them.
packageparser.nim(395)   readPackageInfo
    Error:  Could not read package info file in /home/runner/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble;
        ...   Reading as ini file failed with: 
        ...     Invalid section: .
        ...   Evaluating as NimScript file failed with: 
        ...     /home/runner/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble(2, 12) Error: cannot open file: src/pkgversion

Locally I get:

    Error:  The package inside "/Users/jaremycreechley/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2" is invalid.
  Details:  Could not read package info file in /Users/jaremycreechley/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble;
        ...   Reading as ini file failed with: 
        ...     Invalid section: .
        ...   Evaluating as NimScript file failed with: 
        ...     /Users/jaremycreechley/.nimble/pkgs2/atlas-0.6.3-bcaac15af2ecbfb9b183dd18cc643840f9df89a2/atlas.nimble(2, 12) Error: cannot open file: src/pkgversio

Relax package name restrictions

Atlas errors out on a project I'm trying to include: stack_string. Since stack_strings is a valid module it should be supported.

atlas use does not resolve all dependencies

Running atlas use norm in a workspace does not fetch db_connectors required by Norm's dependencies lowdb.

In order to get norm to compile it is necessary to install db_connectors manually.

Abstract the fetching mechanism

The fetcher is the component that does the following:

• first rewrite the URL
• then depending on the URL, use git or hg or nimgrab or curl or wget ...

Use "shallow" git cloning when pulling dependencies

Consider using git clone --depth=1 for pulling dependencies.

This:

• Saves a non-negligible amount of space dropping what in most cases amounts to dead weight.
• Speeds-up the cloning process.

For further possible interacting with the pulled repos there's git fetch --unshallow.

Re-add namespace support

Notes for re-adding support for namespaces.

It seems the proc's pkgUrlToDirname and extractProjectName both need updating and maybe refactoring. Also, the GitHub search will need re-adding if it's not present. It would be good to review the package resolution in version 0.6.3 to ensure all the edge cases are handled.

Also, it'll likely be a good time to fix up the on disk directory paths to support relative paths from the workspace.

It'd be ideal to write unit tests for this before setting up integration tests.

Readme example does not work.

atlas use malebolgia
[Info] (.) is the current workspace
[Info] (malebolgia) could not resolve by name or repo; searching GitHub
[Error] (malebolgia) package not found by github search
[Warning] (malebolgia) cannot resolve package name

I expected malebolgia to be found. I expected the readme to have a package that is easy to find.

Possible interest support for cross compiling?

I've spent way more time than I care to think about trying to figure out a generic cross compiling scheme. conan and conda sorta offer this but not very well.

All of that said, Nim already has decent cross compiling support but it's just not very centralized and requires each dev to sorta re-invent things. It also depends on each dev to figure out how to get and install cross compilers.

It'd be amazing if atlas could provide a way to provide a sequence like "install cross-xc compilers then use them to build" sequence. From what little I've seen of Atlas this wouldn't seem to impractical too add. Especially if it could be added as just another builder / extractor with possibly a bit of sugar on the nim.cfg side. Perhaps another concept of a "provider" could be added too, e.g. recipe for building and installing a specific Nim compiler or non-Nim compile tools.

I'm creating this issue to find out if others would be interested in as well. It's something I'd be keen on adding if it'd be accepted. Though it'd have to be non-invasive and not complicate the simple build path, otherwise I think it'd be going down the wrong path.

Atlas use should work with SSH for private repository

It would be nice to be able to do "atlas use [email protected]:..." on private packages

`atlas pin` doesn't work when the workspace is in the current project

Found this bug updating one of my earlier projects with a lockfile. Looks like instead of using the URL, atlas for some reason falls back to the local dir.

I plan to add some integration tests for lockfiles as I don't believe we have any currently. Or if we do, there needs to be more.

Expected lockfile:

{
  "items": {
    "stew": {
      "dir": "$deps/nim-stew",
      "url": "https://github.com/status-im/nim-stew",
      "commit": "7afe7e3c070758cac1f628e4330109f3ef6fc853",
      "version": "0.1.0"
    },
    ...
}

But getting:

  "items": {
    "stint": {
      "dir": "$deps/nim-stint",
      "url": "file:///Users/jaremycreechley/projs/status/nim-codex-dht/vendor/nim-stint",
      "commit": "86621eced1dcfb5e25903019ebcfc76ed9128ec5",
      "version": "0.0.1"
    },
    ...
}

`atlas use project2` does not resolve to a `project2` existing in the workspace

I'm trying to make atlas understand local development packages that already exist in the workspace.

I don't want to use URL, the point of the folder structure is to not need url at all
But im having a hard time making atlas understand this structure

The documentation reads:

A workspace contains one or multiple "projects". These projects can use each other and it is easy to develop multiple projects at the same time.
https://github.com/nim-lang/atlas/blob/master/doc/atlas.md#projects

But I'm not quite getting how that is going to happen if atlas use mypkg is not finding the folder right next to itself, which exists inside the workspace structure already.

I know about atlas use /path/to/package.... but that is adding requires "file://path/to/package" to the nimble file, and that is incompatible syntax with nimble, which makes nimble completely break and not able to run at all.
(I may not want to use nimble myself, but I cannot force that choice on everybody, specially with atlas being in such early dev state. So I don't want to make my nimble files unusable just because I chose atlas)

The main problem is that I don't want to have my dependencies in a subfolder. I have to maintain a set of modules like this:

And the list is only growing. Which is why I'm turning to atlas, in the hopes that I can finally use nimble develop like I used to do without issues, but since that feature is very broken in Nimble, I'm hoping that atlas can solve the problem.

I have even tried nim.cfg with both nimble and raw nim c and no nimble at all... and they both fail to resolve the sub-dependencies. Which is extremely frustrating, because it means I have to reupload the package, remove it from local, redownload my own package, and repeat.... for every single line I change, which is just ridiculous.

The main goal is being able to keep my current folder structure of dependencies all stored at the root of the workspace, and make each dependency understand that its dependencies exist in the folder right next to them.
How can I approach this with atlas?

Impossible version range fails silently

Using non-existing version or impossible version constraint doesn't raise an error :

If I have a project with a "myproject.nimble" file containing the following :
requires "arraymancer >= 0.10.0 & < 0.7.20"

And I call "atlas use myproject.nimble", atlas doesn't fail when it is impossible to resolve the version.

a project like baz.cpp is cloned into baz

Need to distinguish between atlas and nimble when running the install hooks

Due to nimble moving the src path, and atlas having the opposite behavior, install hooks not created for atlas in mind will fail.

I can't figure out how to `atlas use` a specific version

I've tried the following:

atlas use argparse==4.0.1
atlas use argparse=4.0.1
atlas use [email protected]
atlas use 'argparse == 4.0.1'
atlas use 'argparse = 4.0.1'

How do tell it to use a particular version?

#20 breaks cloning from url

Reverting this commit fixes the issue. Error log:

 atlas clone https://github.com/planetis-m/raylib-game-template
[Info] (.) is the current workspace
Cloning into 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning intp 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning into 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning into 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
Cloning 'https://github.com/planetis-m/raylib-game-template'...
fatal: protocol '/home/antonisg/code/wnaylib/https' is not supported
[Error] (https://github.com/planetis-m/raylib-game-template) exernal program failed: git clone --recursive https://github.com/planetis-m/raylib-game-template https://github.com/planetis-m/raylib-game-template

Needs CI

• Needs CI.
• Nim needs to ship Atlas just like it ships Nimble, we should offer comparable koch.nim commands.
• Also the documentation should be part of Nim.

Available version detection must consider the contents of the .nimble file

And not just the available git tags.

Needs hg/mercurial support

Regression between atlas in Nim 2.0.2 and Nim 2.0.8 : old hash commit with previous nimble formatting generate no nim.cfg file or empty nim.cfg file

I upgraded from 2.0.2 to 2.0.8 and notice a severe regression : atlas use no longer generate a correct nim.cfg files.

Atlas does not special case a `nim` dependency anymore

• Start with empty workspace.
• Run atlas use ormin and watch Atlas do some stuff with bootstrapping Nim.

No longer respects `--deps`

Presently the following results in micros being installed in the workspace directory instead of .deps.

mkdir workspace
cd workspace
atlas init  --deps=".deps"
atlas use micros

Looking at the atlas.workspace file deps is not populated:

{
  "deps": "",
  "overrides": "",
  "plugins": "",
  "resolver": "SemVer",
  "graph": null
}

Even if one populates deps:".deps" using atlas use micros still installs it in the workspace root.

[request | question] Ability to connect atlas to the root toolchain of the system

I've tried setting up atlas for a project, but I don't know if I like that there is no way (that I know of) to connect an atlas workspace to the root toolchain folder of the host system (or any other arbitrary folder).

I understand if this is by design, and it does make a lot of sense if this is the case.

What I was thinking is having the option to (explicitly, manually) setup paths where atlas can search for dependencies, so that the same libraries don't end up duplicated 10 times because of having 10 different workspaces.

My idea would be:

1. Setup alternative folder to search for (explicitly)
2. `atlas use mydepend`
2.1 atlas searches for the dependency in the alternative folders, if they are any
---> does not redownload the library, just uses the ones in the other folder
---> choose by prioritizing the last folder found, like nim.cfg does (or first, or whatever other rule)
2.2 no alternatives found. atlas continues with the current behavior where the libraries are duplicated once for each workspace

That would be my feature request if this is not possible.
Or just my question if the feature exists and I haven't found it (sorry if that's the case).

Atlas "develop", add local paths to nim.cfg

Atlas should have another option,
to just put local paths in the nim.cfg.

This way we could developing a module and use it directly in the consuming application.

For example it could be:

atlas develop /path/to/module 

This would then add the path to the module to the nim.cfg.
It could also add this to the nimble file?

Improve output format of dependencies graph

Currently, it is possible to output the dependencies with a graph representation using GraphViz.
See: https://github.com/nim-lang/atlas/blob/master/src/atlas.nim#L446

« Ideally Atlas would output a graph representation that is not only useful for GraphViz. Maybe in JSON and then some external tool would render that in the browser. » Araq from this forum post.

Implement better "vendor" style workspace setup

Myself and several others have used the "vendoring" style where a sub-directory is used as the Atlas workspace. This is a very common pattern for many language package managers like Npm or Elixir's mix and each project can have it's own versions checked out. Golang also uses a vendor directory for project specific deps pinning in addition to it's global GOPATH workspace.

For many projects it's convenient to setup the main parent repo as a Nim project that also contains the configs to install dependencies for that project rather than needing to create an explicit workspace. Often this is most useful for folks who want to clone a repo, install the deps locally and try out a project. This is pretty big for people who see a Nim project and just want to "try it out". Though this can be handy for a user who has a workspace but doesn't want to pollute it with some project's dependencies before they check it out. Lastly, it also enables a smooth transition path for Status project which use the nimbus-build-system that uses git submodules in a vendor folder.

The previous implementation of this in Atlas was very hacky and searched directy subdirectories in the current folder for workspaces in normal workspaces weren't found. That's problematic for lots of reasons and caused issues with normal atlas workspaces containing other workspaces. However, there should be a couple of solutions to enabling a vendoring style setup with Atlas properly.

Here's a couple of feature goals:

• make the vendored workspace location explicitly configured
  • possibly a standard predefined location like Go's ./vendor folder
• make it easy for folks to use projects with a vendored workspace with a standard workspace locally without changing the repo's configs
  • possibly always make vendored workspaces "secondary", as if a parent workspace is detected always prefer it

Here's a few approaches that have come up in a previous PR and I've thought of trying:

1. an atlas.config project config file which could specify a default vendored workspace subdirectory in addition to other per-project configs
2. explicit atlas.vendor file that just contains the path to a vendor folder (similar to some of git's setups)
3. an option in a atlas.workspace configuration that specifies a vendored workspace
4. just train users to use atlas --workspace=./vendor xyz
5. modification of previous where a flag like -l uses a pre-defined folder like vendor as the workspace, similar to the atlas -g support for global

I'm partial towards either option 1 or option 5 of having an atlas.config or a -l local option, respectively. They're both explicit.

Having a workspace config in the current project seems sort of confusing with Atlas's standard usage. The atlas.vendor link file are sort of annoying in Nimble and Git.

Cleanup command line options & documentation

Many commands like use and pin and rep can work on a Workspace or an individual project. We should offer project.use and workspace.use as aliases that enforce a respective interpretation. If the qualified command syntax is not used then if the current working directory is a project the command is mapped to project.command and else if the working directory is a workspace it is mapped to workspace.command otherwise it is an error as it's not clear what was meant.

`atlas use` should work even when not in a project

Nimble parsing error

Nimble parser needs to handle this properly:

/Users/jaremycreechley/.atlas/nim-testutils/testutils.nimble(10, 17) Error: assignments to 'srcDir' must be string literals

Caused by # in:

bin           = @["ntu"]
#srcDir        = "testutils"

Provide a section in the readme on how to use atlas to install Nim itself.

This comment:

Choosenim is the best way to have the latest Nim version around while still being able to jump to devel versions or a local copy.

Is it though? Atlas can do the same already.

This comment seems to imply that you can use atlas as a choosenim replacement. There is anything about this in the README? How does it work? What commands does it need to run?

Custom extractors

The parser that process project.nimble is what I call an extractor. Custom extractors allow to extend this mechanism to cover other package managers, including ones for different programming languages. An extractor can be written in any programming language you like.

The entry in the configuration would look somewhat like:

"extractor": {"pattern": "requirements.txt", "language": "python", "parser": "<url to extension>", "resolver": "SemVer"}

Can't seem to add local package

✝ ~/code/timbrel ✝  atlas use file://../saintnim/
[Info] (/Users/saint/code/timbrel) is the current workspace
[Info] (/Users/saint/code/timbrel/timbrel.nimble) updated
[Info] (timbrel) installing dependencies for timbrel.nimble
[Info] (print) updated package to a9df499a441d997ab9c8658b014e84c561f5528a
/Users/saint/code/atlas/src/atlas.nim(583) atlas
/Users/saint/code/atlas/src/atlas.nim(576) main
/Users/saint/code/atlas/src/atlas.nim(474) main
/Users/saint/code/atlas/src/atlas.nim(222) installDependencies
/Users/saint/code/atlas/src/atlas.nim(197) traverseLoop
/Users/saint/code/atlas/src/depgraphs.nim(564) getCfgPath
/Users/saint/.choosenim/toolchains/nim-2.0.2/lib/system/fatal.nim(53) sysFatal
Error: unhandled exception: index out of bounds, the container is empty [IndexDefect]

I can't seem to add it this way at all. I'm able to get it to work by manually editing the nim.cfg and adding --path:"/Users/saint/code/saintnim/src"

Any idea how to fix this? Thanks!

updating of packages must respect versioning

It must be reworked in general, the existing code is not clean enough.

Add support for custom generators

We currently have 2 generators:

1. The subsystem that patches the nim.cfg file.
2. The subsystem that writes to project.nimble

This mechanism could be extended so that it can generate other files like a requirements.txt. Ideally it also supports to setup/edit environment variables or the command that is used for a "builder" (see the ticket about builders).

Refactor AtlasContext into it's own file?

Would it make sense to move AtlasContext into it's own module like context.nim?

It'd let the main atlas.nim be broken up into more self contained modules -- something like a cloning.nim module for handling that functionality. I think it'd help making unit tests as well and to use logging more places.

Restore handling conflicting or overriden package url's using `repo.user.host` format

The update to the dependency graph broke this.

This breaks any projects using Atlas where users want to change to a different git repo for a dependency for testing or waiting for PR's to get merged.

`atlas.workspace` config uses absolute file paths, breaks portability

Workspace configuration saved to atlas.workspace uses absolute file paths which breaks portability/dual boot usage.

Possible solution: save the ondisk property of the workspace as a relative path to the config file, save the ondisk property of the packages as relative paths to the workspace directory.

add support for namespaces

If there are two projects of the same name but of different origin (i.e coming from different URLs), namespace the projects. This means to turn the project name into $project.$author.$github.

Latest atlas cannot be installed via nimble anymore

This happend using nim v2.0.0.

✘-INT Clonkk ~$ nimble install https://github.com/nim-lang/atlas
Downloading https://github.com/nim-lang/atlas using git
  Warning:  No nimblemeta.json file found in /home/rcaillaud/.nimble/pkgs2/nimlsp-0.4.4-b624dd472b343ca6dcb5e825bbd47f7d1388f077
  Verifying dependencies for atlas@0.8.0
 Installing atlas@0.8.0
   Building atlas/atlas using c backend
/tmp/nimble_128084/githubcom_nimlangatlas/src/packagesjson.nim(10, 8) Warning: imported and not used: 'context' [UnusedImport]
/tmp/nimble_128084/githubcom_nimlangatlas/src/sat.nim(559, 3) Warning: unreachable else, all cases are already covered [UnreachableElse]
/tmp/nimble_128084/githubcom_nimlangatlas/src/nimblechecksums.nim(9, 12) Warning: use command `nimble install checksums` and import `checksums/sha1` instead; sha1 is deprecated [Deprecated]
/tmp/nimble_128084/githubcom_nimlangatlas/src/depgraphs.nim(180, 41) Error: 'nimbleContents' is not a mutable location; it cannot be moved
    Prompt: Build failed for 'https://github.com/nim-lang/[email protected]', would you like to try installing 'https://github.com/nim-lang/atlas@#head' (latest unstable)? [y/N]

Add support for builders

Atlas currently lacks any support for a "build" step. Ideally the implementation is flexible enough to integrate cmake etc.

Collect errors and report them all at the end

Instead of "There were problems" collect warnings and errors and output them at the end of a run. Then the user would not have to scroll through the potentially long log to see what went wrong.

Package name should be case insensitive

$ atlas init
$ atlas clone arraymancer
$ atlas use nimjl # Because it uses arraymancer as a dependency

This creates 2 repo arraymancer : "Arraymancer" and "arraymancer" (capital A) difference.

IMO that shouldn't be possible & package name should be case insensitive. I don"t want to have 3 different packages "Tools", "tools", "TOOLS" that are different things

lockfiles must take into account the used Nim compiler version