Need to package icommands for NIX about irods_client_icommands

[kellerb]

We are all for having NIX packages and thank you for offering to help out.

To explain the error you are seeing:

For iRODS 4.1, the icommands source code was housed in the main iRODS repository. When building, the icommands would make use of core iRODS code by using the object files left over from building the server executables.

For iRODS 4.2, we have pulled the icommands source code out of the main repository (obviously). When building, the icommands now make use of core iRODS code by:

• linking against the iRODS shared libraries included in the irods-runtime package
• using headers and cmake configuration files included in the irods-dev package

Both the irods-runtime and irods-dev packages are created by the main iRODS repository.

Note that all of the cmake work on the main iRODS repository is currently on the XBB-2 branch. Before 4.2 is released, we will be moving that work onto the master branch and putting out some documentation on how the new build system works.

[bzizou]

Thank you for your support. I discussed with Antoine at the ugm2016 and he already talked to me about the XBB branch, so I was already on the way, but I did not have the time to work on that since I'm back. I think I'll take a look again at it in the next 2 weeks, and I'll make updates to this issue.
Regard!

[trel]

@bzizou Any news? Things have settled down since June and should work much more smoothly now as we get close to the 4.2.0 release.

[bzizou]

I've spent a couple of hours without success during the summer... I also took some time to install NixOS on my laptop and I'm on the way to deploy it on a production server environment, and then I feel more comfortable with the Nix packaging system. So, if things have evolved for iRods build system, it's time for me to give another try right now. Thks for the news!

[bzizou]

So, I just took a look. Please, tell me if I'm wrong:

• The XBB branch is now merged into the master branch
• To build an irods-client-icommand package for a new distribution, I need to build a irods-runtime and irods-dev package for this distribution
• To build the irods-runtime and irods-dev packages, I need to build the irods_externals build dependencies

So, if I'm right, my job starts with patching externals/install_prerequisites.py to support the NIX distribution?

[trel]

You have unwound the ball correctly.

You could satisfy the prerequisites manually first if you want, but yes, additions to the script would certainly be welcome.

Then build the core, then install dev and runtime, then build the icommands.

[bzizou]

Just to inform that work is in progress into the irods4.2 branch of my nixpkgs repo:
https://github.com/bzizou/nixpkgs/tree/irods4.2
I changed the title of the issue, as it's now a feature request, more than a bug.
You can assign it to me if you want...

[bzizou]

I finally packaged irods and irods-icommands into Nix, and I'm very proud of it! It's currently only in my repo, but I'll make a pull request in the Nix repo as soon as I have enough testing. For the irods servers part, I still have some work for integration into NixOS, but at least the libraries are ok as a dependency for icommands.
Relevant files:
https://github.com/bzizou/nixpkgs/tree/irods4.2/pkgs/tools/filesystems/irods
Relevant part of https://github.com/bzizou/nixpkgs/blob/irods4.2/pkgs/top-level/all-packages.nix:

boost160clang38 = callPackage ../development/libraries/boost/1.60.nix { 
      llvmPackages = llvmPackages_38;
      toolset = "clang";
      enableLibcxx = true;
};

irods = callPackage ../tools/filesystems/irods { 
    llvmPackages = llvmPackages_38;
    gcc = gcc48;
    boost = boost160clang38;
    avro-cpp = avro-cpp.override {
      llvmPackages = llvmPackages_38;
      boost = boost160clang38;
      libcxxSupport = true;
    };
};

irods-icommands = callPackage ../tools/filesystems/irods/icommands.nix { 
    llvmPackages = llvmPackages_38;
    boost = boost160clang38;
};

I also had to adapt the avro-cpp and boost packages, pull requests coming soon into the Nix repo.

[bzizou]

Pull request : NixOS/nixpkgs#19898

[trel]

This is very cool. Let us know if there's anything we can do to help.

We don't need any changes in the iRODS code? It just works?

[bzizou]

There are some minor changes that we can do later to avoid some modifications of the cmake files and the irods_root_path.cpp file. It's all about paths. But let the pull request be merged, and we'll see that for the next increment...
The very hard work was about the boost+avro-cpp+clang 3.8 dependency, that may lead to changes in the deep of nix because of headers conflicts between libc++ >= 3.8 and the stdlib (NixOS/nixpkgs#19859 and NixOS/nixpkgs#19895)
I may have to revert some things done before the PR to be merged...

But yeah, it just works! You did a great job with the cmakification of the build!! And Nix did the rest regarding the "externals" problem as it's a feature of Nix to be able to recompile the dependencies!

[trel]

Yes, I saw that thread and conversation. You jumped into the deep end - and lived.

@kellerb gets the credit for the CMake.

We'll watch progress on the PR - thanks.

[bzizou]

Is there a beta tarball of irods, irods_schema_messaging and irods_client_icommands 4.2 sources somewhere?
Or should we wait for the 4.2 to be released? I'm afraid that my PR is blocked because using direct downloads from github...

[trel]

There isn't a good public place to put such a tarball. Unfortunately, this is one of the limitations of using the submodules.

We've removed one of the submodules for 4.2 - we'll only call in irods_schema_messaging, and it is very lightweight - you might be able to pull it in separately, directly.

The easiest thing will be to wait for 4.2 to be released.

[bzizou]

OK, so just after SC16? ;-)

[trel]

Did we solve this? Because it's been a year :)

[bzizou]

@trel Yes! Sorry for the delay...
We are successfuly using irods-icommands from NIX since year ago ;-)
https://github.com/NixOS/nixpkgs/tree/master/pkgs/tools/filesystems/irods
I'll upgrade this package to irods 4.2.2 soon... (actually already done here, I have to push it upstream)
However, no success for the server part of this package, we end up with a freezed irodsServer process just after the SQL initialization. But this should be another issue...

[trel]

Thanks - yes, the process model is a bit different in 4.2+. Let us know if you think we can help.

[samrose]

Did you all ever resolve this? How do you install and start irods services in nixos?

[bzizou]

No. We gave up launching irods services under nixos... Anyway, we are still using icommands with Nix and Nixos, as Nix is our main way to provide applications to our users.