vc3-project / vc3-builder Goto Github PK

Provide the steps to or a script that builds an image in a singularity using software built with VC3. Ideally this would be building and exporting the environment from VC3 builder into an image.

This would be great to reproducibly creating an image in different distros and across versions.

Suggested user interface:

vc3-pilot --require parrot --require python-2.6 --run "parrot_run cmssim.exe"

Maybe some optional arguments:

--database vc3-software.json
--install /home/joe/vc3-software

Suggest that the software catalog contain checksums for each of the tarballs, and that vc3-pilot verifies them. This will give us (and our users) greater confidence in mirroring tarballs in our own locations.

Word writing in error string is misspelled.

While building CCTools in vagrant for singularity.

[100%] Built target libcvmfs

• : Wed Nov 8 00:35:52 2017
• mkdir -p /vc3-builder/vc3-root/x86_64/generic/libcvmfs/v2.3.0/{lib,include}
• : Wed Nov 8 00:35:52 2017
• cp libcvmfs.a /vc3-builder/vc3-root/x86_64/generic/libcvmfs/v2.3.0/lib
• : Wed Nov 8 00:35:52 2017
• cp ../../cvmfs/libcvmfs.h ../../cvmfs/libcvmfs_int.h /vc3-builder/vc3-root/x86_64/generic/libcvmfs/v2.3.0/include
  cp: target '/vc3-builder/vc3-root/x86_64/generic/libcvmfs/v2.3.0/include' is not a directory
  Cleaning payload with pid: 8405

I am building this in an

Ubuntu:16.04

command:
apt-get -y update
apt-get -y install build-essential git libz-dev libperl-dev python-dev swig
git clone https://github.com/vc3-project/vc3-builder.git
cd vc3-builder
./vc3-builder --require cctools

I stood up a vanilla CentOS 6 container and installed 'make', 'git', and 'gcc'. Here's what I see, as of e99d12a:

$ git clone https://github.com/vc3-project/vc3-core
Initialized empty Git repository in /home/lincoln/vc3-core/.git/
remote: Counting objects: 119, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 119 (delta 4), reused 0 (delta 0), pack-reused 107
Receiving objects: 100% (119/119), 39.46 KiB, done.
Resolving deltas: 100% (58/58), done.
$ ./vc3_pilot_job parrot -- parrot_run python examples/read-file-from-cvmfs.py
Can't locate File/Fetch.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./vc3_pilot_job line 18.
BEGIN failed--compilation aborted at ./vc3_pilot_job line 18.

Now, I've never been quite sure if these containers are truly representative of a base CentOS install (i.e., yum groupinstall @core), or if they're cut down even further. This issue might be moot. From yum I see it comes from the perl-File-Fetch RPM:

# yum provides *File/Fetch.pm
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: chicago.gaminghost.co
 * extras: centos.mirror.nac.net
 * updates: bay.uchicago.edu
base/filelists_db                                                                                                                                          | 6.4 MB     00:00
extras/filelists_db                                                                                                                                        |  38 kB     00:00
updates/filelists_db                                                                                                                                       | 582 kB     00:00
perl-File-Fetch-0.26-141.el6_7.1.x86_64 : Generic file fetching mechanism
Repo        : base
Matched from:
Filename    : /usr/share/perl5/File/Fetch.pm

Some suggested output for the pilot job, as it works, so that the user can quickly grasp the principle of operation:

checking for installed parrot: not installed
parrot requires cvmfs-1.2.3
cvmfs-1.2.3 requires cmake-1.2.3
checking for installed cmake: not installed
downloading cmake from $url ...
building cmake
installing cmake
. . .

Also suggest that you redirect the stderr of the various procedures, so that the output isn't overwhelmed by information that is out of context. (Builds like to emit a lot of stuff on stderr.)