ucl-rits / rcps-buildscripts

Scripts to automate package builds on RC Platforms

rcps-buildscripts

Scripts to automate package builds on RC Platforms

rcps-buildscripts's Issues

Install request: libOSMesa.so.6

IN01325476

http://www.mesa3d.org/install.html

Should be something like this:

./configure --prefix=/path/to/install/dir --enable-osmesa
make
make install

May need to load xorg-utils/X11R7.7

For .so.6 I am assuming you might need this version of the source: ftp://ftp.freedesktop.org/pub/mesa/older-versions/6.x/

(We have a libOSMesa.so.8 in the VisIt module with the libraries that VisIt builds)

Install request: NWChem 6.6

Legion install requested. IN00874137

Install request: libXdmcp

Add libXdmcp to the xorg-utils/X11R7.7 module - required by Amber12 which some people have personal installs of. IN00775845

ftp://ftp.x.org/pub/X11R7.7/src/lib/libXdmcp-1.1.1.tar.bz2

Install request: cancerit suite

IN01315017

https://github.com/cancerit

In particular these:
BRASS (depends on grass, VAGrENT, cgpVcf)
CaVEMan and cgpCaVEManPostProcessing
gcpIndel
ascatNgs

There are a great many dependencies needed for all these, some of which we have as modules already.

Install request: Easy Lausanne

For IN00950423

https://github.com/mattcieslak/easy_lausanne

Prerequisites:

• FreeSurfer
• FSL
• numpy/scipy/nibabel/networkx

Install request: Hammock 1.0.4

IN00937227

http://www.recamo.cz/en/software/hammock-cluster-peptides/

This program works in combination with clustal-omega, hmmer and HHsuite. (Comes with compiled versions, but strongly recommends you to build your own).

Jobs requeueing on new OS

Jobs which complete successfully or end on an error are going back into the queue to be rerun rather than being removed from it. This is a known issue and we are working on it.

OpenFOAM's MPI is not working

It is trying to use the dummy Pstream library instead of the correct one.

mpirun -np 4 interFoam -parallel

--> FOAM FATAL ERROR:
Trying to use the dummy Pstream library.
This dummy library cannot be used in parallel mode

From function UPstream::init(int& argc, char**& argv)
in file UPstream.C at line 37.

FOAM exiting

Local2Scratch doesn't exist on new Legion yet

Scripts relying on #Local2Scratch won't copy anything back until we implement it on the new system. You'll need to put a manual tar line in your script, like this

#9. Preferably, tar-up (archive) all output files onto the shared scratch area
tar zcvf $HOME/Scratch/output/files_from_job_$JOB_ID.tgz $TMPDIR

kmc-2.1.1_install does not download kmc package

If I try to install kmc locally, the install script does not download the package in the temp dir, as in the messages below, but the same wget from the command line works.

$ bash rcps-buildscripts/kmc-2.1.1_install

--2015-11-16 09:33:11-- https://github.com/marekkokot/KMC/archive/2.1.1.tar.gz
Resolving github.com (github.com)... 192.30.252.130
Connecting to github.com (github.com)|192.30.252.130|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/marekkokot/KMC/tar.gz/2.1.1 [following]
--2015-11-16 09:33:11-- https://codeload.github.com/marekkokot/KMC/tar.gz/2.1.1
Resolving codeload.github.com (codeload.github.com)... 192.30.252.145
Connecting to codeload.github.com (codeload.github.com)|192.30.252.145|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 819958 (801K) [application/x-gzip]
Saving to: '2.1.1.tar.gz'

0% [ ] 0 --.-K/s in 0s

Cannot write to '2.1.1.tar.gz' (Success).

Install request: ScaLAPACK

We don't have a standalone ScaLAPACK in the new software stack, only MKL's. Related to IN00902683 for individual installs of CP2K.

Install request: NAMD2.11

IN00979987

Request for new version.

http://www.ks.uiuc.edu/Development/Download/download.cgi?PackageName=NAMD

Install request: xmds

Not on new Legion. IN00777097

http://www.xmds.org/

STAR-CD: Investigate problems with multi-node jobs

STAR-CD using Intel MPI works within a single node but doesn't work across nodes on either
Legion or Grace.

Install request: Amber 16

We've received a request via ISD Purchasing to upgrade Amber on Legion and Grace to version 16.

Install request: Octopus 5.0.1

IN00980150

http://www.tddft.org/programs/octopus/wiki/index.php/Octopus_5.0.1

Current GROMACS does not have working gmx view

IN00824302

Looks like it was missing some parts of X11 when compiled, so trying to run gmx view gives

Compiled without X-Windows - can not run viewer.

Install request: bedtools

Not on new Legion. IN00774818

Install request: WebLogo 3

IN01157939

Requires Python 2.7 with numpy, also ghostscript and pdf2svg.

http://weblogo.threeplusone.com/manual.html
https://github.com/WebLogo/weblogo

CUDA not on new Legion yet

The GPU nodes haven't been moved yet either.

If you need to use the GPU nodes immediately, email rc-support and we can allow you to submit jobs on old Legion.

Install request: VinaLC

IN01261049

A parallelised version of AutoDock Vina, molecular docking.

Licensed under Apache License 2.0.

http://mvirdb1.llnl.gov/static_catsid/vina/

Requires boost, mentions boost-mpi as a possibility but says that it currently uses MPI directly.

Beside the standard installation, you can also turn on the Boost MPI binding. Currently the program uses MPI library directly instead of Boost MPI

errors in blast+-2.2.30_install

Blast+ in not installed due to two errors; the following line has a typo (VERSION) and does not point to $VERSION

ftp://ftp.ncbi.nlm.nih.gov/blast/executables/LATEST/ncbi-blast-$(VERSION)+-src.tar.gz

so it should be changed to

ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/${VERSION}/ncbi-blast-${VERSION}+-src.tar.gz

Also, "../configure" should be "./configure" (configure is in the current dir)

These fixes are available in the fork ICONIC-UCL/rcps-buildscripts

Install request: R.E.D. RESP

Myself, and at least two of my PhD students require an online service called R.E.D. RESP (Restrained Electrostatic Potential). It essentially derives point charges of an atomistic structure for AMBER by first optimising the structure using Gaussian (Gaussian Inc. gave them a 'public' license to dump out the final structure but not any of the energetic or frequency data) and then running the structure through a series of python scripts according to some key-value parameter values provided in two input files. The final product is a zip file full of charge data and coordinate geometry.

Unfortunately, a week ago the R.E.D. Server had gone down. The AMBER community has no idea what's happened. Thankfully, the service providers give away the source code under the GNU license so institutes can establish their own private R.E.D. RESP servers. The link to the scripts is below (rescued by someone before the service died) - as you can guess, any further information has vanished along with the website.

http://lysozyme.medchem.utah.edu/ros/public_html/RED/RED-III.52-Tools-Files/

Investigate if we can install this as a module on Legion. IN00799592

Install request: MOLPRO 2015

Molpro 2015 on Legion to solve problems with multi-core runs within a node.

(Note: no binaries for Infiniband available yet).

IN00856134

Install request: GMSH

IN01800984

A three-dimensional finite element mesh generator with built-in pre- and post-processing facilities

http://gmsh.info/

License: GPL

Numpy 1.10.2 breaks LAPACK/BLAS detection for our build script.

If we try to run either the python 2 or python 3 bundle scripts, we get this error:

Installing collected packages: numpy
  Running setup.py install for numpy
    Complete output from command /shared/ucl/apps/python/bundles/1451921561-3/venv/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-7yjz732n/numpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file
__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-5wekvzaj-record/install-record.txt --single-version-externally-managed --compile --install-headers /shared/ucl/apps/python/bundles/1451921561-3/venv/i
nclude/site/python3.4/numpy:
    blas_opt_info:
    blas_mkl_info:
      libraries mkl,vml,guide not found in ['/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    openblas_info:
      libraries openblas not found in ['/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
      NOT AVAILABLE

    atlas_3_10_blas_threads_info:
    Setting PTATLAS=ATLAS
      libraries tatlas not found in ['/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib', '/usr/lib/sse2', '/usr/lib/']
      NOT AVAILABLE

    atlas_3_10_blas_info:
      libraries satlas not found in ['/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib', '/usr/lib/sse2', '/usr/lib/']
      NOT AVAILABLE

    atlas_blas_threads_info:
    Setting PTATLAS=ATLAS
      libraries ptf77blas,ptcblas,atlas not found in ['/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib', '/usr/lib/sse2', '/
usr/lib/']
      NOT AVAILABLE

    atlas_blas_info:
      libraries f77blas,cblas,atlas not found in ['/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64/sse2', '/usr/lib64', '/usr/lib/sse2', '/usr/lib', '/usr/lib/sse2', '/usr/
lib/']
      NOT AVAILABLE

    blas_info:
    Replacing _lib_names[0]=='blas' with 'openblas'
    Replacing _lib_names[0]=='openblas' with 'openblas'
    C compiler: cc

    creating /tmp/tmpxedlndus/tmp
    creating /tmp/tmpxedlndus/tmp/tmpxedlndus
    compile options: '-I/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib -I/usr/local/include -I/usr/include -I/shared/ucl/apps/python/bundles/1451921561-3/venv/include -c'
    cc: /tmp/tmpxedlndus/source.c
    cc /tmp/tmpxedlndus/tmp/tmpxedlndus/source.o -L/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib -lcblas -o /tmp/tmpxedlndus/a.out
    /bin/ld: cannot find -lcblas
    collect2: error: ld returned 1 exit status
    /bin/ld: cannot find -lcblas
    collect2: error: ld returned 1 exit status
    cc /tmp/tmpxedlndus/tmp/tmpxedlndus/source.o -L/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib -lblas -o /tmp/tmpxedlndus/a.out
    /bin/ld: cannot find -lblas
    collect2: error: ld returned 1 exit status
    Running from numpy source directory.
    /tmp/pip-build-7yjz732n/numpy/numpy/distutils/system_info.py:1651: UserWarning:
        Atlas (http://math-atlas.sourceforge.net/) libraries not found.
        Directories to search for the libraries can be specified in the
        numpy/distutils/site.cfg file (section [atlas]) or by setting
        the ATLAS environment variable.
      warnings.warn(AtlasNotFoundError.__doc__)
    Traceback (most recent call last):
      File "/shared/ucl/apps/python/3.4.3/gnu-4.9.2/lib/python3.4/distutils/unixccompiler.py", line 194, in link
        self.spawn(linker + ld_args)
      File "/tmp/pip-build-7yjz732n/numpy/numpy/distutils/ccompiler.py", line 28, in <lambda>
        m = lambda self, *args, **kw: func(self, *args, **kw)
      File "/tmp/pip-build-7yjz732n/numpy/numpy/distutils/ccompiler.py", line 72, in CCompiler_spawn
        raise DistutilsExecError('Command "%s" failed with exit status %d%s' % (cmd, s, msg))
    distutils.errors.DistutilsExecError: Command "cc /tmp/tmpxedlndus/tmp/tmpxedlndus/source.o -L/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib -lcblas -o /tmp/tmpxedlndus/a.out" failed with exit status 1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/tmp/pip-build-7yjz732n/numpy/numpy/distutils/system_info.py", line 1727, in has_cblas
        extra_postargs=info.get('extra_link_args', []))
      File "/shared/ucl/apps/python/3.4.3/gnu-4.9.2/lib/python3.4/distutils/ccompiler.py", line 734, in link_executable
        debug, extra_preargs, extra_postargs, None, target_lang)
      File "/shared/ucl/apps/python/3.4.3/gnu-4.9.2/lib/python3.4/distutils/unixccompiler.py", line 196, in link
        raise LinkError(msg)
    distutils.errors.LinkError: Command "cc /tmp/tmpxedlndus/tmp/tmpxedlndus/source.o -L/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib -lcblas -o /tmp/tmpxedlndus/a.out" failed with exit status 1

    During handling of the above exception, another exception occurred:
...

Followed by attempts to link against various blas lib names (-lblas etc) all of which fail.

This is despite us explicitly setting $LAPACK and $BLAS to the path to our openblas .so file.

Switching back to Numpy 1.9.2 resolves this issue, but obviously this is not sustainable, so a solution needs to be found. Ominously, the readme for 1.10.2 states:

• gh-6675 Optimized blas detection broken in master and 1.10.

http://sourceforge.net/projects/numpy/files/NumPy/1.10.2/

Which implies they tried to fix it and have broken it for us or that for some reason it is still broken for us. The error suggests that it's not searching for BLAS in the path we've set as it's searching in '/shared/ucl/apps/python/bundles/1451921561-3/venv/lib', '/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/'].

And we have set $BLAS/$LAPACK thus:

LAPACK=${LAPACK:-/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib/libopenblas.so}
BLAS=${BLAS:-/shared/ucl/apps/openblas/0.2.14/gnu-4.9.2/lib/libopenblas.so}

export LAPACK
export BLAS

There are various options for fixing this, one of which might be to create sym-links in the openblas lib directory to libopenblas.so called "libblas.so" and "liblapack.so".

hdf/5-1.8.15-p1-impi requires two conflicting versions of mpi on new legion OS

cceamda@login09 ~]$ module show hdf/5-1.8.15-p1-impi
-------------------------------------------------------------------
/shared/ucl/apps/modulefiles/libraries/hdf/5-1.8.15-p1-impi/intel-2015-update2:

module-whatis    Adds hdf5 1.8.5-p1 (built with Fortran and IntelMPI options) to your environment.
prereq   compilers/intel/2015/update2
prereq   mpi/intel/2015/update3/intel
prereq   mpi/openmpi/1.8.4/gnu-4.9.2

but

module show mpi/intel/2015/update3/intel
-------------------------------------------------------------------
/shared/ucl/apps/modulefiles/libraries/mpi/intel/2015/update3/intel:

module-whatis    adds Intel MPI to your environment variables
conflict     mpi

and

[cceamda@login09 ~]$ module show  mpi/openmpi/1.8.4/gnu-4.9.2
-------------------------------------------------------------------
/shared/ucl/apps/modulefiles/libraries/mpi/openmpi/1.8.4/gnu-4.9.2:

module-whatis    adds OpenMPI 1.8.4 for GCC 4.9.2 compilers to your environment variables
conflict     mpi

Welcome message and quota reporting not on new Legion

In the meantime, can run du -h --max-depth=1 to see the totals for all the top-level directories relative to where you are. (Run separately in Scratch to see things in there).

Install request: HMMER

Sequence analysis. IN00778573

http://hmmer.janelia.org/

Install request: waves2Foam for OpenFOAM (3 or 4?)

IN01847774

https://openfoamwiki.net/index.php/Contrib/waves2Foam

waves2Foam is an additional library and solver for OpenFOAM.

Was going to suggest building it for OpenFOAM 3, but it appears OpenFOAM 4 just came out two days ago - check on waves2Foam compatibility when we start this, as 4 is not currently mentioned.

Install CESM 1.2.2

We've had a request for a 1.2.x version of CESM.

I'll take a look at this.

Install request: CERNLIB

CERNLIB on Legion for use with GNU 4.9.2 and NAG Mark 24.

Was on Old Legion.

IN01032834

Install request: PostGIS

The shipping group need PostGIS installed:

http://postgis.net/source/

Most of the dependencies are already installed but we'll want to do a central install of postgres.

Install request: Opensees

IN01814544

OpenSees is a software framework for developing applications to simulate the performance of structural and geotechnical systems subjected to earthquakes.

http://opensees.berkeley.edu
http://opensees.berkeley.edu/WebSVN/listing.php?repname=OpenSees&path=%2Ftrunk%2FSRC

They provide Windows and Mac binaries only, we'd need to compile from source (and edit a Makefile.def as they don't use configure at present).

Install request: EIGENSOFT

IN01193402

https://data.broadinstitute.org/alkesgroup/EIGENSOFT/

Info:
http://www.hsph.harvard.edu/alkes-price/software/
http://www.hsph.harvard.edu/alkes-price/eigensoft-frequently-asked-questions/

Does not appear to use the Broad license so shouldn't need a separate application group:

SOFTWARE COPYRIGHT NOTICE AGREEMENT
This software and its documentation are copyright (2010) by Harvard University
and The Broad Institute. All rights are reserved. This software is supplied
without any warranty or guaranteed support whatsoever. Neither Harvard
University nor The Broad Institute can be responsible for its use, misuse, or
functionality. The software may be freely copied for non-commercial purposes,
provided this copyright notice is retained.

Registration is voluntary and for update information not license purposes. (From http://www.hsph.harvard.edu/alkes-price/software/)

SOFTWARE REGISTRATION: we encourage EIGENSOFT users to register here. Registration is voluntary, but will allow us to send information about software updates.

Rebuild Pythons with SQLite 3 support

It appears they didn't find the sqlite3 headers when configuring.

Install request: FastQC

IN00824441

Sequence data analysis. http://www.bioinformatics.babraham.ac.uk/projects/fastqc/

Install request: GAMESS-US

IN01255189

http://www.msg.ameslab.gov/gamess/download.html

Need to clarify license:

That page says this:

A site license for GAMESS is available at no cost to both academic and industrial users. You will be asked to agree to this license during the process of downloading GAMESS. The license acknowledges the Gordon group's exclusive distribution rights to GAMESS, and also prohibits you from making copies of the GAMESS code for any purpose except use at your own institution.

Actual license says this:

This license is considered a "Research Group" license. Thus anyone directly associated with your research group is covered by the license and may share your copy of GAMESS on all of your group's computer resources.

Install request: Clustal omega

Sequence aligner. http://www.clustal.org/omega/

Also requires http://argtable.sourceforge.net/

IN00778573

Install request: ANTs

Not on new Legion yet. IN00772718

http://stnava.github.io/ANTs/

MATLAB DCS on new Legion (and remote submission)

MATLAB 2015a is available on new Legion, but the DCS is still being configured. I will update here when it is ready.

Install request: sgplib and pyspglib

On Legion and Grace. Spglib is a C library for finding and handling crystal symmetries.

http://spglib.sourceforge.net/

IN00865871

Install request: CRYSTAL 14 1.0.4

IN01315433

The version contains a fix for a particular bug that I've been experiencing whilst running the current version (v1.0.3) on Grace, to do with restarting vibrational frequency calculations.

Install request: PLINK1.9 (when release available)

IN01051232

Currently still in beta - waiting for first full release before a central install, assuming this happens relatively soon.

https://github.com/chrchang/plink-ng/releases

https://www.cog-genomics.org/plink2/

Install request: AFNI

Request for the AFNI software suite to be installed on Legion, available at afni.nimh.nih.gov.

Ticket IN00770976.

(Website currently not loading).

Install request: GROMACS with Plumed 2.2

Currently we have Plumed 2.1.2

IN00804561

Install request: grace (xmgrace)

http://plasma-gate.weizmann.ac.il/Grace/

Install request: STAR (sequence aligner) and RSEM

IN01803209

https://github.com/alexdobin/STAR
https://github.com/bli25ucb/RSEM_tutorial

Install request: proovread

https://github.com/BioInf-Wuerzburg/proovread/releases

Depends on Log::Log4perl, File::Which, blast+ and samtools.

blast+/2.2.30/intel-2015-update2 and samtools/1.2/gnu-4.9.2 are available as modules and Log4perl is installed with perl/5.22.0 so just needs File::Which

IN00852851

Install request: sklearn for Python3

IN01106911

Can use pip3, is called scikit-learn

http://scikit-learn.org/stable/index.html