Coder Social home page Coder Social logo

diracos's Introduction

DIRACOS

pipeline status documentation status

DIRACOS aims at bringing in one archive all the dependencies needed by DIRAC to run Agents, Services and clients. It is not intended to perform interactive tasks (debugging, file editing, etc). We are aware than some “basic” tools won’t work (less, emacs, etc). They won’t be fixed.

The documentation for DIRACOS can be found here.

Disclaimer

DIRACOS is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. We are aiming at supporting SLC6 and CC7, whilst we test also on other platforms we do not provide support for those.

diracos's People

Contributors

aldbr avatar andresailer avatar chaen avatar chrisburr avatar diracos avatar fstagni avatar marcelovilaca avatar sfayer avatar

Stargazers

avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

chaen marcelovilaca petricm diracos andresailer chrisburr roiser sfayer aldbr

diracos's Issues

Remove runit from DIRAC

As agreed during today's BILD meeting, as of v7r0, runit will not be distributed with DIRAC anymore and should thus be removed from DIRACOS

Broken/wrong links

If you untar the bundle into a bare container you will find some broken links

find . -xtype l
./diracos/usr/lib64/python2.7/site-packages/PyQt4/uic/pyuic.py
./diracos/usr/lib64/python2.7/UserDict.py
./diracos/usr/lib64/python2.7/_abcoll.py
./diracos/usr/lib64/python2.7/_weakrefset.py
./diracos/usr/lib64/python2.7/abc.py
./diracos/usr/lib64/python2.7/codecs.py
./diracos/usr/lib64/python2.7/copy_reg.py
./diracos/usr/lib64/python2.7/fnmatch.py
./diracos/usr/lib64/python2.7/genericpath.py
./diracos/usr/lib64/python2.7/linecache.py
./diracos/usr/lib64/python2.7/locale.py
./diracos/usr/lib64/python2.7/ntpath.py
./diracos/usr/lib64/python2.7/os.py
./diracos/usr/lib64/python2.7/posixpath.py
./diracos/usr/lib64/python2.7/re.py
./diracos/usr/lib64/python2.7/sre.py
./diracos/usr/lib64/python2.7/sre_compile.py
./diracos/usr/lib64/python2.7/sre_constants.py
./diracos/usr/lib64/python2.7/sre_parse.py
./diracos/usr/lib64/python2.7/stat.py
./diracos/usr/lib64/python2.7/types.py
./diracos/usr/lib64/python2.7/warnings.py
./diracos/usr/lib64/libcom_err.so
./diracos/usr/lib64/libkeyutils.so
./diracos/usr/share/doc/runit-2.1.2/debian
./diracos/usr/share/doc/nordugrid-arc-5.4.0/arc.conf.reference
./diracos/usr/bin/voms-proxy-list
./diracos/usr/lib/jvm-exports/java-1.7.0-oracle.x86_64
./diracos/usr/lib/jvm-exports/java-1.8.0-oracle.x86_64
./diracos/etc/fonts/conf.d/58-dejavu-lgc-sans-mono.conf
./diracos/etc/fonts/conf.d/20-unhint-small-dejavu-lgc-sans-mono.conf
./diracos/etc/fonts/conf.d/57-dejavu-sans-mono.conf
./diracos/etc/fonts/conf.d/20-unhint-small-dejavu-sans-mono.conf

all the python stuff links to /usr/lib64 which I don't think is correct

ll diracos/usr/lib64/python2.7/UserDict.py
lrwxrwxrwx. 1 root root 32 Oct 24 00:49 diracos/usr/lib64/python2.7/UserDict.py -> /usr/lib64/python2.7/UserDict.py

Multithreaded depackers

maybe adding this to the cfg?

config_opts['chroot_setup_cmd'] = 'install @buildsys-build /usr/bin/pigz /usr/bin/lbzip2'
config_opts['macros']['%__gzip'] = '/usr/bin/pigz'
config_opts['macros']['%__bzip2'] = '/usr/bin/lbzip2'

Silence Warnings in diracosrc 

/opt/dirac/diracos/usr/lib64/python2.7/distutils/__init__.py:14: UserWarning: The virtualenv distutils package at %s appears to be in the same location as the system distutils?
  "The virtualenv distutils package at %s appears to be in the same location as the system distutils?")

can be solved with the following line in the diracosrc

PYTHONWARNINGS="ignore"

Remove libselinux.so.1 for CentOS 8 support

As there is now a docker image for CentOS 8 I thought I'd quickly try it and unsurprisingly everything is horribly broken. More surprisingly it's trivial to fix enough that the client tests work.

The error when installing is:

2019-10-02 11:16:38 UTC dirac-install [NOTICE]  Defaults written to defaults-DIRAC.cfg
2019-10-02 11:16:38 UTC dirac-install [NOTICE]  Executing /admin/runit-2.1.2/scripts/dirac-externals-requirements...
/usr/bin/coreutils: symbol lookup error: /usr/bin/coreutils: undefined symbol: mode_to_security_class

It is due to LD_LIBRARY_PATH is causing coreutils to take libselinux.so.1 from DIRAC OS. Ideally LD_LIBRARY_PATH shouldn't be used at all but as that isn't an option I think it can be taken from the host instead of being included in DIRACOS.

CentOS 8 Installation Steps

$ docker run --rm -it centos:8 bash
dnf -y groupinstall "Development Tools"
dnf -y config-manager --set-enabled PowerTools
dnf -y install glibc-static wget freetype fontconfig pixman libXrender psmisc
wget http://smarden.org/runit/runit-2.1.2.tar.gz
tar xvf runit-2.1.2.tar.gz
cd admin/runit-2.1.2/
package/install
curl -O  -L https://github.com/DIRACGrid/DIRAC/raw/integration/Core/scripts/dirac-install.py
chmod +x dirac-install.py
dnf -y install python2
# Must set python to be python2 for dirac-install.py to work
alternatives --set python /usr/bin/python2
./dirac-install.py -r v7r0 -t client

Check srpm cache before downloading package

That will save a few download for SRPM

In [2]: srpm = '/eos/project/l/lhcbwebsites/www/lhcb-rpm/dirac/DIRACOS/SRPM/dcap-2.47.12-4.el6.src.rpm'

In [3]: urlSrpm = 'http://mirror.switch.ch/ftp/mirror/epel/6/SRPMS/Packages/d/dcap-2.47.12-4.el6.src.rpm'


In [7]:  rpmUtils.miscutils.splitFilename( os.path.basename(srpm))
Out[7]: ('dcap', '2.47.12', '4.el6', '', 'src')

In [8]:  rpmUtils.miscutils.splitFilename( os.path.basename(urlSrpm))
Out[8]: ('dcap', '2.47.12', '4.el6', '', 'src')

Issues with vanilla container

If you want to use DIRACOS in the vanilla cern/slc6-base container you would execute

curl -O http://diracos.web.cern.ch/diracos/master/diracos-master.tar.gz
curl -O https://raw.githubusercontent.com/DIRACGrid/DIRACOS/master/tests/integration/test_import.py
yum install tar -y
tar xf diracos-master.tar.gz -C /tmp
export DIRACOS=/tmp/diracos
source $DIRACOS/diracosrc

then if you run the test pytest test_import.py it fails with

# pytest test_import.py 
=============================================================== test session starts ===============================================================
platform linux2 -- Python 2.7.13, pytest-3.9.2, py-1.7.0, pluggy-0.8.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/.hypothesis/examples')
rootdir: /, inifile:
plugins: cov-2.6.0, hypothesis-3.79.3
collected 109 items                                                                                                                               

test_import.py ..............................F.............................s................................................                [100%]

==================================================================== FAILURES =====================================================================
________________________________________________________________ test_module[git] _________________________________________________________________

moduleName = 'git'

    @parametrize('moduleName', moduleNames)
    def test_module(moduleName):
      """ Try to import a module and check whether it is located in DIRACOS.
    
          Modules that are in the ALLOWED_TO_FAIL list are shown as skipped and generate a warning
    
          Modules that require graphic libraries on the system (GRAPHIC_MODULES) are skipped on container
      """
    
      try:
        module = __import__(moduleName)
    
        # Test whether it is correctly imported from DIRACOS
    
        try:
          modulePath = module.__file__
          # return true, if the common prefix of both is equal to directory
          # e.g. /a/b/c/d.rst and directory is /a/b, the common prefix is /a/b
          assert os.path.commonprefix([modulePath, diracosPath]) == diracosPath, \
              "ERROR %s not from DIRACOS: %s" % (moduleName, modulePath)
    
        # builtin modules like sys have no path
        except AttributeError as e:
          print "WARNING no path for %s" % moduleName
    
      except ImportError as e:
        msg = "could not import %s: %s" % (moduleName, repr(e))
        print traceback.print_exc()
    
        if moduleName in ALLOWED_TO_FAIL:
          warnings.warn(msg)
          pytest.skip("WARN: " + msg)
        elif moduleName in GRAPHIC_MODULES:
          warnings.warn(msg + "(Possibly due to system graphic libraries not present)")
          pytest.skip("WARN: " + msg + "(Possibly due to system graphic libraries not present)")
        else:
>         pytest.fail("ERROR: " + msg)
E         Failed: ERROR: could not import git: ImportError('Failed to initialize: Bad git executable.\nThe git executable must be specified in one of the following ways:\n    - be included in your $PATH\n    - be set via $GIT_PYTHON_GIT_EXECUTABLE\n    - explicitly set via git.refresh()\n\nAll git commands will error until this is rectified.\n\nThis initial warning can be silenced or aggravated in the future by setting the\n$GIT_PYTHON_REFRESH environment variable. Use one of the following values:\n    - quiet|q|silence|s|none|n|0: for no warning or exception\n    - warn|w|warning|1: for a printed warning\n    - error|e|raise|r|2: for a raised exception\n\nExample:\n    export GIT_PYTHON_REFRESH=quiet\n',)

test_import.py:238: Failed
-------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------
None
-------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------
Traceback (most recent call last):
  File "/test_import.py", line 212, in test_module
    module = __import__(moduleName)
  File "/tmp/diracos/usr/lib64/python2.7/site-packages/git/__init__.py", line 85, in <module>
    raise ImportError('Failed to initialize: {0}'.format(exc))
ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh()

All git commands will error until this is rectified.

This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
    - quiet|q|silence|s|none|n|0: for no warning or exception
    - warn|w|warning|1: for a printed warning
    - error|e|raise|r|2: for a raised exception

Example:
    export GIT_PYTHON_REFRESH=quiet

================================================= 1 failed, 107 passed, 1 skipped in 2.54 seconds =================================================

Shrink size of DIRACOS

I already tried various compression level, but we are at the maximum for that.
By default, we also get ride of all the docs and debug info in the cfg

      "excludePatterns": [
         ".*-doc-.*",
         ".*-debuginfo-.*"
      ],

So we will have to trim things down one way or another.
I do not want any dirty hack in any case.

MySQL 8.0 support

Installation using fullInstall in DIRAC/tests/Jenkins/dirac_ci.sh fails when the script wants to create a new DB in MySQL. It seems like MySQL 8 changes default authentication plugin from mysql_native_password to caching_sha2_password, as mentioned here.
Install log is here: log.txt.

The following environment variables were used:

DEBUG=True
DIRAC_RELEASE=v6r22-pre3
DB_USER=Dirac
DB_PASSWORD=Dirac
DB_ROOTUSER=admin
DB_ROOTPWD=password
DB_HOST=172.18.0.4
DB_PORT=3306
DIRACOSVER=v1r1

NoSQLDB_HOST=172.18.0.5
NoSQLDB_PORT=9200

repository=DIRACGrid
branch=rel-v6r22
DIRACBRANCH=rel-v6r22

as well as having MySQL 8.0 and ElasticSearch 6.6.0 in separate Docker containers. The server was also installed in a Docker container. The admin user in docker was created using

CREATE USER 'admin'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'admin'@'%' WITH GRANT OPTION;
FLUSH PRIVILEGES;
quit

Solutions to this include:

  1. Upgrading MySQL client in DIRACOS to 8.0 or higher and implement backwards compatibility for 5.X. (DIRACOS bundling + DIRAC code)
  2. Changing authentication method for both Dirac and admin users using ALTER USER admin IDENTIFIED WITH mysql_native_password BY 'password'; (DIRAC code)

Git depends on having a host installation

Currently the DIRACOS installation of git depends on the host's helper binaries:

$ git --exec-path
/usr/libexec/git-core

As DIRACOS needs to be runtime relocatable without editing the binaries there are several ways of working around this:

  1. Add export GIT_EXEC_PATH=$DIRAC/diracos/usr/libexec/git-core to the bashrc
  2. Patch the spec file to enable a relocatable build. I've tried this and it works with the following patch.
diff --git a/git.spec b/git.spec
index f1482ff..baa9339 100644
--- a/git.spec
+++ b/git.spec
@@ -248,6 +248,7 @@ htmldir = %{_docdir}/%{name}-%{version}
 prefix = %{_prefix}
 ASCIIDOC8 = 1
 ASCIIDOC_NO_ROFF = 1
+RUNTIME_PREFIX = YesPlease
 EOF # Filter bogus perl requires
--
2.21.0

Unfortunately 2. has the caveat that git 1.7.1 only works if you use an absolute path to git:

$ git --exec-path
git: exec_cmd.c:23: system_path: Assertion `argv0_path' failed.
Aborted

$ $(which git) --exec-path
/tmp/git-build/usr/libexec/git-core

This can be accepted as a "feature" or avoided by:

  1. Replace git with a wrapper script that calls exec "${DIRAC}/something/git" "$@"
  2. Update to a newer version of git (2.18+) which uses /proc to read the exe symlink

Updating to a newer git seems like the cleanest option but I'm not sure how difficult it would be.

Wrong version comparison in cache 

[root@41a16d20ab28 /]# dos-build-package /tmp/DIRACOS/config/diracos.json rpm
INFO:root:Building http://ftp.redhat.com/pub/redhat/linux/enterprise/6Client/en/os/SRPMS/rpm-4.8.0-59.el6.src.rpm with mockConfig /tmp/DIRACOS/mockConfigs/mock-build-diracos.cfg
DEBUG:root:SRPM file is an URL, download it first
DEBUG:root:Beginning file download: http://ftp.redhat.com/pub/redhat/linux/enterprise/6Client/en/os/SRPMS/rpm-4.8.0-59.el6.src.rpm
DEBUG:root:Writing to /tmp/rpm-4.8.0-59.el6.src.rpm
DEBUG:root:Downloading with curl -o /tmp/rpm-4.8.0-59.el6.src.rpm -L http://ftp.redhat.com/pub/redhat/linux/enterprise/6Client/en/os/SRPMS/rpm-4.8.0-59.el6.src.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3734k  100 3734k    0     0   786k      0  0:00:04  0:00:04 --:--:--  877k
INFO:root:The repo already contains a build, not rebuilding: [u'/diracos_repo/buildOnly/rpm-4.8.0-55.el6.py27.usc4.src.rpm']

Add documentation about release procedure

Specify how to get the static version list for the requirements.txt

for i in $(awk -F '[>=]' {'print $1'} requirements.txt | grep -v git); do latest=$(pip install $i==0.0.0 2>&1 | grep 'from versions'  | awk -F '[,:]' {'print $NF'} | sed -e 's/)//g' -e 's/ //g'); echo "$i==$latest"; done

virtualenv does not work

Initial issue is that the shebang of virtualenv is

DIRAC-PROD>virtualenv /tmp/toto
/usr/bin/env: python2.7 -s: No such file or directory

The problem is this line. Only one argument is passed to the command in the shebang (https://www.in-ulm.de/~mascheck/various/shebang/)

#!/usr/bin/env python2.7 -s

This is done by the diracos bundle script, originally it is #!/usr/bin/python2.7 -s

The -sconfuses things

But even when this is fixed, it does not work

DIRAC-PROD>virtualenv /tmp/toto
Using real prefix '/usr'
New python executable in /tmp/toto/bin/python2.7
Also creating executable in /tmp/toto/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /tmp/toto/bin/python2.7 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
ImportError: No module named pkgutil
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/opt/dirac/diracos/usr/bin/virtualenv", line 3, in <module>
    virtualenv.main()
  File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 711, in main
    symlink=options.symlink)
  File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 944, in create_environment
    download=download,
  File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 900, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/opt/dirac/diracos/usr/lib/python2.7/site-packages/virtualenv.py", line 795, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /tmp/toto/bin/python2.7 - setuptools pip wheel failed with error code 1

Can not submit ARC job

Trying to submit job we have in result

VERBOSE: Running command: arcsub -d DEBUG -c hepgrid5.ph.liv.ac.uk job2.xrsl
VERBOSE: System configuration file (/etc/arc/client.conf or /usr/etc/arc/client.conf) does not exist.
INFO: Cannot copy example configuration (/usr/share/arc/examples/client.conf), it is not a regular file
INFO: No configuration file could be loaded.
INFO: Using proxy file: /home/dirac/testARC/x509up_u15872
INFO: Using CA certificate directory: /etc/grid-security/certificates
DEBUG: Certificate format is PEM
DEBUG: Trying to check X509 cert with check_cert_type
DEBUG: Module Manager Init
ERROR: Invalid JobDescription:
&(executable="test.sh")
(inputFiles=(test.sh "test.sh"))
(stdout="test.out")
(stderr="test.err")
(outputFiles=("test.out" "") ("test.err" ""))

No job description parsers available

It seems ARC does not use plugins despite they are formally "installed".

At lxplus we have during job submission

....
DEBUG: Module Manager Init
DEBUG: Loaded /usr/lib64/arc/libaccJobDescriptionParser.so
DEBUG: Loaded JobDescriptionParserPlugin ARCJSDLParser
DEBUG: Loaded JobDescriptionParserPlugin JDLParser
DEBUG: Loaded JobDescriptionParserPlugin XRSLParser
VERBOSE: String successfully parsed as nordugrid:xrsl.
....

It seems ARC looks for plugins at /usr/lib64/arc only by default.

Resume:
ARC should be configured somehow.

Firefox in dependency list 

Found duplicates for libpango-1.0.so.0: ['/tmp/diracos/usr/lib64/libpango-1.0.so.0.2800.1', '/tmp/diracos/usr/lib64/firefox/bundled/lib64/libpango-1.0.so.0.4000.4']
Traceback (most recent call last):
  File "/tmp/set_RPATH.py", line 176, in <module>
    main(sys.argv[1])
  File "/tmp/set_RPATH.py", line 171, in main
    check_duplicates(so_names)
  File "/tmp/set_RPATH.py", line 98, in check_duplicates
    raise NotImplementedError(v, hashes)
NotImplementedError: (['/tmp/diracos/usr/lib64/libpango-1.0.so.0.2800.1', '/tmp/diracos/usr/lib64/firefox/bundled/lib64/libpango-1.0.so.0.4000.4'], {'536e9298b19719476b4ed03d150734da16fc57e17c067f6624b1d751b57ee904', '7b7caade480e9b2379686a2cceee98d8d2a5959d1b388ddc8f5ec2e7a492b017'})
Traceback (most recent call last):
  File "/tmp/bundlelib.py", line 251, in <module>
    main()
  File "/tmp/bundlelib.py", line 245, in main
    manualDependencies=manualDependencies)
  File "/tmp/bundlelib.py", line 224, in _doBundleDIRACOS
    subprocess.check_call(bundleCmd)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/tmp/bundleDiracOS.sh']' returned non-zero exit status 1
Finish: shell
Traceback (most recent call last):
  File "/usr/bin/dos-bundle", line 9, in <module>
    load_entry_point('diracos==1.0.0', 'console_scripts', 'dos-bundle')()
  File "/usr/lib/python2.6/site-packages/diracos/scripts/bundleDiracOS.py", line 17, in main
    diracoslib.bundleDIRACOS(cfg)
  File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 605, in bundleDIRACOS
    subprocess.check_call(bundleCmd)
  File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mock', '-r', u'/builds/CLICdp/iLCDirac/DIRACOS/mockConfigs/mock-install-diracos.cfg', '--shell', 'python /tmp/bundlelib.py']' returned non-zero exit status 1

Rename tests 

Sorry to come late on this, just a comment. pytest by default looks for files named "test_*.py", and inside it for test items in functions named test_*. Here you go against the practice.

Document procedure when 404 

404 not found

INFO:root:Building http://dl.fedoraproject.org/pub/epel/6Server/SRPMS/Packages/l/lcgdm-1.9.2-1.el6.src.rpm with mockConfig /DIRACOS/mockConfigs/mock-build-diracos.cfg
DEBUG:root:SRPM file is an URL, download it first
DEBUG:root:Beginning file download: http://dl.fedoraproject.org/pub/epel/6Server/SRPMS/Packages/l/lcgdm-1.9.2-1.el6.src.rpm
DEBUG:root:Writing to /tmp/lcgdm-1.9.2-1.el6.src.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
128   257  128   257    0     0    750      0 --:--:-- --:--:-- --:--:--  1511
DEBUG:root:Checking existance of /DIRACOS/patches/lcgdm.patch
DEBUG:root:Rebuild SRPM with ['/usr/bin/mock', '-r', u'/DIRACOS/mockConfigs/mock-build-diracos.cfg', '--rebuild', u'/tmp/lcgdm-1.9.2-1.el6.src.rpm']
INFO: mock.py version 1.3.5 starting (python version = 2.6.6)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
ERROR: Cannot find/open srpm: /tmp/lcgdm-1.9.2-1.el6.src.rpm. Error: error reading package header
Traceback (most recent call last):
  File "/usr/bin/dos-build-package", line 9, in <module>
    load_entry_point('diracos==0.0.1', 'console_scripts', 'dos-build-package')()
  File "/usr/lib/python2.6/site-packages/scripts/buildPackage.py", line 20, in main
    diracoslib.buildPackage(package)
  File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 463, in buildPackage
    _buildFromSRPM(packageCfg)
  File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 326, in _buildFromSRPM
    _mockRebuild(srpmFile, mockConfig)
  File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 355, in _mockRebuild
    subprocess.check_call(cmd)
  File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/mock', '-r', u'/DIRACOS/mockConfigs/mock-build-diracos.cfg', '--rebuild', u'/tmp/lcgdm-1.9.2-1.el6.src.rpm']' returned non-zero exit status 1

When this happens (I will put it on the doc):

check the website if there was just a minor version upgrade. If it is, fine, let's just use that one.
You can see there is lcgdm-1.10 so it is NOT a minor version.

In that case, just google for the exact RPM on google, and you can take it from any repo.
For example ftp://ftp.icm.edu.pl/vol/rzm8/linux-fedora/linux/epel/6/SRPMS/Packages/l/lcgdm-1.9.2-1.el6.src.rpm

So you edit the json file and replace the link (good test though, never tried an ftp link)

yum utils version mismatch

yum-utils-1.1.30-41(base) vs yum-utils-1.1.30-40(lhcb)

[root@90c8bd9bc220 ~]# dos-bundle DIRACOS/config/diracos.json
INFO:root:Bootstraping packaging of diracos
INFO: mock.py version 1.3.5 starting (python version = 2.6.6)...
Start: init plugins
INFO: selinux disabled
Finish: init plugins
Start: run
Start: chroot init
INFO: calling preinit hooks
INFO: enabled root cache
INFO: enabled yum cache
Start: cleaning yum metadata
Finish: cleaning yum metadata
INFO: enabled HW Info plugin
Start: yum install
Error: Package: yum-utils-1.1.30-41.el6.noarch (base)
           Requires: python(abi) = 2.6
           Available: python-2.7.13-2.el6.i686 (local-py2.7)
               python(abi) = 2.7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
ERROR: Command failed: 
 # /usr/bin/yum --installroot /var/lib/mock/epel-6-x86_64-install/root/ install @buildsys-build yum yum-utils python-devel xz-devel gcc git
Error: Package: yum-utils-1.1.30-41.el6.noarch (base)
           Requires: python(abi) = 2.6
           Available: python-2.7.13-2.el6.i686 (local-py2.7)
               python(abi) = 2.7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Traceback (most recent call last):
  File "/usr/bin/dos-bundle", line 9, in <module>
    load_entry_point('diracos==0.0.1', 'console_scripts', 'dos-bundle')()
  File "/usr/lib/python2.6/site-packages/scripts/bundleDiracOS.py", line 17, in main
    diracoslib.bundleDIRACOS(cfg)
  File "/usr/lib/python2.6/site-packages/diracos/diracoslib.py", line 609, in bundleDIRACOS
    subprocess.check_call(bundleCmd)
  File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mock', '-r', u'/root/DIRACOS/mockConfigs/mock-install-diracos.cfg', '--shell', 'python /tmp/bundlelib.py']' returned non-zero exit status 30

Add ldap

Or not? give a good reason.

It's used by, e.g., for querying the BDII.

Specify number of jobs to build

Mock can be instructed to use N cores to build with

config_opts['macros']['%_smp_mflags'] = "-j4"

it might be worth mentioning this or even better to add the option

dos-build-all-rpms -j4 /tmp/DIRACOS/config/diracos.json

Missing dependencies for ssh and rrdtool

On a raw SLC6 docker image:

[root@a97b0495c41e releases]# rrdtool 
rrdtool: error while loading shared libraries: libfreetype.so.6: cannot open shared object file: No such file or directory
[root@a97b0495c41e releases]# ssh
ssh: error while loading shared libraries: libfipscheck.so.1: cannot open shared object file: No such file or directory

For rrdtool, the issue is that libfreetype is declared as a dependency for building, not for running.
For ssh, it's not declared at all..

Generate a diracosrc

This allows to build the LD_LIBRARY_PATH and PATH at construction time. the bashrc from DIRAC should then just source this one.

Missing files in DIRACOS

Dumping before I go home and forget, nss related dependencies need to be a consistent version and are not currently included in DIRACOS. This causes curl and git to be unable to use SSL. Example:

tar xvf diracos-v1r5.tar.gz
# Doesn't work
singularity exec --bind /cvmfs  /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc -c 'source diracos/diracosrc; curl -v https://github.com > /dev/null'
# Binding /lib64 into the container fixes it
singularity exec --bind /lib64 --bind /cvmfs  /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc -c 'source diracos/diracosrc; curl -v https://github.com > /dev/null'
# Hackily update (found using strace | grep "\"/lib64/")
cp /lib64/libfreebl3.so diracos/lib64/
cp /lib64/libsoftokn3.so diracos/lib64/
cp /lib64/libfreeblpriv3.so diracos/lib64/
cp /lib64/libfreeblpriv3.chk diracos/lib64/
cp /lib64/libfreeblpriv3.so diracos/lib64/
cp /lib64/libsoftokn3.chk diracos/lib64/
cp /lib64/libsoftokn3.so diracos/lib64/
cp /lib64/libnsssysinit.so diracos/lib64/
cp /lib64/libnssdbm3.so diracos/lib64/
cp /lib64/libnssdbm3.chk diracos/lib64/
cp /lib64/libnssdbm3.so diracos/lib64/
# Now works without binding
singularity exec --bind /cvmfs  /cvmfs/cernvm-prod.cern.ch/cvm4/ bash --norc -c 'source diracos/diracosrc; curl -v https://github.com > /dev/null'

fedpkg broken in epel

If you follow the instructions for installation you will currently get an error when executing

$ yum install -y mock rpm-build fedora-packager createrepo python-pip

---> Package policycoreutils.x86_64 0:2.0.83-30.1.el6_8 will be installed
--> Finished Dependency Resolution
Beginning Kernel Module Plugin
Finished Kernel Module Plugin
Error: Package: fedpkg-1.34-3.el6.noarch (epel)
           Requires: python2-openidc-client >= 0.6.0
           Installing: python2-openidc-client-0-3.20170523git77cb3ee.el6.noarch (epel)
               python2-openidc-client = 0-3.20170523git77cb3ee.el6
Error: Package: fedpkg-1.34-3.el6.noarch (epel)
           Requires: python2-openidc-client >= 0.6.0
           Available: python2-openidc-client-0-3.20170523git77cb3ee.el6.noarch (epel)
               python2-openidc-client = 0-3.20170523git77cb3ee.el6
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

This is due to the update of fedpkg from 1.34-1 to 1.34-3. The dependent library python2-openidc-client.noarch 0:0.6.0-1.20180605gitcd8d91c.el6 is still in epel-testing and should be moved moved to epel soon, but according to discussion with IT they don't know when. In the meantime you have to enable epel-testing

yum-config-manager --enable epel-testing

Bump versions 

Dear all

we have released a new version of FTS (3.8.0)

http://cern.ch/go/z8RJ

gfal2 (2.16.0)

http://dmc.web.cern.ch/release/gfal2-2160

and Davix (0.6.9)

http://dmc.web.cern.ch/release/davix-069

They are now available on our RC repos and installed on our FTS pilot nodes.

Apart from bug fixes the main new features are:

- Automatic Session Reuse

http://fts3-docs.web.cern.ch/fts3-docs/docs/session_reuse.html#automatic-session-reuse

- Macaroons and Scitokens support ( thanks to Brian Bockelman)

http://fts3-docs.web.cern.ch/fts3-docs/docs/3rdpartycopy.html#macaroons-and-scitokens-support

We are going to enable them on the pilot by this week so they can be
validated

regards

Andrea

Run DIRAC integration tests in CI

Now that the tests are self contained for GitHub actions it should be simple to add to the GitLab CI pipeline.

I'm happy to implement it but thought I should ask first.

Any thoughts?

Compilation fail: No such file or directory: u'/diracos_repo/x86_64'

I get this error when trying to build the OS

docker run --privileged --cap-add=SYS_ADMIN --oom-kill-disable=true -it cern/slc6-base bash

yum install -y mock rpm-build fedora-packager createrepo python-pip

export DIRACOS_REPO=/cvmfs/clicdp.cern.ch/DIRAC/OS
mkdir -p $DIRACOS_REPO/i386 $DIRACOS_REPO/i686 $DIRACOS_REPO/src $DIRACOS_REPO/noarch $DIRACOS_REPO/x86_64 $DIRACOS_REPO/buildOnly
createrepo $DIRACOS_REPO  

pip install git+https://github.com/DIRACGrid/DIRACOS.git
git clone https://github.com/DIRACGrid/DIRACOS.git

dos-build-all-rpms DIRACOS/config/diracos.json >& log.txt

log.txt

Add doc on python 2.6 

Getting requirements for yum-3.2.29-81.el6.py27.usc4.src
 --> python-2.7.13-2.el6.x86_64
 --> gettext-0.17-18.el6.x86_64
 --> intltool-0.41.0-1.1.el6.noarch
 --> python-nose-0.10.4-3.1.el6.py27.usc4.noarch
 --> python-2.7.13-2.el6.x86_64
 --> rpm-python-4.8.0-59.el6.x86_64
 --> Already installed : rpm-4.8.0-59.el6.x86_64
 --> python-iniparse-0.3.1-2.1.el6.py27.usc4.noarch
 --> python-2.7.13-2.el6.x86_64
 --> python-urlgrabber-3.9.1-11.el6.py27.usc4.noarch
 --> yum-metadata-parser-1.1.2-16.el6.py27.usc4.x86_64
 --> pygpgme-0.1-18.20090824bzr68.el6.py27.usc4.x86_64
Error: Package: rpm-python-4.8.0-59.el6.x86_64 (base)
           Requires: libpython2.6.so.1.0()(64bit)
Error: Package: rpm-python-4.8.0-59.el6.x86_64 (base)
           Requires: python(abi) = 2.6
           Installing: python-2.7.13-2.el6.x86_64 (local-py2.7)
               python(abi) = 2.7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

List of shipped python packages

The list of shipped python packages is not complete. For example, M2Crypto is shipped because it is a dependency of fts3, but it is not mentioned.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.