Tier -0 deploy/manage and RPMs needed for patch review about t0

[vkuznet]

valya: I added a prototype of t0.spec for RPMs. Since there is no yet T0 tag in SVN, I can't test it properly. I also don't have enough info about T0 dependencies, and only put python and python-sphinx. The build steps I used as in DAS, which relies on WMCore, builds sphinx docs, etc.

Dirk please add list of dependencies, create a tag (it doesn't matter if it is valid or not) and I can test the spec file.

The deploy/manage scripts I can develop once I know details how T0 will be installed (is it going to cmsweb or will be deployed outside of it, etc.). Dirk if you can add details about deployment it will be clear to me how to organize deploy/manage scripts.

[hufnagel]

hufnagel: WMCore should not be checked out in the spec file. T0 should use WMCore, not include it. As such, there should be wmcore dependencies (core and oracle is good enough for now). The wmcore packages should then pull in all other dependencies, T0 needs nothing additional AFAIK.

Eventually T0 will go to cmsweb, but not in the beginning, so there needs to be a way to set it up on a standard vobox in a directory of your choice.

Tried creating a tag, but couldn't figure out a way to push a git tag to SVN. How do I do that ? Would be good to have some tagging instructions on the wiki.

[vkuznet]

valya: Ok, I"ll work out with WMCore dependencies.

Regarding tagging SVN with git, you can use the following command: git svn tag X.Y.Z

For more info: git svn help

[hufnagel]

hufnagel: Had to ask for a git upgrade on the Tie0development machines, version was too old and did not support the "git svn tag" syntax.

Please use tag 1.9.0

[vkuznet]

valya: Please review newly attached t0.spec. I add dependency on WMCore, commented out not yet ready fields and was able to successfully build T0 RPM.

Please note, so far T0 code does not provide standard python setup.py file to perform its build and installation. I think it should do that asap. All CMS python packages, including WMCore, provide such file. It is a standard way to build python packages and encouraged by HTTP to handle CMS python packages. The WMCore/setup.py file is a comprehensive example and probably you need to look at more concise DAS setup.py. Both can be found in TRAC under browse code links with appropriate package. If you want me to create one I can do that too (probably need another ticket for that).

Valentin.

[vkuznet]

valya: Dirk did you have chance to look at submitted changes. I can revisit this activity.

[vkuznet]

valya: I was able to build T0 RPMs, here is the RPM content (I cut if off):

{{{
valya@vocms106(21:20:43)> rpm -qpliR w/RPMS/slc5_amd64_gcc461/cms+t0+1.0.0-1-1.slc5_amd64_gcc461.rpm
Name : cms+t0+1.0.0 Relocations: /build/vk/build_stuff/w
Version : 1 Vendor: CMS
Release : 1 Build Date: Mon 06 Feb 2012 09:17:17 PM CET
Install Date: (not installed) Build Host: vocms106.cern.ch
Group : cms Source RPM: cms+t0+1.0.0-1-1.src.rpm
Size : 327440 License: "As required by the orginal provider of the software."
Signature : (none)
Packager : CMS [email protected]
Summary : CMS Experiment package SpecChecksum:b4eae72be6c9f2884248bc2d5dc64550
Description :
No description
/bin/sh
/bin/sh
/bin/sh
cms+wmcore+0.8.26pre1-comp
external+bz2lib+1.0.5-comp3
external+curl+7.21.6-comp2
external+db4+4.4.20-comp3
external+expat+2.0.1-comp2
external+gcc+4.6.1-comp3
external+gdbm+1.8.3-comp2
external+openssl+0.9.8e-comp3
external+py2-docutils+0.7-comp2
external+py2-httplib2+0.7.1-comp3
external+py2-jinja+2.5.2-comp2
external+py2-pycurl+7.19.0-comp2
external+py2-pygments+1.3.1-comp2
external+py2-setuptools+0.6c9-comp3
external+py2-simplejson+1.9.2-comp3
external+py2-sphinx+1.0.4-comp2
external+py2-sqlalchemy+0.5.2-comp3
external+python+2.6.4-comp3
external+sqlite+3.7.10
external+zlib+1.2.3-comp3
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/etc
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/etc/profile.d
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/etc/profile.d/dependencies-setup.csh
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/etc/profile.d/dependencies-setup.sh
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/etc/profile.d/init.csh
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/etc/profile.d/init.sh
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/lib
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/lib/python2.6
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/lib/python2.6/site-packages
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/lib/python2.6/site-packages/T0
/build/vk/build_stuff/w/slc5_amd64_gcc461/cms/t0/1.0.0/lib/python2.6/site-packages/T0/JobSplitting
.....
}}}

Final piece is version of WMCore which should be used for T0. Once this will be known I can build/upload RPMs into COMP repository.

[hufnagel]

hufnagel: With the changes I just committed for the Tier0Feeder, you should use WMCore 0.8.26pre1. There is still an outstanding WMCore bug there, but it only affects job submission, which is kind of irrelevant as there is also still a bug in passing Tie0created workloads in the JobCreator.

So if you make a new SVN tag of T0 and use WMCore 0.8.26pre1 you could build some rpms and upload them. That's good enough for now and it's certainly good enough to work out deployment issues.

[vkuznet]

valya: Done, new tag is 1.0.1, new RPM is cms+t0+1.0.1. You can try it using standard cmsweb deployment procedure.

[hufnagel]

hufnagel: Testing a first version of the Tie0deployment scripts which are basically a much simplified and slightly modified version of the WMAgent standalone deployment scripts. Fails with

++ apt-get -y install cms+t0+1.0.1
Reading Package Lists...
Building Dependency Tree...
E: Couldn't find package cms+t0+1.0.1

Are you sure you uploaded the rpm ? My scripts work because if I replace the package name with cms+wmagent and the version with 0.8.26pre3, everything works (well, until it gets to the customization parts that rely on the Tie0code).

[vkuznet]

valya: Dirk,
according to cmsweb rules, developers never upload RPMs into official COMP repository. This is done by Diego, once they produce production tag. Instead RPMs can be uploaded into comp.pre repository. Status of all packages available over there can be found here https://twiki.cern.ch/twiki/bin/viewauth/CMS/DMWMBuildsStatusPreReleases. The cms+t0+1.0.1 RPM is there.

You can test this RPM by following instruction from DMWM build page: https://twiki.cern.ch/twiki/bin/view/CMS/DMWMBuilderAgent#Testing_the_Pre_Released_RPMs

Basically you need to bootstrap your area using comp.pre repository, e.g.
sh ./bootstrap.sh -architecture $SCRAM_ARCH -path $PWD -repository comp.pre setup
and then you apt-install will find appropriate RPM.

[hufnagel]

hufnagel: Tie0is not cmsweb, neither is WMAgent. I doubt WMAgent follows these rules. But it's no problem to do a standalone deployment from comp.pre. With that I get past these problems, only to run into the next.

Basically, the dependency list of the cms+t0 package is not complete. I guess cms+wmcore does not pull in everything we need. We should switch to cms+wmagent.

[vkuznet]

valya: It is not about cmsweb or wmagent, it is more about keeping common repository clean. If everyone will throw untested stuff into COMP it will quickly become messy. So rules are more common sense rather then specific to system. comp.pre is perfectly fine for testing. It does not break anything and follows the same rules as COMP installation.

Regardless of cms+wmcore. Since many packages in COMP uses different versions of WMCore I followed the same rules as we applied to any other system, i.e. WMCore built in into T0 RPM. Once you install RPM it will install the version of WMCore you requested, the environment will be accordingly set, etc.

Could you please be more specific what is missing? I don't care if we add cms+wmagent. It is more a question to verify what is needed rather than actual constraint.

[hufnagel]

hufnagel: The errors I saw were about couch and some slqalchemy mysql stuff. As the Tie0is based on the WMAgent, to first order anything needed by the WMAgent is also needed by the Tie0 I really do not want to go through a cycle of finding out a new list of dependencies, because even if the list for the Tie0would be smaller now, in principle anything needed in the WMAgent could become needed in the Tie0at some future time as/when we expand the Tie0features.

Therefore best would be to install the wmagent rpm as a dependency. That also insures we stay in sync with WMAgent deployments.

I tried that already (see version 1.6 of t0.spec), but it does not build for some reason and the instructions how to check the build logs seem to be outdated, vocms82 does not allow logins.

Feels free to take a look and make changes.

[hufnagel]

hufnagel: Alternative to include wmagent as dependency is to go through the wmagent spec file and spec files for any wmcore related dependencies of wmagent and get a list of externals they rely on. I am not sure that is a good idea in the long run though because it adds maintenance overhead. Too easy to get out of sync later again.

[vkuznet]

valya: The vocms82 has been retired and all builds are moved to vocms106. Diego must update twiki. Apart from the node change I think nothing change in procedure though.

Now, I fixed t0.spec where the source for its tarball had wrong number.

I've added wmagent deps which mostly wmcore components (db-couch, db-mysql, db-oracle, webtools) and dbs/dbs3/dls clients. I'm not sure if T0 needs the later. But for time being I left t0 spec with a single dependency, wmagent.

New RPM is ready in comp.pre repository. Please try it out.

[hufnagel]

hufnagel: Gave it a try, everything works. Closing this, modifications can be tracked in other tickets. The deploy/manage scripts for tie0are tracked in #3091