Migration of usercode repository of larger team about cmssw

[Dr15Jones]

Hi Gerrit,

Giulio wrote the FAQ and I'm afraid he's on vacation so I'll do my best to interpret his intent.
On Aug 1, 2013, at 8:52 AM, gvonsem [email protected] wrote:

Our team in Brussels (about 15 members) has currently a common repository on cvs, UserCode/TopBrussels/, and we would like to migrate this to GIT. In the usercode migration instructions (http://cms-sw.github.io/cmssw/usercode-faq.html), it is mentioned that an 'organization' is more useful for a repository for larger groups (> 5 people). Do you think such an organization for our team can be created, or do you still recommend to move it to a personal repository (since we are not representing a POG/PAG, only an institute team)?
From the github site's description of an organization repository, it seems to me like a perfect match for you. It seems to me that the difference between an personal github account and an organization is the organization makes it possible to assign different permissions to different people. Also, a personal github is supposed to be tied to one person while an organization allows multiple administrators.

For instance, it is not clear to me if code of such an 'official' CMS organization would become part of CMSSW, or if it stays outside of CMSSW like was the case on cvs.
It would stay outside of CMSSW. Only code in cms-sw/cmssw repository is considered to be in CMSSW.

Chris

Christopher Jones
Fermi National Accelerator Laboratory
[email protected]

[gvonsem]

Dear Chris,

Many thanks for your reply. Do you know what the following steps are to create such an 'organization'? The FAQ mentions it should be approved, but I don't know if this also means that you (or someone else) should create such an organization, and who takes care of the actual migration of the repository.

Best regards,
Gerrit

[Dr15Jones]

Hi Gerrit,

On Aug 1, 2013, at 11:46 AM, gvonsem [email protected] wrote:

Many thanks for your reply. Do you know what the following steps are to create such an 'organization'? The FAQ mentions it should be approved, but I don't know if this also means that you (or someone else) should create such an organization, and who takes care of the actual migration of the repository.

The FAQ points to the github documentation about organizations
https://github.com/blog/674-introducing-organizations

As for 'approval', I interpreted that as only if this is part of the 'official' CMS code (e.g. CMSSW or computing's workflow tools). Yours is an 'unofficial CMS group' and I see no reason for you to need approval (just like old UserCode didn't need approval). Therefore you can make the organization yourself and do the migration.

Chris

Christopher Jones
Fermi National Accelerator Laboratory
[email protected]

[ktf]

Indeed, the idea is to try to give opt-in centralized support for things which are more official than someone's private UserCode areas. I've rephrased the FAQ to be hopefully more clear.

http://cms-sw.github.io/cmssw/usercode-faq.html#i_need_to_collaborate_with_some_fellow_physicists

[ktf]

Please close the issue if you are happy with the reply.

[gvonsem]

Thanks for the replies. I made an organization, but now I have a technical problem in the migration of the repository. I follow the steps on http://cms-sw.github.io/cmssw/usercode-faq.html to try to get UserCode/TopBrussels/TopTreeAnalysis into a repository. At the last step (git push --mirror -u origin), I get some warnings about some files that are too big, but also errors (*). I am particularly worried about the messages like:

! [remote rejected] CMSSW_53X -> CMSSW_53X (pre-receive hook declined)

What do they mean and how can this be solved? And is there a way to skip the migration of the files that are too large?
I actually succeeded in migrating a similar package UserCode/TopBrussels/TopTreeProducer in our repository without problems, so I am puzzled why it doesn't work for UserCode/TopBrussels/TopTreeAnalysis.

Many thanks for your help,

Gerrit

(*)
-bash-4.1$ git push origin master
git: /afs/cern.ch/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_11/external/slc6_amd64_gcc472/lib/libz.so.1: no version information available (required by git)
git: /afs/cern.ch/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_11/external/slc6_amd64_gcc472/lib/libz.so.1: no version information available (required by git)
Counting objects: 9, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (9/9), 2.30 KiB, done.
Total 9 (delta 4), reused 9 (delta 4)
To [email protected]:TopBrussels/TopTreeAnalysis.git

• [new branch] master -> master
  -bash-4.1$ git push --mirror -u origin
  git: /afs/cern.ch/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_11/external/slc6_amd64_gcc472/lib/libz.so.1: no version information available (required by git)
  git: /afs/cern.ch/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_11/external/slc6_amd64_gcc472/lib/libz.so.1: no version information available (required by git)
  Counting objects: 7183, done.
  Delta compression using up to 8 threads.
  Compressing objects: 100% (2282/2282), done.
  Writing objects: 100% (7183/7183), 233.12 MiB | 3.39 MiB/s, done.
  Total 7183 (delta 4861), reused 7182 (delta 4861)
  remote: Error code: 627cc90820bc5da1a55845440c0e21c9
  remote: warning: Error GH413: Large files detected.
  remote: warning: See http://git.io/iEPt8g for more information.
  remote: warning: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_1_zjets.txt is 63.15 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_2_data.txt is 78.56 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_1_zjets.txt is 63.13 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_2_data.txt is 78.53 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_2_data.txt is 80.33 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_1_zjets.txt is 54.99 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_1_zjets.txt is 55.07 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_1_zjets.txt is 63.14 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_1_zjets.txt is 63.12 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_2_data.txt is 63.37 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: warning: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_2_data.txt is 62.45 MB; this is larger than GitHub's recommended maximum file size of 50 MB
  remote: error: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_1_data.txt is 108.99 MB; this exceeds GitHub's file size limit of 100 MB
  remote: error: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_1_data.txt is 108.94 MB; this exceeds GitHub's file size limit of 100 MB
  remote: error: File Isis-SingleTop-tW/information/lepveto_run_lumi_event_1_data.txt is 108.92 MB; this exceeds GitHub's file size limit of 100 MB
  remote: error: File Isis-SingleTop-tW/information/lepsel_info_run_lumi_event_1_data.txt is 108.97 MB; this exceeds GitHub's file size limit of 100 MB
  To [email protected]:TopBrussels/TopTreeAnalysis.git
  ! [remote rejected] CMSSW_42X -> CMSSW_42X (pre-receive hook declined)
  ! [remote rejected] CMSSW_52X -> CMSSW_52X (pre-receive hook declined)
  ! [remote rejected] CMSSW_53X -> CMSSW_53X (pre-receive hook declined)
  ! [remote rejected] origin/master -> origin/master (pre-receive hook declined)
  Branch master set up to track remote branch master from origin.
  error: failed to push some refs to '[email protected]:TopBrussels/TopTreeAnalysis.git'

[davidlt]

@gvonsem these are extremely big plain-text data files. The limit is 50MB per file I believe. You need to compress them (gzip, bzip2, xz) and change the code to programmatically read compressed files.

[gvonsem]

Ok, we will make sure to make these files smaller. But do you think this is what also causes the error
! [remote rejected] CMSSW_53X -> CMSSW_53X (pre-receive hook declined) ?

[ktf]

@davidlt that's not enough, you need to actually exclude them from the migration, otherwise they'll be in somewhere in the history.

@gvonsem I suggest you rsync your usercode repository locally, remove those files and do the conversion. Then you can compress them as David suggests and add them back in the branches where you need them.

[gvonsem]

Hi again,
I was wondering if someone could help; I tried to remove the large files from cvs, by doing 'cvs remove', but as you may know this just moves the files to the Attic directory, and does not delete them from the history. As a consequence the migration to GIT still does not work. Do you know how to remove the files completely from CVS, or to exclude one specific directory in the conversion?

[gvonsem]

As far as I understand from some googling research, there is no way to remove files from the Attic (and thus from the history) if you don't have admin right over the CVS repository. If I try to remove a file in /afs/cern.ch/project/cvs/reps/CMSSW/UserCode/TopBrussels/TopTreeAnalysis/Isis-SingleTop-tW/information/Attic, I obviously get 'permission denied' (as should be, otherwise everyone could just remove files from everyone). Do you know who I should contact to remove files from the Attic, and if this is the way to go or not?
Thanks!

[gvonsem]

Deleting the problematic files from the local GIT repository before pushing could have been a solution as well (https://help.github.com/articles/working-with-large-files), but in the end Giulio just deleted the files on cvs. The migration works now and I'm closing this issue.