From b4ff0023df79528e8e47f67e7251ba35d88813e9 Mon Sep 17 00:00:00 2001

From Ticket #207

Replying to metson:

* I think exposing tracebacks to the user isn't a good idea

I think before going to users will need some testing and bugs discovery...if a not expected traceback is raised me, Daniele, Eric, whoever want can easily see what's happened. Then, for user this wont be in there!

I saw a nice way around this at PyCon involving several exception and log handlers that gets a very succinct message to the user while still putting the full traceback in the equivalent of the crab.log file.

I'll implement this when I get access to the slides from the talk.

Aim to wrap server negotiation, myproxy, upload of input. This ticket not include workflow creation which depend on CMSSW interaction (which has a dedicated ticket).
The script must implement :

• user code (default.tgz for those who are familiar with CRAB2) upload
• user cfg upload
• cmssw pset upload (config cache interaction)
• check status
• output retrieval

ERROR: testC_databaseIsPersistent

Traceback (most recent call last):
File "/home/bbslave/buildslave/full-python26-mysql/build/test/python/WMCore_t/BossLite_t/DBObjects_t.py", line 594, in testC_LoadObjects
task.load(db)
File "/home/bbslave/buildslave/full-python26-mysql/build/src/python/WMCore/BossLite/DbObjects/Task.py", line 153, in load
result = db.objLoad(self)
File "/home/bbslave/buildslave/full-python26-mysql/build/src/python/WMCore/BossLite/DbObjects/BossLiteDBWM.py", line 202, in objLoad
return self.objAdvancedLoad(obj = obj, binds = binds)
File "/home/bbslave/buildslave/full-python26-mysql/build/src/python/WMCore/BossLite/DbObjects/BossLiteDBWM.py", line 40, in wrapper
raise DbError(msg)
DbError: 'Failure in BossLiteDBWM class(ProgrammingError) (1146, "Table 'testdb.bl_task' doesn't exist") 'SELECT id as id, \n name as name, \n dataset as dataset, \n start_dir as startDirectory, \n output_dir as outputDirectory, \n global_sandbox as globalSandbox, \n cfg_name as cfgName, \n server_name as serverName, \n job_type as jobType, \n total_events as total_events,\n user_proxy as user_proxy, \n outfile_basename as outfileBasename, \n common_requirements as commonRequirements\n FROM bl_task\n WHERE id = 1 ' ()'
-------------------- >> begin captured logging << --------------------
root: DEBUG: Transaction::commit(175468048) had null transaction object
--------------------- >> end captured logging << ---------------------

Support specifying the good lumi list from one of the standard certification team supplied URLs

Split off from #1552.

The protocol is not strictly in the FWJR. All I can do is parse the URI looking for managerv1/managerv2 to choose the protocol. This is done completely client side.

I have a patch ready for this, but I'm hesitating because I seem to recall something in the past with managerv1/managerv2 not always corresponding to srmv1/srmv2.

Implementing the user input sandbox upload into the user sandbox cache component through the CRABServerInterface.

From Simon comment:

Medium term; evaluate if other jobtypes are needed, try to minimise the number used into a few abstract classes (e.g. upload_config_and_input etc. rather than a 1:1 mapping with the Workflows/specs available in the server)

Longer term; look into downloadable plugins from the server (probably another ticket is needed for that...)

REST responses will need to be translated into a more user friendly message for users. Could use some kind of online dictionary to make adding new messages simpler?

Alternatively we translate DAS calls in the Server. This has the advantage of the client being simple and the messages being centrally updateable, but potentially puts significant load on the Server (e.g. it would serve DAS-style data to itself, process it, then serve to the client).

Depends on #44 being done.

Clarify authentication (closes #137) and pythonise the stg output parsing (closes #138)
Create a ticket if one isn't specified

Based on second round of feedbacks

Precursor to fixing #1409. This adds the capability to upload to the UserTarball class.

The StorageSite parameter is needed to propagate the storage site of user outputs to the AsyncStageOut.

Unless I missed it there is currently no way to communicate these things to the server and then on to the WMSpec.

Change default config name + remove dest storage + typo

From Simon comment:

The point is that the exceptions should in the majority of cases come from the server (e.g. if you can talk to the server and get a response any exception will be an HTTP error from the server). I don't think you need to wrap that further. Also, exceptions aren't that useful in the client - as you say, if one is raised it's basically going to print a message and exit, there's not going to be much beyond that in terms of exception handling - so having a bunch of specific exceptions is somewhat code bloat.

Basically I'd see the first version of the client have no exceptions (e.g. just use ones from pythons standard library. ValueError?, HTTPError etc.) and only add specific exceptions in very complicated, rare cases. I wouldn't be surprised if the final client had no specialised exceptions in either...

The CredentialInteractions is trapping and re-raising generic exceptions. As we pointed out over the chat there is no valid reason to do that...
We need to make this part of the code definitely more robust.

123

Implementing the server info command that will return information about the server. Starting with first round of needed information.

Use the api provided in ticket #1551 to do the checksum of the file transferred and the protocol type for the transfer.

We need to upload the user sandbox to CRABInterface and retrieve the hash string for the sandbox

Implemented in #3374

client must be able to interact with CMSSW user local area, discover which code to use and pack it.
This is what basically correspond to crab2 creation.

Handling user role

dumpConfig does the old style config and hasn't been used in years.

Please review

Checking python version plus config adjustment

CRAB Client minor minor fix lost in last patches

Per #214, validate the user supplied parameters for the CMSSW JobType at the Submit stage.

Include unit tests.

currently we depend on python 2.6. If the cmsenv is set it came for free but shouldn't we be independent from CMSSW?

This is going to be needed for the checksumming. There are two functions in this file for calculating checksums.

Please review. This patch adds the functionality to upload the CMSSW cfg.py file to the server and retrieves the Couch DocID which is placed in the task definition.

Client side for ticket #1549

Adding configurable parameters

Few fixes + minor changes

Implement client functionality to get back log files.
-- Since the functionality should be exposed to the user I think the log to get back should be just the executable one (e.g. CMSSW.log, FJR..). Do we should take care to report to the user the Wrapper's log? or this latter should be needed just to the Ops for debugging ?

If there is no handler on the logger, the user will see a message to this effect. It's confusing, so add a null handler to the logger. Later a handler that writes to the crab.log file also gets added.

This can be simplified when we migrate to python 2.7

Eventually aim to support this

https://hypernews.cern.ch/HyperNews/CMS/get/generators/1036.html

https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideEWKUtilities#How_to_compile_link_run

Could be CRABServer or core, dunno. In any case, it looks like a minor tweak to the CMSSW setup and the ability to download files (via squid, I would hope) would allow us to run these jobs in an automated way.

Based on the workflow crab3 is going to propose, if user wants output back to its workstation a copy from the storage is needed.
The client should wrap this. This wrapper should probably integrate some kind of API (SE API) which will be in some way related to #681.

Aiming at having something soon I guess we can agree to start with a basic "wrapping" and evolve soon later to the final approach (again: based on the way the #681 is going to evolve).

As subject.

Implement client functionality to get back output files.
It is not needed to develop something like SEAPI, but something more light that just copy back files and does a checksum; starting with lcg-cp then if needed adding others (srm-cp); implementation can be something like this dummy example:

def copyLcg(input, output):
print 'copy the file: from ', input, ' to the ', output
print 'if succeeded check the checksum in ', output

def copySrm(input, output):
pass

copymap = {
'lcg': copyLcg,
'srm': copySrm
}

copymap'lcg'

Check if pset exists + typo in config template

An example with the status command

[lxplus403] ~/scratch0/TEST_RELEASE/CLIENT_CRAB3 $ crab status
TypeError: 'NoneType' object is unsubscriptable See logfile for more info

First fixes

Input from Simon attached.

Add to the status the possibility to have number of jobs per status.