dmwm / crabclient Goto Github PK
View Code? Open in Web Editor NEWrunrange
runrange
From b4ff0023df79528e8e47f67e7251ba35d88813e9 Mon Sep 17 00:00:00 2001
From Ticket #207
Replying to metson:
* I think exposing tracebacks to the user isn't a good idea
I think before going to users will need some testing and bugs discovery...if a not expected traceback is raised me, Daniele, Eric, whoever want can easily see what's happened. Then, for user this wont be in there!
I saw a nice way around this at PyCon involving several exception and log handlers that gets a very succinct message to the user while still putting the full traceback in the equivalent of the crab.log file.
I'll implement this when I get access to the slides from the talk.
Aim to wrap server negotiation, myproxy, upload of input. This ticket not include workflow creation which depend on CMSSW interaction (which has a dedicated ticket).
The script must implement :
Traceback (most recent call last):
File "/home/bbslave/buildslave/full-python26-mysql/build/test/python/WMCore_t/BossLite_t/DBObjects_t.py", line 594, in testC_LoadObjects
task.load(db)
File "/home/bbslave/buildslave/full-python26-mysql/build/src/python/WMCore/BossLite/DbObjects/Task.py", line 153, in load
result = db.objLoad(self)
File "/home/bbslave/buildslave/full-python26-mysql/build/src/python/WMCore/BossLite/DbObjects/BossLiteDBWM.py", line 202, in objLoad
return self.objAdvancedLoad(obj = obj, binds = binds)
File "/home/bbslave/buildslave/full-python26-mysql/build/src/python/WMCore/BossLite/DbObjects/BossLiteDBWM.py", line 40, in wrapper
raise DbError(msg)
DbError: 'Failure in BossLiteDBWM class(ProgrammingError) (1146, "Table 'testdb.bl_task' doesn't exist") 'SELECT id as id, \n name as name, \n dataset as dataset, \n start_dir as startDirectory, \n output_dir as outputDirectory, \n global_sandbox as globalSandbox, \n cfg_name as cfgName, \n server_name as serverName, \n job_type as jobType, \n total_events as total_events,\n user_proxy as user_proxy, \n outfile_basename as outfileBasename, \n common_requirements as commonRequirements\n FROM bl_task\n WHERE id = 1 ' ()'
-------------------- >> begin captured logging << --------------------
root: DEBUG: Transaction::commit(175468048) had null transaction object
--------------------- >> end captured logging << ---------------------
Support specifying the good lumi list from one of the standard certification team supplied URLs
Split off from #1552.
The protocol is not strictly in the FWJR. All I can do is parse the URI looking for managerv1/managerv2 to choose the protocol. This is done completely client side.
I have a patch ready for this, but I'm hesitating because I seem to recall something in the past with managerv1/managerv2 not always corresponding to srmv1/srmv2.
Implementing the user input sandbox upload into the user sandbox cache component through the CRABServerInterface.
REST responses will need to be translated into a more user friendly message for users. Could use some kind of online dictionary to make adding new messages simpler?
Alternatively we translate DAS calls in the Server. This has the advantage of the client being simple and the messages being centrally updateable, but potentially puts significant load on the Server (e.g. it would serve DAS-style data to itself, process it, then serve to the client).
Depends on #44 being done.
Clarify authentication (closes #137) and pythonise the stg output parsing (closes #138)
Create a ticket if one isn't specified
Based on second round of feedbacks
Precursor to fixing #1409. This adds the capability to upload to the UserTarball class.
The StorageSite parameter is needed to propagate the storage site of user outputs to the AsyncStageOut.
Unless I missed it there is currently no way to communicate these things to the server and then on to the WMSpec.
Change default config name + remove dest storage + typo
The point is that the exceptions should in the majority of cases come from the server (e.g. if you can talk to the server and get a response any exception will be an HTTP error from the server). I don't think you need to wrap that further. Also, exceptions aren't that useful in the client - as you say, if one is raised it's basically going to print a message and exit, there's not going to be much beyond that in terms of exception handling - so having a bunch of specific exceptions is somewhat code bloat.
The CredentialInteractions is trapping and re-raising generic exceptions. As we pointed out over the chat there is no valid reason to do that...
We need to make this part of the code definitely more robust.
123
Implementing the server info command that will return information about the server. Starting with first round of needed information.
Use the api provided in ticket #1551 to do the checksum of the file transferred and the protocol type for the transfer.
We need to upload the user sandbox to CRABInterface and retrieve the hash string for the sandbox
Implemented in #3374
client must be able to interact with CMSSW user local area, discover which code to use and pack it.
This is what basically correspond to crab2 creation.
Handling user role
dumpConfig does the old style config and hasn't been used in years.
Please review
Checking python version plus config adjustment
CRAB Client minor minor fix lost in last patches
Per #214, validate the user supplied parameters for the CMSSW JobType at the Submit stage.
Include unit tests.
currently we depend on python 2.6. If the cmsenv is set it came for free but shouldn't we be independent from CMSSW?
This is going to be needed for the checksumming. There are two functions in this file for calculating checksums.
Please review. This patch adds the functionality to upload the CMSSW cfg.py file to the server and retrieves the Couch DocID which is placed in the task definition.
Client side for ticket #1549
Adding configurable parameters
Few fixes + minor changes
Implement client functionality to get back log files.
-- Since the functionality should be exposed to the user I think the log to get back should be just the executable one (e.g. CMSSW.log, FJR..). Do we should take care to report to the user the Wrapper's log? or this latter should be needed just to the Ops for debugging ?
If there is no handler on the logger, the user will see a message to this effect. It's confusing, so add a null handler to the logger. Later a handler that writes to the crab.log file also gets added.
This can be simplified when we migrate to python 2.7
Eventually aim to support this
https://hypernews.cern.ch/HyperNews/CMS/get/generators/1036.html
https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideEWKUtilities#How_to_compile_link_run
Could be CRABServer or core, dunno. In any case, it looks like a minor tweak to the CMSSW setup and the ability to download files (via squid, I would hope) would allow us to run these jobs in an automated way.
Based on the workflow crab3 is going to propose, if user wants output back to its workstation a copy from the storage is needed.
The client should wrap this. This wrapper should probably integrate some kind of API (SE API) which will be in some way related to #681.
Aiming at having something soon I guess we can agree to start with a basic "wrapping" and evolve soon later to the final approach (again: based on the way the #681 is going to evolve).
As subject.
Implement client functionality to get back output files.
It is not needed to develop something like SEAPI, but something more light that just copy back files and does a checksum; starting with lcg-cp then if needed adding others (srm-cp); implementation can be something like this dummy example:
def copyLcg(input, output):
print 'copy the file: from ', input, ' to the ', output
print 'if succeeded check the checksum in ', output
def copySrm(input, output):
pass
copymap = {
'lcg': copyLcg,
'srm': copySrm
}
copymap'lcg'
Check if pset exists + typo in config template
An example with the status command
[lxplus403] ~/scratch0/TEST_RELEASE/CLIENT_CRAB3 $ crab status
TypeError: 'NoneType' object is unsubscriptable See logfile for more info
First fixes
Input from Simon attached.
Add to the status the possibility to have number of jobs per status.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.