Error handling in CMOR 3.1 about cmor

[ehogan]

Possibly related to this, in the CMIP6 example the following command is used:

error_flag = cmor.dataset_json("Test/test_python_CMIP6_experimentID.json")

In the CMOR2 User Guide, both Fortran and C return 0 upon success, whereas Python returns None. Is there a way for the exit code to be returned to pywrapper.py in the same way as Fortran and C? I understand there are cases where CMOR should explicitly fail, but if the underlying function returns the exit code rather than exiting itself, couldn’t the code in pywrapper.py check the exit code and then raise an exception / return the correct exit code, depending on what you want CMOR3 to do in each case?

[dnadeau4]

• Example 1 I created this message for you. I return as soon as I detect that you did not call cmor.setup(). cmor.setup() set all memory to 0, so it is very important
• Example 2 I just check the code and fclose() is called only by cmor_close(), I will look at change this behavior so that the file is open/close for each error. This should not be too difficult.
• No exit to python, not sure, I am sending a SIGINT signal, I tried SIGTERM, but you get the "Abort" message on red hat. I tried other signal, but I need to program to stop, already, I have noticed that python will run the function and only send the signal when all functions have ``return`(ed). That creates weird behavior. I just did not want to "kill -9" the process the way it used to do.
• I am not sure what you mean by (3). I do not have control of the "Keyboard Interrupt" message after the SIGINT signal has been sent. I can look, but that seems hard.
• I think returning (0) instead of "None" in python is a great idea. It will help my unittest a lot. I'll try to do this next week.

[ehogan]

Example 1: I created this message for you. I return as soon as I detect that you did not call cmor.setup(). cmor.setup() set all memory to 0, so it is very important

No exit to python, not sure, I am sending a SIGINT signal, I tried SIGTERM, but you get the "Abort" message on red hat. I tried other signal, but I need to program to stop, already, I have noticed that python will run the function and only send the signal when all functions have `return(ed). That creates weird behavior. I just did not want to "kill -9" the process the way it used to do.

The Python interpreter is terminated when CMOR detects that cmor.setup() was not called because an exit(1) is used (https://github.com/PCMDI/cmor/blob/master/Src/cmor.c#L598) rather than SIGINT. Is there any chance that a SIGINT signal can be sent in this case, so that the Python interpreter is not terminated?

Example 2: I just check the code and fclose() is called only by cmor_close(), I will look at change this behavior so that the file is open/close for each error. This should not be too difficult.

Great, thank you! :)

I am not sure what you mean by (3). I do not have control of the "Keyboard Interrupt" message after the SIGINT signal has been sent. I can look, but that seems hard.

I haven't had much luck with figuring out if there is a way to send the log message to the exception :( I did find a way to change the type of exception though :) The following changes can be made to https://github.com/PCMDI/cmor/blob/master/Lib/pywrapper.py#L5 to raise a specific exception when a SIGINT signal is caught:

class CMORError(Exception):
    def __init__(self, message):
        super(CMORError, self).__init__(message)

def sig_handler(signum, frame):
    message = 'Problem with CMOR. Please check the logfile (if defined).'
    raise CMORError(message)

signal.signal(signal.SIGINT, sig_handler)

I ran some quick tests and this seems to work for me (i.e., I can catch all CMORError exceptions). I think this would be better than a KeyboardInterrupt exception :)

I think returning (0) instead of "None" in python is a great idea. It will help my unittest a lot. I'll try to do this next week.

Great, thank you! :)

[ehogan]

I haven't had much luck with figuring out if there is a way to send the log message to the exception :(

Unless @doutriaux1 comment can help?

[dnadeau4]

Example 1: I created this message for you. I return as soon as I detect that you did not call cmor.setup(). cmor.setup() set all memory to 0, so it is very important

No exit to python, not sure, I am sending a SIGINT signal, I tried SIGTERM, but you get the "Abort" message on red hat. I tried other signal, but I need to program to stop, already, I have noticed that python will run the function and only send the signal when all functions have `return(ed). That creates weird behavior. I just did not want to "kill -9" the process the way it used to do.

The Python interpreter is terminated when CMOR detects that cmor.setup() was not called because an exit(1) is used (https://github.com/PCMDI/cmor/blob/master/Src/cmor.c#L598) rather than SIGINT. Is there any chance that a SIGINT signal can be sent in this case, so that the Python interpreter is not terminated?

That is right, I decided to call exit() since there will be no log, cmor.setup() was not called. Without it, no log file is opened to write messages. I just abandon boat and print this message to the user. If I send a SIGINT, all functions need to return which creates a lot of unnecessarily and confusing messages. cmor.setup() is a must and have to be called before anything. Think of it like a __init__() in Python.

Example 2: I just check the code and fclose() is called only by cmor_close(), I will look at change this behavior so that the file is open/close for each error. This should not be too difficult.

Great, thank you! :)

I am not sure what you mean by (3). I do not have control of the "Keyboard Interrupt" message after the SIGINT signal has been sent. I can look, but that seems hard.

I haven't had much luck with figuring out if there is a way to send the log message to the exception :( I did find a way to change the type of exception though :) The following changes can be made to https://github.com/PCMDI/cmor/blob/master/Lib/pywrapper.py#L5 to raise a specific exception when a SIGINT signal is caught:

class CMORError(Exception):
    def __init__(self, message):
        super(CMORError, self).__init__(message)
   def sig_handler(signum, frame):
       message = 'Problem with CMOR. Please check the logfile (if defined).'
       raise CMORError(message)
   signal.signal(signal.SIGINT, sig_handler)

I ran some quick tests and this seems to work for me (i.e., I can catch all CMORError exceptions). I think this would be better than a KeyboardInterrupt exception :)

I am fine with the Keyboard Interrupt, if CMOR3 user input file is fine, we won't have a critical error.

At this moment, I am writing the "CMIP6 Validator" using the Control Vocabulary, but I'll try it when I get a chance.

I think returning (0) instead of "None" in python is a great idea. It will help my unittest a lot. I'll try to do this next week.

Great, thank you! :)

[ehogan]

Closing in favour of #86 and #87.