Server Error - no save file & empty dataframe about adobe-analytics-api-2.0

[hayden-blair]

UPDATE:

Removing the "if lastPage not in report:" safegaurd allowed my request to reach and execute the "if errorCode in report.keys()" statement. The error message that was printed allowed me to diagnose the problem as an issue with the rsid I was trying to access, and I have now resolved my current problem.

Thanks for your help!

[loldenburg]

The instability of the Adobe API Response (1 in 6 responses is empty) was escalated to Adobe.

[hayden-blair]

Definitely! Let me give you what I know:

• I had an error in my request
• The error was I was trying to access a report suite that I did not have permissions for
• The error message your module gave me was from the lastPage safeguard "Server Error - no save file & empty dataframe"
• Once I bypassed the safeguard, your module gave me the more helpful error message:

"Error with your statement
User does not have access to this report suite - Error caused by exceptionMessage=User does not have access to this report suite"

which was generated by printing: report['errorDescription']

-This improved error message allowed me to diagnose the problem correctly

[pitchmuc]

This enhancement is now available with the new release (v 0.2.3 -> c2e94d6).
I am closing the issue.
Let see if we need to re-open it.

[loldenburg]

When I use verbose, the error is:

{
"error": [
"Request Error"
]
}

I get only the header columns with no data in the obj that is returned.

[pitchmuc]

Hello Lukas,

Yeah, I had to deal with that instability but normally the engineer were supposed to fix this.

Context
This happens when there is no "lastPage" key in the object return by the Server.
It should be as described in the response format here : https://adobedocs.github.io/analytics-2.0-apis/#/reports/runReport

This information allows me to know if I should start to loop for the results or not, and also when to stop.
I was having 2 problems from it:

• empty data returned
• infinite loop

If Savik wants to look at it, it is at line 1035 in the aanalytics.py file : https://github.com/pitchmuc/adobe_analytics_api_2.0/blob/9572079afdb5b8191c0e4460cb00561bc1ec38ba/aanalytics2/aanalytics2.py#L1035-L1048

Current Problem
As said before, when I encountered this issue, there was no data returned at all. So there is an empty dataframe returns in the "data" key. You can add the debug=True parameter that will create some files (request and response) to verify the response.

I think the problem comes from the fact that you try to use that data to merge it later on and because it is an empty dataframe, it doesn't merge correctly:
My guess comes from this message : "Index 0 is out of bounds for axis 0 with size 0"

Next steps
In case I am into something, you can try to check if the dataframe is empty (df.empty == False) before merging it.

Other
Could you try to also re-run it with the debug mode and let me know if there are some data returned in the response file ?
I am not particularly happy with the way I handle it, I may need to change the logic.

Also, if there is no data (as assumed), please open a Support Ticket and send them both files.

btw :
You did good to add the throttling (time.sleep) in your loop.
Within one report request (eg. ags.getReport(aa_req_click2order, n_result='1000')) I take care of the throttling myself with my internal loop but it may be an issue between 2 calls.

[pitchmuc]

Also the fact that you receive this statement with verbose :

{
"error": [
"Request Error"
]
}

It is because I cannot convert the response to a json (so the server definitely answers with an issue)
https://github.com/pitchmuc/adobe_analytics_api_2.0/blob/9572079afdb5b8191c0e4460cb00561bc1ec38ba/aanalytics2/aanalytics2.py#L189-L195

So it is probably an error returned by the server.

However, it means that the tipp I gave you to get the error files will not work.
It will probably give you nothing for the response.

You just will be able to know which of the request is actually causing the issue.

I will probably update this so you can see the response.text to better identify the server response.

[loldenburg]

"I think the problem comes from the fact that you try to use that data to merge it later on and because it is an empty dataframe, it doesn't merge correctly:"
-> Yes, but I was referring to the "empty file" problem that you have now detailed as well, thanks for that.

I am doing it this way now:

    def aa2_get_report_retry(req, tries=10, aa_inst=ags, n_res='100'):
        """uses the AA v2 API (aanalytics2 module) and returns
        the requested report. Retries several times ("tries" parameter) if output is empty (happens often due to API V2.0 instabilities)
        
        :param req: request as JSON string
        :param tries: number of tries
        :param aa_inst: adobe analytics v2 instance
        :param n_res: n_result parameter (number of results)
        :return: dataframe with report data
        :rtype: pandas.DataFrame
        """
        result_df = []
        i = 0
        while (len(result_df) == 0) & (i < tries):
            if (i > 0):
                time.sleep(60) # take a 60-second break before querying right away again
            logging.info("trying to get data, round " + str(i + 1))
            result_raw = aa_inst.getReport(req, n_result=n_res)
            result_df = result_raw["data"]
            i = i + 1
        if (i >= tries):
            logging.error("Could not get data from Adoba after " + str(tries) + "tries. Quitting.")
            return result_df
        else:
            return result_df

And then:

        search_vol_df = aa2_get_report_retry(aa_req_search_vol, tries=10, aa_inst=ags, n_res='1000')

This seems to work for me. I still get the Server Error about every 3rd or 4th try, but then I try up to 10 times again.
Maybe you could add a similar logic as a "retry" parameter to the getReport function.

[pitchmuc]

OK. So there was no error on your statement, this is adobe analytics server not stable enough, am I correct ?

The retry parameter is a good idea!
I'll add it on the next release.

To be clear on the implementation :

You will have a retry parameter, and if after the number of retry it still fails, I will keep on with returning empty dataframe.
Would that be the expected behavior ?

[loldenburg]

Yeah that would be it, thanks!
I don't know if the 60s timeout is needed after getting an empty dataframe, still trying to find that out, but if you know the answer already, go for it!

[pitchmuc]

As discussed the issue #24 seems to come from the same origin. Instability of Adobe Analytics API (or network) and would "require" the same fix, or way to handle it : integrating a "retry" parameter.

I am considering adding a new connection layer to handle all connections to Adobe Analytics more elegantly.
it may take a bit of time as I don't want to do breaking changes.

[pitchmuc]

Hello @loldenburg,
I have published a new version of the API wrapper, version 0.1.0 that should solve the issue.
The methods would need to change a bit for getting the companyId.

Now you can use the Loggin class this way :

import aanalytics as api2

api2.importConfigFile('blabla.json')
logger = api2.Loggin(retry=3)
cid = logger.getCompanyId()

note that you can now directly create the Analytics instance with that Loggin class.

myCompany = logger.createAnalyticsConnection('companyId',retry=3)

Also, if you already know which companyId to use, you can use the Analytics class directly.
No need to call the getCompanyId method

myCompany = api2.Analytics('companyId',retry=3)

The old method to get CompanyId still exist but will not be updated to have the retry parameter.
I will keep the 0.0.10 version avaiable on pypi.
I will update the getStarted documentation to reflect those changes but the last release already incorporate several new documentations (see twitter ;) )

[loldenburg]

@pitchmuc Thanks, that is great, the retry for getting the Analytics instance should already solve some of the API instability issues of Adobe. I hope I can test that out later this week.
I would recommend to call the class "Logging", not "Loggin". Maybe you can update this before I try it.

[loldenburg]

update: newest version by Julien calls it "Login", has to do with the Login Company, I misread that. Thanks!

[pitchmuc]

closing this issue

[loldenburg]

thx @pitchmuc it solves it for getting the company, but the problem is for any method (eg getDimensions, getReport etc.), so in theory the retry parameter should be there all the time. But fine for now, I am doing my own retry logic in my scripts now.

[pitchmuc]

Have you seen that there is a retry parameter as well in the Analytics class instanciation ?
So you can do something like this :

myCompany = aanalytics2.Analytics('companyId',retry=3) ## will retry 3 times (30 s intervals) when there is an error for GET methods (the server returns an error)

Also, if you have specified a retry parameter on the Login class and use the createAnalyticsConnection to create your Analytics class instance.
The number of retry you have specified in the Login class will be passed to the Analytics class instance creation.

Unfortunately, I can only manage when there is a real error (I cannot generate a JSON format from the server response).
I cannot handle all of the different formats and fields expected for each requests.

[loldenburg]

Ah I didn't get that, sorry. So if I instantiate the analytics class with the retry parameter, it is valid for any GET request (but not PUT or POST, right (those usually are not the problem)?

…

On Wed, Nov 11, 2020 at 2:51 PM Julien ***@***.***> wrote: Have you seen that there is a retry parameter as well in the Analytics class instanciation ? So you can do something like this : myCompany = aanalytics2.Analytics('companyId',retry=3) ## will retry 3 times when there is an error for GET methods (the server returns an error) Also, if you have specified a retry parameter on the Login class and use the createAnalyticsConnection to create your Analytics class instance. The number of retry you have specified in the Login class will be passed to the Analytics class instance creation. Unfortunately, I can only manage when there is a real error (I cannot generate a JSON format from the server response). I cannot handle all of the different formats and fields expected for each requests. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD34SUASMWPIMHW2EZZSQMLSPKJETANCNFSM4RWX52ZQ> .

[pitchmuc]

Yes exactly.
Also, it can also be that your PUT or POST request is the problem, it is not coming from Adobe server.
I would need to verify format and information before asking multiple times Adobe servers to deal with an erronous request.

[mrkobk]

Hi there,

quickly wanted to check, if there was any update (on Adobe end) on whether this error is something that is planned to be addressed for good? understand from the comments above that it's caused by a missing LastPage key. I come across this issue from time to time so wondering if Adobe intends to ensure the Key is always present in the near future?

[hayden-blair]

Would like to echo the response of mrkobk. Currently requesting a report and receiving this message every time I run the request:

Warning : Server Error - no save file & empty dataframe.

The json file for my report request was generated from a fully populated workspace using the techniques described in this video released by Adobe: https://www.youtube.com/watch?v=j1kI3peSXhY&feature=emb_logo

Seems like there should be no issue...

[pitchmuc]

I know it is super frustrating.
A ticket has been opened on Adobe Analytics customer Care.

For this issue, it seems that this issue is not on my end because the lastPage is documented in the swagger and for consistency,it should always be in the response.
See swagger response

The issue is, as you may have guessed, that if I don't get that key, there is no way for me to know if the data are complete or no.
Also, I had issue with report having issue when the key was missing.

Therefore it has been developed as a safe-guard...

However...
Following @hayden-blair comment, it seems that you can easily reproduce that issue (never been consistent on my end).
I can propose to parameterize this safeguard.
Explanation:

• I would add a parameter (unsafe - default to False). But if you set it to true, the getReport method will assume that this is the last page and try to return the data returned by the server.

it may break if there is an actual error and I cannot ensure that the data are complete but using that parameter, you will know these implications / possibilities.
Would that be OK with you ?

I would love if this is OK for you that @hayden-blair can test it.
I can do a new release this week for this.

[hayden-blair]

Thanks for the response, really appreciate your input! I'll pull the repo and add the new lastPage parameter to the ???Analytics.getReport() function??? and see what kind of results I generate. Just to make sure I understood your suggestion clearly, you recommended that I:

1. Pull the aanalytics repo
2. alter the safegaurd at line 1035 in aanalytics2.py

I am understanding correctly?

[pitchmuc]

Hello @hayden-blair
Thanks for your feedback ! You were faster than me on this.
I just did a commit that gives you a new parameter in the getReport() method. It is "unsafe" parameter.
So by running getReport(...., unsafe=True) you will pass over the safeguard lastPage.
(basically what you did manually).

New signature of the method:

getReport(
            self,
            json_request: Union[dict, str, IO],
            limit: int = 1000,
            n_result: Union[int, str] = 1000,
            save: bool = False,
            item_id: bool = False,
            unsafe: bool = False,
            verbose: bool = False,
            debug=False
    ) -> object:
        """
        Retrieve data from a JSON request.Returns an object containing meta info and dataframe. 
        Arguments:
            json_request: REQUIRED : JSON statement that contains your request for Analytics API 2.0.
            limit : OPTIONAL : number of result per request (defaut 1000)
            The argument can be : 
                - a dictionary : It will be used as it is.
                - a string that is a dictionary : It will be transformed to a dictionary / JSON.
                - a path to a JSON file that contains the statement (must end with ".json"). 
            n_result : OPTIONAL : Number of result that you would like to retrieve. (default 1000)
                if you want to have all possible data, use "inf".
            item_id : OPTIONAL : Boolean to define if you want to return the item id for sub requests (default False)
            unsafe : OPTIONAL : If set to True, it will not check "lastPage" parameter and assume first request is complete. 
                This may break the script or return incomplete data. (default False).
            save : OPTIONAL : If you would like to save the data within a CSV file. (default False)
            verbose : OPTIONAL : If you want to have comment display (default False)
            
        """

I am curious to know if the issue was from the request or in my handling of the response ?

[pitchmuc]

Trying to understand if I need to fix something on my module

[pitchmuc]

Thanks a lot @hayden-blair
This is very valuable feedback (and pretty easy to reproduce & "fix").
Now you will have this when running the getReport with this error:

Hopefully, it plays nicely with other errors as well.
I will keep the "unsafe" mode as well, in case it solve the issue for someone.

I will also keep the fact that it returns empty dataframe on "safe mode" (no raise Exception)
So in case you are running a loop, you don't have it break.

I will publish the new version of the module this week and close this ticket when it is done.
It will be version 0.2.3

[mrkobk]

hi,

to clarify: "my" issue is not related to access issues and only appears sporadically with 1 and the same request working today, but failing tomorrow. appreciate the new flag, but if I properly read through this, its simply suppresses the error message (symptom) instead of tackling the actual cause of the problem leaving me with the option of a) deal with the error, b) have no error but technically not know if the data I get returned is actually complete. Is that correct?

In any case many thanks for providing the option in the first place. Aware that the issue itself is not with this library but rather with Adobe itself. Thanks

[pitchmuc]

Yes exactly @mrkobk

That is the kind of issue that I was / am a bit hopeless for... I will encourage you to reach to Adobe via customer care.
I know @loldenburg opened a ticket already.
I pushed some times ago about that issue internally. At some point, I hope they will be able to solve it.

On their side, it is hard because as you said (and I experienced it as well) it is very sporadic so hard to reproduce.

Hopefully, the error message will be better now and we may be able to spot the underlying issue.
I will add a statement to print the response JSON when the issue arise (not "lastPage" in the response) and you decided to run the getReport in "unsafe" mode.

So even if it breaks later, we will see the response statement and be able to transfer it to Engineers.

If you run it locally, I have a "debug" parameter that also save the request and the response in JSON file. I especially built this parameter to debug this issue long time ago to provide engineer with information.
I know that some are running in the cloud and don't want to save file in their instance, hence the new print coming may help.

Trying my best to help you help other by gathering as much info as possible on this bug... but the last word will probably not come from me.

Open to other / any ideas.

[mrkobk]

Many Thank again! Appreciate your input on the subject

[loldenburg]

Thanks a lot for making the error message clearer. In my case, it is definitely not a rights issue (not having rights to a certain Report Suite for example), because the exact same request works most of the time, but sometimes gets this error. But I will upgrade to the new version as soon as it is there to see whether I can thus identify the error better.

…

On Thu, Feb 11, 2021 at 8:21 AM mrkobk ***@***.***> wrote: Many Thank again! Appreciate your input on the subject — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD34SUCCV2WDHW6UBPN2YHTS6OAPHANCNFSM4RWX52ZQ> .