storerjeremy / python-semrush Goto Github PK

This repo, uses the requests library to hit the api URL and manage parameters. One issue with this is that when using display_filter, the api expects:

+|Ur|Co|electrical-supplies
(include|URL|Containing|electrical-supplies)

to be:

%2B|Ur|Co|electrical-supplies

if you pass this string to the client.domain_organic method, it will be converted to:

%252B%7CUr%7CCo%7Celectrical-supplies

when Requests builds the URL request.

a workaround is to do the following:

url = "electrical-supplies"
display_filter = quote("+|Ur|Co|" + url, safe='|/', encoding=None, errors=None)    
client.api_url = 'http://api.semrush.com/?display_filter='+display_filter
data = client.domain_organic(domain="domain.com", database='us',display_limit=5,export_columns='Ur,Po,Ph,Nq')

The solution would be to simply have the retrieve method of the SemrushClient class handle a kwargs key of 'display_filter' directly on the api URL and let requests handle the remaining params.

There may be some other way to hook into urlencode, or the Requests library to skip encoding '|' characters, but I am not aware of any.

I realize this is a couple-year-old repo, so understood if not wanting to look at. I wanted to leave this here in case someone ran into the same issue.

Here:

Change to:

 lines = data.splitlines()

Here:

Add:

if 'export_columns' in kwargs:
    kwargs['export_columns'] = ",".join([c.strip() for c in kwargs['export_columns'].split(',')])

AttributeError: 'SemrushClient' object has no attribute 'traffic_summary'

There's an issue with parsing returned data with URLs that include a semicolon:

From /python_semrush/semrush.py

     84             result = {}
     85             for i, datum in enumerate(line.split(';')):
---> 86                 result[columns[i]] = datum.strip('"\n\r\t')
     87             results.append(result)

As an example this URL was passed from a call to the organic_phrase function:
http://www.hilton.com/en/hotels/content/SPTSHHF/media/pdf/Tangerine_Bar_2.pdf;jsessionid=DTE5TAZBV525MCSGBI12VCQ

Resulting in list index out of range error. To get around this (just in case others find the same problem) I modified my script to declare export_escape=1 in the arguments to force double-quotes; I then updated the parser to split on '";"' instead of ";"

The new code looks like this:
(lines 75-89 of /python_semrush/semrush.py)

    @staticmethod
    def parse_response(data):
        results = []
        data = data.decode('unicode_escape')
        lines = data.split('\r\n')
        lines = list(filter(bool, lines))
        columns = lines[0].split(';')

        for line in lines[1:]:
            result = {}
            for i, datum in enumerate(line.split('";"')):
                result[columns[i]] = datum.strip('"\n\r\t')
            results.append(result)

        return results

I'm sure there is a better way to do this, but in the meantime, this is a workaround that works!