metron-project / simyan Goto Github PK

Currently the IssueResultSchema has the has_staff_reviewfield defined as a boolean, but really it appears to be a Nested field which contains a id, name, api_detail_url, and site_detail_url fields. So, when a value is present it will kick out a marshmallow.exceptions.ValidationError.

It's easy enough to fix, tho I'm not 100% sure it's worth the effort since it seems to be data that would really only be pertinent for ComicVine internal use, and I'd be incline to just comment out the field and leave it as a placeholder.

To reproduce this bug, just add the following test:

def test_issue_list_staff_boolean(talker):
    wolv = talker.issue_list({"filter": f"volume:{85930}"})
    assert wolv[0].has_staff_review == True

Endpoints:

• /team
• /teams

Comicvine API ID: 4060

Comicvine has added a new field to the Team obj isssues_disbanded_in, there is already a disbanded_in_issues field, so the 2 should be merged.

Keep getting
no such column: query_date
error when trying to search comic vine, using the latest 1.2.0 version as well. Double checked my api_key value as well and it's good.

session = Comicvine(api_key=api_key, cache=SQLiteCache())

As per the title. I'm getting lots of validation errors in the following format:

1 validation error for ParsingModel[List[simyan.schemas.story_arc.StoryArc]]
__root__ -> 0 -> publisher
none is not an allowed value (type=type_error.none.not_allowed)

I have had a one-off similar occurrence of a failure due to lack of value for 'first_appeared_in_issue' when doing a filter by name search for 'Lo, this Monster'.

The code which is triggering this issue:
results = _cvSession.story_arc_list(params={"filter": "name:%s" % (name)})

Endpoints:

• /characters
• /character

Comicvine API ID: 4005

Comicvine calls these Objects, but Object is an annoying keyword to work around, so I suggest renaming it to Item

Endpoints:

• /object
• /objects

Comicvine API ID: 4055

The volume and volume_list endpoint tests now return a first issue, either:

• Update the schema to make first issue non-optional
• Find new test data

Context : Currently the cache stores the expiry date as calculated from the expiry input parameter of simyan.sqlite_cache.SQLiteCache.

Request : While not a huge deal, it would be slightly more flexible if the query date is stored instead of the expiry date. The oldest accepted query date could then be calculated by subtracting the expiry parameter from the current date.

Reason : At present, there's no way to identify when a query was carried out so it's impossible to make any meaningful adjustments to the expiry parameter for existing data. With this change, if I modify the preferred expiry parameter in the future Simyan will be able to calculate the expected expiry date for the content in response to my adjustment.

I am grabbing matches for quite a few series (~3000) and have noticed that occasionally the returned result on a volume search has issues with some of the attributes showing up as a VolumeResult object instead of a string. I'm not sure if this is enough info, but the last line of output from my script is the error I'm getting.

There was an error processing Volume search for Doctor Who Magazine (1979):
sequence item 0: expected str instance, VolumeResult found

The only attributes I'm using from the VolumeResult are name, start_year, publisher.name, id_, issue_count.

I'm not sure if this is a bug or a user error, but I can't see what I'm missing that would make 1 out of every 50-100 series searches return this.

The volume and volume_list endpoint tests now return a last issue, either:

• Update the schema to make first issue non-optional
• Find new test data

Currently you can only pass in a params dict for list/search endpoints but its unclear and undocumented what the valid parameters actually are.

I can't see this functionality anywhere presently? I need to run a query in the following format:

https://comicvine.gamespot.com/api/search/?api_key=5555555555&format=json&resources=volume&limit=100&query=logans+run

I am finding that using volume_list with a name filter is failing to return some matches that aren't an exact fit. My current query using Simyan is:

comicvine.volume_list(params={"filter": "name:%s" % (seriesName)})

An example of my problem is the series "Logan's Run". If I search for "Logans Run" (no apostrophe) the volume_list with a name filter returns 0 results. The same search using the URL at the top returns the correct series in the results.

I am assuming that the filter by name only returns exact string matches, which is the cause of my problems?

Currently Simyan ratelimits 20 req/min for any resource. Comicvine requires 200 req/resource/hour.

Having IssueEntry in Issues and GenericEntries causes confusion. Look into renaming the Generic entries to be less confusing.

Currently last_issue and first_issue are required and may not be null, however it appears Comicvine allows them to be null.

Example: https://comicvine.gamespot.com/api/volume/4050-92409/?api_key=***&format=json

Error Message:

simyan.exceptions.APIError: {'last_issue': ['Field may not be null.'], 'first_issue': ['Field may not be null.']}

Glancing real quick at this, the insert method for the SqliteCache class was:

    def insert(self, query: str, response: str):
        expiry = datetime.now() + timedelta(days=self.expiry)
        self.cur.execute(
            "INSERT INTO queries(query, response, expiry) VALUES(?, ?, ?);",
            (query, json.dumps(response), expiry.strftime("%Y-%m-%d")),
        )
        self.con.commit()

but with this pull request the expiry parameter can be None, so it probably needs to be changed to something like this to head off any potential bugs. I'm not on my dev machine, so you should definitely double check this solution. ;)

    def insert(self, query: str, response: str):
        if self.expiry:
            expiry = datetime.now() + timedelta(days=self.expiry)

            self.cur.execute(
                "INSERT INTO queries(query, response, expiry) VALUES(?, ?, ?);",
                (query, json.dumps(response), expiry.strftime("%Y-%m-%d")),
            )
        else:
            self.cur.execute(
                "INSERT INTO queries(query, response) VALUES(?, ?);",
                (query, json.dumps(response)),
            )
        self.con.commit()

Originally posted by @bpepple in #34 (comment)

Endpoints:

• /location
• /locations

Comicvine API ID: 4020

Looks like Comic Vine doesn't use a standard datetime format on their data, and when it deviates from the "%Y-%m-%d %H:%M:%S" format it will raise a APIError.

To reproduce this error, add the following creator test:

def test_creator_with_dob(talker):
    kirby = talker.creator(5614)
    assert kirby.date_of_birth == datetime.datetime(1917, 8, 28, 0, 0)

This will raise an exception since the json contains the following for the date of death:

{'date': '1994-02-06 00:00:00.000000', 'timezone_type': 3, 'timezone': 'America/Los_Angeles'}

My guess is you need to add multiple tests to every resource (Creators, Issues, Publishers, etc) that has a date field to verify the date formatting is correct.

The volume and volume_list endpoint tests now return a publisher field, either:

• Update the schema to make publisher non-optional
• Find new test data

Currently all requests cached or not are rate limited, the function call should be split so the cache can be checked without being rate limited.

Endpoints:

• /concept
• /concepts

Comicvine API ID: 4015

I've had WiFi drop out issues at my place, and I've noticed a few times that it will sit for 5-10 minutes before returning a result (WiFi dropout is 5-10 seconds tops). Interrupting the script returns the following:

  File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 228, in volume_list
    results = self._retrieve_all_responses(endpoint="/volumes", params=params)
  File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 398, in _retrieve_all_responses
    response = self._get_request(endpoint=endpoint, params=params)
  File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 145, in _get_request
    response = self._perform_get_request(url=url, params=params)
  File "/opt/homebrew/lib/python3.9/site-packages/ratelimit/decorators.py", line 113, in wrapper
    return func(*args, **kargs)
  File "/opt/homebrew/lib/python3.9/site-packages/ratelimit/decorators.py", line 80, in wrapper
    return func(*args, **kargs)
  File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 98, in _perform_get_request
    response = get(url, params=params, headers=self.headers)
  File "/opt/homebrew/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/opt/homebrew/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/opt/homebrew/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/opt/homebrew/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/opt/homebrew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/opt/homebrew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/opt/homebrew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()

Not sure if this is anything that can be helped in your script? Maybe reduce the timeout for returning a response?

• Probably don't need the PyPI downloads badge (I think it might be a bit broken)
• The Github Action - Formatting badge should be updated to the Github Action - Testing
• Why choose the name Code-Testing change it to Testing or Tests
• Could add the Development Status as a badge

I've started to add documentation support (Readthedocs), which can be tracked at this branch, and should have most of it ready for review by the end of the week:

https://github.com/bpepple/Simyan/tree/documentation

Endpoints:

• /origin
• /origins

Comicvine API ID: 4030

I'm having an issue with the new 'search' endpoint for resources. Every query I enter in the below format freezes my program during runtime. No errors or anything, it just simply sits there without making any apparent progress.

Comicvine.search(resource=simyan.comicvine.ComicvineResource.VOLUME,query='string here')

Endpoints:

• /power
• /powers

Comicvine API ID: 4035

Looks like the create_session() func use a non-optional api_key parameter. It would be easier to use downstream if you make that optional and if the information is not available throw an exception. For example, if your are using a settings file for you project it make sense to generate one on initial use, and then allow to provide the information at a later instance.

https://github.com/Buried-In-Code/Simyan/blob/76ed857b899712be2912bb6b64ed0add3898e09e/simyan/__init__.py#L14

it would be much nicer to use something like this:

def create_session(api_key: Optional[str] = None, cache: Optional[SQLiteCache] = None) -> Session:

I have encountered the same issue as #86 with the 'cover_date' field in Issue_List. Not exactly the same, but close enough? Unfortunately I was ~1hr into running a script and didn't have logging enabled, so can't point you to the Volume that caused this until I rerun with error handling fixed.

Would it perhaps make sense to have this exception handled in a way that doesn't kill the script that I'm running? Maybe a default value for each field when this occurs?

The line in my code:

results = session.issue_list(params={"filter": "volume:%s" % (comicID)})

The traceback:

  File "/Users/user/Downloads/CBL-ReadingLists-main/tools/simyan/session.py", line 252, in issue_list
    return IssueList(results)
  File "/Users/user/Downloads/CBL-ReadingLists-main/tools/simyan/issue_list.py", line 103, in __init__
    raise APIError(error)
simyan.exceptions.APIError: {'cover_date': ['Field may not be null.']}

Investigate why the Story Arc tests take significantly longer than any other test