metron-project / simyan Goto Github PK
View Code? Open in Web Editor NEWA Python wrapper for the Comicvine API.
Home Page: https://pypi.org/project/simyan
License: GNU General Public License v3.0
A Python wrapper for the Comicvine API.
Home Page: https://pypi.org/project/simyan
License: GNU General Public License v3.0
Currently the IssueResultSchema has the has_staff_review
field defined as a boolean, but really it appears to be a Nested field which contains a id
, name
, api_detail_url
, and site_detail_url
fields. So, when a value is present it will kick out a marshmallow.exceptions.ValidationError.
It's easy enough to fix, tho I'm not 100% sure it's worth the effort since it seems to be data that would really only be pertinent for ComicVine internal use, and I'd be incline to just comment out the field and leave it as a placeholder.
To reproduce this bug, just add the following test:
def test_issue_list_staff_boolean(talker):
wolv = talker.issue_list({"filter": f"volume:{85930}"})
assert wolv[0].has_staff_review == True
Endpoints:
/team
/teams
Comicvine API ID: 4060
Comicvine has added a new field to the Team obj isssues_disbanded_in
, there is already a disbanded_in_issues
field, so the 2 should be merged.
Keep getting
no such column: query_date
error when trying to search comic vine, using the latest 1.2.0 version as well. Double checked my api_key value as well and it's good.
session = Comicvine(api_key=api_key, cache=SQLiteCache())
As per the title. I'm getting lots of validation errors in the following format:
1 validation error for ParsingModel[List[simyan.schemas.story_arc.StoryArc]]
__root__ -> 0 -> publisher
none is not an allowed value (type=type_error.none.not_allowed)
I have had a one-off similar occurrence of a failure due to lack of value for 'first_appeared_in_issue' when doing a filter by name search for 'Lo, this Monster'.
The code which is triggering this issue:
results = _cvSession.story_arc_list(params={"filter": "name:%s" % (name)})
Endpoints:
/characters
/character
Comicvine API ID: 4005
Comicvine calls these Objects, but Object is an annoying keyword to work around, so I suggest renaming it to Item
Endpoints:
/object
/objects
Comicvine API ID: 4055
The volume and volume_list endpoint tests now return a first issue, either:
Context : Currently the cache stores the expiry date as calculated from the expiry input parameter of simyan.sqlite_cache.SQLiteCache.
Request : While not a huge deal, it would be slightly more flexible if the query date is stored instead of the expiry date. The oldest accepted query date could then be calculated by subtracting the expiry parameter from the current date.
Reason : At present, there's no way to identify when a query was carried out so it's impossible to make any meaningful adjustments to the expiry parameter for existing data. With this change, if I modify the preferred expiry parameter in the future Simyan will be able to calculate the expected expiry date for the content in response to my adjustment.
I am grabbing matches for quite a few series (~3000) and have noticed that occasionally the returned result on a volume search has issues with some of the attributes showing up as a VolumeResult object instead of a string. I'm not sure if this is enough info, but the last line of output from my script is the error I'm getting.
There was an error processing Volume search for Doctor Who Magazine (1979):
sequence item 0: expected str instance, VolumeResult found
The only attributes I'm using from the VolumeResult are name, start_year, publisher.name, id_, issue_count.
I'm not sure if this is a bug or a user error, but I can't see what I'm missing that would make 1 out of every 50-100 series searches return this.
The volume and volume_list endpoint tests now return a last issue, either:
Currently you can only pass in a params dict for list/search endpoints but its unclear and undocumented what the valid parameters actually are.
I can't see this functionality anywhere presently? I need to run a query in the following format:
I am finding that using volume_list with a name filter is failing to return some matches that aren't an exact fit. My current query using Simyan is:
comicvine.volume_list(params={"filter": "name:%s" % (seriesName)})
An example of my problem is the series "Logan's Run". If I search for "Logans Run" (no apostrophe) the volume_list with a name filter returns 0 results. The same search using the URL at the top returns the correct series in the results.
I am assuming that the filter by name only returns exact string matches, which is the cause of my problems?
Currently Simyan ratelimits 20 req/min for any resource. Comicvine requires 200 req/resource/hour.
Having IssueEntry in Issues and GenericEntries causes confusion. Look into renaming the Generic entries to be less confusing.
Currently last_issue
and first_issue
are required and may not be null, however it appears Comicvine allows them to be null.
Example: https://comicvine.gamespot.com/api/volume/4050-92409/?api_key=***&format=json
Error Message:
simyan.exceptions.APIError: {'last_issue': ['Field may not be null.'], 'first_issue': ['Field may not be null.']}
Glancing real quick at this, the insert
method for the SqliteCache
class was:
def insert(self, query: str, response: str):
expiry = datetime.now() + timedelta(days=self.expiry)
self.cur.execute(
"INSERT INTO queries(query, response, expiry) VALUES(?, ?, ?);",
(query, json.dumps(response), expiry.strftime("%Y-%m-%d")),
)
self.con.commit()
but with this pull request the expiry
parameter can be None
, so it probably needs to be changed to something like this to head off any potential bugs. I'm not on my dev machine, so you should definitely double check this solution. ;)
def insert(self, query: str, response: str):
if self.expiry:
expiry = datetime.now() + timedelta(days=self.expiry)
self.cur.execute(
"INSERT INTO queries(query, response, expiry) VALUES(?, ?, ?);",
(query, json.dumps(response), expiry.strftime("%Y-%m-%d")),
)
else:
self.cur.execute(
"INSERT INTO queries(query, response) VALUES(?, ?);",
(query, json.dumps(response)),
)
self.con.commit()
Originally posted by @bpepple in #34 (comment)
Endpoints:
/location
/locations
Comicvine API ID: 4020
Looks like Comic Vine doesn't use a standard datetime format on their data, and when it deviates from the "%Y-%m-%d %H:%M:%S"
format it will raise a APIError.
To reproduce this error, add the following creator test:
def test_creator_with_dob(talker):
kirby = talker.creator(5614)
assert kirby.date_of_birth == datetime.datetime(1917, 8, 28, 0, 0)
This will raise an exception since the json contains the following for the date of death:
{'date': '1994-02-06 00:00:00.000000', 'timezone_type': 3, 'timezone': 'America/Los_Angeles'}
My guess is you need to add multiple tests to every resource (Creators, Issues, Publishers, etc) that has a date field to verify the date formatting is correct.
The volume and volume_list endpoint tests now return a publisher field, either:
Currently all requests cached or not are rate limited, the function call should be split so the cache can be checked without being rate limited.
Endpoints:
/concept
/concepts
Comicvine API ID: 4015
I've had WiFi drop out issues at my place, and I've noticed a few times that it will sit for 5-10 minutes before returning a result (WiFi dropout is 5-10 seconds tops). Interrupting the script returns the following:
File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 228, in volume_list
results = self._retrieve_all_responses(endpoint="/volumes", params=params)
File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 398, in _retrieve_all_responses
response = self._get_request(endpoint=endpoint, params=params)
File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 145, in _get_request
response = self._perform_get_request(url=url, params=params)
File "/opt/homebrew/lib/python3.9/site-packages/ratelimit/decorators.py", line 113, in wrapper
return func(*args, **kargs)
File "/opt/homebrew/lib/python3.9/site-packages/ratelimit/decorators.py", line 80, in wrapper
return func(*args, **kargs)
File "/opt/homebrew/lib/python3.9/site-packages/simyan/comicvine.py", line 98, in _perform_get_request
response = get(url, params=params, headers=self.headers)
File "/opt/homebrew/lib/python3.9/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/opt/homebrew/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/homebrew/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
resp = self.send(prep, **send_kwargs)
File "/opt/homebrew/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
r = adapter.send(request, **kwargs)
File "/opt/homebrew/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
resp = conn.urlopen(
File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/homebrew/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "/opt/homebrew/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/opt/homebrew/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/homebrew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/opt/homebrew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
self.do_handshake()
File "/opt/homebrew/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
Not sure if this is anything that can be helped in your script? Maybe reduce the timeout for returning a response?
Code-Testing
change it to Testing
or Tests
Development Status
as a badgeI've started to add documentation support (Readthedocs), which can be tracked at this branch, and should have most of it ready for review by the end of the week:
Endpoints:
/origin
/origins
Comicvine API ID: 4030
I'm having an issue with the new 'search' endpoint for resources. Every query I enter in the below format freezes my program during runtime. No errors or anything, it just simply sits there without making any apparent progress.
Comicvine.search(resource=simyan.comicvine.ComicvineResource.VOLUME,query='string here')
Endpoints:
/power
/powers
Comicvine API ID: 4035
Looks like the create_session() func use a non-optional api_key parameter. It would be easier to use downstream if you make that optional and if the information is not available throw an exception. For example, if your are using a settings file for you project it make sense to generate one on initial use, and then allow to provide the information at a later instance.
it would be much nicer to use something like this:
def create_session(api_key: Optional[str] = None, cache: Optional[SQLiteCache] = None) -> Session:
I have encountered the same issue as #86 with the 'cover_date' field in Issue_List. Not exactly the same, but close enough? Unfortunately I was ~1hr into running a script and didn't have logging enabled, so can't point you to the Volume that caused this until I rerun with error handling fixed.
Would it perhaps make sense to have this exception handled in a way that doesn't kill the script that I'm running? Maybe a default value for each field when this occurs?
The line in my code:
results = session.issue_list(params={"filter": "volume:%s" % (comicID)})
The traceback:
File "/Users/user/Downloads/CBL-ReadingLists-main/tools/simyan/session.py", line 252, in issue_list
return IssueList(results)
File "/Users/user/Downloads/CBL-ReadingLists-main/tools/simyan/issue_list.py", line 103, in __init__
raise APIError(error)
simyan.exceptions.APIError: {'cover_date': ['Field may not be null.']}
Investigate why the Story Arc tests take significantly longer than any other test
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.