scrappycocco / howlongtobeat-pythonapi

A simple Python API to read data from howlongtobeat

Hi, I am Michele 🎮

Hi, my name is Michele, but you can call me Scrappy. I graduated with a bachelor’s degree in computer science at the University of Padua. I did a final year Internship and "thesis", about volumetric rendering of medical data (Italian only).

I have always been passionate about game development and game engines, so in my free time during University I studied the basics of Vulkan (ScrapEngine). I like to help and mentor other people when I can, because of this I always encourage to open an Issue if you have a question or if you want to discuss something you don't understood, I will reply as soon as possible!

Since the end of 2019 I joined Project Borealis as volunteer programmer to help the team creating a fan-made Half-Life game in Unreal Engine. I am currently working full time as Online (Game) Programmer developing racing games, while I keep helping Project Borealis and creating small projects in my spare time.

howlongtobeat-pythonapi's Issues

Search returning "None" for every search as of 5/19/21

Hello,

Last year I put together a little script for myself using this library so that I could search for HLTB times via the terminal without having to fire up a browser. My last change to my script was in December 21 - I haven't updated it since, but I have been using it since then with no issues.

However, today when I tried to use it, I'm getting "None" for every search I try, even when I know there should be results (my script uses the default matching threshold of 0.4). For example, "Pokemon" returns "None", and then there's nothing to display of course.

I dug into the source code of the library itself a little bit. In HTMLRequests.py it looks like using "search_results.php" in the SEARCH_URL (as is the standard the library comes with) returns a 404 response code. So I'm guessing something has changed on HLTB's side?

I was able to get a 200 response code by updating SEARCH_URL to use "#search" instead, but then I was still not getting any search results no matter what I tried. So whatever's going on is a little beyond what I can solve for myself, unfortunately.

Hoping this doesn't mean the end of this library. Any help would be greatly appreciated.

Error with minimum similarity 0

This seems to have caused a side effect where if you search for a game that is not found, you get a HowLongToBeatEntry with all None values. But only if you specify input_minimum_similarity=0.

from howlongtobeatpy import HowLongToBeat, HowLongToBeatEntry
results = HowLongToBeat().search('Thief: Gold')
print(len(results))
> 0
results = HowLongToBeat(input_minimum_similarity=0.4).search('Thief: Gold')
print(len(results))
> 0
results = HowLongToBeat(input_minimum_similarity=0).search('Thief: Gold')
print(len(results))
> 1
print(vars(results[0]))
> {'game_id': -1,
 'game_name': None,
 'game_image_url': None,
 'game_web_link': None,
 'gameplay_main': -1,
 'gameplay_main_unit': None,
 'gameplay_main_label': None,
 'gameplay_main_extra': -1,
 'gameplay_main_extra_unit': None,
 'gameplay_main_extra_label': None,
 'gameplay_completionist': -1,
 'gameplay_completionist_unit': None,
 'gameplay_completionist_label': None,
 'similarity': 0}

Originally posted by @kparal in #3 (comment)

Incorrect precision in completion times

Currently, the completion time fields in a HowLongToBeatEntry are ints instead of floats. This is because of the following lines in JSONResultParser.parse_json_element:

# Add a few times elements as help for the user
current_entry.main_story = round(input_game_element["comp_main"] // 3600, 1)
current_entry.main_extra = round(input_game_element["comp_plus"] // 3600, 1)
current_entry.completionist = round(input_game_element["comp_100"] // 3600, 1)
current_entry.all_styles = round(input_game_element["comp_all"] // 3600, 1)

The problem is that // is used instead of /, which results in integer division instead of float division. Because of the round function here, which doesn't actually do anything at the moment, I'm assuming this is a bug and not a feature.

To fix it, you can change these lines to:

# Add a few times elements as help for the user
current_entry.main_story = round(input_game_element["comp_main"] / 3600, 2)
current_entry.main_extra = round(input_game_element["comp_plus"] / 3600, 2)
current_entry.completionist = round(input_game_element["comp_100"] / 3600, 2)
current_entry.all_styles = round(input_game_element["comp_all"] / 3600, 2)

(I also recommend using two digits of rounding precision instead of one — alternatively, you could forgo rounding altogether and let the user choose the rounding precision)

Non-letter characters missing when searching for a game name

> from howlongtobeatpy import HowLongToBeat, HowLongToBeatEntry 
> results = HowLongToBeat().search('Half-Life: Opposing Force')                                                  
> print(results[0].game_name)                                                                                    
HalfLife Opposing Force

It seems the HTML output changed and now some attributes don't have special characters in them (like title), but it is still possible to find the proper game name in that html. This is what send_web_request() returns:

<h3 class='global_padding shadow_box back_blue center'>We Found 3 Games for "Half-Life: Opposing Force"</h3>
<ul>
<div class="clear"></div>
<li class="back_darkish" 							style="background-image:linear-gradient(rgb(31, 31, 31), rgba(31, 31, 31, 0.9)), url('https://howlongtobeat.com/gameimages/250px-Half-Life_Opposing_Force_box.jpg')"> <div class="search_list_image">
<a aria-label="HalfLife Opposing Force" title="HalfLife Opposing Force" href="game?id=4256">
<img alt="Box Art" src="https://howlongtobeat.com/gameimages/250px-Half-Life_Opposing_Force_box.jpg" />
</a>
</div> <div class="search_list_details"> <h3 class="shadow_text">
<a class="text_green" title="HalfLife Opposing Force" href="game?id=4256">Half-Life: Opposing Force</a>
</h3> <div class="search_list_details_block"> <div>
<div class="search_list_tidbit text_white shadow_text">Main Story</div>
<div class="search_list_tidbit center time_100">5&#189; Hours </div>
<div class="search_list_tidbit text_white shadow_text">Main + Extra</div>
<div class="search_list_tidbit center time_100">6&#189; Hours </div>
<div class="search_list_tidbit text_white shadow_text">Completionist</div>
<div class="search_list_tidbit center time_100">7&#189; Hours </div>
</div> </div>
</div> </li>
<li class="back_darkish" 							style="background-image:linear-gradient(rgb(31, 31, 31), rgba(31, 31, 31, 0.9)), url('https://howlongtobeat.com/gameimages/30301_Dark_Operations.jpg')"> <div class="search_list_image">
<a aria-label="Dark Operations" title="Dark Operations" href="game?id=30301">
<img alt="Box Art" src="https://howlongtobeat.com/gameimages/30301_Dark_Operations.jpg" />
</a>
</div> <div class="search_list_details"> <h3 class="shadow_text">
<a class="text_white" title="Dark Operations" href="game?id=30301">Dark Operations</a>
</h3> <div class="search_list_details_block"> <div>
<div class="search_list_tidbit text_white shadow_text">Main Story</div>
<div class="search_list_tidbit center time_00">--</div>
<div class="search_list_tidbit text_white shadow_text">Main + Extra</div>
<div class="search_list_tidbit center time_00">--</div>
<div class="search_list_tidbit text_white shadow_text">Completionist</div>
<div class="search_list_tidbit center time_00">--</div>
</div> </div>
</div> </li> <div class="clear"></div>
<li class="back_darkish" 							style="background-image:linear-gradient(rgb(31, 31, 31), rgba(31, 31, 31, 0.9)), url('https://howlongtobeat.com/gameimages/30300_Military_Duty.jpg')"> <div class="search_list_image">
<a aria-label="Military Duty" title="Military Duty" href="game?id=30300">
<img alt="Box Art" src="https://howlongtobeat.com/gameimages/30300_Military_Duty.jpg" />
</a>
</div> <div class="search_list_details"> <h3 class="shadow_text">
<a class="text_white" title="Military Duty" href="game?id=30300">Military Duty</a>
</h3> <div class="search_list_details_block"> <div>
<div class="search_list_tidbit text_white shadow_text">Main Story</div>
<div class="search_list_tidbit center time_00">--</div>
<div class="search_list_tidbit text_white shadow_text">Main + Extra</div>
<div class="search_list_tidbit center time_00">--</div>
<div class="search_list_tidbit text_white shadow_text">Completionist</div>
<div class="search_list_tidbit center time_00">--</div>
</div> </div>
</div> </li> <div class="clear"></div>
</ul> 

Can you please fix? Thank you!

howlongtobeatpy-0.1.9

All searches returning None (all requests returning 404)

I was hoping to integrate this into Goodplays, unfortunately I haven't been able to get it working. Every search returns None, apparently due to a 404.

For example:

>>> from howlongtobeatpy import HowLongToBeat
>>> HowLongToBeat(0.0).search("Outer Wilds")

returns None. When I modify HTMLRequests.py to spit out the response text on non-200 responses I get a 404 page:

<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width"/><title>HowLongToBeat.com | Game Lengths, Backlogs and more!</title><meta name="theme-color" content="#000000"/><meta name="description" content="How long are your favorite video games? HowLongToBeat has the answer. Create a backlog, submit your game times and compete with your friends!"/><meta name="robots" content="noodp, noydir"/><meta name="thumbnail" content="https://howlongtobeat.com/img/hltb_brand2.png"/><link rel="canonical" href="https://howlongtobeat.comundefined"/><meta property="twitter:url" content="https://howlongtobeat.comundefined"/><meta property="og:url" content="https://howlongtobeat.comundefined"/><meta property="og:title" content="HowLongToBeat.com | Game Lengths, Backlogs and more!"/><meta property="og:type" content="website"/><meta property="og:image" content="https://howlongtobeat.com/img/hltb_brand2.png"/><meta property="og:description" content="How long are your favorite video games? HowLongToBeat has the answer. Create a backlog, submit your game times and compete with your friends!"/><meta name="twitter:card" content="summary"/><meta name="twitter:description" content="How long are your favorite video games? HowLongToBeat has the answer. Create a backlog, submit your game times and compete with your friends!"/><meta property="twitter:domain" content="howlongtobeat.com"/><meta name="twitter:site" content="@HowLongToBeat"/><meta name="twitter:image" content="https://howlongtobeat.com/img/hltb_brand2.png"/><meta name="next-head-count" content="19"/><link rel="apple-touch-icon" sizes="57x57" href="https://howlongtobeat.com/img/icons/apple-touch-icon-57x57.png"/><link rel="apple-touch-icon" sizes="60x60" href="https://howlongtobeat.com/img/icons/apple-touch-icon-60x60.png"/><link rel="apple-touch-icon" sizes="72x72" href="https://howlongtobeat.com/img/icons/apple-touch-icon-72x72.png"/><link rel="apple-touch-icon" sizes="76x76" href="https://howlongtobeat.com/img/icons/apple-touch-icon-76x76.png"/><link rel="apple-touch-icon" sizes="114x114" href="https://howlongtobeat.com/img/icons/apple-touch-icon-114x114.png"/><link rel="apple-touch-icon" sizes="120x120" href="https://howlongtobeat.com/img/icons/apple-touch-icon-120x120.png"/><link rel="apple-touch-icon" sizes="144x144" href="https://howlongtobeat.com/img/icons/apple-touch-icon-144x144.png"/><link rel="apple-touch-icon" sizes="152x152" href="https://howlongtobeat.com/img/icons/apple-touch-icon-152x152.png"/><link rel="apple-touch-icon" sizes="180x180" href="https://howlongtobeat.com/img/icons/apple-touch-icon-180x180.png"/><link rel="icon" type="image/png" href="https://howlongtobeat.com/img/icons/favicon-32x32.png" sizes="32x32"/><link rel="icon" type="image/png" href="https://howlongtobeat.com/img/icons/android-chrome-192x192.png" sizes="192x192"/><link rel="icon" type="image/png" href="https://howlongtobeat.com/img/icons/favicon-96x96.png" sizes="96x96"/><link rel="icon" type="image/png" href="https://howlongtobeat.com/img/icons/favicon-16x16.png" sizes="16x16"/><link rel="manifest" href="/manifest.json"/><link rel="preconnect" href="https://howlongtobeat.com"/><link rel="preload" as="script" href="https://cdn.ziffstatic.com/pg/howlongtobeat.js"/><script type="text/javascript" id="pogo" src="https://cdn.ziffstatic.com/pg/howlongtobeat.js" async=""></script><link rel="stylesheet" href="https://cdn.ziffstatic.com/pg/howlongtobeat.css"/><link rel="preload" as="script" href="https://cdn.ziffstatic.com/pg/howlongtobeat.prebid.js"/><script crossorigin="true" src="https://cdn.ziffstatic.com/jst/zdconsent.js" async=""></script><link rel="preload" href="/_next/static/css/228c4b801d7021a9.css" as="style"/><link rel="stylesheet" href="/_next/static/css/228c4b801d7021a9.css" data-n-g=""/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script src="/_next/static/chunks/webpack-2efae08fba7c9b69.js" defer=""></script><script src="/_next/static/chunks/framework-d51ece3d757c7ed2.js" defer=""></script><script src="/_next/static/chunks/main-c908647eeec19f62.js" defer=""></script><script src="/_next/static/chunks/pages/_app-c2db2970f545cae6.js" defer=""></script><script src="/_next/static/chunks/pages/404-c06b07ed3ab0773c.js" defer=""></script><script src="/_next/static/JCeqWkOAftOy_JRL-i2FJ/_buildManifest.js" defer=""></script><script src="/_next/static/JCeqWkOAftOy_JRL-i2FJ/_ssgManifest.js" defer=""></script></head><body><div id="__next"><script>!function(){try{var d=document.documentElement,n='data-theme',s='setAttribute';var e=localStorage.getItem('theme');if('system'===e||(!e&&true)){var t='(prefers-color-scheme: dark)',m=window.matchMedia(t);if(m.media!==t||m.matches){d.style.colorScheme = 'dark';d[s](n,'dark')}else{d.style.colorScheme = 'light';d[s](n,'light')}}else if(e){d[s](n,e|| '')}if(e==='light'||e==='dark')d.style.colorScheme=e}catch(e){}}()</script><div class="Layout_container__V2eEE"><header class="MainNavigation_header__WuiTa"><nav class="MainNavigation_nav__LkHHd"><a class="MainNavigation_brand__8YjKY" aria-label="HowLongToBeat" href="/"></a><ul class="MainNavigation_list__xBZrm"><li><a href="/forum">Forum</a></li><li><a href="/stats">Stats</a></li><li><a href="/submit">Submit</a></li></ul><ul class="MainNavigation_login__KE7zX"><li><a class="text_primary" href="/login">Login</a></li><li class="MainNavigation_join_link__4bsgx"><a class="mobile_hide text_primary" href="/login/signup">Join</a></li></ul><div class="MainNavigation_search__kw6St"><input class="MainNavigation_search_box__jDUWW back_form" aria-label="Search" tabindex="2" type="text" placeholder="Search Your Favorite Games..." autoComplete="off"/></div></nav></header><main class="Layout_main__NgJgX"><div class="back_dark" id="global_site" style="display:block;border-top:1px solid transparent"><div class="contain_out"><div class="contain_in index_padding"><div class="content_100 center"><div class="global_padding_big"><h1 class="global_padding"><span class="mobile_hide">Error</span> 404 - Not Found</h1></div></div><div class="content_100 center"><img src="https://howlongtobeat.com/img/404/pong.gif" style="width:100%" alt="404"/></div><div class="content_100 center"><p class="in">Sorry! The page you are looking for does not exist. Try going back or visiting a different link.</p></div></div></div></div></main><footer class="Footer_footer__2MMdT back_primary"><div class="Footer_footer_inside__UfjFE"><div class="Footer_footer_links__GA8Gc"><h3>HowLongToBeat</h3><ul><li><a href="/feedback">Contact Us</a></li><li><a href="/conduct">Code of Conduct</a></li><li><a href="/privacy">Privacy Policy</a></li></ul></div><div class="Footer_footer_links__GA8Gc"><h3>Social</h3><ul><li><a href="https://discord.gg/v5F26Dk" target="_blank" rel="noreferrer">Discord</a></li><li><a href="https://facebook.com/HowLongToBeat/" target="_blank" rel="noreferrer">Facebook</a></li><li><a href="https://twitter.com/HowLongToBeat/" target="_blank" rel="noreferrer">Twitter</a></li></ul></div><div class="Footer_footer_copyright__TDPg5"><h3>© 2022 Ziff Davis, LLC</h3><ul><li>Powered By Community, Built With Love</li><li><a href="#" class="showConsentTool"><img alt="AdChoices Icon" src="https://c.evidon.com/pub/icong1.png" class="evidon-consent-link-image" style="vertical-align:middle" width="14" height="18"/> <!-- -->AdChoices</a></li><li></li></ul></div></div></footer></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{"pageMetadata":{"noTopAd":true}},"__N_SSG":true},"page":"/404","query":{},"buildId":"JCeqWkOAftOy_JRL-i2FJ","isFallback":false,"gsp":true,"scriptLoader":[]}</script></body></html>

(I replicated this by manually posting against the URL.)

From what I can tell, it looks like they've changed the posting endpoint to https://www.howlongtobeat.com/api/search, reformatted it slightly, and are actually returning JSON now, which should make parsing it a hell of a lot easier. I haven't been following this closely, but perhaps this change corresponds to the GamePass integration they've just implemented.

Unfortunately, I'm unable to get it to actually authorize (everything I post to the endpoint gives me a 401). For example:

>>> import requests
>>> requests.post('https://www.howlongtobeat.com/api/search', json={'searchType': 'games', 'searchTerms': ['OUTER', 'WILDS'], 'searchPage': 1, 'size': 20}, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.102 Safari/537.36 OPR/90.0.4480.54', 'content-type': 'application/json', 'origin': 'https://howlongtobeat.com', 'referrer': 'https://howlongtobeat.com/', 'accept': '*/*'})
<Response [401]>

Python version:

Python 3.6.10 (default, Dec 19 2019, 23:04:32)
[GCC 5.4.0 20160609] on linux

Test workflow

Add game year into HowLongToBeatEntry

Hi, HLTB changed recently and the game year is no longer a part of the game title. For example, search for God of War, you'll now see two games with the same title, and the year is in a different color on the website. Can you please expose the year in HowLongToBeatEntry, so that we can access this information? Thanks a lot!

Error in async requests

Exception: Cannot connect to host howlongtobeat.com:443 ssl:default [[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:852)]

Why this happens is currently unknown

Support retrieving stats by game ID

This library currently only supports searching by game title. However, in some cases, I already know the game ID (for example when I already searched for this title and just want to update the information). If I know the ID, I would like to retrieve the game stats directly, instead of searching for the title again and having to deal with other titles that might be named the same way or similarly. Can you please support something like this?

result = HowLongToBeat().game_id('62935')

That would involve parsing this web page:
https://howlongtobeat.com/game.php?id=62935

Thanks!

Setup.py fake_useragent==0.1.11

When I installed the package it also installed fake_useragent==0.1.11.
Unfortunately, I kept on getting the following error:
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached
Updating fake_useragent to the latest version (1.1.0) solved this issue for me and I haven't had any other issues since.

similarity_case_sensitive confusion in readme

From README:

Also remember that by default the similarity check is case-sensitive between the name given and the name found, if you want to ignore the case you can use:
results = HowLongToBeat(0.0).search("Awesome Game", similarity_case_sensitive=False)

This is very confusing to me, because it is setting similarity_case_sensitive while also using 0.0 as similarity value. That means similarity checking is disabled, and so... setting similarity_case_sensitive makes no sense in this case?

I think the example code should be rewritten e.g. to this:

results = HowLongToBeat(0.7).search("Awesome Game", similarity_case_sensitive=False)

Searching only returns 1 game when the web search returns 6

When I search for "Grip" on HLTB web, I get 6 results:

GRIP: Combat Racing 
Hot Shots Tennis: Get a Grip 
GripShift 
Iron Grip: Warlord 
Griptape Backbone 
Iron Grip: Marauders 

However, using the API, I get just one result:

> from howlongtobeatpy import HowLongToBeat, HowLongToBeatEntry 
> results = HowLongToBeat().search('Grip')                                                                        
> print(len(results))
1
> print(results[0].game_name)
GripShift

Am I doing something wrong?

howlongtobeatpy-0.1.10

please support search modifiers to include/isolate DLCs

howlongtobeat.com search form changed and now it doesn't show DLCs in search results, unless you open Search Options and change Modifiers to Include (or Isolate) DLC.

You can test this by searching e.g. for "Hearts of Stone". You'll see no results until you include DLCs in search options.

Please support search modifiers in HowLongToBeat.search() and async_search() methods.

For example:

def search(self, game_name: str, dlc: str = None):
:param dlc: possible values are None to not modify search options, 'include' to also include DLC names in search results, and 'isolate' to search in DLC names only

Thank you!

Github Action doesn't work

Looks like Python 3.6 is disappearing from the Github Action that I use, and I could consider to upgrate the project to a newer python version to resume the action

Case-insensitive similarity

Hello!
Sometimes the name of the game in some source (e. g. Steam) and on HLTB completely coincides, not counting the case, but the similarity is very low. For example, game RED HOT VENGEANCE has only 0.3 similarity.

It would be great if you can add case insensitive search capability.
It might look something like this:
HowLongToBeat(0.9).search("game_name", case_sensitive = False)

Thank you!