hmasdev / pyjpboatrace

pyjpboatrace :speedboat: provides you with useful tools for data analysis and auto-betting for boatrace.

Hi, I'm hmasdev 👋

I am working as a Machine Learning Engineer in Tokyo, Japan. I am interested in Python, Machine Learning.

💻 My Projects

Python Projects

Libraries

Projects	Techs and Features
	🔧Techs: Python, Selenium, Beautiful Soup 4 💡Features: Data scraping API and Betting API for the official boat race website Various webdrivers available Suitable for a wide range of users, from beginners to advanced
	🔧Techs: Python, langchain 💡Features: Simple Runnable with Loop Techniques like "Self-Consistent" are available
	🔧Techs: Python, venv 💡Features: Portable Python Virtual Environment You can use this library in the almost same way as venv

Applications

Projects	Techs and Features
	🔧Techs: Python, OpenAI API, langchain, pynput 💡Features: CLI-based Typing Application The typing targets are generated by LLM Your typing is recorded in a json file, which helps you analyze your typing skill.
	🔧Techs: Python, langchain, pyautogen, docker compose 💡Features: CLI-based Werewolf Game You can play with LLMs like ChatGPT The mode without human is available, that is, a werewolf game with only LLMs

VSCode-related

Projects	Techs and Features
	🔧Techs: JavaScript, OpenAI API 💡Features: Simple Shortcut to fix typo in a line: Ctrl+K Ctrl+N You can set API endpoint, model name and system prompt. The fixed positions are highlighted.
	🔧Techs: bash, bats 💡Features: Get the latest version of a vscode extension Download the vsix file

⌨️ Keyboards

Projects	Techs and Features
	🔧Techs: C, QMK Firmware, QMK MSYS 💡Features hand-fitted key allocation 🙌 splitted 34 keys
	🔧Techs: C, QMK Firmware, QMK MSYS, Vial 💡Features hand-fitted key allocation 🙌 34 keys Available with Vial

Utilities

Projects	Techs and Features
	🔧Techs: bash 💡Features: Aliases for shell commands Aliases for git Aliases for python and pyenv Aliases for npm Aliases for docker Alias for vscode

📫 How to reach me

• X.com: @hmdev3
• Qiita: @hmasdev
• Note: hmasdev

pyjpboatrace's Issues

Recommendations for scheduled testing

Abstract

The codes of pyjpboatrace should be tested periodically.

AsIs: Background

On 2022/01/22, the codes of pyjpboatrace have not been tested since 2021/08/13 (See https://github.com/hmasdev/pyjpboatrace/actions/runs/1124482660/workflow).
This is because .github/workflows/pytest.yaml is executed only when there is a push action or there is a pull request opened or synchronized.

ToBe: Describe the solution you'd like

The codes of pyjpboatrace should be tested periodically, e.g. once a month or once a week.

Add tests with Python 3.10 to pytest workflow in github actions

Abstract

To modify pytest workflow in github actions in order to test pyjpboatrace with Python 3.10

AsIs: Background

Pytest workflow in guthub actions tests pyjpboatrace with Python 3.7, 3.8 and 3.9.

ToBe: Describe the solution you'd like

Pytest workflow in github actions tests pyjpboatrace with Python 3.7, 3.8, 3.9 and 3.10.

Additional context

Solving this issue guarantees that pyjpboatrace works with Python 3.10.

Add arguments (date, stadium and race) to response dictionary

Abstract

The response dictionaries from scraping functions e.g. get_stadiums (StadiumsScraper.get) and get_odds_trifecta (TrifectaOddsScraper.get), should contain the arguments: date, stadium number and race number.

AsIs: Background

A use case of pyjpboatrace is to use pyjpboatrace scrapers to create boatrace database, where date, stadium number and race number are likely to be primary keys. However, the response dictionaries from scraping functions don't contain the arguments, so it is required to add the arguments, like date, stadium number and race number, to the response dictionaries after scraping.

The following is an example:

from datetime import date
import json
from pyjpboatrace import PyJPBoatrace

# initialize
boatrace_tools = PyJPBoatrace()

# get data
d=date(2020,12,2)
stadium=1
race=4
response = boatrace_tools.get_odds_trifecta(d=d, stadium=stadium, race=race)

# add date, stadium and race to response
# because response is a dictionary like {"1-2-3": 7.0, ..., "6-5-4": 9999}
response["date"] = d
response["stadium"] = stadium
response["race"] = race

# close (you can use 'with' statement)
boatrace_tools.close()

ToBe: Describe the solution you'd like

I want the response dictionaries from pyjpboatrace's scrapers to contain the arugments like date, stadium number and race number. For example, the reponse from get_odds_trifecta should be like

{
  "date": date(2020, 1, 1),  # This is a new element.
  "stadium": 1,    # This is a new element.
  "race": 1,    # This is a new element.
  "1-2-3": 5.6,
  "1-2-4": 14.4,
  ...,
  "6-5-4": 8650.0
}

Request to automate updates to README.md.

Abstract

Request to automate updates to README.md.

AsIs: Background

The current README.md contains the PyJPBoatrace API specifications and sample output. However, when changes are made to the code, especially when there are updates to the output of each API, the sample output must be obtained manually and the README.md updated.

ToBe: Describe the solution you'd like

If there are updates to the output of each API, the sample output updates in README.md are automatically committed to the main branch when the corresponding pull request is merged into the main branch.

Additional context

• Step.1: to add a script to use the current code to update sample outputs;
• Step.2: to create a workflow to execute the script above to update README.md and commit the change to main.

MEMO

This problem because apparent in #61 (comment)

[BUG] PyPI workflow in actions does not work

Describe the bug

PyPI workflow in actions, which runs when released, returns exit code 1 during uploading .whl and .tar.gz with twine.

https://github.com/hmasdev/pyjpboatrace/actions/runs/1103843756

Environment

• OS: Ubuntu 20.04.2 LTS
• Python Version: 3.7.11 (actions/setup-python@v1)

To Reproduce (Probabily)

Steps to reproduce the behavior:

1. Go to https://github.com/hmasdev/pyjpboatrace/releases
2. Click on "Draft a new release"
3. Input tag version, title and description.
4. Click on "Publish release", which triggers PyPI workflow.
5. Error during PyPI workflow in Upload step as follows:

   Traceback (most recent call last):
      File "/home/runner/.local/share/virtualenvs/pyjpboatrace-cg8vEFYy/bin/twine", line 5, in <module>
        from twine.__main__ import main
      File "/home/runner/.local/share/virtualenvs/pyjpboatrace-cg8vEFYy/lib/python3.8/site-packages/twine/__init__.py", line 27, in <module>
        import importlib_metadata
   ModuleNotFoundError: No module named 'importlib_metadata'
   Error: Process completed with exit code 1.

Expected behavior

Twine succeeds in uploading .whl and .tar.gz to pypi server.

Uploading distributions to https://upload.pypi.org/legacy/
Uploading pyjpboatrace-0.2.0-py3-none-any.whl
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 63.5k/63.5k [00:02<00:00, 24.2kB/s]
Uploading pyjpboatrace-0.2.0.tar.gz
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50.3k/50.3k [00:01<00:00, 31.2kB/s] 

View at:
https://pypi.org/project/pyjpboatrace/0.2.0/

Problems

If this bug is not fixed, we can not upload this package to pypi automatically, that is, manual operation is required when uploading.

Reasons (Optional)

Checking the log, I found pipenv uses python 3.8.10 as follows:

Creating a virtualenv for this project...
Pipfile: /home/runner/work/pyjpboatrace/pyjpboatrace/Pipfile
Using /usr/bin/python3.8 (3.8.10) to create virtualenv...
created virtual environment CPython3.8.10.final.0-64 in 694ms

This is probably a reason for this bug.

See https://stackoverflow.com/a/59734959/16567832

[BUG] Scheduled tests failed for stadiums, race_info and just_before_info scrapers

Describe the bug

Pytest failed.

Environment

• github actions (See https://github.com/hmasdev/pyjpboatrace/actions/runs/1766385345)
• Python Version: 3.7, 3.8 and 3.9

To Reproduce

I have confirmed that the following test fails in my environment as well:

$ pipenv install --dev
$ ./download_html_for_test.sh
$ pipenv run pytest

Screenshots or codes

See https://github.com/hmasdev/pyjpboatrace/actions/runs/1938733501

Expected behavior

The test should pass.

Problems

Reasons (Optional)

TBD

[BUG] If grade is SG, parsing fails.

If grade is SG, parsing fails.
https://www.boatrace.jp/owpc/pc/race/index?hd=20221030

from datetime import date
import json
from pyjpboatrace import PyJPBoatrace
boatrace_tools = PyJPBoatrace()
dic = boatrace_tools.get_stadiums(d=date(2022,10,30))
with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(dic, f, ensure_ascii=False)
boatrace_tools.close()

...
'Always': {'status': '-',
'grade': ['[FailedToParse]is-SGa'],
'timeframe': '',
'title': 'The 69th Boat Race Derby',
'period': ['2022-10-25', '2022-10-30'],
'day': 'Last day'},
...

[BUG]同着が発生したとき、3連単の値を一つしか取得できない

Describe the bug

同着が発生したとき、3連単の値を一つしか取得できない。
https://boatrace.jp/owpc/pc/race/raceresult?rno=1&jcd=12&hd=20210614

Add timestamp to the responses from odds scrapers

Abstract

The response dictionaries from odds scrapers, like get_odds_trifecta (TrifectaOddsScraper.get), should contain the timestamp when odds number are updated.

AsIs: Background

Odds scrapers for a race return the final odds after the race is over.
However, they return uncertain odds before the race is not over.

The numbers varies depending on when the scrapers are called.
For example, get_odds_trifecta returns one-min-ahead trifecta odds when you call get_odds_trifecta one minute before the deadline.
On the other hand, it returns five-min-ahead trifecta odds when you call get_odds_trifecta five minutes before the deadline.
The problem is that you can not find the timestamp when odds have been updated in the reponse from the scraper.

The following is an example:

from datetime import date, datetime
import json
from pyjpboatrace import PyJPBoatrace

# initialize
boatrace_tools = PyJPBoatrace()

# get data
d=date(2020,12,2)
stadium=1
race=4
timestamp = datetime.now()
response = boatrace_tools.get_odds_trifecta(d=d, stadium=stadium, race=race)

# add timestamp to response
# because response is a dictionary like {"1-2-3": 7.0, ..., "6-5-4": 9999}
response["timestamp"] = timestamp

# close (you can use 'with' statement)
boatrace_tools.close()

ToBe: Describe the solution you'd like

I want odds scrapers to response the timestamp in addition to odds values.
Altanatively, I want odds scrapers to scrape the update timestamp from the target website, because odds pages in pyjpboatrace.jp contain a timestamp when the odds are updated.

すべての選手がフライングだった際のget_race_resultのエラーについて

すべての選手がフライングだった際においてget_race_resultがエラーを吐く様子です。
https://www.boatrace.jp/owpc/pc/race/raceresult?rno=3&jcd=21&hd=20180101

dic = boatrace_tools.get_race_result(d=date(2018,1,1), stadium=21, race=3)

[BUG] Scheduled Test Failed

Describe the bug

Pytest failed.

Environment

• github actions (See https://github.com/hmasdev/pyjpboatrace/actions/runs/1766385345)
• Python Version: 3.7, 3.8 and 3.9

To Reproduce

I have confirmed that the following test fails in my environment as well:

$ pipenv install --dev
$ ./download_html_for_test.sh
$ pipenv run pytest

Screenshots or codes

See https://github.com/hmasdev/pyjpboatrace/actions/runs/1766385345

Expected behavior

The test should pass.

Problems

Reasons (Optional)

Comparing the websites of boatrace.jp, e.g. https://www.boatrace.jp/owpc/pc/race/beforeinfo?rno=7&jcd=14&hd=20200825, with the expected outputs of the related functions of pyjpboatrace, e.g. https://github.com/hmasdev/pyjpboatrace/tree/main/tests/data ,
I have found the ID of the direction of each boat race track had changed from before.

I guess that this is the reason why Pytest failed.