twin-te / twinte-parser

twinte内部で使用されるKdBパーサ

Twin:te

Local Development - only docker

build images

docker compose build db db-migration back front sponsorship proxy-docker parser codegen

environment variables on backend

cp back/.env back/.env.local // please edit it

db migration

docker compose run --rm db-migration bash -c 'make migrate-up db_url=${DB_URL}'
docker compose run --rm db-migration bash -c 'make migrate-up db_url=${TEST_DB_URL}'

update courses based on kdb

docker compose run -u root --rm parser python ./download_and_parse.py --year 2024 --output-path kdb_2024.json
mv ./parser/kdb_2024.json ./back/kdb_2024.json
docker compose run -u root --rm back go run .  update-courses-based-on-kdb --year 2024 --kdb-json-file-path kdb_2024.json
rm ./back/kdb_2024.json

start services

docker compose up db back front sponsorship proxy-docker

Access to http://localhost:4000 or http://localhost:4000/sponsorship

Local Development - docker + host machine

Only proxy-host, db and db-migration are running in the docker container.

Version

• Go : 1.22.x
• Nodejs : nodejs 22.x.x
• Python : 3.12.x

Please install direnv. Remember to set hook.

Example in Mac

brew install direnv
echo 'eval "$(direnv hook zsh)"' >> ~/.zshrc
source ~/.zshrc

Please Install Bun.

Example in Mac

brew install oven-sh/bun/bun

Build Images

docker compose build db db-migration proxy-host

Terminal workspace

docker compose up -d db db-migration
docker compose run --rm db-migration bash -c 'make migrate-up db_url=${DB_URL}'
docker compose run --rm db-migration bash -c 'make migrate-up db_url=${TEST_DB_URL}'

Terminal parser

pip install -r requirements.txt
python download_and_parse.py --year 2024 --output-path kdb_2024.json

Terminal back

cd back

// setup environment variables
cp .env .env.local // please edit .env.local file
direnv allow .

// update courses based on kdb
go run .  update-courses-based-on-kdb --year 2024 --kdb-json-file-path ../parser/kdb_2024.json

// hot reload
go install github.com/air-verse/air@latest
air

Terminal front

cd front
bun install
bun run dev

Terminal sponsorship

cd sponsorship
bun install
bun run dev

Terminal workspace

docker compose up proxy-host

Access to http://localhost:4000 or http://localhost:4000/sponsorship

twinte-parser's Issues

取りこぼしてる情報を取得する

単位数や、授業概要など

idが振られていない場合の対処

twinte-parser/src/parser.ts

Line 89 in d6a7641

id: columns[0],

上の画像のような場合に、(病理学に対してここではHB32127が来るべき)
欠損値をできるだけ無くしたほうが良いかもしれない