akenaide / wsoffcli

POC ws-tcg scrapper

wsoffcli

Collect data from https://ws-tcg.com/

Synopsis

Collect data from https://ws-tcg.com/.

Create a json file for each card with most information.

Example: 'wsoffcli fetch -n IMC' will fetch all cards with a code starting with 'IMC'

If you want more than one use '##' as seperator like 'wsoffcli fetch -n BD##IM'

'--serie' use a hidden number in the official site, this number is increment for each new set (e.g Kadokawa is number 259, Goblin 260 ...).

To use environ variable, use the prefix 'WSOFF'.

Options

  -h, --help           help for wsoffcli
  -n, --neo string     Neo standar by set
  -s, --serie string   serie number

SEE ALSO

• wsoffcli completion - Generate the autocompletion script for the specified shell
• wsoffcli fetch - Fetch cards
• wsoffcli gendoc - Generate doc with Cobra
• wsoffcli products - Get products information

Auto generated by spf13/cobra on 16-Nov-2023

wsoffcli's Issues

Fix code scanning alert - Log entries created from user input

Tracking issue for:

• https://github.com/Akenaide/wsoffcli/security/code-scanning/1

Iter is broken

Add page_num option

• Add page_num argument in parser
• read page_num from specific config or in generic config
• Parse from the given page if present, otherwise start from 0
• Save the last page at the end, so we can have an incremantal update

Improve baserarity option

パラレル | パラレルを除く

Form key: parallel: 0
We should set it to "1"

Retry don't always works

Fetch products page

We want to collects products informations.

Format will be

[ ...
{
  "setName": "あやかしトライアングル",
  "releaseDate": "2023/11/23",
  "setCode": "AYT",
  "imageURL": "https://ws-tcg.com/wordpress/wp-content/uploads/20231013111510/BP%E3%80%8CAYT%E3%80%8DBOX.png"
}
...]

Booster cards in one unique file

Create an option to concat all cards in one unique file

---

--export (default card)

• card: export each card in a single file
• booster: export each booster in a single file
• all: export in a single file

Add sentry

https://sentry.io

Fix code scanning alert - Log entries created from user input

Tracking issue for:

• https://github.com/Akenaide/wsoffcli/security/code-scanning/2

Add new field

Add imageCardURL from ws-tcg.com. It will contains the image URL.

Kanseru can skip pages

for {
	select {
	case retryLink := <-retry:
		jobs <- retryLink
		log.Printf("Retry: %v", retryLink)
	default:
		jobs <- fmt.Sprintf("%v?page=%d", Baseurl, page)
		page = page + 1
	}

	if kanseru {
		log.Println("Kanseru at: ", page)
		break
	}
}

We should put a condition on the break.

-s expansion is broken

We found 0 pages when this is used.

Support english website

https://en.ws-tcg.com

Iter option is not consistant.

Read config from environ

We would like to run the cron more frequently. To achieve this, we need to configure the cron, and the cleaner way to do so is by using an environment variable.

To accomplish this, we should:

• Configure Viper to read the correct variable name from the environment.
• Add ARGS to the Docker file so that we can configure it even when using Docker.