pennydreadfulmtg / penny-dreadful-tools

A suite of tools for the Penny Dreadful MTGO community

Penny Dreadful Tools

Repository for the tools used by the Penny Dreadful Community.

View individual subdirectories for details

Modules

analysis is for Magic Online game log analysis.

decksite is the code for pennydreadfulmagic.com.

discordbot is the Discord chatbot.

github_tools are some GitHub integration utillities.

logsite is the code for logs.pennydreadfulmagic.com.

logsite_migrations are alembic database migrations for logsite.

magic is for information about Magic – cards, decklists, card images, legality, etc.

maintenance is for useful scripts, usually run via cron.

modo_bugs is for integration with https://github.com/PennyDreadfulMTG/modo-bugs/issues.

price_grabber builds a database of card prices.

rotation_script is for the script that handles Penny Dreadful rotation (card legality changes each season).

shared contains a bunch of general purpose helper classes. Things that could be used in any project.

shared_web contains a bunch of web-specific helper classes. It also contains our React code for "live" data tables.

Contributing

Contributions are very welcome. Please join the Discord at https://pennydreadfulmagic.com/discord/ and feel free to ask questions in #code.

Development Environment Setup

• Install Docker (https://www.docker.com/get-started)
• git clone https://github.com/PennyDreadfulMTG/Penny-Dreadful-Tools.git
• docker-compose build
• docker-compose up

If you plan on running things outside of the containers (eg: dev.py or logsite):

• Install python 3.10
• Install pipenv
• Install npm
• git clone https://github.com/PennyDreadfulMTG/Penny-Dreadful-Tools.git
• cd Penny-Dreadful-Tools
• pipenv install
• pipenv run python dev.py build

Configuring Environment

• Add a bot at https://discordapp.com/developers/applications/me
• Add a bot user for the bot
• Add the bot to your server with https://discordapp.com/oauth2/authorize?client_id=<your client id here>&scope=bot
• Go to the Bot section.
• Click to reveal the token (not secret) on https://discordapp.com/developers/applications/me
• Copy .env.example to .env and alter the value for "token" to this value. (or to "token" in config.json if not in docker setup)
• Do the same for the discord client_id and client_secret
• Optionally take a look at shared/configuration.py and enter any required non-default information into .env
• You will want to investigate the various targets in dev.py that acts as a Makefile. Some of these utilities use GitHub's commandline git-enchancer, hub: https://github.com/github/hub

Manual Development Environment Setup (Non-docker instructions)

• Install MariaDB 10.0+
• Install python 3.10
• Install pipenv
• Install npm
• git clone https://github.com/PennyDreadfulMTG/Penny-Dreadful-Tools.git
• cd Penny-Dreadful-Tools
• pipenv install
• pipenv run python build.py
• Using the values from your .env issue the following commands in MySQL (you don't need to create the databases):
  • CREATE USER '<mysql_user>'@'<mysql_host>' IDENTIFIED BY '<mysql_passwd>';
  • GRANT ALL ON <decksite_database>.* TO '<mysql_user>'@'<mysql_host>'; GRANT ALL ON <decksite_test_database>.* TO '<mysql_user>'@'<mysql_host>';
  • GRANT ALL ON <prices_database>.* TO '<mysql_user>'@'<mysql_host>';
  • GRANT ALL ON <magic_database>.* TO '<mysql_user>'@'<mysql_host>';
  • GRANT ALL ON <logsite_database>.* TO '<mysql_user>'@'<mysql_host>';
• Download a copy of the production decksite database (with personal information stripped):
  • mysql -u <mysql_user> -p<mysql_passwd> -e "CREATE DATABASE <decksite_database>"
  • curl https://pennydreadfulmagic.com/static/dev-db.sql.gz >/tmp/dev-db.sql.gz
  • gunzip /tmp/dev-db.sql.gz
  • mysql -u <mysql_user> -p<mysql_passwd> <decksite_database> </tmp/dev-db.sql
  • mysql -u <mysql_user> -p<mysql_passwd> -e "CREATE DATABASE <decksite_test_database>"
• Some very minor parts of the bot (the "modofail" command) use libopus and ffmpeg which are not in pip and must be installed in a your-OS-specific way separately. Very optional.

Running Decksite (pennydreadfulmagic.com)

• pipenv run python run.py decksite
• Visit http://localhost:5000/

Running Logsite (logs.pennydreadfulmagic.com)

• pipenv run python run.py logsite
• Visit http://localhost:5001/

Running Discordbot

• pipenv run python run.py discordbot
• Visit your Discord server.

Running the tests

There are various levels of granularity but in general use you want:

• pipenv run python dev.py test # Runs the unit tests, type checking, lint.

Check the dev.py source code for the full set of options including unit, types, lint (covered by test above) as well as functional (integration tests), perf (performance tests). release will take you all the way from your committed change to a PR via the tests (needs GitHub's commandline gh/hub installed).

Working on React components

• Run logsite
• pipenv run python dev.py watch # Builds bundle.js after every file change.

penny-dreadful-tools's Issues

magidex.com link is not sent as part of message when responding to !search

We should bundle it with the image and legality text so that it doesn't get separated from the main response.

Potentially delete/redownload old images

Between new card images coming out (in better quality), and card combos that will never be used again, it might be worth deleting some of the older images that are accruing in the cache.

Let's think about ways to do that.

Database.escape doesn't cope very well with floating point numbers (isdigit only works on positive integers).

isdigit only works on positive integers.

If you try and escape 7.5 you'll get '7.5' back.

Fix database Release Date

• We don't convert date to seconds-since-epoch but we have release_date as an INTEGER type. sqlite will let us insert it anyway but that's hella broken.

Card Variations (Alternative arts from same set)

• When storing card info have not accounted for 'variations' - an array of variant versions of a card by multiverse id (like Fallen Empires multiple arts). Also all the fields in "extras".

Multiverse ID is not correctly populated

multiverse_id = cards[0].multiverse_id is not working with cards retrieved from the db so fallback images will fail. Multiverse id doesn't seem to be populated in the db.

NOT doesn't work entirely correctly in !search

There's a bug when the first part of a query is NOT. NOT in general is a little broken and could be replaced by '-' as per magidex. -- do_test('NOT c:g', 'NOT (id IN (SELECT card_id FROM card_color WHERE or_id = 5))')

Ability to request specific printing of a card

• [bottle gomes|mrd] <-- support some syntax (possibly this) for getting the specific version of an image.

If a search is not valid it should give you feedback as to why not

Example:

!search c:x

will fail silently but should say:

"Invalid search. Unknown color 'x'"

Favor PD-legal cards in results

!search and [Elvish] should either be restricted to PD-legal or (better?) sort PD-legal-first favoring legal cards.

strip, lstrip and rstrip don't do what I thought they did - check all usages are sane.

I thought they chopped that exact string off the left/right/both sides. But they treat the string more like a list of characters to chop.

Standardize quotes

• Standardize on single quotes except where single quotes are contained within the string.

!Random tweaks

• Random should take a numeric argument and default to 1.

Database should return dicts, not lists

• We probably want to make the rows returned from database into dicts instead of lists. If we do that we must change Card(r) to Card(*r) in Oracle.

Convert search/tests.py to use pytest and be triggered by a pytest run/checked by jenkins

Use Fetcher more

• Do all HTTP requests with Fetcher

We do the `'SELECT ' + (', '.join(property for property in oracle.Oracle.properties())).rstrip(', ')` thing in two places. Unify.

It would be nice to have the github link in the bot's description

Don't fetch from the internet unnecessarily

Check the timestamp of legal_cards.txt and don't refetch if it's less than an hour old. Same for AllCards.json.

!price command that gets mtggoldfish current low mtgo price for speculation

Create database more efficiently

Build server takes an hour to create the database so switch to an executeMany or even just big long INSERTs with many entries.

Rarity search does not work

r:rare
r:r

should both work.

We use integers for rarity but try and search on first letter?

Switch from AllCards.json to AllSets.json

AllCards doesn't include printing-specific info.

See discussion in #25 #28 #35
Also required for #14, should we end up doing that.

!search should tack on a magidex URL when more_text is shown.

!status command for mtgo status

We have both Template().substitute and "blah" + "foo" and %-based string interpolation. And we should probaby be using .format()!

- Ability to search like magidex syntax

More than one image at once gives "no image available" when falling back to gatherer images

• More than one image at once gives "no image available" when falling back to mtgsdk images because the SDK only supports single images. We could send each separately or only send the first rather than "no image available"?

We should be even more generous in what we accept in value_lookup. If we have half a clue what you meant we should do the right thing.

Clean up imports

• Use import not from x import y – http://effbot.org/zone/import-confusion.htm

Write more unit tests

Because more coverage is always a good thing.

The bot should record any ! commands it sees that it doesn't understand

And we should interpret them as feature requests.

Backs of Double Faced Cards can show up too much

[Lambholt] gives you multiple copies of reverse sides but there's no immediately obvious way to make it ok.

Normalize Image sizes

• EMN and other smaller images look stupid when joined. Normalize size before joining on the image server.

Support :10:, :UU:, :2W: emojis

@binarySwordsman has added :00:, :10:, :11:, :12:, :13:, :15:, :16:, :01:, :2B:, :2G:, :2R:, :2U:, :2W:, :02:, :03:, :04:, :05:, :06:, :07:, :08:, :09:, :BG:, :BP:, :BR:, :BB:, :CC:, :EE:, :GP:, :GU:, :GW:, :GG:, :RG:, :RP:, :RW:, :RR:, :SS:, :TT:, :UB:, :UP:, :UR:, :UU:, :WB:, :WP:, :WU:, :WW:, :XX:

The rule is "remove curly braces and slashes. If single letter, double up. If single number prepend with 0."

Note there is currently no emoji for {E} (Energy), {Y}, {Z}, {Q} (untap), {P} (generic Phyrexian mana, only on one card's rules text), {20} (only in one card's rules text), nor any of the un-set specific symbols - {100}, {1,000,000}, {∞}, {hr}, {½}, {hw}.

Fuzzy matching

[Ad Nauseum] should load Ad Nauseam. [all sun's dawn] and [all suns dawn] should load All Suns' Dawn.

Providing the full name of Split Cards [Fire // Ice] doesn't work

• Split cards. The bot expects one half [Fire] or the other [Ice] but the image server expects both. So either the bot doesn't respond or if it does it doesn't get an image. We should support: [Fire], [Fire / Ice] and [Fire // Ice] and variations ignoring whitespace.

Local mode

• Add --local commandline option that causes the bot to run locally and receive commands from commandline instead of (as well as?) connecting to discord.

Generalize color_where to all subtables (color identity, subtypes, etc.) and add support for those searches.

Fetch the zipped version of AllSets.json

Will be quicker on slow connections.

Zero Byte uploads

• As of Sep 26th we are still uploading zero byte files sometimes despite the checks we have -- what's going on?

We raise a generic Exception in four places in search. We should raise something more specific (possibly custom).

Support negation operation in !search (for example "c!rug")

c!rug is not yet supported. do_test('(c!rug OR c:c) "Beyond lies the wub" o:trample', "((cost LIKE '%rug%') OR (cost LIKE '%c%')) AND (name LIKE '%Beyond lies the wub%' OR type LIKE '%Beyond s the wub%' OR text LIKE '%Beyond lies the wub%') AND (text LIKE '%trample%')");

Form help text programatically from list of commands

As more commands get added they should automatically appear in help.

At one extreme of this kind of approach is docopt (see http://docopt.org/) which is, let's be honest, amazing.

A simple solution would be to have a command-making class or function that expects a name and a string description and we use that for all commands.

Image names shouldn't *start* with a dash, really.

Reduce Max file length

• Max 255 chars for filename length is cutting it too fine given that we have to add an extension so cut off at a lower number.

Searching for multiple colors does not work

Examples:

!search c:ub # find all cards that are blue or black or both
!search c:ubm # find all cards that are blue AND black

The database should know as much as it can about data to catch issues early

• Because of the way we create the card table every column ends up nullable but some (like name) should not be.

There is some kind of general problem with Aether.

The constants in value_lookup should be loaded from the db and cached not hardcoded.