pokeapi / ditto

Tool for meta operations over PokéAPI data

A RESTful API for Pokémon - pokeapi.co

Beta GraphQL support is rolling out! Check out the GraphQL paragraph for more info.

Setup  

• Download this source code into a working directory, be sure to use the flag --recurse-submodules to clone also our submodules.

• Install the requirements using pip:

  make install
  # This will install all the required packages and libraries for using PokeAPI

• Set up the local development environment using the following command:

  make setup

• Run the server on port 8000 using the following command:

  make serve

Database setup

To build or rebuild the database by applying any CSV file update, run

make build-db

Visit localhost:8000/api/v2/ to see the running API!

Each time the build-db script is run, it will iterate over each table in the database, wipe it, and rewrite each row using the data found in data/v2/csv.

If you ever need to wipe the database use this command:

make wipe-sqlite-db

If the database schema has changed, generate any outstanding migrations and apply them

make make-migrations
make migrate

Run make help to see all tasks.

Docker and Compose  

There is also a multi-container setup, managed by Docker Compose V2. This setup allows you to deploy a production-like environment, with separate containers for each service, and is recommended if you need to simply spin up PokéAPI.

Start everything by

make docker-setup

If you don't have make on your machine you can use the following commands

docker compose up -d
docker compose exec -T app python manage.py migrate --settings=config.docker-compose
docker compose exec -T app sh -c 'echo "from data.v2.build import build_all; build_all()" | python manage.py shell --settings=config.docker-compose'

Browse localhost/api/v2/ or localhost/api/v2/pokemon/bulbasaur/ on port 80.

To rebuild the database and apply any CSV file updates, run

make docker-build-db

If the database schema has changed, generate the migrations and apply those

make docker-make-migrations
make docker-migrate

GraphQL  

When you start PokéAPI with the above Docker Compose setup, an Hasura Engine server is started as well. It's possible to track all the PokeAPI tables and foreign keys by simply

# hasura cli needs to be installed and available in your $PATH: https://hasura.io/docs/latest/graphql/core/hasura-cli/install-hasura-cli.html
# hasura cli's version has to greater than v2.0.8
make hasura-apply

When finished browse http://localhost:8080 and you will find the admin console. The GraphQL endpoint will be hosted at http://localhost:8080/v1/graphql.

A free public GraphiQL console is browsable at the address https://beta.pokeapi.co/graphql/console/. The relative GraphQL endpoint is accessible at https://beta.pokeapi.co/graphql/v1beta

A set of examples is provided in the directory /graphql/examples of this repository.

Kubernetes  

Kustomize files are provided in the folder https://github.com/PokeAPI/pokeapi/tree/master/Resources/k8s/kustomize/base/. Create and change your secrets:

cp Resources/k8s/kustomize/base/secrets/postgres.env.sample Resources/k8s/kustomize/base/secrets/postgres.env
cp Resources/k8s/kustomize/base/secrets/graphql.env.sample Resources/k8s/kustomize/base/secrets/graphql.env
cp Resources/k8s/kustomize/base/config/pokeapi.env.sample Resources/k8s/kustomize/base/config/pokeapi.env
# Edit the newly created files

Configure kubectl to point to a cluster and then run the following commands to start a PokéAPI service.

kubectl apply -k Resources/k8s/kustomize/base/
kubectl config set-context --current --namespace pokeapi # (Optional) Set pokeapi ns as the working ns
# Wait for the cluster to spin up
kubectl exec --namespace pokeapi deployment/pokeapi -- python manage.py migrate --settings=config.docker-compose # Migrate the DB
kubectl exec --namespace pokeapi deployment/pokeapi -- sh -c 'echo "from data.v2.build import build_all; build_all()" | python manage.py shell --settings=config.docker-compose' # Build the db
kubectl wait --namespace pokeapi --timeout=120s --for=condition=complete job/load-graphql # Wait for Graphql configuration job to finish

This k8s setup creates all k8s resources inside the Namespace pokeapi, run kubectl delete namespace pokeapi to delete them. It also creates a Service of type LoadBalancer which is exposed on port 80 and 443. Data is persisted on 12Gi of ReadWriteOnce volumes.

Wrappers

Official wrapper	Repository	Features
Node server-side	PokeAPI/pokedex-promise-v2	Auto caching
Browser client-side	PokeAPI/pokeapi-js-wrapper	Auto caching, Image caching
Java/Kotlin	PokeAPI/pokekotlin
Python 2/3	PokeAPI/pokepy	Auto caching
Python 3	PokeAPI/pokebase	Auto caching, Image caching

Wrapper	Repository	Features
.Net Standard	mtrdp642/PokeApiNet	Auto caching
Dart	prathanbomb/pokedart
Go	mtslzr/pokeapi-go	Auto caching
PHP	lmerotta/phpokeapi	Auto caching, lazy loading
PowerShell	Celerium/PokeAPI-PowerShellWrapper
Python	beastmatser/aiopokeapi	Auto caching, asynchronous
Ruby	rdavid1099/poke-api-v2
Rust	lunik1/pokerust	Auto caching
Scala	juliano/pokeapi-scala	Auto caching
Spring Boot	dlfigueira/spring-pokeapi	Auto caching
Swift	kinkofer/PokemonAPI
Typescript server-side/client-side	Gabb-c/Pokenode-ts	Auto caching

Donations

Help to keep PokéAPI running! If you're using PokéAPI as a teaching resource or for a project, consider sending us a donation to help keep the service up. We get 1+ billion requests a month!

Thank you to all our backers! Become a backer

Join Us On Slack!

Warning Currently no maintainer has enough free time to support the community on Slack. Our Slack is in an unmaintained status.

Have a question or just want to discuss new ideas and improvements? Hit us up on Slack. Consider talking with us here before creating a new issue. This way we can keep issues here a bit more organized and helpful in the long run. Be excellent to each other 😄

Sign up easily!

Once you've signed up visit PokéAPI on Slack

Contributing

This project exists thanks to all the people who contribute

All contributions are welcome: bug fixes, data contributions, and recommendations.

Please see the issues on GitHub before you submit a pull request or raise an issue, someone else might have beat you to it.

To contribute to this repository:

• Fork the project to your own GitHub profile

• Download the forked project using git clone:

  git clone --recurse-submodules [email protected]:<YOUR_USERNAME>/pokeapi.git

• Create a new branch with a descriptive name:

  git checkout -b my_new_branch

• Write some code, fix something, and add a test to prove that it works. No pull request will be accepted without tests passing, or without new tests if new features are added.

• Commit your code and push it to GitHub

• Open a new pull request and describe the changes you have made.

• We'll accept your changes after review.

Simple!

Deprecation

As of October 2018, the v1 API has been removed from PokéAPI. For more information, see pokeapi.co/docs/v1.html.

ditto's Issues

Optimize the generated schema

There's a ton of duplication in the schema generated by ditto analyze. If we can write something that'll factor out common sub-models, it would be super useful for auto-generating API clients.

Bonus points for some clever naming heuristic.

Add support for named resources

From PokeAPI/pokeapi#369.

Ditto needs to produce named files - eg /pokemon/bulbasaur - in the transform step.

Alternative file structure

To support migrating PokeAPI to static files, the file structure will have to change from

data/
|- api/
|    |- v2/
|    |    |- language/
|    |    |    |- 1/
|    |    |    |    |- index.json
...

to

data/
|- api/
|    |- v2/
|    |    |- language/
|    |    |    |- 1
|    |    |    | ...
|    |    |- language

, where each sub-directory also has a file of the same name, and all index.json instances are removed.

It can be either a complete change or a flag if we don't want to change how ditto serve works.

Investigate performance of clone

It may be possible to use some parallelism to speed up ditto clone.

One informal clone on my machine running Windows took just under 30mins.

Where to host the ditto updater bot

From PokeAPI/pokeapi#353:

@sargunv:

IIRC there's also a DigitalOcean account being used to host the API right now
...
I do plan on finding somewhere to host the ditto updater bot, so that's probably something worth transferring if you don't want to depend on something only I control. It should run fine on their smallest container (I plan on testing that today).

Set up a regular job to update data automatically

It would be nice to set up some job somewhere that commits an update to the api/schema data on a regular basis. That way the static site deployment wouldn't need to clone every time.

Configure CI and add tests

Use CircleCI, add tests, make them a push blocker

Different Pokemon Forms dont have formatted Names.

While every species has their Localized name, like as example, Sprigatito or Felori, depending on language, this doesn't apply to Formes.

I dont get a formatted string for every forme, like "Zoroark (Hisui)" or "Shaymin (Sky Forme)"

Needed CLI flags

• --version
• --quiet or maybe make quiet the default and add --log

Docker Compose build fails using python alpine image

Running docker-compose up --build results in an error during poetry install.

My guess is that the alpine image is missing essential build tools like gcc. Swapping this image out for python:3-bullseye results in a successful build, at the cost of a larger container size.

[+] Building 40.6s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 463B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 66B                                                                                   0.0s
 => [internal] load metadata for docker.io/library/python:3-alpine                                                 2.6s
 => [1/5] FROM docker.io/library/python:3-alpine@sha256:ba6cfcca463537621aac63ffda4f93cd73e1f3dea59a83287603fbebd  2.9s
 => => resolve docker.io/library/python:3-alpine@sha256:ba6cfcca463537621aac63ffda4f93cd73e1f3dea59a83287603fbebd  0.0s
 => => sha256:530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc 2.80MB / 2.80MB                     0.4s
 => => sha256:ba6cfcca463537621aac63ffda4f93cd73e1f3dea59a83287603fbebd02444e4 1.65kB / 1.65kB                     0.0s
 => => sha256:fc633a94d6239c161ead1fd3a192b5853e5807da6239f1ac9cf20c31391af3ab 1.37kB / 1.37kB                     0.0s
 => => sha256:eb1dfcc4f9e9884135653d74000b3a174fd77cc76a557601e9b763a064ba67ef 7.04kB / 7.04kB                     0.0s
 => => sha256:cc8c14b1a767335de44f2bc926cb52487979a0fe602ac5429643dbb238f297fb 666.77kB / 666.77kB                 0.5s
 => => sha256:bd99fa58365b603cbeb71c93c19182410b640606f36158c10d7ee09d63c1a4f6 12.18MB / 12.18MB                   1.3s
 => => extracting sha256:530afca65e2ea04227630ae746e0c85b2bd1a179379cbf2b6501b49c4cab2ccc                          0.5s
 => => sha256:777a82aef5431a7852c25deef1cc9e479a3be2c2a9f49e41306b9da78184f1ef 231B / 231B                         0.6s
 => => sha256:1ef14832cf3c6d891726d458ea44fb7be8662b778048035236efe234a7c2699f 2.87MB / 2.87MB                     1.1s
 => => extracting sha256:cc8c14b1a767335de44f2bc926cb52487979a0fe602ac5429643dbb238f297fb                          0.4s
 => => extracting sha256:bd99fa58365b603cbeb71c93c19182410b640606f36158c10d7ee09d63c1a4f6                          0.5s
 => => extracting sha256:777a82aef5431a7852c25deef1cc9e479a3be2c2a9f49e41306b9da78184f1ef                          0.0s
 => => extracting sha256:1ef14832cf3c6d891726d458ea44fb7be8662b778048035236efe234a7c2699f                          0.3s
 => [internal] load build context                                                                                  1.6s
 => => transferring context: 59.06MB                                                                               1.5s
 => [2/5] RUN pip install poetry                                                                                  18.0s
 => [3/5] ADD . /ditto                                                                                             0.2s
 => [4/5] WORKDIR /ditto                                                                                           0.1s
 => ERROR [5/5] RUN poetry install                                                                                16.5s
------
 > [5/5] RUN poetry install:
#0 1.441 Creating virtualenv pokeapi-ditto-FLGrt-Xt-py3.10 in /root/.cache/pypoetry/virtualenvs
#0 2.508 Installing dependencies from lock file
#0 2.924
#0 2.924 Package operations: 27 installs, 0 updates, 0 removals
#0 2.924
#0 2.926   • Installing appdirs (1.4.3)
#0 2.928   • Installing atomicwrites (1.4.0)
#0 2.928   • Installing attrs (19.3.0)
#0 2.931   • Installing certifi (2020.4.5.1)
#0 2.937   • Installing chardet (3.0.4)
#0 2.942   • Installing click (7.1.2)
#0 2.947   • Installing entrypoints (0.3)
#0 2.950   • Installing idna (2.9)
#0 2.955   • Installing mccabe (0.6.1)
#0 2.957   • Installing multidict (4.7.5)
#0 2.961   • Installing more-itertools (8.2.0)
#0 2.967   • Installing pluggy (0.13.1)
#0 7.981   • Installing py (1.8.1)
#0 8.028   • Installing pycodestyle (2.5.0)
#0 8.084   • Installing pyflakes (2.1.1)
#0 8.123   • Installing six (1.14.0)
#0 8.137   • Installing toml (0.10.0)
#0 8.142   • Installing urllib3 (1.25.9)
#0 16.25
#0 16.25   EnvCommandError
#0 16.25
#0 16.25   Command ['/root/.cache/pypoetry/virtualenvs/pokeapi-ditto-FLGrt-Xt-py3.10/bin/pip', 'install', '--no-deps', '/root/.cache/pypoetry/artifacts/bf/d3/e2/eaa2fd0d0109e16c0a1c14ddd0619083d61bc4d6b33076433fab541d97/multidict-4.7.5.tar.gz'] errored with the following return code 1, and output:
#0 16.25   Processing /root/.cache/pypoetry/artifacts/bf/d3/e2/eaa2fd0d0109e16c0a1c14ddd0619083d61bc4d6b33076433fab541d97/multidict-4.7.5.tar.gz
#0 16.25     Installing build dependencies: started
#0 16.25     Installing build dependencies: finished with status 'done'
#0 16.25     Getting requirements to build wheel: started
#0 16.25     Getting requirements to build wheel: finished with status 'done'
#0 16.25     Preparing metadata (pyproject.toml): started
#0 16.25     Preparing metadata (pyproject.toml): finished with status 'done'
#0 16.25   Building wheels for collected packages: multidict
#0 16.25     Building wheel for multidict (pyproject.toml): started
#0 16.25     Building wheel for multidict (pyproject.toml): finished with status 'error'
#0 16.25     error: subprocess-exited-with-error
#0 16.25
#0 16.25     × Building wheel for multidict (pyproject.toml) did not run successfully.
#0 16.25     │ exit code: 1
#0 16.25     ╰─> [72 lines of output]
#0 16.25         /tmp/pip-build-env-8_0t84dp/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:463: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
#0 16.25           warnings.warn(msg, warning_class)
#0 16.25         **********************
#0 16.25         * Accellerated build *
#0 16.25         **********************
#0 16.25         running bdist_wheel
#0 16.25         running build
#0 16.25         running build_py
#0 16.25         creating build
#0 16.25         creating build/lib.linux-x86_64-cpython-310
#0 16.25         creating build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/_multidict_py.py -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/_abc.py -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/_multidict_base.py -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/_compat.py -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/__init__.py -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         running egg_info
#0 16.25         writing multidict.egg-info/PKG-INFO
#0 16.25         writing dependency_links to multidict.egg-info/dependency_links.txt
#0 16.25         writing top-level names to multidict.egg-info/top_level.txt
#0 16.25         reading manifest file 'multidict.egg-info/SOURCES.txt'
#0 16.25         reading manifest template 'MANIFEST.in'
#0 16.25         warning: no previously-included files matching '*.pyc' found anywhere in distribution
#0 16.25         warning: no previously-included files found matching 'multidict/_multidict.html'
#0 16.25         warning: no previously-included files found matching 'multidict/*.so'
#0 16.25         warning: no previously-included files found matching 'multidict/*.pyd'
#0 16.25         warning: no previously-included files found matching 'multidict/*.pyd'
#0 16.25         no previously-included directories found matching 'docs/_build'
#0 16.25         adding license file 'LICENSE'
#0 16.25         writing manifest file 'multidict.egg-info/SOURCES.txt'
#0 16.25         /tmp/pip-build-env-8_0t84dp/overlay/lib/python3.10/site-packages/setuptools/command/build_py.py:153: SetuptoolsDeprecationWarning:     Installing 'multidict._multilib' as data is deprecated, please list it in `packages`.
#0 16.25             !!
#0 16.25
#0 16.25
#0 16.25             ############################
#0 16.25             # Package would be ignored #
#0 16.25             ############################
#0 16.25             Python recognizes 'multidict._multilib' as an importable package,
#0 16.25             but it is not listed in the `packages` configuration of setuptools.
#0 16.25
#0 16.25             'multidict._multilib' has been automatically added to the distribution only
#0 16.25             because it may contain data files, but this behavior is likely to change
#0 16.25             in future versions of setuptools (and therefore is considered deprecated).
#0 16.25
#0 16.25             Please make sure that 'multidict._multilib' is included as a package by using
#0 16.25             the `packages` configuration field or the proper discovery methods
#0 16.25             (for example by using `find_namespace_packages(...)`/`find_namespace:`
#0 16.25             instead of `find_packages(...)`/`find:`).
#0 16.25
#0 16.25             You can read more about "package discovery" and "data files" on setuptools
#0 16.25             documentation page.
#0 16.25
#0 16.25
#0 16.25         !!
#0 16.25
#0 16.25           check.warn(importable)
#0 16.25         copying multidict/__init__.pyi -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/_multidict.c -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         copying multidict/py.typed -> build/lib.linux-x86_64-cpython-310/multidict
#0 16.25         creating build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         copying multidict/_multilib/defs.h -> build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         copying multidict/_multilib/dict.h -> build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         copying multidict/_multilib/istr.h -> build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         copying multidict/_multilib/iter.h -> build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         copying multidict/_multilib/pair_list.h -> build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         copying multidict/_multilib/views.h -> build/lib.linux-x86_64-cpython-310/multidict/_multilib
#0 16.25         running build_ext
#0 16.25         building 'multidict._multidict' extension
#0 16.25         creating build/temp.linux-x86_64-cpython-310
#0 16.25         creating build/temp.linux-x86_64-cpython-310/multidict
#0 16.25         gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/root/.cache/pypoetry/virtualenvs/pokeapi-ditto-FLGrt-Xt-py3.10/include -I/usr/local/include/python3.10 -c multidict/_multidict.c -o build/temp.linux-x86_64-cpython-310/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
#0 16.25         error: command 'gcc' failed: No such file or directory
#0 16.25         [end of output]
#0 16.25
#0 16.25     note: This error originates from a subprocess, and is likely not a problem with pip.
#0 16.25     ERROR: Failed building wheel for multidict
#0 16.25   Failed to build multidict
#0 16.25   ERROR: Could not build wheels for multidict, which is required to install pyproject.toml-based projects
#0 16.25
#0 16.25   [notice] A new release of pip available: 22.1.2 -> 22.2
#0 16.25   [notice] To update, run: pip install --upgrade pip
#0 16.25
#0 16.25
#0 16.25   at /usr/local/lib/python3.10/site-packages/poetry/utils/env.py:1195 in _run
#0 16.34       1191│                 output = subprocess.check_output(
#0 16.34       1192│                     cmd, stderr=subprocess.STDOUT, **kwargs
#0 16.34       1193│                 )
#0 16.34       1194│         except CalledProcessError as e:
#0 16.34     → 1195│             raise EnvCommandError(e, input=input_)
#0 16.34       1196│
#0 16.34       1197│         return decode(output)
#0 16.34       1198│
#0 16.34       1199│     def execute(self, bin, *args, **kwargs):
#0 16.34
------
failed to solve: executor failed running [/bin/sh -c poetry install]: exit code: 1

Publish to PyPi

This should be the easiest way for the CI system to use ditto (PokeAPI/pokeapi#354).

Name conflict

The name ditto conflicts with the name of a program in MacOS. Should change it to ditto-api or something.

Configure code review flow

Replace dynamic URL placement with a guided solution

Instead of guessing the URL based on headers, add a runtime flag to ditto serve to specify replacement URL. Fall back to relative links when it's not specified.

We should also ensure external links don't fall prey to replacement (sprites are broken because of that atm).

Netlify is returning an object describing the site instead of our API result

I don't understand this Netlify bug(?) at all.

Test call: /api/v2/pokemon/

Response from my sandbox site: https://pastebin.com/raw/ZqyHfBT7
Response from the pokeapi site: https://pastebin.com/raw/LSxzc5Sb

Both sites are running identical code. What's up with that?

@neverendingqs you seem familiar with Netlify, you got any idea? I'd like to have more info to report to Netlify than just "this is broken sometimes".

Supporting v1 of PokeAPI

From PokeAPI/pokeapi#353:

I am also thinking to the v1 users. @phalt once said that a lot of calls are directed to the v1 endpoint. Is it possible to make ditto scan also the v1 endpoints?

Use pathlib wherever appropriate

Replace all raw path manipulation with pathlib to avoid future platform compatibility issues.

Generate an API spec

We should be able to use the collection of generated schemas to generate an always up-to-date API spec. This spec would be used to generate API clients and maybe even documentation.

This looks like promising format: https://swagger.io/specification/

CORS Headers

JSONs minification

It would be nice to minify the JSON files that Ditto retrieve from Pokeapi.

This can be achieved by json.dumps()