alltheplaces / alltheplaces.xyz

The website for All The Places.

All the Places

A project to generate point of interest (POI) data sourced from websites with 'store location' pages. The project uses scrapy, a popular Python-based web scraping framework, to execute individual site spiders that retrieve POI data, publishing the results in a standard format. There are various scrapy tutorials on the Internet and this series on YouTube is reasonable.

Getting started

Development setup

Windows users may need to follow some extra steps, please follow the scrapy docs for up to date details.

Ubuntu

These instructions were tested with Ubuntu 22.04.1 LTS on 2024-02-21.

1. Install Python 3 and pip:

   $ sudo apt-get update
   $ sudo apt-get install -y python3 python3-pip python-is-python3
   

2. Install pyenv and ensure the correct version of Python is available. The following is a summary of the steps, please refer to the pyenv documentation for the most up-to-date instructions.

   $ sudo apt-get install -y build-essential libssl-dev zlib1g-dev \
         libbz2-dev libreadline-dev libsqlite3-dev curl git \
         libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev \
         libffi-dev liblzma-dev
   $ curl https://pyenv.run | bash
   $ echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bashrc
   $ echo 'eval "$(pyenv init --path)"' >> ~/.bashrc
   $ echo 'eval "$(pyenv init -)"' >> ~/.bashrc
   $ exec "$SHELL"
   $ pyenv install 3.11
   

3. Install pipenv and check that it runs:

   $ pip install --user pipenv
   $ pipenv --version
   pipenv, version 2023.12.1
   

4. Clone a copy of the project from the All the Places repo (or your own fork if you are considering contributing to the project):

   $ git clone [email protected]:alltheplaces/alltheplaces.git
   

5. Use pipenv to install the project dependencies:

   $ cd alltheplaces
   $ pipenv sync
   

6. Test for successful project installation:

   $ pipenv run scrapy
   

   If the above runs without complaint, then you have a functional installation and are ready to run and write spiders.

macOS

These instructions were tested with macOS 14.3.1 on 2024-02-21.

1. Install Python 3 and pip:

   $ brew install python@3
   

2. Install pyenv and ensure the correct version of Python is available. The following is a summary of the steps, please refer to the pyenv documentation for the most up-to-date instructions.

   $ brew install pyenv
   $ echo 'eval "$(pyenv init --path)"' >> ~/.zshrc
   $ echo 'eval "$(pyenv init -)"' >> ~/.zshrc
   $ exec "$SHELL"
   $ pyenv install 3.11
   

3. Install pipenv and check that it runs:

   $ brew install pipenv
   $ pipenv --version
   pipenv, version 2023.12.1
   

4. Clone a copy of the project from the All the Places repo (or your own fork if you are considering contributing to the project):

   $ git clone [email protected]:alltheplaces/alltheplaces.git
   

5. Use pipenv to install the project dependencies:

   $ cd alltheplaces
   $ pipenv sync
   

6. Test for successful project installation:

   $ pipenv run scrapy
   

   If the above runs without complaint, then you have a functional installation and are ready to run and write spiders.

Codespaces

You can use GitHub Codespaces to run the project. This is a cloud-based development environment that is created from the project's repository and includes a pre-configured environment with all the tools you need to develop the project. To use Codespaces, click the button below:

Docker

You can use Docker to run the project. This is a container-based development environment that is created from the project's repository and includes a pre-configured environment with all the tools you need to develop the project.

1. Clone a copy of the project from the All the Places repo (or your own fork if you are considering contributing to the project):

   $ git clone [email protected]:alltheplaces/alltheplaces.git
   

2. Build the Docker image:

   $ cd alltheplaces
   $ docker build -t alltheplaces .
   

3. Run the Docker container:

   $ docker run -it alltheplaces
   

Contributing code

Many of the sites provide their data in a standard format. Others export their data via simple APIs. We have a number of guides to help you develop spiders:

• What should I call my spider?
• Using Wikidata and the Name Suggestion Index
• Sitemaps make finding POI pages easier
• Data from many POI pages can be extracted without writing code
• What is expected in a pull request?
• What we do behind the scenes

The weekly run

The output from running the project is published on a regular cadence to our website: alltheplaces.xyz. You should not run all the spiders to pick up the output: the less the project "bothers" a website the more we will be tolerated.

Contact us

Communication is primarily through tickets on the project GitHub issue tracker. Many contributors are also present on OSM US Slack, in particular we watch the #poi channel.

License

The data generated by our spiders is provided on our website and released under Creative Commons’ CC-0 waiver.

The spider software that produces this data (this repository) is licensed under the MIT license.

alltheplaces.xyz's Issues

alltheplaces.xyz/map is not linking back

I just shared https://www.alltheplaces.xyz/map/#8.11/51.92/0.447 and realised that I need to share also https://www.alltheplaces.xyz/ as map does not link back (or count on people being able to edit URL to get back to main site)

attribution could be also more clear and explain that dots are from ATP and basemap from OSM

CSS preventing access to link

On the main page of the website https://www.alltheplaces.xyz/ I am unable to click the "alltheplaces" link in the bottom left because the :hover CSS causes the link to move before it can be clicked:

Apparently there is this CSS rule:

a:hover, a :focus {
    color: #069;
    font-weight: bold;
}

and the change to bold forces the line to wrap. Replacing the font-weight: bold with something like text-decoration: underline should solve the problem. I couldn't find this CSS in the repo -- I guess it is part of jekyll-theme-minimal?

Map seems to show no POIs

Maybe it just something bad with my phone but https://www.alltheplaces.xyz/map/ seems to but show POIs anymore

(As usual, clearing cache, restart and changing browser has not helped but opening issue fixed bug :) )

Consider building dataset compared with OpenStreetMap

Of potential interest:

• listing objects without nearby OSM match, revealing locations potentially worth surveying
• listing OSM objects marked as branded with brand present in ATP and no nearby ATP match

and publishing both as a map.

Both can end detecting bad data in ATP rather than missing/outdated shops in OSM, definitely would not be usable for direct import but I think it would be of interest and useful for further reprocessing.

It is possible that I would be able to implement something like that if there would be interest in running and publishing it.

"Check details for the latest build" has no details for latest build

Check details for the latest build at the https://www.alltheplaces.xyz/ links https://www.alltheplaces.xyz/spiders

Main page and https://data.alltheplaces.xyz/runs/latest.json reports that last build completed at 2024-02-26T01:05:44Z

But https://www.alltheplaces.xyz/spiders has no entry for latest run

note: I tried disabling adblock just in case it is hit by wildcard block on latest or similar.

Some pages are broken

https://www.alltheplaces.xyz/spiders and https://www.alltheplaces.xyz/wikidata pages does not work anymore, maybe after this commit: 6c37dbc

Console log on spiders page:

Add PMTiles links to the previous builds page and a link to the latest PMTiles output on the home page

Can PMTiles output be advertised on alltheplaces.xyz to make it discoverable? For example, adding a link to the PMTiles URL of the latest build (e.g. https://data.alltheplaces.xyz/runs/2023-12-23-13-31-59/output.pmtiles) on the home page, and adding a new column with PMTiles links in the previous runs page?

Add links to the logs

Currently getting to the logs is a pain, as far as I know the only way is by knowing the url. Adding a link to them would help (maybe spiders.html on the features count?)