tilezen / tileserver

A lightweight tileserver to share code paths with tilequeue for tile generation.

tileserver

A lightweight tileserver to share code paths with tilequeue for tile generation.

Installation

We recommend following the vector-datasource installation instructions.

There is a requirements file that can be used to install.

pip install -Ur requirements.txt

Then:

python setup.py develop

Installation (detailed)

In addition to the dependencies in requirements.txt, tileserver requires

• PostgreSQL client-side development library and headers (for psycopg)
• GEOS library

These can be installed on Debian-based systems with

sudo apt-get install libpq-dev libgeos-c1v5

Then install the python requirements with

pip install -Ur requirements.txt

Then:

python setup.py develop

Usage

cp config.yaml.sample config.yaml
python tileserver/__init__.py config.yaml

tileserver's Issues

Remove custom layer handling logic

Custom layer handling logic is done by xonacatl now.

error trying to start tileserver - ImportError: cannot import name on_invalid_geometry_make_valid

I am getting this error when attempting to start tilserver:

I get this when using git tags 0.5.0, 0.5.1, 0.6.1 and on Ubuntu 14.04 and 16.04

Traceback (most recent call last):
  File "/tileserver/tileserver/__init__.py", line 5, in <module>
    from tilequeue.command import make_queue
  File "/usr/local/lib/python2.7/dist-packages/tilequeue/command.py", line 9, in <module>
    from tilequeue.format import lookup_format_by_extension
  File "/usr/local/lib/python2.7/dist-packages/tilequeue/format/__init__.py", line 3, in <module>
    from tilequeue.format.mvt import encode as mvt_encode
  File "/usr/local/lib/python2.7/dist-packages/tilequeue/format/mvt.py", line 1, in <module>
    from mapbox_vector_tile.encoder import on_invalid_geometry_make_valid
ImportError: cannot import name on_invalid_geometry_make_valid

Is this a circular dependency of some sort?

Thanks.

Add tapalcatl support

Store file format results as ZIP.

See: https://github.com/tilezen/tapalcatl

ImportError: No module named tileserver.cache

I updated from tileserver v2.0.0 and tilequeue v1.7.0 to tileserver v2.1.0 and tilequeue v1.8.0 and now when trying to run the server I get:

root@vps405830:/home/usr/tileserver# python tileserver/__init__.py config.yaml
Traceback (most recent call last):
  File "tileserver/__init__.py", line 14, in <module>
    from tileserver.cache import CacheKey
ImportError: No module named tileserver.cache

I am using an ansible script which worked fine before. After changing versions I get the error. I can also not import tileserver when in Python.

Python versions

What python versions are supported?

Enforce max zoom

Some requests for high zooms [1] end up producing overflows and generate errors [2]. Can we start enforcing a max zoom, and return 404s for zooms higher than that?

@nvkelso what would be reasonable? 20? 25?

[1] https://tile.mapzen.com/mapzen/vector/v1/all/9999/1/1.json
[2]

File "/usr/local/lib/python2.7/dist-packages/tilequeue/tile.py", line 319, in calc_meters_per_pixel_dim (2 ** (zoom + 8))) | OverflowError: long int too large to convert to float

psycopg2.ProgrammingError

I tried to run tileserver on my machine by following this tutorial: https://github.com/tilezen/vector-datasource/wiki/Mapzen-Vector-Tile-Service . Everything was installed successfully but when I try to retrieve tile from this server I get an error psycopg2.ProgrammingError: ProgrammingError: function jsonb_build_object(unknown, unknown, unknown, bigint, unknown, unknown, unknown, real) does not exist LINE 8: jsonb_build_object( ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. from file query.py from tilequeue library. I am also attaching screenshot:

For all libraries I am using master branch as when I tried to use versions recommended in that tutorial I get another errors in Python.

I am not sure whether to post this issue into tilequeue or vector-datasource repository.

Be more resilient to storage errors

While S3 was down, we failed a bunch of requests because we try to read the tile from S3 as a short-cut before querying it from the database. Unfortunately, there are many things that can go wrong with fetching a tile over the network.

I've grabbed a selection of errors from the logs at the bottom of this issue.

The request could have been served by querying the database, but these exceptions thrown mean we return 500 errors instead. Throwing exceptions seems appropriate for tilequeue, where we want to make sure that the job isn't marked complete unless the tile is stored safely. For tileserver, where there's a client waiting on the other end of a connection, perhaps we want to catch and log the exceptions and then continue as if the tile were missing?

After the tile has been rendered, we store it. It looks like if that raises an exception then we just log it and move on rather than stopping, since it happens on a background thread.

Traceback (most recent call last): 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 204, in __call__    response = self.handle_request(request) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 249, in handle_request    tile_data = self.reformat_from_stored_json(request_data, layer_data) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 331, in reformat_from_stored_json    tile_data = self.read_tile(coord) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 389, in read_tile    raw_data = self.store.read_tile(coord, fmt, 'all') 
   File "/usr/local/lib/python2.7/dist-packages/tilequeue/store.py", line 60, in read_tile    key = self.bucket.get_key(key_name) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 192, in get_key    key, resp = self._get_key_internal(key_name, headers, query_args_l) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 199, in _get_key_internal    query_args=query_args) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request    retry_handler=retry_handler 
   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1068, in make_request    retry_handler=retry_handler) 
   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 942, in _mexe    request.body, request.headers) 
   File "/usr/lib/python2.7/httplib.py", line 1017, in request    self._send_request(method, url, body, headers) 
   File "/usr/lib/python2.7/httplib.py", line 1051, in _send_request    self.endheaders(body) 
   File "/usr/lib/python2.7/httplib.py", line 1013, in endheaders    self._send_output(message_body) 
   File "/usr/lib/python2.7/httplib.py", line 864, in _send_output    self.send(msg) 
   File "/usr/lib/python2.7/httplib.py", line 826, in send    self.connect() 
   File "/usr/lib/python2.7/httplib.py", line 1216, in connect    self.timeout, self.source_address) 
   File "/usr/lib/python2.7/socket.py", line 571, in create_connection    raise err 
 timeout: timed out

Traceback (most recent call last): 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 204, in __call__    response = self.handle_request(request) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 249, in handle_request    tile_data = self.reformat_from_stored_json(request_data, layer_data) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 331, in reformat_from_stored_json    tile_data = self.read_tile(coord) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 389, in read_tile    raw_data = self.store.read_tile(coord, fmt, 'all') 
   File "/usr/local/lib/python2.7/dist-packages/tilequeue/store.py", line 60, in read_tile    key = self.bucket.get_key(key_name) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 192, in get_key    key, resp = self._get_key_internal(key_name, headers, query_args_l) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 199, in _get_key_internal    query_args=query_args) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request    retry_handler=retry_handler 
   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1068, in make_request    retry_handler=retry_handler) 
   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1025, in _mexe    raise BotoServerError(response.status, response.reason, body) 
 BotoServerError: BotoServerError: 500 Internal Server Error

Traceback (most recent call last): 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 204, in __call__    response = self.handle_request(request) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 249, in handle_request    tile_data = self.reformat_from_stored_json(request_data, layer_data) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 331, in reformat_from_stored_json    tile_data = self.read_tile(coord) 
   File "/usr/local/lib/python2.7/dist-packages/tileserver/__init__.py", line 389, in read_tile    raw_data = self.store.read_tile(coord, fmt, 'all') 
   File "/usr/local/lib/python2.7/dist-packages/tilequeue/store.py", line 60, in read_tile    key = self.bucket.get_key(key_name) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 192, in get_key    key, resp = self._get_key_internal(key_name, headers, query_args_l) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/bucket.py", line 199, in _get_key_internal    query_args=query_args) 
   File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request    retry_handler=retry_handler 
   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1068, in make_request    retry_handler=retry_handler) 
   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1025, in _mexe    raise BotoServerError(response.status, response.reason, body) 
 BotoServerError: BotoServerError: 503 Service Unavailable

Could not run v1.3.0

Hello,
I've followed the install instructions located here

However when trying to run tileserver I get the following error:

Traceback (most recent call last):
File "tileserver/init.py", line 19, in
from tilequeue.metatile import make_single_metatile, extract_metatile
ImportError: cannot import name make_single_metatile

I didn't get any error during install, so I really have no idea about what went wrong.

Start meta-tiling at zoom 16 (from 18)

• https://github.com/mapzen/tileserver/blob/c60fb611fa1de31fe45dcf2b5bc9a3c7f00762e5/tileserver/__init__.py#L302-L323

pip intall error

Hello team :)

I have an error when i am trying to install tile server,

pip install -Ur tileserver/requirements.txt
Collecting AppDirs==1.4.3 (from -r tileserver/requirements.txt (line 1))
  Using cached appdirs-1.4.3-py2.py3-none-any.whl
Requirement already up-to-date: argparse==1.2.1 in /usr/lib/python2.7 (from -r tileserver/requirements.txt (line 2))
[...]
  Could not find a version that satisfies the requirement raw_tiles (from tilequeue->-r tileserver/requirements.txt (line 27)) (from versions: )
No matching distribution found for raw_tiles (from tilequeue->-r tileserver/requirements.txt (line 27))

can you help me to resolve this ?.

Thanks to you :)

CORS problem due to missing min_zoom in layer water

Hello,

in case of error situation:

env/lib/python2.7/site-packages/tilequeue/process.py

min_zoom = output_props.get('min_zoom')
assert min_zoom is not None,
'Missing min_zoom in layer %s' % layer_name

the CORS „Access-Control-Allow-Origin” header is not generated...

Sincerely
Gregers

Not seeing any output through tileserver appears to be running. Any advice?

Hi,

I am fairly new to mapping software and am trying to set up the tileserver on AWS so that we can host our own tiles. I have followed the installation instructions on how to build everything from source that are listed on the repo. The tileserver appears to work but I am not able to access it.

I can confirm that my EC2 instance is accessible to the public. I created a node.js script that just writes something to the screen and I can access that via my servers public URL.

The tile server, however appears to not return any results.

• What did you see?

I can start the tileserver as per the instructions. After activating my env I get:

$ python tileserver/__init__.py config.yaml
Shapely speedups enabled
* Running on http://localhost:8080/ (Press CTRL+C to quit)
* Restarting with stat
Shapely speedups enabled
* Debugger is active!
* Debugger PIN: 335-198-251

• What did you expect to see?

When I visit my instance's public URL at 8080, I see: This site can’t be reached. When I try to access any of the test URL's that are provided in the instructions (i.e. http://localhost:8080/buildings/16/19293/24641.json) I get no response either.

• What map location are you having problems with?

Are there any instructions online on how to use the debugger?

Relevant parts of my yaml config file for the tileserver, the username for postgres has superuser privleges...

postgresql:
  host: localhost
  port: 5432
  # multiple databases can be specified, and these are iterated
  # through to balance query loads. This is useful when connecting to
  # pgbouncer, which can dispatch to different back end databases
  # based on the name.
  dbnames: [osm]
  user: _secret!_
  password: _secret!_
queries:
  config: ../vector-datasource/queries.yaml
  template-path: ../vector-datasource/queries
  # whether to reload the jinja query templates. This should be off in
  # production.
  reload-templates: true

# format extensions to support
# buffered Mapbox Vector Tiles are also possible by specifying mvtb
formats: [json, topojson, mvt]
# buffer configuration, see tilequeue sample config for details
buffer: {}

server:
  host: localhost
  port: 8080
  debug: true
  reload: true
  threaded: false

Any advice / links on how to further troubleshoot this would be appreciated! Not sure what to do at the moment other than dig through the source code...

syntax error

i am getting an syntax error when i run tileserver:

Tileserver is not accepting remote connections

Hi,

I installed successfully the tileserver, and it handles request correctly, but only if I try from local machine (ex. curl http://localhost:8080/all/16/19293/24641.json). If I try from a remote machine (ex. http://[remoteip]:8080/all/16/19293/24641.json, the tileserver doesn't accept remote connection.
All stuffs related to firewall and ports are properly configured.
There is some parameter to tweak in order to handle remote requests?

Thanks

Always query for all layer data

Some of the post processing transform steps can dynamically create layers, and have implicit dependencies on source layer data to be present. To keep things simple, we should always get the query data for all layers, regardless of the request, and just prune the list of layers afterwards as necessary.

Unknown WKB type 49

Hi!

I get this error and backtrace when I try requesting tiles from my mapzen tile server.

127.0.0.1 - - [19/Feb/2016 15:13:17] "GET /all/15/9644/12322.json HTTP/1.1" 500 -
Traceback (most recent call last):
File "/Users/divya/Documents/oneconcern/Vector-tile-server/Mapzen/tileserver/tileserver/init.py", line 187, in call
response = self.handle_request(request)
File "/Users/divya/Documents/oneconcern/Vector-tile-server/Mapzen/tileserver/tileserver/init.py", line 283, in handle_request
[])
File "/usr/local/lib/python2.7/site-packages/tilequeue-0.7.1-py2.7.egg/tilequeue/process.py", line 323, in process_coord
feature_layers = _preprocess_data(feature_layers, shape_padded_bounds)
File "/usr/local/lib/python2.7/site-packages/tilequeue-0.7.1-py2.7.egg/tilequeue/process.py", line 26, in _preprocess_data
shape = loads(wkb)
File "/usr/local/lib/python2.7/site-packages/shapely/wkb.py", line 16, in loads
return reader.read(data)
File "/usr/local/lib/python2.7/site-packages/shapely/geos.py", line 427, in read
raise ReadingError("Could not create geometry because of errors "
ReadingError: Could not create geometry because of errors while reading input.
ParseException: Unknown WKB type 49

Why am I getting this? Does this mean something is wrong with my data? I am using psql (PostgreSQL) 9.5.0 as my db.
Thanks!

Is there a default SRID?

I'm wondering if there is a default SRID assumed for the underlying queries (bounding boxes etc?) And somewhat related, whether there is documentation available for the queries and template files?

I am trying a basic setup including a queries.yaml file as such:

all:
  - blocks_data

layers:
  blocks_data:
    start_zoom: 0
    area-inclusion-threshold: 1
    template: poly.jinja2
    clip: False
    clip_factor: 1
    geometry_types: [Polygon]
    transform: []
      # see https://github.com/tilezen/vector-datasource/blob/master/vectordatasource/transform.py
    sort: []
      # see: https://github.com/tilezen/vector-datasource/blob/master/vectordatasource/sort.py
    simplify_before_intersect: False
    simplify_start: 0

post_process: []

And a corresponding poly.jinja2 file as follows:

SELECT
    "id" AS __id__,
    "param1",
    "param2",
    {% filter geometry %}geom{% endfilter %} AS __geometry__
FROM
    blocks_data
WHERE
    {{ bounds['polygon']|bbox_filter('geom') }}

I've tried this on a table with geometry represented in SRID 4326, which didn't return anything, so also tried on a table with the geometry stored in 3857, but no luck yet.

Thanks.

Server crashes when receiving little bit of load

Hello again, so basically like the title says, is only me using the server but it crashes and throws errors, is there any suggestion on how to tune it up to make it more performant and resilient?

ERROR when run __init__.py

root@mapsapi:~/tileserver# python tileserver/init.py config.yaml
Traceback (most recent call last):
File "tileserver/init.py", line 546, in
tile_server = create_tileserver_from_config(config)
File "tileserver/init.py", line 475, in create_tileserver_from_config
os.path.dirname(queries_config_path))
File "/usr/local/lib/python2.7/dist-packages/tilequeue/command.py", line 413, in parse_layer_data
resources = _parse_postprocess_resources(post_process_item, cfg_path)
File "/usr/local/lib/python2.7/dist-packages/tilequeue/command.py", line 338, in _parse_postprocess_resources
*sys.exc_info()))))
Exception: Unable to init resource 'matcher' with function 'vectordatasource.transform.CSVMatcher' due to Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tilequeue/command.py", line 332, in _parse_postprocess_resources
fn = resolve(init_fn_name)
File "/usr/local/lib/python2.7/dist-packages/zope/dottedname/resolve.py", line 30, in resolve
found = import(used)
ImportError: No module named vectordatasource

Redis caching problem on last config file

Anyone can help me?
Thanks

# to cache tiles locally, enable a cache. This can be useful when
# experiencing timeouts for low or mid zoom ranges.

cache:
   type: redis
   redis:
     url: redis://localhost:6379
#    file:
#      prefix: /home/map/mapzen/tilequeue/tiles #directory_prefix

Remove reading from store logic

With the new tapalcatl and xonacatl microservices in place, the logic in tileserver to try and read the json format from s3 should be removed.

settings for multiple vector tile sources

I am looking at how to combine multiple vector tile files. Could you explain what settings are needed to have a single entry point containing vector tile data of both (or multiple) sources?

[release] Bump version up to v1.0.0

custom layers in postgis

I have custom layers stored in postgres with geometry. How can I use tileserver to serve mvt from these layers?
Also, Is addition of more layers dynamically supported?

Handle SIGTERM gracefully

Upon receiving SIGTERM, tileserver should:

1. Start responding to health check requests with an error.
2. Wait a configurable grace period, or until all outstanding requests have finished.
3. Shut down.

This allows tileserver to work with ELB connection draining or HAProxy to terminate while not dropping any requests. If this is done along with staggering shutdowns / upgrades so that only part of the cluster is down at any one time, then no requests are lost.

Add tileserver to pypi

Is it possible to add tileserver to pypi? I could take this on, but would much rather a maintainer does so.

My Server seems very slow in serving tiles,pls explain how to add tilequeue for buffering

Hi ,
Thank a lot for providing such an awesome opensource ... you guys are awesome ..
I configured everything as per tutorial,but my server seems very slow in delivering tiles.Pls help us to make it smooth..

pip install fails

(ve) pnorman@pippin:~/osm/tilezen/ve$ pip install tileserver==2.1.0
Collecting tileserver==2.1.0
  Using cached tileserver-2.1.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-HUCgBa/tileserver/setup.py", line 5, in <module>
        with open(version_path) as fh:
    IOError: [Errno 2] No such file or directory: '/tmp/pip-build-HUCgBa/tileserver/VERSION'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-HUCgBa/tileserver/

This is in a clean virtualenv

ImportError: No module named meta.python

Hello! Many thanks for all this works!
while I was installing tileserver, I run into this error:
Traceback (most recent call last): File "tileserver/__init__.py", line 404, in <module> tile_server = create_tileserver_from_config(config) File "tileserver/__init__.py", line 375, in create_tileserver_from_config output_calc_mapping = make_output_calc_mapping(yaml_config) File "/home/gloria/env/local/lib/python2.7/site-packages/tilequeue/command.py", line 554, in make_output_calc_mapping from vectordatasource.meta.python import make_function_name_props ImportError: No module named meta.python

I have followed instruction reported here: https://github.com/tilezen/vector-datasource/wiki/Mapzen-Vector-Tile-Service

the error I got is the output of python tileserver/__init__.py config.yaml.
I am using a virtual machine with Ubuntu 14.04.
What should I do to fix it?

Cut zoom content for 17+ requests from z16 tiles

Instead of doing a fresh DB request for the zoom 17+ tile bounds, we should get the parent zoom 15 16 tile and from it cut the zoom 16+ 17+ content (since 2x2 metafiles at 512px size are starting to be a thing).

Error with python setup.py develop, install directory is hard-coded (or sudo is required)

Running python setup.py develop produces the following error:

running develop
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

   [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-31955.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

   /usr/local/lib/python2.7/dist-packages/

Does this not defeat the purpose of using a virtual environment? Should sudo be required?

Republish tilejson on dev

And update the deploy checklist docs

multi-threded operation in tileserver

Hello,

Just playing around with tileserver as a hobby on the home pc, and I noticed the server spawned by the werkzeug libraries chokes heavily on a single cpu core, while trying to hit the postgis database and render them as json tiles, etc

I wonder whether it would generally benefit if you allow the threaded parameter be read from config.yaml

1. Slightly modify the constructor adding threaded parameter explicitly:

run_simple(server_config['host'], server_config['port'], tile_server, threaded=server_config.get('threaded', False),
               use_debugger=server_config.get('debug', False),
               use_reloader=server_config.get('reload', False))

2. Allow the threaded entry in config.yaml, e.g.

server:
  host: localhost
  port: 8099
  debug: true
  reload: true
  threaded: true

I personally noticed a much better distribution of CPU load among cores after that.

Regards
silviu

AssertionError: Missing min_zoom in layer water

I installed vector-datasource as per instruction. Then I import data into database. after that I installed tileserver and ran the server. no matter what I requests it always shows error on different layers saying that missing min_zoom in layer. How can I solve this?

init.py

I have errors in scrypt init.py:

Traceback (most recent call last): File "/home/user01/tileserver/tileserver/__init__.py", line 195, in __call__ response = self.handle_request(request) File "/home/user01/tileserver/tileserver/__init__.py", line 290, in handle_request coord, self.layer_config.all_layers) File "/usr/local/lib/python2.7/dist-packages/tilequeue/query.py", line 206, in __call__ raise async_exception ProgrammingError: column "way_area" does not exist LINE 6: way_area::bigint AS area, ^

127.0.0.1 - - [12/Sep/2016 14:26:02] "GET /all/16/39619/20489.json HTTP/1.1" 500 - Traceback (most recent call last): File "/home/user01/tileserver/tileserver/__init__.py", line 195, in __call__ response = self.handle_request(request) File "/home/user01/tileserver/tileserver/__init__.py", line 290, in handle_request coord, self.layer_config.all_layers) File "/usr/local/lib/python2.7/dist-packages/tilequeue/query.py", line 206, in __call__ raise async_exception ProgrammingError: relation "water_polygons" does not exist LINE 16: FROM water_polygons t

Should we specify fixed versions in tagged releases?

We currently specify the master branch for tilezen software dependencies in requirements.txt, although we specify fixed, non-master versions in the vector-datasource release notes. This is confusing, since the installation instructions say to install requirements from requirements.txt. We should fix versions in requirements.txt or provide different installation instructions which install fixed versions.

See #75 for context.

Writing to cache/store after generation

Thank you for the great software!

Do I understand correctly, that at this time it is not possible to write vtile generated by tileserver on demand to cache/store. So that if the same request from the client comes in after some time, same tile would not have to be generated again.

Looks like currently cache/store is only filled by running tilequeue script and generating specified tiles (and it is hard to know which tiles would be requested, very low percentage of high zoom level tiles are ever requested so there is no point to generate them all and it takes a lot of time). Tiles not found in cache/store and generated on demand by tileserver are thrown away?

Thank you

Not returning all information like roads, buildings, etc

Hello I have successfully setup the tile server and imported the data from the docs, but when I run it against a javascript client the information that I receive from the endpoint does not contain streets or anything so my app draws an incomplete map.
Do you have any idea why this would be happening? Am I doing something wrong?
Thanks.

serve requests for custom (multi) layers more efficiently

Instead of issuing queries for custom layers, we can instead check the s3 cache to see if a particular tile exists, and then filter out the relevant layers to serve the response.

Question about scaling

Hello, I am not sure if this is a correct place, but I just have question regarding tileserver, tilepacks and vector datasource. I have a setup on Ubuntu 14.04 of vector datasource for creating mbtiles files. I had it working on desktop computer with 4-core CPU with 16 GB of DDR3. Now I have updated computer to 12-core CPU with 32 GB of DDR4. It is on NVMe SSD Samsung 960 PRO 2 TB. Creating mbtiles on this new computer is even slower, I hoped that it will be faster as it seemed that using more core CPU this could scale properly. In tileserver I have enabled threading and disabled debug. Could you think of something that I could do wrongly? Thank you a lot.

ImportError: cannot import name DataFetcher in v2.1.0, wrong tilequeue version in requirements

I have tileserver v2.1.0 checked out and correspondingly tilequeue v1.8.0 and vector-datasource v1.4.0 checked out. I created a new python virtualenv, installed the requirements from tileserver, and then went to tilequeue v1.8.0 and vector-datasource v1.4.0 and ran python setup.py develop (btw, should I not be passing a develop argument if using numbered releases?).

When I run python tileserver/__init__.py config.yaml I get

Traceback (most recent call last):
  File "tileserver/__init__.py", line 11, in <module>
    from tilequeue.query import DataFetcher
ImportError: cannot import name DataFetcher

Solution

I noticed the very bottom of the tileserver requirements.txt file has git+https://github.com/mapzen/tilequeue@master#egg=tilequeue. So I tried pip uninstall tilequeue and got

 pip uninstall tilequeue
Uninstalling tilequeue-1.9.0:
etc....

and then

cd ../tilequeue/
python setup.py develop
cd ../tileserver/
python tileserver/__init__.py config.yaml

Did not produce that error.

Add generate script

To preseed tiles in a given area, similar to https://github.com/tilezen/joerd/blob/master/scripts/generate.py or TileStache seed.

Cat readme into setup.py long description

Like we do here for tilequeue:

• https://github.com/tilezen/tilequeue/blob/master/setup.py#L12
• long_description=open('README.md').read(),

table missing in the postgresql database

hello everyone,

I am just starting and I want to install the mapzen tileserver on my raspberry pi 2.
I have folllowed the process described on the mapzen tileserver Wiki .
At the end, when starting my server locally, and typing this request( http://localhost:8080/all/16/19293/24641.json), still have the same error "ProgrammingError: column "mz_transit_level" does not exist LINE 26: AND mz_transit_level <= 16 ^

Performance question

I have successfully deployed tileserver on my local machine for creating mbtiles locally. On that machine I have Samsung 960 PRO 2 TB sdd, 32 GB of RAM and Intel i5 6600 cpu. I have also currently imported Europe extract into postgresql database and tile requests are really slow on this machine. I have didn't enabled caching as it caused some problems when I enabled it. I was thinking about upgrading system to some multicore AMD Ryzen cpu as this current cpu has only 4 cores. Maybe AMD Threadripper 1950X (16 cores 32 threads) when it will be released. Does tileserver speed of responding to requests benefit from cpu with multiple cores? Do you have some benchmarks? I can also see that responses are faster when there are less objects on current tile. Thank you.

No 'Access-Control-Allow-Origin' header is present on the requested resource.

Is there a way to set the access control headers for tileserver?

I am getting these errors:
No 'Access-Control-Allow-Origin' header is present on the requested resource.

Caching and reformatting

Thanks a lot for providing a tilezen stack. Love it.
I am curious about one thing: are you guys actually using this tileserver(master) code as it is in production?

I noticed that mapzen returns .mvt tiles quite fast. However when I run tileserver as it is with caching enabled, the cached .mvt will return pretty slow especially for mid-zoom levels that are largest in size. I looked into the code and found that reformat_from_stored_json will be called even I have .mvt tiles available in cache. Once modified to read without reformat it returns instantly as expected.

My question is - what is reformatting used for and why it is needed on each non-json tile request?

Provide binary via pip.

When installing tileserver with pip (or similar tools) you get a binary for tilequeue, but not tileserver.

I believe entry_points needs to be specified for tileserver.

I've not made use of entry_points before, but from my reading there needs to be a 'tileserver = tileserver.__init__:something' in a dict.