mapbox / mapbox-sdk-py

Python SDK for Mapbox APIs **DEVELOPMENT IS TEMPORARILY PAUSED, SEE CONTRIBUTING.md**

mapbox-sdk-py

A Python client for Mapbox web services. The Mapbox Python SDK is a low-level client API, not a Resource API such as the ones in . Its methods return objects containing HTTP responses from the Mapbox API.

Services

• Analytics V1 examples, website

  • API usage for services by resource.
  • available for premium and enterprise plans.

• Directions V5 examples, website

  • Profiles for driving, walking, and cycling
  • GeoJSON & Polyline formatting

• Distance V1 DEPRECATED

• Geocoding V5 examples, website

  • Forward (place names ⇢ longitude, latitude)
  • Reverse (longitude, latitude ⇢ place names)

• Map Matching V4 examples, website

  • Snap GPS traces to OpenStreetMap data

• Static Maps V4 examples, website

  • Generate standalone images from existing Mapbox mapids (tilesets)
  • Render with GeoJSON overlays

• Static Styles V1 examples, website

  • Generate standalone images from existing Mapbox styles
  • Render with GeoJSON overlays
  • Adjust pitch and bearing, decimal zoom levels

• Surface V4 examples, website

  • Interpolates values along lines. Useful for elevation traces.

• Uploads V1 examples, website

  • Upload data to be processed and hosted by Mapbox.

• Datasets V1 examples, website

  • Manage editable collections of GeoJSON features
  • Persistent storage for custom geographic data

• Tilequery V4 examples, website

  • Retrieve data about specific features from a vector tileset

• Maps V4 examples, website

  • Retrieve an image tile, vector tile, or UTFGrid in the specified format
  • Retrieve vector features from Mapbox Editor projects as GeoJSON or KML
  • Retrieve TileJSON metadata for a tileset
  • Retrieve a single marker image without any background map

Please note that there may be some lag between the release of new Mapbox web services and releases of this package.

Documentation

Please see https://mapbox-mapbox.readthedocs-hosted.com/en/latest/

Installation

$ pip install mapbox

Testing

pip install -e ".[test]"
python -m pytest

To run the examples as integration tests on your own Mapbox account

MAPBOX_ACCESS_TOKEN="MY_ACCESS_TOKEN" python -m pytest --doctest-glob='*.md' docs/*.md

See Also

• Mapbox API Documentation: https://www.mapbox.com/api-documentation/
• Javascript SDK: https://github.com/mapbox/mapbox-sdk-js
• Mapbox API command line interface: https://github.com/mapbox/mapbox-cli-py

mapbox-sdk-py's Issues

Coordinate format for reverse geocoding input

I'm always looking for symmetry on the command line, and since a JSON array is the most straightforward output of forward geocoding, it would be great to be able to consume it when reverse geocoding. For example, this (using jq) produces JSON:

$ mapbaux geocode "Bodega Bay, CA" --access-token MY_TOKEN | \
>  jq '.features[0].center' -c
[-123.0481,38.3332]

And the round trip could be

$ mapbaux geocode "Bodega Bay, CA" --access-token MY_TOKEN | \
> jq '.features[0].center' -c | \
> mapbaux geocode --rev --access-token MY_TOKEN

There's also precedent in rio-sample for JSON formatted coordinate pairs [x, y] or [long, lat]

https://github.com/mapbox/rasterio/blob/master/docs/cli.rst#sample

but it seems like it would be relatively easy to also accept x y or x, y. There's a pattern that could be expressed as a simple regex.

/cc @dnomadb @sbma44

Move --access-token option from command to group

New commands won't have to implement the option if it's on the group.

This means mbx --access-token MY_TOKEN geocode instead of mbx geocode --access-token MY_TOKEN.

Support for the token environment variable will be unchanged.

0.1 release plan

We can have a first release with geocoding only, but the CLI's name conflict with another mapbox needs to be resolved.

Create a test user

A Mapbox test user specifically for this module's tests.

Config file for mbx command

A config file is something quality tools have. It would allow developers to have per-directory access tokens among other things. .ini style is (see Python's distutils or https://pip.pypa.io/en/latest/user_guide.html#config-file) one way to go.

Conventional unix style names for environment variables

Environment variables are part of the API and I prefer the good old unix style MAPBOX_ACCESS_TOKEN to MapboxAccessToken. I don't see any conflicts.

/cc @sbma44 @tmcw @willwhite

Move profile parameter from Distance class to request

As we did for Directions.

Add type filtering feature

https://www.mapbox.com/developers/api/geocoding/#filter-type

We can maintain a list of types in the module and surface a choice from the list in the CLI.

CLI design principle and strategy

We're developing a simple and easy to use CLI for the mapbox client module both as a service to customers and as a check on the usability of the module.

• The CLI should be 99% about arg/opt definition and interoperation with other programs in the shell.
• The CLI will have first class support for JSON and GeoJSON.
• We'll rely on other programs like sed, awk, jq for finer parsing of responses from APIs at first, absorbing some of this when it makes sense.

There is a name conflict with another critical mapbox program. I wish we'd reserved mapbox for customer-facing programs and libraries, but we've crossed that bridge. Let's think of a better name for the CLI than "mapbaux" soon.

lon/lat parameters in forward() have no effect

Discovered while running doctests against the API.

=================================== FAILURES ===================================
_________________________ [doctest] docs/geocoding.md __________________________
125
126 Place results may be biased toward a given longitude and latitude.
127
128 ```python
129
130 >>> response = geocoder.forward(
131 ...     "1600 pennsylvania ave nw", lon=-118.220, lat=34.049)
132 >>> response.status_code
133 200
134 >>> response.geojson()['features'][0]['place_name']
Expected:
    "1600 Pennsylvania Ave, Los Angeles, California 90033, United States"
Got:
    '1600 Pennsylvania Ave NW, Washington, District of Columbia 20006, United States'

/Users/sean/code/mapbox-sdk-py/docs/geocoding.md:134: DocTestFailure
=========================== 1 failed in 0.45 seconds ===========================

https://github.com/mapbox/mapbox-sdk-py/blob/master/tests/test_geocoder.py#L164 isn't a sufficient test.

Support HTTP-centric and Geojson returns

We need a pattern for the sdk-py so we have methods that

• return raw responses (i.e don't hide the network)
• return custom geojson or other appropriate data structure parsed from the response (i.e. principle of least surprise, geojson in -> geojson out)

How do we do that? Possibilities...

1. Have paired methods with some naming convention
   • route and route_geojson or
   • _route and route
2. return type as a keyword arg (return_geojson=True)
3. man in the middle "attack" to massage the API response
4. generic functions or methods that accept the response and do stuff to it
5. ???

Review github3.py

Ian Cordasco's https://github.com/sigmavirus24/github3.py has some cool features and the Python community likes it. We should review and consider its model.

Particularly interesting is its login pattern

from github3 import login

g = login(user, password)

and the example of updating a user in two difference ways at https://github3py.readthedocs.org/en/master/examples/github.html#changing-your-user-information. One less resource-oriented:

new_name = 'J. Smith'
blog = 'http://www.example.com/'
company = 'Vandelay Industries'
bio = """# J. Smith

A simple man working at a latex factory
"""

if g.update_user(new_name, blog, company, bio=bio):
    print('Profile updated.')

and one more resource-oriented:

me = g.user()
if me.update(new_name, blog, company, bio=bio):
    print('Profile updated.')

Add proximity feature to forward geocoding

https://www.mapbox.com/developers/api/geocoding/#proximity

Rename Uploader.extract to Uploader.create

For more harmony with the JS SDK and the app server's own method.

Add Datasets API

The Data API isn't on https://www.mapbox.com/developers/api/ but let's get it in the module at preview time.

Move profile parameter to the request method

So that users can switch profiles without creating new service objects. I think one instance of a particular service per application ought to be the main use case.

name kwarg to uploads

Add Styles API

https://www.mapbox.com/developers/api/styles/

Add Uploads API

I feel like the most useful thing this module can do for a user is abstract over the staging requirement https://www.mapbox.com/developers/api/uploads/#Stage.a.file.on.Amazon.S3.

Would be nice if this fit well with Rasterio (raster datasets as a native concept). Shouldn't depend on Rasterio.

0.3 release

New APIs:

• Directions
• Distance
• Surface

TODO:

• update README
• update remove CLI
• close remaining issues
• tag -- Travis will make our PyPI releases for us

Tentative date for release: 10/29.

Simplest thing that could possibly work

#3 is rather fancy. More basic would be something like the class and tests in the demo module below:

# tests.py
# usage: py.test tests.py

import responses
import requests


# TODO: the Dataset class is here in the tests only to make a single
# file demonstration possible. Should be moved to a proper module.

class Dataset:
    """A very simple Dataset proxy

    This class is pretty close to the simplest thing that can possibly
    work. There's no state and no hiding of the network; proxy methods
    return `requests.Response`."""

    def __init__(self, name, **kwds):
        """Create a proxy for the named dataset"""
        self.name = name
        # N.B. this is not the correct API endpoint URI.
        self.baseuri = 'https://api.example.com/data/%s' % self.name

    @property
    def metadata(self):
        """Return a dict containing dataset metadata"""
        return requests.get('%s/metadata' % self.baseuri)


# Tests of the class follow.

def test_name():
    """Dataset's name is set and is accessible"""
    assert Dataset('lolwut').name == 'lolwut'


def test_baseuri():
    """Dataset's base uri is set and is accessible"""
    assert Dataset('lolwut').baseuri == 'https://api.example.com/data/lolwut'


@responses.activate
def test_metadata():
    """Dataset's metadata is accessible"""

    responses.add(
        responses.GET, 'https://api.example.com/data/lolwut/metadata',
        body='{"name": "lolwut"}', status=200,
        content_type='application/json')

    resp = Dataset('lolwut').metadata
    assert resp.status_code == 200
    assert resp.json()['name'] == 'lolwut'

An interface like that in #3 could be built upon this more explicitly client-server style of interface, which suggests that this is the actual one to implement.

Add Directions API

https://www.mapbox.com/developers/api/directions/

There's probably some higher-level functions that could be included (converting the route to a valid GeoJSON FeatureCollection comes to mind). Not sure if the sdk is the right place for those helper functions or if they should just be a cookbook/wiki/example script.

Rename Directions.route() to directions()

The JS SDK has getDirections() but that's not great for Python. How about directions() instead of route() @perrygeo?

Make doctests more resilient to API data changes

This doctest is suddenly failing, most likely due to changes in the geocoder data.

=================================== FAILURES ===================================
_________________________ [doctest] docs/geocoding.md __________________________
150 ## Reverse geocoding
151 
152 Places at a longitude, latitude point may be found using `Geocoder.reverse()`.
153 
154 ```python
155 
156 >>> response = geocoder.reverse(lon=-73.989, lat=40.733)
157 >>> response.status_code
158 200
159 >>> for f in response.geojson()['features']:
Differences (unified diff with -expected +actual):
    @@ -1,5 +1,5 @@
     Atlas Installation, 124 E 13th St, New York, New York 10003, United States: poi.2701346205
     Gramercy-Flatiron, New York, 10003, New York, United States: neighborhood.21161
    -New York, New York, United States: place.37501
    +New York, New York, United States: place.34530
     10003, New York, United States: postcode.2254639497
     New York, United States: region.628083222
/home/travis/build/mapbox/mapbox-sdk-py/docs/geocoding.md:159: DocTestFailure
===================== 1 failed, 1 skipped in 0.19 seconds ======================

Fixing manually for now but we should anticipate this happening fairly regularly and make the tests more resilient.

Add all API services to mbx

Catch credential failures in upload()

If the credentials can't be generated, the result is passed along and generates a crytic key error about AWS keys. We should hide all the AWS staging details and prevent the abstraction from leaking.

If the credentials fail, inform them of the need for a mapbox access key with upload:* scope.

Naming

We've got modules, service classes and methods to name:

from mapbox.services.<module> import <ServiceClass>
<ServiceClass>(...).<method>()

Let's come up with a way to keep them consistent with the public API website.

Discussed on phone call:
<module> comes from the url e.g. https://www.mapbox.com/developers/api/<module>/

Related to https://github.com/mapbox/mapbox-api-documentation/issues/1

Static Map API

Send User-Agent header with an SDK product token

Requests' default is

>>> import requests
>>> requests.utils.default_user_agent()
'python-requests/2.5.3 CPython/3.4.2 Darwin/13.4.0'

The thing to do is to prepend mapbox-sdk-py/$version.

Rename CLI to mbx

mapbaux -> mbx

Test AWS S3 interaction

The upload API touches AWS S3 services. We should use mocks to unit test this interaction. Initial attempts with moto were not successful due to this error

Raise exceptions on HTTP errors

http://www.python-requests.org/en/latest/user/quickstart/#response-status-codes

response.raise_for_status() will raise an HTTPError if not successful, returns None otherwise

Remove CLI

That's a separate project now: https://github.com/mapbox/mbx-cli.

Map Matching API

0.2 release

New features, new API versions.

• Close issues
• Bump version, update change log
• Tag
• Make and upload dists

Get user from token

As in https://github.com/mapbox/mapbox-sdk-js/blob/master/lib/get_user.js. This allows us to eliminate a duplicate parameter when creating dataset and upload service instances.

Functions to encode features

The directions and surface APIs both require ; delimited lon,lat coordinate pairs. The directions API takes a json payload with a coordinates array. Future APIs may require different encodings for geometry inputs.

But python users are going to be dealing with different data structures: geojson-like objects, shapely geometries, fiona datasources, geodataframes, etc. All of these implement the __geo_interface__ so that's a big win.

So Question 0: should we provide encoding functions or just leave it up to the user? If we assume the former, let's spec it out and define the behavior a bit more precisely.

Question 1 Should the encoding happen external to the API wrapper or internally?

# external
waypoints_string = encode_points(features)
resp = directions.route(waypoints_string, ...)

# internal
direction.route(features, ...)

Question 2 What can features consist of?

• An open fiona datasource?
• A single feature collection
• An iterable of features
• An iterable of geojson geometries
• An iterable of shapely geometries
• A single geo object
• WKT? WKB?

I wrote a vector data abstraction function for rasterstats that solves a similar problem - pass it an object and it will try it's best to get an iterable of feature-like dicts out of it. Maybe a good model, maybe not depending on how liberal/strict we'd want to be with inputs.

Question 3 Validation.
Should the encoding enforce known limitation on the API such as min/max number of points, geometry types (e.g. points only), precision requirements or others? Will geometries be "cast" i.e. multipoint encoded as a series of single points?

Just an idea at this point. But if we do it (and do it right) it could make interop with other python geo tools ridiculously smooth.

Add Distance API

https://www.mapbox.com/developers/api/distance/

Currently in preview but never too early to plan.

Testing

Testing against an API is not the way to go: it's slow, annoys devops, and you can't test offline. A talk I saw at PyCon (ref forthcoming) recommended https://github.com/getsentry/responses for mocking Requests, the library we're going to use to make actual requests.

The mock responses are well explained in the project README; the gist of it is that you create mock response objects with JSON bodies, status codes, headers, etc and register them to a URL and a HTTP method.

Add Surface API

https://www.mapbox.com/developers/api/surface/

README

JFDI, I know, but let's write a solid start of a README before we write code for this.

Problem with polyline module dependencies

@perrygeo I've traced our issues with failing tests on Travis to the polyline module requiring exactly one version of six. As shown by pipdeptree:

polyline==1.1 -> six [required: ==1.8.0, installed: 1.8.0]

Everybody would be better off if the requirement were six>=1.8.0. I'm going to pursue this upstream and pin the version locally for testing purposes.

Test examples in README

If we converted the examples in the README to doctests, we could use https://pytest.org/latest/doctest.html.

Some extra work will be needed to fake the access tokens.

Testing requests with non-deterministic json.dumps

The static maps API takes urlencoded GeoJSON data. We want to use responses to test that our URL is being constructed correctly. But python dictionaries have non-deterministic order so their json.dumps string may not match our mocked URL.

Do we:

• enforce sort order with sort_keys=True? Seems heavy handed to do that just for the sake of testing.
• create multiple mocked URLs for all permutations of the dictionary order? Not very appealing.
• somehow get responses to accept wildcards? Not currently possible it appears.

Any other options?

cc @sgillies

Explaining the SDK

Here's an important paragraph from http://aws.amazon.com/sdk-for-python/:

Boto3 has two distinct levels of APIs. Client (or "low-level") APIs provide one-to-one mappings to the underlying HTTP API operations. Resource APIs hide explicit network calls but instead provide resource objects and collections to access attributes and perform actions.

The Mapbox Python SDK is by this measure, low level. Let's keep it low level and not mix in higher level abstractions. Not that higher level is bad, but it's another project or sub project.

0.4 release

Distance API change (#68) and User-Agent header so we can get usage stats.

Python usage strawman

What's the #1 Python library for spatial features? Fiona 😄, so this API client should aim to fit that well. I'm going to stand up a usage example for us to knock around, starting with vector data since I'm familiar with that part of the API and have done some vector library design. Geocoding and other API client modules will be a bit different but the design style could be similar to this one.

I'm really into the built in Python types and protocols (free documentation! free interoperability!) and less into writing new ones. I'd like to see how far we can go without exposing users to new interfaces.

Feature data

A feature object is basically deserialized GeoJSON. It could have other methods, but at the core it's this.

{'id': 'lolwut', 'type': 'Feature', ...}

Feature datasets

A Mapbox dataset is among other things a Python mapping. There are already classes like this in the Python universe (Zope and other B-tree implementations).

You get a dataset object by calling mapbox.dataset() with a dataset identifier and auth parameters.

Uploading features to Mapbox

Adding new features or updating existing features is done by calling its update() method.

import fiona
import mapbox

with mapbox.dataset(...) as dst, fiona.open('example.shp') as src:
    dst.update((ftr['id'], ftr) for ftr in src)

Fetching a feature from Mapbox

Fetching a feature is done with everyday indexing syntax.

with mapbox.dataset(...) as dst:
    ftr = dst['lolwut']
    # ftr: {'id': 'lolwut', 'type': 'Feature', ...}

Deleting a feature from Mapbox

As with a mapping.

with mapbox.dataset(...) as dst:
    del dst['lolwut']

0.3.1 release

I botched 0.3.0 and PyPI doesn't allow update of releases.

Write usage examples for Mapbox API documentation

The examples should slot right in to https://www.mapbox.com/developers/api/.

Add batch geocoding feature

https://www.mapbox.com/developers/api/geocoding/#batch

A POST endpoint for these batch requests would be a little easier to use, but no biggie.