tortilla / tortilla

Wrapping web APIs made easy.

Tortilla

Wrapping web APIs made easy.

Installation via PIP:

pip install tortilla

Quick usage overview:

>>> import tortilla
>>> github = tortilla.wrap('https://api.github.com')
>>> user = github.users.get('octocat')
>>> user.location
'San Francisco'

The Basics

Tortilla uses a bit of magic to wrap APIs. Whenever you get or call an attribute of a wrapper, the URL is appended by that attribute's name or method parameter. Let's say we have the following code:

id, count = 71, 20
api = tortilla.wrap('https://api.example.org')
api.video(id).comments.get(count)

Every attribute and method call represents a part of the URL:

api         -> https://api.example.org
.video      -> /video
(id)        -> /71
.comments   -> /comments
.get(count) -> /20
Final URL   -> https://api.example.org/video/71/comments/20

The last part of the chain (.get()) executes the request. It also (optionally) appends one last part to the URL. Which allows you to do stuff like this:

api.video.get(id)
# instead of this
api.video(id).get()

So to summarize, getting attributes is used to define static parts of a URL and calling them is used to define dynamic parts of a URL.

Once you've chained everything together, Tortilla will execute the request and parse the response for you.

At the moment, Tortilla only accepts JSON-formatted responses. Supporting more formats is on the roadmap for future Tortilla versions.

The parsed response will be bunchified which makes dictionary keys accessible through attributes. So, say we get the following JSON response for the user 'john':

{"name": "John Doe"}

If we request this with an already created wrapper, we can access the response data through attributes:

>>> user = api.users.get('john')
>>> user.name
'John Doe'

Headers

A common requirement for accessing APIs is providing authentication data. This usually has to be described in the headers of each request. Tortilla makes it very easy for you to describe those recurring headers:

api.config.headers.token = 'secret authentication token'

You can also define custom headers per request:

api.endpoint.get(headers={'this': 'that'})

These headers will be appended to the existing headers of the wrapper.

Parameters

URL parameters can be defined per request in the params option:

api.search.get(params={'q': 'search query'})

Caching

Some APIs have a limit on the amount of requests you can make. In these cases, caching can be very helpful. You can activate this with the cache_lifetime parameter:

api = tortilla.wrap('https://api.example.org', cache_lifetime=100)

All the requests made on this wrapper will now be cached for 100 seconds. If you want to ignore the cache in a specific situation, you can use the ignore_cache parameter:

api.special.request.get(ignore_cache=True)

The response will now be reloaded.

URL Extensions

APIs like Twitter's require an extension in the URL that specifies the response format. This can be defined in the extension parameter:

api = tortilla.wrap('https://api.twitter.com/1.1', extension='json')

This option can be overridden with every request or subwrap:

api.special.endpoint.extension = 'xml'
api.special.endpoint.get(extension='xml')

URL Suffix

Some APIs uses a trailling slash at the end of URLs like in example below:

https://api.example.org/resource/

You can add the trailling slash with suffix="/" argument when wrapping the API or getting the URL with .url(suffix="/") method:

api = tortilla.wrap('https://api.example.org', suffix="/")
api.video(71).comments.url()

Will return the following URL:

api         -> https://api.example.org
.video      -> /video
(id)        -> /71/
Final URL   -> https://api.example.org/video/71/

Debugging

Activating debug mode can be done with the debug parameter:

api.debug = True
# OR
api = tortilla.wrap('https://api.example.org', debug=True)

You can override the debug parameter per request:

api.stuff.get(debug=False)
api.other.stuff.get(debug=True)

An example using the GitHub API:

>>> user = github.users.get('octocat')
Executing GET request:
    URL:     https://api.github.com/users/octocat
    headers: {}
    query:   None
    data:    None

Got 200 OK:
    {u'public_repos': 5, u'site_admin': ...

Enjoy your data.

tortilla's Issues

Use of "colorclass" breaks IPython on Windows

When I import Tortilla on Windows, my console breaks:

Any pyreadline-assisted features, like tab-completion and history browsing (which uses the up & down keys), don't work at this point. This also happens with colorama and appears to have something to do with IPython going crazy when it loses control of stdout.

When I remove this part in wrappers.py:

if os.name == 'nt':
    colorclass.Windows.enable()  # <--

It works again, but debug trace colouring is (understandably) broken:

In [7]: api = tortilla.wrap("http://api.steampowered.com", debug=True)

In [8]: api.ISteamUser.ResolveVanityURL.v1.get("?vanityurl=smileybarry")
←[34mExecuting GET request:←[39m
←[90m    URL:   http://api.steampowered.com/ISteamUser/ResolveVanityURL/v1/?vanityurl=smileybarry
    headers: {}
    query: None
    data:  None
←[39m
←[31mGot 400 Bad Request (NOT JSON):←[39m
←[90m    <html><head><title>Bad Request</title></head><body><h1>Bad Request</h1>Please verify that all requir...
←[39m

Can you disable colouring & use of colorclass on Windows, at least for the time being?

Does tortilla support authentication cookies?

The API I want to use sets a cookie with a first call to route /login.
Then the cookie must be sent with every subsequent request (user must be logged in to access resources).
Does tortilla support cookies?

adding delay between each request?

The API I am trying to use will reject my HTTP requests if they are too frequent (say, more than 5 requests per second).

Would it be possible to provide some kind of delay between each call when creating the wrapper?

api = tortilla.wrap('https://github.com', delay=0.2)  # 200ms between each request

I am willing to write a pull request for this but I am not sure if I need to modify the Client or the Wrap class...

We could make a call to self._pause() defined below before each call:

def pause(self):
    if self.delay > 0:
        current_time = time.time()
        elapsed = current_time - self.previous_call
        if elapsed < self.delay:
            time.sleep(self.delay - elapsed)
        self.previous_call = current_time

What do you think?

Another option would be to use an adapted version of the retry decorator defined here: https://wiki.python.org/moin/PythonDecoratorLibrary#Retry

dependency on colorclass

Please make the colorclass dependency optional.

It is useless in a server environment.
And services like AppEngine don't allow the use of ctypes which colorclass depends on.

Add support for BasicAuth

I am trying to use your library to access a Jama rest service.

This however requires HTTP Basic Auth.

It would be great if your library could add support for this, maybe by forwarding the auth object to the
request library as described in:
http://docs.python-requests.org/en/master/user/authentication/

So` client code could become like

from requests.auth import HTTPBasicAuth
import tortilla
api = tortilla.wrap('https://api.example.org')
api.config.auth = HTTPBasicAuth('user', 'pass')

or maybe
from tortilla import HTTPBasicAuth

Add path formatting flags

Requesting endpoints that contain hyphens is currently not very straight-forward nor user-friendly. A new hyphenated option could be added to the wrapper. When enabled, underscores in the attribute names will be replaced with hyphens when forming the URL:

# before
api('hyphenated-endpoint').get()

# after
api = tortilla.wrap('https://api.example.org', hyphenated=True)
api.hyphenated_endpoint.get()  # GET https://api.example.org/hyphenated-endpoint

Other option flags, such as camelcased, could be implemented as well to make code more PEP-8 compliant:

# before
api.camelCasedEndpoint.get()

# after
api = tortilla.wrap('https://api.example.org', camelcased=True)
api.camel_cased_endpoint.get()  # GET https://api.example.org/camelCasedEndpoint

I think that this formatting should only happen in __getattr__ to keep __call__ open for potential special situations.

Docs are outdated

Several features are either incorrectly documented or not covered in the docs.

These include:

• Request throttling
• Caching using Redis
• Using custom request and response formats
• Using OAuth
• The workings of Tortilla's dynamic configuration

PyPI release

Thanks for creating a great library! Any chance of cutting a new PyPI release that includes all of the recent changes in master? I'm particularly interested in #30.

Tortilla doesn't remove accidental trailing slashes in "tortilla.wrap"

If you accidentally call tortilla.wrap with a URL that has a(ny) trailing slash(es) (like http://api.steampowered.com/), the resulting HTTP requests always start with that extra (amount of) slash(es), breaking them. (For example, that domain returns a 404 if I call //ISteamUser/ResolveVanityURL/v1 instead of /ISteamUser/ResolveVanityURL/v1)

Client.request format impose same conversion for requests load and response

There may be cases where a binary-data load must be sent along with the request, and a json response is received (for example, see http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html).

It may be worth splitting the format option into req_format and resp_format, both having "json" as default.

I guess, that having the response parsed from yml is not a very common use case, and that the need to pare responses different from json may point to XML, otherwise it could be possible to only specify the format for requests, and always parse response from json.

Can't open a PR

Hey,
I'm trying to push a new branch to open a PR for a small change.
I do not have permission to do so.

May I have permission please @redodo :) ?

Thanks.

Hyphen character in API url

Package very interesting,
I seem to see a problem with the APIs that offer resources with a name containing the hyphen-minus.

How i can use it with a resource called like this: /api/my-resource/1 ?

GETting a response which has a key called 'self' raises an exception

If an endpoint yields JSON with a 'self' key somewhere in the body, tortilla is unable to .get() it, and raises an exception:

Traceback (most recent call last):
  File "test_tortilla.py", line 69, in test_json_response
    assert api.has_self.get() == endpoints['/has_self']['body']
  File "/home/vagrant/dev/tortilla/tortilla/wrappers.py", line 357, in get
    return self.request('get', *parts, **options)
  File "/home/vagrant/dev/tortilla/tortilla/wrappers.py", line 353, in request
    return self._parent.request(method=method, **options)
  File "/home/vagrant/dev/tortilla/tortilla/wrappers.py", line 353, in request
    return self._parent.request(method=method, **options)
  File "/home/vagrant/dev/tortilla/tortilla/wrappers.py", line 215, in request
    return bunchify(parsed_response)
  File "/home/vagrant/dev/tortilla/tortilla/utils.py", line 34, in bunchify
    return Bunch(**obj)
TypeError: __init__() got multiple values for keyword argument 'self'

I've captured this in a test in markrian@b0f441b.

what kind of APIs are these?

I've never seen an api that has something like his:

https://api.github.com

What is this called?

How can I POST json data?

For instance, I would like to send {'username': 'god', 'password': 'p4s5w0Rd'} to route /login
How should I proceed?

import tortilla
camomile = tortilla.wrap('https://my.api.com/')
camomile.login.post({"username": "god", "password": "p4s5w0Rd"})
camomile.login.post('{"username": "god", "password": "p4s5w0Rd"}')

vk.com API

Hello. vk.com API have method users.get.

import tortilla

vk = tortilla.wrap('https://api.vk.com/')
vk.method.users.get.get(params={'user_ids':'1'})
>>> AttributeError: 'function' object has no attribute 'get'

How to make request?

Formatting flags are not working for single endpoints

Formatting flags like hyphenate are only working when the formatter is set on the base Wrapper

base = tortilla.wrap('api.example.org', hyphenate=True)
base.hyphenated_endpoint  # GET api.example.org/hyphenated-endpoint

# however
base = tortilla.wrap('api.example.org')
base.hyphenated_endpoint(hyphenate=True)  # GET api.exaple.org/hyphenated_endpoint
base('hyphenated_endpoint', hyphenate=True)  # same result

I think it would be a good idea to add support of formatting options to single endpoints as well.

Link to the docs.

Please link to http://tortilla.readthedocs.org/en/latest/ from either the README or from the github repo options.

Logo proposal

Greetings,

My apologies in case this is not the right channel to address this. I'm a designer in development and an open source enthusiast, who exploring GitHub found your project and decided to propose a logo design for it. It's (of course) totally free and we would be working together to create the design that fits best. In case you agree, you could share some ideas you may have (colours, shapes, etc) so I have something to start with.

Kind regards and keep up the great work!

Simplification

Wow, such fun, many cool, I already wrote "tortilla" a few month ago while writing a trello client, and I'm verry happy to see I'm not alone to need that :-)

But your implementation seems convoluted, using dict and running up to the original parent at query time to forge back the URI.

My implem (very basic, with almost no option, but it was covering my needs) seems a lot simplier (and I like simple !):

class RESTException(BaseException):
    def __init__(self, url, response, *args, **kwargs):
        self.response = response
        self.url = url

    def __str__(self):
        return "<RESTException {} on {}>".format(
            self.response.status_code, self.url)

class REST(object):                                                     
    """                                                                                                                                          
    Basic "REST to object mapper" client.                                                                                                        
    Usage:                                                                                                                                       

    > client = REST("http://example.com")                                                                                                        

    > client.foo.bar()                                                                                                                           
    # Calls http://example.com/foo/bar                                                                                                           
    {'some': 'response', 'from': 'example.com'}                                                                                                  

    > foo, bar = 'foo', 'bar'                                                                                                                    
    > client[foo][bar]()                                                                                                                         
    # Calls http://example.com/foo/bar                                                                                                           
    {'some': 'response', 'from': 'example.com'}                                                                                                  

    > client.foo.bar.baz(params={'users': 'me'})                                                                                                 
    # Calls http://example.com/foo/bar/baz?users=me                                                                                              
    # Response are given in JSON, HTTP error are raised as exceptions                                                                            
    """                                
    def __init__(self, url, auth=None):
        self.auth = auth
        self.url = url

    def __call__(self, method='GET', path='', **kwargs):    
        response = requests.request(method, self.url + path,
                                    headers={'Accept': 'application/json'},
                                    auth=self.auth,
                                    **kwargs)
        if response.status_code != 200:
            raise RESTException(response=response, url=self.url + path)
        return response.json()

    def __getattr__(self, path):
        return REST(self.url + '/' + path, self.auth)

    __getitem__ = __getattr__

My question:

• Do you have a specific need to use your actual implementation, instead of simply concatenating path segments ?

RFI: Allow index access in addition to the chain of callables

First of all, I really like the direction you took on the implementation.

I would like to suggest a couple of potential improvements to an API.

1) There are cases of URLS which are build like this:

http://somebooksite.com/accounts/<account_id>/<bookshelf_id>/<genre_id>/<book_id>'

Currently the following code is required to access it:

api = tortilla.wrap('http://somebooksite.com/')
responce = api.accounts(account_id)(bookshelf_id)(genre_id).get(book_id)

( there is an option to build the path from outside and pass it to accounts() - but it's a defeatist tactics :) )

I would suggest to also allow index access and override __getitem__. This will allow for the following access style:

api = tortilla.wrap('http://somebooksite.com/')
responce = api.accounts[account_id][bookshelf_id][genre_id].get(book_id)

This probably will look a bit more pythonic, then the long chain of callables. Bunch itself allows for both types of access, which makes the call look nicer.

Missing Dependency: lunch

It appears that the dependency lunch was removed from pypi (and github).

Failed command

(tortilla-venv)$ pip install tortilla
[snip]
Downloading/unpacking lunch (from tortilla)
  Could not find any downloads that satisfy the requirement lunch (from tortilla)
Cleaning up...
No distributions at all found for lunch (from tortilla)
Storing debug log for failure in [snip]/pip.log

pip.log

Downloading/unpacking lunch (from tortilla)
  Getting page https://pypi.python.org/simple/lunch/
  Could not fetch URL https://pypi.python.org/simple/lunch/: 404 Client Error: Not Found
  Will skip URL https://pypi.python.org/simple/lunch/ when looking for download links for lunch (from tortilla)
  Getting page https://pypi.python.org/simple/
  URLs to search for versions for lunch (from tortilla):
  * https://pypi.python.org/simple/lunch/
  Getting page https://pypi.python.org/simple/lunch/
  Could not fetch URL https://pypi.python.org/simple/lunch/: 404 Client Error: Not Found
  Will skip URL https://pypi.python.org/simple/lunch/ when looking for download links for lunch (from tortilla)
  Could not find any downloads that satisfy the requirement lunch (from tortilla)

Lunchtime!

Your setup.py and requirements are asking for "lunch". From your wrappers file, I think that's supposed to be "bunch" version 1.0.1

Documentation about headers is wrong

In the documentation, we can see an example of header set for all queries:
api.headers.token = 'secret authentication token'

Actually, it looks like it should be:
api.config.headers.token = 'secret authentication token'

Recurring params

It would be cool if api.param could recur like api.headers. The Tumblr and Github API's can accept the access key in the query string.

Currently, when I pass an access key I will:

api = tortilla.wrap('https://api.example.org')
api.video(id).comments.get(count, params={ 'token' : token})

I think it might be nice to be able to say when you may need to make a lot of requests

api = tortilla.wrap('https://api.example.org')
api.params.token = token
api.video(id).comments.get(count)

Any way to access response headers?

I'm looking into implementing pagination using the Github API, where information about pagination is provided in the Link header of an API call.

Using requests I can so something like this:

r = requests.head('https://api.github.com/repos/foo/bar/issues',auth=auth)
r.links

Is there an easy way to do this using tortilla?

How to set parameters of the API?

I am doing calls to an api which I would like to wrap with tortilla:

In:

import tortilla
api= tortilla.wrap('https://api.site.com/api')
input = 'hi'
process = api.key('93840928jdsdsosdp920399').get(text)
print(process)

Out:
{'status': {'code': '100', 'msg': 'Operation denied', 'credits': '0'}}

I also tried to:

import tortilla
api= tortilla.wrap('https://api.site.com/api',
                              param1 ='92893084kjhsd90128sd8f090sd', \
                              param2 = 'json',\
                              param3 = input, \
                              paramN = 'es')
input = 'hi'
print(process)

Nevertheless, I can't access to the json's parameters. How can I set the parameters and get some keys of the json?.

RFI: HTTP Status Code and Error handling

Currently, there is no convenient way to handle HTTP return codes and errors, this information is lost:

#
#... Skip...
 r = requests.request(method, url, params=params,
                             headers=request_headers, data=data, **kwargs)
#... Skip ...
        if has_body:
            return bunch.bunchify(json_response)
#... Skip ...

There has to be a way to handle errors. There are several possible ways out of it, that I could think of:

1. Create a Response object to inherit from Bunch , with a ResponseMixin that will define status_code method ( or property ) .
   This has a disadvantage if you want to return user-defined objects with dump/load callable.

2. Put a property , like _status_code_ on an object (of any type) before returning in. This has almost no disadvantages, besides some obscure case of return type already having _status_code_ defined, in which case you can raise an exception

3. Have a method on an endpoint which will return last requests endpoint.last_status_code ( have to consider multiple instances with various async libs, when requests on the same endpoint are executed in parallel - but I didn't really think hard about this yet. )

Options (2), and (3) are probably better.

RFI: Allow for defining external loads / dumps instead of Bunch.

Could be set the same way headers are set for an endpoint, as an option.

This is useful for people using 'schematics' and the similar libraries to define validation and type coercion models.
There is no problem to run the model on a Bunch objects, but passing the loads/dumps has the following advantages:

1. Will remove an unnecessary level of an indirection.
2. Will allow to pass objects to post/patch directly in their native validated format, without conversion to the dictionary ( will be performed by serializer the tortilla customer defines )
3. Will allow you not to support formats ( like XML and whatnot ) directly, passing the request straight to the wrapper/unwrapper callables.

This change can be backward compatible, setting JSON->Bunch as a default loads and Dict->JSON as dumps ( will match current behaviour ).

Output format will be up to tortilla customer, so providing he knows what API receive he will be able to provide the appropriate callables.

Example with schematics:

from schematics.models import Model, 
import tortilla
import json

register = tortilla.wrap('https://people_register.com')

Class Person(Model):
    name = StringType()
    age = IntType()

    def dumps(self):
        """ Dump JSON from Person """
        return json.dumps(self.to_primitive())

    @classmethod
    def loads(cls, json):
        """ Provided JSON return Person object """
        return Person(json.loads(json))

# Wrapper receives response and converts 
register.loads = Person.loads
register.dumps = Person.dumps

# Here we pass native object !
register.post(data=Person(name='John Doe', age=25))

# Here we will receive back Person object, validated !
person = register.get()

What do you think?