interagent / http-api-design Goto Github PK

Would simplify as presently we duplicate content and will have to update separately.

As mentioned in the gitbook, Returning response with uuid is a better practice. But backend developer usually use database to store data.As we know,It is not recommended to use uuid in the database,So how to solve the problem or haveing good suggestion.

Hi Heroku and other folks,

1. To what extent is the http-api-design compatible with jsonapi.org?
2. To what extent can these two projects be made to be compatible or complementary in the future?

202: Request succeeded for a POST, DELETE, or PATCH call that will complete asynchronously

According to the http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html 202

The request has been accepted for processing, but the processing has not been completed.

So that doesn't means Request succeeded. The Request has successfully reached the server, and accepted for processing.

I was also looking at https://developer.yahoo.com/social/rest_api_guide/http-response-codes.html

I wondered why one should not return a 204: No content in response of a successful Patch or Delete request.

Since when you delete a resource, you eighter already have the full resource, or could easilly get it before deleting. The same goes for a patch request.

By doing this, the response can easily be a few kb lighter, which is always good, no?

I am about to implement a business process which can be more or less modeled as a state machine. In abstract terms, here is how the state machine would look like

[Entity1] --process1--> [state1]--process2-->[state2]--process3-->[final state]

Every state transition is atomic operation. The reason we want to model it as a state machine is that complete operation can take long time to run and we do not want the caller to be blocked. So idea is that we would accept the request from caller and return 202 Accepted is request looks ok.

After that, a scheduled process would pick up the request from database and trigger process1 by calling a REST endpoint. Same for process2 and process3.

There are two ways that this can be modeled. First is implement a single REST endpoint like below

PUT http://api.com/entity/{id}/process

Because we know the current state of the entity, we can determine which process to execute next. But then I feel this API is not expressive and following would look better

PUT http://api.com/entity/{id}/process1
PUT http://api.com/entity/{id}/process2
PUT http://api.com/entity/{id}/process3

In the above, there is a distinct endpoint for every process that can be triggered on the entity. If you trigger process1 on an entity which is in final state then you would get back an error.

I am not sure which one is right or if both of these approaches are wrong. Anyone has any experience of doing something like this in past?

From https://github.com/interagent/http-api-design
"See Summary for Table of Contents." links to:
https://github.com/interagent/http-api-design/blob/master/en/SUMMARY.md
it's OK;

but from Gitbook https://geemus.gitbooks.io/http-api-design/content/en/
this Summary link rewrites to https://geemus.gitbooks.io/http-api-design/content/en/SUMMARY.md
and it's a 404.

Same for the "We welcome contributions to this guide." link:
https://geemus.gitbooks.io/http-api-design/content/CONTRIBUTING.md (404)

In addition to id/message, consider dev vs user errors, etc.

see: https://github.com/WhiteHouse/api-standards#error-handling

To quote the existing README

Use the plural version of a resource name unless the resource in question is a singleton within the system (for example, in most systems a given user would only ever have one account). This keeps it consistent in the way you refer to particular resources.

This IMO is a bad example - as a system has many users, which could make the entire example difficult to interpret. Perhaps a clearer example of a singleton API is /status, or /search.

I translated this guide into Traditional Chinese, and made the structure GitBook compatible. Since we already have translations in two (or more) languages, maybe we can add a section to list these works. Like the followings:

In other languages

• Traditional Chinese version (based on 232f8dc), by @kcyeu
• Korean version (based on f38dba6), by @yoondo

Looks that I'm not the only one who wants to know why it's recommended to do something in that way or another.

• #8
• #9
• #15
• #36

So it'll be super helpful to have explanation under each paragraph, cause it helps to understand things and not only blindly follow someone's tips. And we don't need right explanation (I think there's no one), only what the author thinks, why he does so.

from @joe-mojo #16 (comment)

other role (no update) like sending a message to a resource (handler) or annotating en existing resource.

this raises a point i've been meaning to bring up... using URLs for actions (like /runs/{run_id}/actions/stop) is something that doesn't seem like the most correct thing to do. would it be more appropriate to recommend using something like JSON-RPC and POST /runs/{run_id} where the JSON-RPC request indicates the action to be performed?

A great example is Version with Accepts header. I wouldn't say it's obvious why putting the version in the accept header is valuable as opposed to versioning in the URL.

We commonly include an envelope containing foreign keys for parent objects. To simplify and avoid n+1 queries, it can often be very useful to ask for those envelopes to be expanded to serialize the entire parent object. We've started experimenting with this at Heroku and I wanted to at least note it here and link to a raw doc around it. I want to let the experiment run a bit longer before I settle on it and include it, but wanted to give some early ideas.

Copy of rough spec here: https://gist.github.com/geemus/6b0d59b5813d765c5267

I'm missing a paragraph on how authentication should be done. The paragraph on "Provide executable examples" mentions using a token with the comment "acquire from dashboard".

Thanks!

The current advice implicitly in this doc and explicitly in the Heroku API reference breaks large result sets into ranges, and uses field values as constraints to be applied to the query for the rest of the result set. It isn't hard to stumble upon scenarios where a result set is sorted by a non-unique field (ratings, popularity, counts, times, etc), wherein ranges requested by the specified constraints would yield overlapping result sets.

Unfortunately, the Content-Range scheme doesn't account for these scenarios (likely, because we haven't yet had such a scenario in the Heroku API). As this is document aims to serve as general API design advice and non-unique sort+pagination is not an uncommon situation, some thought towards this problem may be merited by the document authors. Additionally, our team (at Heroku) has come across some scenarios we'd like to introduce which would benefit from an answer.

@geemus @brandur have either of you thought about this much before?

@mathias and I have discussed the idea of using "composite keys" which include at least one unique field as a component in search to achieve something similar in spirit to the current advice, but nothing clearly simple emerges.

ie https://github.com/WhiteHouse/api-standards does a good job of listing examples (and counter-examples) for a number of points, which may help clarify.

It's JavaScript Object Notation after all...

I have started looking into another project that I am supposed to add some features to. This project is using 409 Conflict to convey that a unique constraint has failed. In past I have used 400 BadRequest and now 422 Unprocessable entity to indicate failures of unique constraint. I am not sure if use of 409 Conflict for this purpose is right?

I thought it might help to add a Github page for the Guide, so I took the liberty of generating a super simple one, with one of Github's default page templates.

https://github.com/karlcoelho/http-api-design/tree/gh-pages

Hi,

I wasn't clear how you would handle a case where you want to pull in X amount of results, for example in response to a search query without worrying about field names.

The Range header requires a field, so what if you just want to retrieve results regardless of fields. e.g. using a wildcard

Range: * 1..10

Could you elaborate?

I tried to implement some of these principles and found that it involved certain other resources. For ASP.net for example it seems like CacheCow is the best available way to implement ETags for Caching. Maybe adding some links to similar resources for specific server/client languages would serve as a good jumping off point for people who find this useful.

Does there exist any way to test a server for its compliance with the recommendations (and specifications?) within this repository?

If someone posts well formed JSON that is invalid, what http error status is returned and what is the format of the error response?

For purposes of an example, assume that app_name does not allow spaces, what would be the response and status to this?

POST /apps
{"app_name": "foo bar":}

I am currently working on enhancing user service which would be consumed by several other services. Other than offering basic CRUD functionality around users this service is also supposed to act as a SAML enabled id provider and so it needs to be able to authenticate users somehow. The current implementation looks like below

PUT /authenticate

{
Userame: {username},
Passowrd: {password}
}

This returns a true or false depending on success or failure of authentication. I am thinking of changing this to be more RESTful. I am thinking on these lines.

Authentication is a process so there is real resource involved here. Should I define a new virtual resource? What should this resource be? Something that gets created when authentication succeeds? A LoggedInSession? A SecurityContext? SecurityContext sounds better. So let's design the API as below

POST /securitycontext

{
Userame: {username},
Passowrd: {password}
}

Not sure if I am thinking in the right direction.

The link to the Heroku platform docs helps to a certain degree, but it doesn't explain what the benefits are to using this approach to pagination. Given that there's not much tooling available for using the Range header this way, it might be good to describe why it's better than just, for example, embedding page details and next/previous URLs in the JSON response.

Also, some additional detail on how pagination would work in different scenarios would be useful. The platform docs only mention how to sort and specify a maximum value (and what does ".." mean in this context?). It'd be good to see how to specify a range bounded on both ends, and also how to respond when the returned range isn't the same as what was requested (either too many results were requested or there's not enough data to fill the request range).

I expect those docs were written with the expectation that we'd play with the Heroku platform API, but for a broader-purpose doc like this, it'd be good to see some real-world use cases without having to resort to experimenting with a specific implementation.

To better communicate the purpose/intention of attributes it can be useful to use semantic hints (similar in a sense to type hints). They generally appear as suffixes to attribute names, a couple examples we use consistently:

• #{past_tense_verb}_at for timestamps
• #{past_tense_verb}_on for dates (time can be ignored and/or should be midnight)

Another case we have discussed, but that I'm not quite as convinced about yet. I'm not sure how universal it is, but I think it does make it more human-readable and a bit easier to reason about:

• #{noun}_enabled for boolean values. Historically we have a mix of these and just #{noun} for boolean attributes, it seems as though choosing one or the other at least would be better.

JSON allows for both escaped or non-escaped non-ascii characters.

It'd be useful for this document to include guidance on which style is preferred, or if there is no preference.

For example, the following is valid JSON:

{
     "unicode black star": "\u2605"
}

As is the unescaped variant:

{
    "unicode black star": "★"
}

I could see valid arguments for either case.

Happy of course if you consider this out-of-scope, but I know it's something I'd value knowing another team's design preferences.

I think when we work with REST API, we have to handling authentication (session) data. OAuth and others based on JWT.

For example: https://github.com/vanioinformatika/poc-angularjs-jwt-rest

Hello,
I'm having trouble finding a way to use some PLSON functionality that helps me validate a string and escape characters that do not conform to the following:

any-Unicode-character-
except-"-or--or-
control-character

I would like something that if in my string find something other than that I substitute for something.

Tks

Give each resource an id attribute by default. Use UUIDs unless you have a very good reason not to.
Don’t use IDs that won’t be globally unique across instances of the service or other resources in the service, especially auto-incrementing IDs.

Why is that? Why not use an auto-incrementing ID if the resource have to be identified by a resource name in the URL? The resource is identified by its path, I believe. So /orgs/1 it is complete unique and completely different from the /users/1
With RESTful & microservice architecture, having an unique ID for a resource is too harsh because I can have several distinct databases, fine tuned to each kind of service, and to guarantee uniqueness in this kind of situation within the scope of the entire system... It just doesn't seems to payoff. There is a specific reason for this?

I'm asking mostly because Swagger is being mentioned here. It seems you were aware of Swagger but you took a different path. It would be interesting to know the thought process that made you arrive to JSON Schemas instead.

Having things at the top level in most cases improves consistency and helps avoid (at least in some cases) n+1 issues. That said, nested things make since to provide a scope for returning lists in particular (and some things can not easily be de-nested because they do not have globally unique identifiers, but this should probably be avoided also). Anyway... There are some gotchas to this approach, but overall we have tried it a few times and have been finding it useful, so I'd like to expound upon the virtues and usage within the guide (and wanted to start by adding some notes here).

I perennially run into difficulty trying to draw a line around which properties it's reasonable for a resource to return. I'd love it if you could address that in your docs, or talk about how you settled these issues when building your UI on top of the REST API.

Consider a system with users and orgs. I'd assert that it's reasonable for the UI to present a table of all the orgs, with some vital stats about each. If the table must show the user count for each org, there are two obvious solutions for acquiring the user count:

• add userCount to the org resource

curl https://service.com/orgs

[
  {
    "id": "01234567-89ab-cdef-0123-456789abcdef"
    "name": "SPECTRE",
    "userCount": 21
  },
  // ... additional org descriptors
]

• force clients to fetch the list of users for each org and count them

// repeat for each record returned by the org resource
curl https://service.com/orgs/{org_id}/users

[
  {
    id: "7e876304-bcd9-4ce6-9119-126be52f4486",
    username: "eblofeld",
    "created_at": "1908-05-25T12:00:00Z",
    "updated_at": "1981-06-24T13:00:00Z",]
  },
  // ... plus 20 more user descriptors
]

Approach 1 can lead to death by a thousand cuts: the server code backing each resource grows with every new property required by UI, increasing CPU cost but reducing API traffic. And per your versioning strategy, the only way to withdraw support for any such property is to cut a new version (while maintaining the old version).

Approach 2 results in both client and server doing lots of work that neither is really interested in; in this case, the content of each user descriptor is discarded, because the only data needed is the length of the array. For collections with many members or whose members have many properties, you also end up with heavier payloads. It also results very quickly in a huge number of requests: 1 + n * c, where n is the number of rows returned and c is the number of "vital statistics" that aren't part of the base response.

I'm reluctant to draw the line based on the underlying database schema, since that feels like an implementation detail that ought to be abstracted at or by this layer.

Your comments are very much appreciated.

Suggestion

The given timestamp format is RFC5424, a subset of ISO8601

https://tools.ietf.org/html/rfc5424#section-6.2.3

A more liberal approach - though more restrictive than iso8601 - is RFC3339

For example, what is the best restful url for the following purpose?

* sub collection, eg. users in a specific country.  
  * GET /countries/:country/users     #seems too hacky, since no country resource indeed.
  * GET /users/by-country/:country  #may conflict with /users/:name where name=by-country
  * GET /users/country/:country
  * GET /users/collections/country/:country  # borrow ideas from resource/actions/:action
  * GET /users-by-country/:country  #no conflict, by seems not belongs to users resource.
  * GET /user/by-country/:country    #no confilict, use user instead of users but strange
  * GET /users;country=:country      # use matrix params
  * GET /users/:country

* query by field, eg. find user by unique auth token
  * GET /users/by-auth-token/:token
  * GET /users-by-auth-token/:token

sure we can use url params, but when /users/by-country/:country is heavily used, a specific url is nicer.

I started this conversation with @geemus via an email. We decided to make our discussion public for feedback.

I've been developing some API Design notes internally for our team at KISSmetrics. I wrote out our base design patterns, research notes, as well as resources for further information.

I had a question for you. In the section Next foreign key relations, you mention utilizing nesting versus owner_id.

In my experience building API clients I have appreciated both approaches:

• When a client expands the full relation, I prefer the approach you mentioned.
• When a client gives reference to the relation without providing the full body I prefer the owner_id approach.

This way the client doesn't have to inspect the relation envelope to see if it has the full relation or if it needs to look it up via an other request.

I still make the object available in the links envelope. This way a client can manage expansion or firing off another request if necessary. The option is left up to the client for performance.

Here is an example that may better describe my question.

@geemus responded with:

Good question. I err on the side of consistency where I can, so I'm in favor of always nesting (since it works in both cases). It does provide some possible ambiguity between foreign key and full representations. In our case they are (presently) only ever the foreign key(s), we have a rough plan to allow for users to ask for expanded representations in to the existing envelope in this case (probably via the user passing a header or query string). Since you would need to explicitly ask for the expanded version, I think the ambiguity of foreign keys vs full representation should be somewhat reduced at least. Does that help/make sense? I'm certainly up for having further discussion and would certainly be interested to here more about your guide as it develops.

My (@nateklaiber) response:

I agree with the desire for consistency. I also agree that if you are asking for expanded versions that the ambiguity should be reduced. In practice I have found this a little tough, as many times I wrap the JSON responses in an object model. Here's an example when working with Stripe.

As I mentioned there, there may be better approaches which I am open to.

I have copied parts of our guide - it's a work in progress.

@geemus then responded with:

Yeah, I guess since we expect you would always have to explicitly expand, that it isn't so bad. In our case most of our serializations are fairly streamlined, so most of the overhead is in DB lookup anyway, so just returning the full thing seemed easiest (and helps prevent inadvertent n+1 stuff from naive clients). I guess my inclination would be toward defaulting to this easier to consume/harder to do wrong version and then perhaps allowing you to narrow the returned value as an optimization or something.

That said, I do think expanding a nested thing from foreign key to full serialization is also a good thing to allow opting-in to for optimization purposes.

It definitely depends a lot on specifics though, I'm not totally convinced we are universally correct there or anything, but it is working well for us and I think could work well for others (but we have other assumptions rolled in there I'm guessing).

Mobaily

https://github.com/interagent/http-api-design#version-with-accepts-header

What benefits does this have over having the version in the URL? I'm just curious, I'm not a huge advocate of using the version in the URL.

But with the URL, it's easier to see (logs etc) which version is being used.

I agree that "sensitive data will already have been exposed during the first call". But sensitive data also have been exposed even if respond 403.

For example, I post some data via non-TLS requests. And server respond 403. The sensitive data in request body have been exposed during the request.

Overview and Model Layer

We have models using SmallInt fields to store references to their more verbose string counterparts. Lets take a look at our User model for example:

# django `models.py` file -- apologies for the framework specific example
class User(models.Model):
    """A representation of a user account."""
    GENDER_UNSPECIFIED = 0
    GENDER_MALE = 1
    GENDER_FEMALE = 2    
    GENDER_CHOICES = (
        (GENDER_UNSPECIFIED, 'unspecified'),
        (GENDER_MALE, 'male'),
        (GENDER_FEMALE, 'female')
    )
    gender = models.SmallIntegerField(choices=GENDER_CHOICES, default=GENDER_UNSPECIFIED)

# Create a new user and specify its gender
user = User.objects.create(gender=User.GENDER_MALE)
user.gender == 1

# Django has utility that implements `get_FOO_display` to return the 
# more verbose string counterpart of the currently set value
user.get_gender_display() == 'male'

Behind the scenes the data for the gender field is stored as an integer (for a multitude of reasons). When we render this field, we tend to expose the more "human readable" string counterpart, but when we manipulate the data, we do so by using its "key", like so:

user.gender = 'male'
user.save()  # would raise a ValueError

user.gender = User.GENDER_FEMALE
user.save()
user.gender == 2

API Design Questions

We're trying to figure out the best way to expose "choice" fields like this over our API and had a few questions:

1. Is it reasonable to return/accept gender as an Integer in our requests -- or should we be using the more human readable string counterpart instead?
2. What is the best way to convey all of the available gender options to a consumer?
3. What is the best way to handle changes to the key/val mapping? eg: What if we create a fourth gender option called Squid?

Currently, a GET request to our endpoint looks like this:

GET /api/users/
HTTP 200 OK
Content-Type: application/json ;utf-8

[
    {
        "url": "http://project.com/api/users/1/", 
        "gender": 2
    }
]

I think there should be a list of common error statuses and when to use them. Right now there's only a prescribed 401 for unauthenticated users, and 406 for users without permissions.

It'd be nice to have something like:

• 429 Too many requests - for rate limiting APIs
• 422 Unprocessable entity - for POST/PUT validation errors
• 500 Internal server error - for errors that are your backend's fault

ie vs summaries for list views and/or mechanisms to specify desired fields.

pulling this in as a distinct issue from #8

/cc @creynders

it'd be great to mention that POST is typically used for creating things (POST /articles), and PUT/PATCH for editing (PUT /articles/123).

Would be nice to just enabled the Gitpage/website.

Why?
The user want to search/find this resource and read on mobile.

The guide doesn't preclude this, but IMHO it's worth including. RateLimit-Remaining is stated as returning the remaining number of request tokens, but it doesn't say how to specify how much time is left in the current 'window', which I think is worth including as a number of seconds.

• For the case of not having exhausted a rate limit window yet, a client may want to be nice and spread out the requests evenly among the window. 30 requests left and 60 seconds left in the window, for instance, could lead the client to making a request every 2 seconds. Encoded into SDK's or examples, this could lead to more even load on services over time and less bursts.
• For the case of having exhausted the window, the remaining request count of 0 is useful, but telling the client how long until the window ends gives them a more actionable message instead of leaving them to do something like a binary exponential backoff algorithm.
• 'time left' as seconds instead of a timestamp avoids all the time-synchronization issues

I think Twitter's approach here is good, even if codifying this into the guide is considered a bad idea, it might be worth a link as a potential approach?

https://dev.twitter.com/docs/rate-limiting/1.1

• X-Rate-Limit-Limit: the rate limit ceiling for that given request
• X-Rate-Limit-Remaining: the number of requests left for the 15 minute window
• X-Rate-Limit-Reset: the remaining window before the rate limit resets in UTC epoch seconds

Section 1.6 and the Heroku API reference both recommend using Range for pagination.

Using the Range/Next-Range mechanism from HTTP accidentally exposes in the API that the data is (likely) obtained by computing the requested data within a selected window; for instance, for a query that exposes data from a SQL database:

SELECT * FROM table WHERE condition ORDER BY field LIMIT (#page * size), size

Moreover, this implementation pattern cannot provide a consistent view of the data when concurrent actions can introduce new elements at arbitrary indexes. Consider the following sequence of events:

1. client requests the data, gets a partial answer with the first 10 elements and Next-Range: 20 ..; let's call the elements e0 to e9;
2. a concurrent query triggers the insertion of elements x and y at indexes 9 and 14;
3. the client requests the next page, and receives e10 = e9, e11 ... e19: e9 has been seen twice by the client (unless you use id-based pagination), and it saw y but not x (despite them having been added simultaneously, regardless of whether you use id-based pagination).

The right solution is to provide the API client with a consistent view of the data; for APIs returning results of DB queries, this is easily done using either cursors or materialized views, the second having the additional advantage of supporting later refinement queries.

At the API level, this should be materialized using the Link header, with (at least) a reference tagged rel=next. Using Link allows the API implementer to store there whatever is required to designate the right query result (for instance, a DB cursor id).