GET API contains a perfect storm for DoS about augur

[sgoggins]

Thanks, @Seltyk . Rate limiting and pagination are on our road map. These are helpful points for us to keep in mind as we build them out. We of course would welcome contributions in pull request form as well! :)

[Seltyk]

Glad we're on the same page. I'll continue developing my pagination implementation for now; hopefully I'll have a PR to present in the near future.

[sgoggins]

@Ulincsys : Can you share your thoughts about rate limiting and pagination here. I know you did a bit of looking into this already.

[Ulincsys]

Yes, we've been discussing pagination and rate-limiting for the last several months.

Currently Augur does not keep track of requests made to the frontend or the backend. All requests are served blindly and without context as to what other requests a particular client has made within a given time period. My intent moving forward is for Augur to implement an event stream internally so that we can query the behavior patterns for clients while serving requests.

We have three levels of authentication for our endpoints at the moment:

• Unauthenticated
  • These endpoints do not require an API key or user authorization to access.
  • Most of our metrics endpoints fall into this category
• Client API key required
  • These endpoints require a Client [key] authorization header to access
  • The Oauth endpoints fall into this category
• User authorization required
  • These endpoints require a Client [key], Bearer [token] authorization header to access
  • All endpoints that handle user data fall into this category

For the authenticated endpoints, it shouldn't be too difficult to implement rate-limiting once the event system is in place, since we have identity information for the requester. We can set a limit per key per hour (like GitHub), or set limits per endpoint per key per hour. We have not decided on the particulars of the limits yet, but we're definitely open to input on what seems reasonable.

As for the unauthenticated endpoints, we need to find a system that can identify unique agents based solely on the information provided in basic HTTP requests. Is an IP address alone enough? What about multiple legitimate agents making requests from the same host?

• My thought is that we should strictly rate-limit unauthenticated endpoints based on IP address (and any other useful identifiers), and make it optional to pass an API key along with requests to these endpoints. This way, we can say that if an agent wants to make a large volume of requests to unauthenticated endpoints, then they will need to register their application so they can authenticate with an API key.

[Seltyk]

My experience is in embedded development, so rate limiting is something I haven't thought very critically of before. OpenWeatherMap, if memory serves, enforces very strict rate limits on unauthenticated clients, but just making an account and generating an API key significantly loosens the limits. This seems to be the model you were describing in the last paragraph, and at a glance it looks like a simple yet effective way to stop bots from hogging the server while allowing legitimate clients to leverage Augur. The threat of an IP ban isn't going to stop the most malicious attackers, as they could simply hook their bot up to a VPN, but having to make a new account may slow them down enough to discourage that behavior.

I'll pass this thread along to some web and security guys I know, their input is far more valuable than mine.