- Understand the difference between transmission of data vs data and presentation.
- Understand the value of APIs.
Let's say you're building an app that allows users to find the nearest coffee shop that is most likely to have an open table where they can sit and work on their novel.
You need to be able to search for coffee shops and find out things like how many people have recently gone there.
One of the great things about the web is that we can find all kinds of data and use it in new and interesting ways. In this case, we know that Foursquare deals with exactly that sort of thing, so we'll try to get data from them.
One thing we could do is screen-scrape Foursquare. Screen scraping is a technique where you request a URL in the same way a web browser would, but you do so in code so that instead of seeing rendered HTML, you have to parse the HTML to see what you want.
So, we could go to Foursquare and use their search feature and look at the URL generated by the search to figure out how to create our own request. In this case, searching for coffee shops in New York would generate a URL like this:
https://foursquare.com/explore?mode=url&near=New%20York&q=coffee%20shops
We could then use Nokogiri and Ruby to request that page and parse the data.
# irb
doc = Nokogiri::HTML(open('https://foursquare.com/explore?mode=url&near=New%20York&q=coffee%20shops'))This will give us the full search results page from Foursquare. We could inspect that source and figure out that the results are all in <li> elements with a singleRecommendation class, and do something like:
# irb
doc.css('li.singleRecommendation').each do |recommendation|
# ...
endAnd basically keep drilling in from there, trying to figure out how to reliably extract a name, address, ID, and so on, from each result. Many hours later we can feel good about getting this data.
Now what happens when Foursquare does a minor CSS tweak and changes .singleRecommendation to .recommendationCard.
Our whole app is broken! And we won't know about it until a user gets an error. And Foursquare has no responsibility to us because scraping their page is not an intended use of their system, they wouldn't know that we're doing it, and wouldn't be able to tell us about the change even if they wanted to.
Okay, instead of spending hours and hours parsing their new HTML, let's see if they have an API.
An API (Application Programming Interface) is a way for one system to interact with another via a well-defined interface. An interface is any endpoint that can be used to take actions and consume data on a given application. The HTML response of every web site is an interface, for instance.
The web interface provides HTML, which is a combination of data (the dynamic values like venue names, addresses, and check-ins) and presentation (HTML tags, CSS classes, JavaScript effects) meant to be consumed by a web browser. This is great for end users who want to see something visually compelling.
Some applications, knowing that their data might be of use to other applications, go a step further, providing an API as well as the standard web interface.
The benefit of an API is that it's put together explicitly for the purpose of being used by another system, rather than being viewed by a human, so the data that it returns is unencumbered by the presentation, preventing the consuming code from having to wade through HTML just to find values. A good API just provides data in the easiest format possible for code to digest.
So if a web interface provides HTML, what does an API provide?
Most modern web APIs use JavaScript Object Notation, or JSON, as a standard way of describing and defining data.
Note: Some APIs, usually older ones, will use XML rather than JSON to describe data, though that is increasingly rare. However, the remnants of that great stone age are found in AJAX, which stands for Asynchronous JavaScript and XML, XML being the standard when AJAX hit the scene. When we use AJAX today we're almost always using JSON rather than XML, but we haven't changed the name because AJAJ would be a weird acronym.
JSON looks a lot like a Ruby hash, and as a result, figuring out how to access a given piece of data is a snap.
{
"name": "Cuppa Joe's",
"address": "123 Not A Fake Address Street",
"city": "Real City",
"state": "New York",
"zip": "10015"
}Much better than trying to find all that in a bunch of HTML tags, right?
//Flat-fact: JSON is canonically pronounced like the name "Jason", because it makes it easy to carve up data.
Many websites have APIs that you can consume with your applications. The easiest way to find out if the site you want to use provides an API is to Google sitename api, or just go to the site and look for a link to a "Developers" or "API" section.
For instance, if we go to Foursquare and look in the navigation links at the bottom, we'll see a Developer link that we can start to explore, and from there we can find a venue search API endpoint that will allow us to do exactly what we were trying to do above with our screen scraping.
Most good APIs will have extensive documentation about how to use the API, what functions are available, and what data you can access. Be sure to read through the documentation very carefully. Every API is a little bit different, and documentation quality varies, so you'll need to be analytical and methodical, and prepared for some trial and error, when you're learning to consume a new API.
Most good APIs also use REST, just like Rails, so finding your way around should start to feel familiar and consistent as you learn what resources are available. Keep in mind, though, that you will only have access to what the API creator allows. Not all applications open up all features for API access, so make sure what you need is available ahead of time.
We've looked at getting data from other websites via screen scraping, and how cumbersome and brittle it is trying to integrate data with presentation, like an HTML page. We've learned what an API is, how it differs from an interface meant to be consumed by a browser, and how to find out if a site offers an API.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent