Tag: web apis

Learn curl

Dear new developer,

If you interact with web APIs at your job, learning curl is a good investment. Curl is a flexible, powerful, command line utility which lets you make HTTP requests.

In my current job, I build a lot of API requests. Side note: I’m going to use the shorthand API for web based, JSON API; there are many other kinds of APIs, but curl is mostly useful for this type of API.

I’ve found familiarity with curl to be very helpful as I create, share and debug these API calls.

Why learn curl?

With curl, I can easily script up API interaction. In the next section, I’ll discuss the common parameters I use.

Curl works across multiple platforms. It’s a lowest common denominator tool and is present everywhere. It doesn’t require anyone to install anything. (Okay, maybe not true on vanilla windows, but as soon as you have any unix based system, you have curl.) Update 7/21: curl is present on vanilla windows 10. Open up a cmd window and run curl and you’ll see curl: try 'curl --help' for more information.

You can share scripts with team members, add them to issue comments for provide bug reproduction steps, and version control them if you want to improve them over time. I find them especially helpful for issue reports, as there’s no ambiguity with a script.

Curl pairs nicely with jq (which you should also learn); you can pipe the output of curl directly into jq to get understandable formatted JSON.

I find when I use a low level tool like curl, I’m forced to have a deeper understanding of the problem than I would if I used a higher level tool. It’s kind of like a text editor in that way.

I haven’t run into a situation yet where curl couldn’t interact with an API in a way I wanted, though sometimes multi step interactions like an OAuth authorization code grant can be a bit cumbersome. You can set headers, provide cookies, send a request using any HTTP method, and provide data either inline or via a file.

Basics of curl for APIs

Here are the main switches I use with curl for API interaction.

-X <method> lets me specify the HTTP method, so -X GET is a get request and -X POST is a post request. Sometimes the HTTP method is implied by other switches, but I prefer to be explicit.

-H <header> lets me specify HTTP headers. You can use this many times in one command. Often if I’m sending JSON in a post with an authorization header, it’ll look something like -H 'Content-type: application/json' -H 'Authorization: Bearer APIKEYHERE'

-d payload lets me specify a payload for a post or put request. This will often be JSON: -d '{"foo":"bar"}' It can be multi-line if needed. I use single quotes around the JSON to avoid having to escape double quotes. If I need to have a shell variable inline, so that I can extract some text which changes often, I do something like this: -d '{"foo":"'$BAR'"}'

-d @payload lets me put the payload in a file and have curl read from that file. I use this switch less often, because I’ll usually put everything in a shell script anyway.

-vvv lets me see the back and forth of the HTTP requests and response, and also status codes.

As previously mentioned, I also usually wrap everything in a shell script so it is repeatable. On occasion, I’ll even check the shell script in, so that others can use it. This is useful if it is a part of a run book or troubleshooting guide, for example.

Here’s a typical curl shell script, in all its glory:

API_KEY=...
USER_ID=...
curl -vvv -H "Authorization: $API_KEY" -H "Content-type: application/json" -XPOST 'https://api.example.com/user/'$USER_ID -d '{
  "user" : {
    "name": "Dan",
    "blogger": true,
    ...
  }
}'

There’s a lot more available with curl; the whole man page is worth a read. But I hope I’ve convinced you that you can use curl to interact with APIs in a way that is:

  • repeatable
  • shareable
  • versionable

Alternatives

There are many alternatives. I am a bit familiar with postman, which is a web based application that lets you organize and call APIs. There is also a command line runner for postman called newman. Insomnia is another option that some of my team members use. It’s another GUI client with a corresponding CLI client called inso.

While I haven’t dug deeply into these alternatives, at first glance it seems to me that they give you tooling power and richer collaboration. But nothings free; you lose ubiquity and simplicity.

curl is a powerful tool, available everywhere, for examining and interacting with APIs. Learn it well.

Sincerely,

Dan

Web APIs for new developers

Dear new developer,

Chances are high that you’ll be working with other code as a developer.

I remember when I was first starting out and saw the acronym API everywhere. I had no idea what it stood for. In case you are in the same boat, it means Appliciation Programming Interface. What that means is it is a defined way for your code to call into another piece of code.

One of the most common ways to do so in an interconnected time is via a web API. A web API (as I define it) provides significant functionality for your application, across the internet, over HTTP. You may or may not control the API, and you may or may not use an SDK to access the API. (There are, of course, many other ways to integrate other functionality into your application, depending on what you’re doing.)

This article does a good job of discussing some of what you should be aware of when you are accessing web APIs. In particular:

First, a clarification: the term API is used to refer to lots of different things, and its emphasis has shifted a great deal over the years. If you’ve learned anything about Object Oriented development, you may be familiar with APIs as code components that have well-defined interfaces, allowing “customer” code to access the features and functionalities within a system. These APIs usually involve objects and methods / functions. This is not the type of API we’re going to talk about in this article. 😅

In this article, we’re dealing with the kind of API you use to manage access to data and services over the web. If you’ve come across the terms HTTP API or REST API, that’s what we’re talking about. With this type of API, client apps make requests to endpoints to carry out operations on data. An example would be a database for a shop that has a web app and a mobile app – with an API managing access to the data from both apps.

The article goes on to discuss different concerns you should have as a developer interacting with web APIs.

The power of web APIs is that you can gain access to a quite complicated system via a simple API call. Examples of this include:

  • Looking up stores or restaurants that are close to you (Google Maps)
  • Making a charge against a credit card (Stripe)
  • Sending a text message (Twilio)

These all used to be tremendously complicated tasks, sometimes taking weeks to months to set up. Now you can access them and put together an application that does all three in hours. We stand on the shoulders of giants, indeed.

You don’t just benefit from the API integration one time. The API provider has an incentive (especially you are paying them in data or money) to improve their service. And often you can get the benefits of that improvement without lifting a finger.

If you have a large task that you want to do as part of an application, it’s worth googling around a bit to see if someone has already encapsulated it in an API.

Sincerely,

Dan