Learn curl

Dear new developer,

If you interact with web APIs at your job, learning curl is a good investment. Curl is a flexible, powerful, command line utility which lets you make HTTP requests.

In my current job, I build a lot of API requests. Side note: I’m going to use the shorthand API for web based, JSON API; there are many other kinds of APIs, but curl is mostly useful for this type of API.

I’ve found familiarity with curl to be very helpful as I create, share and debug these API calls.

Why learn curl?

With curl, I can easily script up API interaction. In the next section, I’ll discuss the common parameters I use.

Curl works across multiple platforms. It’s a lowest common denominator tool and is present everywhere. It doesn’t require anyone to install anything. (Okay, maybe not true on vanilla windows, but as soon as you have any unix based system, you have curl.) Update 7/21: curl is present on vanilla windows 10. Open up a cmd window and run curl and you’ll see curl: try 'curl --help' for more information.

You can share scripts with team members, add them to issue comments for provide bug reproduction steps, and version control them if you want to improve them over time. I find them especially helpful for issue reports, as there’s no ambiguity with a script.

Curl pairs nicely with jq (which you should also learn); you can pipe the output of curl directly into jq to get understandable formatted JSON.

I find when I use a low level tool like curl, I’m forced to have a deeper understanding of the problem than I would if I used a higher level tool. It’s kind of like a text editor in that way.

I haven’t run into a situation yet where curl couldn’t interact with an API in a way I wanted, though sometimes multi step interactions like an OAuth authorization code grant can be a bit cumbersome. You can set headers, provide cookies, send a request using any HTTP method, and provide data either inline or via a file.

Basics of curl for APIs

Here are the main switches I use with curl for API interaction.

-X <method> lets me specify the HTTP method, so -X GET is a get request and -X POST is a post request. Sometimes the HTTP method is implied by other switches, but I prefer to be explicit.

-H <header> lets me specify HTTP headers. You can use this many times in one command. Often if I’m sending JSON in a post with an authorization header, it’ll look something like -H 'Content-type: application/json' -H 'Authorization: Bearer APIKEYHERE'

-d payload lets me specify a payload for a post or put request. This will often be JSON: -d '{"foo":"bar"}' It can be multi-line if needed. I use single quotes around the JSON to avoid having to escape double quotes. If I need to have a shell variable inline, so that I can extract some text which changes often, I do something like this: -d '{"foo":"'$BAR'"}'

-d @payload lets me put the payload in a file and have curl read from that file. I use this switch less often, because I’ll usually put everything in a shell script anyway.

-vvv lets me see the back and forth of the HTTP requests and response, and also status codes.

As previously mentioned, I also usually wrap everything in a shell script so it is repeatable. On occasion, I’ll even check the shell script in, so that others can use it. This is useful if it is a part of a run book or troubleshooting guide, for example.

Here’s a typical curl shell script, in all its glory:

API_KEY=...
USER_ID=...
curl -vvv -H "Authorization: $API_KEY" -H "Content-type: application/json" -XPOST 'https://api.example.com/user/'$USER_ID -d '{
  "user" : {
    "name": "Dan",
    "blogger": true,
    ...
  }
}'

There’s a lot more available with curl; the whole man page is worth a read. But I hope I’ve convinced you that you can use curl to interact with APIs in a way that is:

  • repeatable
  • shareable
  • versionable

Alternatives

There are many alternatives. I am a bit familiar with postman, which is a web based application that lets you organize and call APIs. There is also a command line runner for postman called newman. Insomnia is another option that some of my team members use. It’s another GUI client with a corresponding CLI client called inso.

While I haven’t dug deeply into these alternatives, at first glance it seems to me that they give you tooling power and richer collaboration. But nothings free; you lose ubiquity and simplicity.

curl is a powerful tool, available everywhere, for examining and interacting with APIs. Learn it well.

Sincerely,

Dan

2 thoughts on “Learn curl

  1. Great article, but you can be more rigorous about quoting your shell variables.

    Consider `BAR=”Some value with spaces”` — then `-d ‘{“foo”:”‘$BAR'”}’` does not preserve the data. You can see this with `printf ‘%s\n’ -d ‘{“foo”:”‘$BAR'”}’` — that _should_ output exactly 2 lines, but there are more.

    I’d recommend using jq to build the JSON data:

    data=$(jq -n -c –arg bar “$BAR” ‘{“foo”: $bar}’)

    Now, you do: `curl … -d “$data”`

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.