Learn curl

Dear new developer,

If you interact with web APIs at your job, learning curl is a good investment. Curl is a flexible, powerful, command line utility which lets you make HTTP requests.

In my current job, I build a lot of API requests. Side note: I’m going to use the shorthand API for web based, JSON API; there are many other kinds of APIs, but curl is mostly useful for this type of API.

I’ve found familiarity with curl to be very helpful as I create, share and debug these API calls.

Why learn curl?

With curl, I can easily script up API interaction. In the next section, I’ll discuss the common parameters I use.

Curl works across multiple platforms. It’s a lowest common denominator tool and is present everywhere. It doesn’t require anyone to install anything. (Okay, maybe not true on vanilla windows, but as soon as you have any unix based system, you have curl.) Update 7/21: curl is present on vanilla windows 10. Open up a cmd window and run curl and you’ll see curl: try 'curl --help' for more information.

You can share scripts with team members, add them to issue comments for provide bug reproduction steps, and version control them if you want to improve them over time. I find them especially helpful for issue reports, as there’s no ambiguity with a script.

Curl pairs nicely with jq (which you should also learn); you can pipe the output of curl directly into jq to get understandable formatted JSON.

I find when I use a low level tool like curl, I’m forced to have a deeper understanding of the problem than I would if I used a higher level tool. It’s kind of like a text editor in that way.

I haven’t run into a situation yet where curl couldn’t interact with an API in a way I wanted, though sometimes multi step interactions like an OAuth authorization code grant can be a bit cumbersome. You can set headers, provide cookies, send a request using any HTTP method, and provide data either inline or via a file.

Basics of curl for APIs

Here are the main switches I use with curl for API interaction.

-X <method> lets me specify the HTTP method, so -X GET is a get request and -X POST is a post request. Sometimes the HTTP method is implied by other switches, but I prefer to be explicit.

-H <header> lets me specify HTTP headers. You can use this many times in one command. Often if I’m sending JSON in a post with an authorization header, it’ll look something like -H 'Content-type: application/json' -H 'Authorization: Bearer APIKEYHERE'

-d payload lets me specify a payload for a post or put request. This will often be JSON: -d '{"foo":"bar"}' It can be multi-line if needed. I use single quotes around the JSON to avoid having to escape double quotes. If I need to have a shell variable inline, so that I can extract some text which changes often, I do something like this: -d '{"foo":"'$BAR'"}'

-d @payload lets me put the payload in a file and have curl read from that file. I use this switch less often, because I’ll usually put everything in a shell script anyway.

-vvv lets me see the back and forth of the HTTP requests and response, and also status codes.

As previously mentioned, I also usually wrap everything in a shell script so it is repeatable. On occasion, I’ll even check the shell script in, so that others can use it. This is useful if it is a part of a run book or troubleshooting guide, for example.

Here’s a typical curl shell script, in all its glory:

API_KEY=...
USER_ID=...
curl -vvv -H "Authorization: $API_KEY" -H "Content-type: application/json" -XPOST 'https://api.example.com/user/'$USER_ID -d '{
  "user" : {
    "name": "Dan",
    "blogger": true,
    ...
  }
}'

There’s a lot more available with curl; the whole man page is worth a read. But I hope I’ve convinced you that you can use curl to interact with APIs in a way that is:

  • repeatable
  • shareable
  • versionable

Alternatives

There are many alternatives. I am a bit familiar with postman, which is a web based application that lets you organize and call APIs. There is also a command line runner for postman called newman. Insomnia is another option that some of my team members use. It’s another GUI client with a corresponding CLI client called inso.

While I haven’t dug deeply into these alternatives, at first glance it seems to me that they give you tooling power and richer collaboration. But nothings free; you lose ubiquity and simplicity.

curl is a powerful tool, available everywhere, for examining and interacting with APIs. Learn it well.

Sincerely,

Dan

Meetings are work

Dear new developer,

I remember when I started out, I thought meetings were a waste of time.

All those people in a room (now a Zoom), discussing an issue. Folks would go back and forth, and at the end of the meeting you’d hopefully have a decision. But often the output was just more research or discussion. You might have to go back to the client, the team, or vendor and ask more questions. This would be followed, you guessed it, another meeting to relay the results.

I just wanted to code! I wanted to be told what to do, or if that wasn’t an option, take my best reasonable guess, and program.

Meetings sure seemed like a waste of time.

Now, many years and meetings later, I realize that meetings are really work. They help align teams, ensure you are building the correct things, and share knowledge. These are key parts of delivering software and making sure that all the programming isn’t for the lost cause.

Yes, you can do many things asychronously. I enjoy flexible work hours and chat tools and pull requests and email lists. But whenever I see an email or other async conversation look like it is going to go off the rails, I jump in and suggest a meeting. It may be a pain, but it will let you cut through days of unclear emails. For example, I remember meeting with a vendor in India at 8pm at night, because that was the time that was least inconvenient for the both of us. We were able to resolve questions we’d been exchanging emails about for days with thirty minutes of conversation.

Meetings define and push the work forward every bit as much as programming.

Sincerely,

Dan