Use your shell’s history

Dear new developer,

The command line is a powerful tool, and writing shell scripts lets you write a series of commands once and replay them any time you like.

But sometimes you will write a series of commands without putting them into a script. This may be because you are exploring a problem or because you haven’t bothered to put together a script. No sense in making something repeatable if you aren’t sure exactly how you are going to repeat it.

But you may want to look back over past commands you have run, whether to run them again, modify them or even just remind yourself what you’ve done. For this, the shell history is very handy. Another time I often look at the shell history is when I am signing into a machine that I don’t visit very often. Perhaps there’s a command to look at some log files that I know, distantly, in the back of my mind, that I ran four weeks ago. I could have documented it, but maybe I didn’t. If I look in my command line history, I can see it.

All of the examples below are for the bash shell. Why? This is a very common shell, more modern shells like zsh are roughly compatible, and most importantly, this is the shell I know.

To use the shell history, first you have to know it exists and then you have to see it. You can also view the history by typing history in your terminal or at the shell prompt. Here’s an example from one of my shells:

  520  vi text.adoc
  521  git status
  522  git add text.adoc
  523  git commit -m "Initial pass at examples and documentation."
  524  vi text.adoc
  525  git add text.adoc
  526  git commit -m "Added one more example." text.adoc
  527  git push origin add-example-docs
  528  history

Each line is one command and is numbered. From here, I can repeat a command by number. So if I type !524 the command vi text.adoc will be run. I can also search for a certain string by using grep:

history | grep 'ssh' # will show all my ssh commands

The above commands are all pretty simple, but you can also link commands together. I often will do something like this:

cd directory && mvn jar && cp file.jar ~/place/jar/goes && cd otherdirectory

Once I get this command right, I can run it over and over again by referencing its line number. The && means that each command has to succeed before subsequent commands are run. So if the mvn command fails, the cp will not run.

This is fine for running tasks that are not going to change, but what about if I want to edit my previous commands. You can navigate the history by using the arrow keys (there are other ways as well, but arrow keys are the default and most accessible). Each up arrow takes you one step further back in your history, and each down arrow takes you one step forward. At any point you can edit the command you are currently on and hit the enter key and run it.

The bang shortcuts

If you know you want to edit whatever command you ran previously, you can use some bang operator shortcuts (so called because “bang” refers to a !). Let’s say I want to get the latest main branch from my git repo.

Suppose I run the command:

gti fetch origin main

I see an error message because I misspelled git. Whoops. I can type:

git !*

!* expands to everything except the first argument in the previous line, so this is the command that is run:

git fetch origin main

Now I want to check out the main branch

git checkout !$

!$ refers to the last argument of the command, so main in this case. The command which is run is then:

git checkout main

Knowing these shortcuts will help you avoid tediously hitting the arrow keys, editing the previous command and re-running it. There’s one more bang shortcut which is very useful. Suppose I want to install something on a linux box:

apt-get install pkgname

Whoops again! I’ll get an error message because I’m not root. Easily fixed by typing:

sudo !!

!! expands to the entire previous command, so what is actually run is:

sudo apt-get install pkgname

I use this very often when working with linux on the command line.

Controlling what is saved

You can control how much history is kept. Sometimes on my development machine I keep an infinite amount of it. I’ve also been on production instances where there was no history kept, I believe as a security measure. Remember, any command you run, including opening up a database connection using a command line client, will be kept in the history buffer. If you put a password on the command line, it will be recorded in this history.

To avoid that, don’t put the password on the command line. Another trick is to set the HISTCONTROL variable. When this is set to ignoreboth any command you type with a space in front won’t be stored in the history.

export HISTCONTROL=ignoreboth # previously set
echo "hello" # included in history
 echo "goodbye" # not included

You can also control explicitly how many commands to store in your history with the HISTSIZE setting:

export HISTSIZE=2000 # will save 2000 commands

Finally, be aware that all the bash shells opened share the same history file. Unless you take special care (as outlined here) the last one closed will win.

Being able to quickly re-run commands you’ve built previously, whether unchanged or not, can be helpful when you are navigating around the command line in pursuit of your software goals. Knowing a few history commands lets you do so.

Sincerely,

Dan

The Surprising Number Of Programmers Who Can’t Program

Dear new developer,

This came up in a Hacker News comment thread recently:

I’ve been working since the 90s and I never attempted to do FizzBuzz. Is it really relevant? Maybe to screen junior developers out of college?

And the response

So, as someone who spends maybe 20% of their time hiring, it’s still a very effective screen. You wouldn’t believe how many people can’t do it. People at big companies, respected places. It’s surprising.

I find it truly surprising. Here’s a post from 2007 about the same issue, so it’s been around a while. From that post:

…I’m starting to get a little worried. I’m more than willing to cut freshly minted software developers slack at the beginning of their career. Everybody has to start somewhere. But I am disturbed and appalled that any so-called programmer would apply for a job without being able to write the simplest of programs.

As a new developer, realize that

  1. You are going to stand out among other applicants if you can program.
  2. If you can’t program, you need to fix that asap.
  3. There are some folks that are just not effective programmers who somehow are in the profession (or seeking employment). You may end up working with and for some of these people.

So, make sure you practice programming. My letters have been about some of the other aspects of software development, but if you can’t do the basic work, you’re going to have a tough time. That’s like a baseball player who can’t run. Doesn’t matter how good someone is at catching a ball or hitting, if you can’t run, you’re going to be fundamentally disadvantaged at baseball.

Sincerely,

Dan

Programming Is Terrible, So Learn To Enjoy It

Dear new developer,

I appreciated this post which talks to people who are interested in being a developer, rather than someone who is newly a developer. I still think a lot of things apply.

This especially resonated:

…programming is terrible, so learn to enjoy it. If you are “on the net” learning about what programming is like, listening to the voices that get the most attention, you are probably getting the “Hollywood version” of what it means to work with software day in and day out. Now don’t get me wrong, I love programming. My best days are when I can take off the many other hats I wear and just focus on a development project and writing tests to go with it. But, it’s not all roses, and you should be prepared for that.

Here are some things that are terrible about programming (and counterpoint).

  • The tool set is always changing. What you learned two years ago may or may not be useful to you now. But some technologies age better than others, and you always have a chance to learn something new.
  • There are two modes of development that are radically different (deep thinking and communication), and you have to switch between them constantly. But you get to do deep thinking!
  • Error messages don’t make sense. But Google can help with that. And so can you, by blogging about the error after you figure it out.
  • With modern applications, there’s a ton of complexity and you can’t possibly understand everything. But you can put together an amazing application that would have taken a team of developers by yourself or with a small team.
  • Sometimes, you may be the expert in your problem, which can make it hard to find help. I remember writhing on the floor trying to figure out one of the most difficult problems I’ve ever solved (figuring out an editing interface and rules for recurring events on a calendar with child events that needed to have different durations than the parent event) and realizing that I could ask no one about it. But, you get the chance to be an expert!

Of course, every job has its warts. And development is no different. But programming and development are in a rapid state of flux. This means that you must (and get to) learn every day.

Sincerely,

Dan

Egoless programming

Dear new developer,

This post is worth reading in full, but is advice that holds for all developers, not just folks starting out.

I especially liked

Fight for what you believe, but gracefully accept defeat. Understand that sometimes your ideas will be overruled. Even if you do turn out to be right, don’t take revenge or say, “I told you so” more than a few times at most, and don’t make your dearly departed idea a martyr or rallying cry.

and

You are not your code. Remember that the entire point of a review is to find problems, and problems will be found. Don’t take it personally when one is uncovered.

But all of them are good. Go on, go read it.

Note that these principles (though not the post) are from a book published in 1971. Technology changes yet people remain the same.

Sincerely,

Dan

Learn your standard library

Dear new developer,

If you want to be good at interviews, learn your algorithms. Loads of companies treat algorithm knowledge as a proxy for general problem solving ability. It makes a certain sort of sense–you have to break down a problem into pieces, turn it into something software can digest, and implement it in code. You can practice these at places like HackerRank. Like any proxy, they can be gamed, which is why tech interviewing feels like a fundamentally broken process. (See this great post about humility in interviewers for another piece of the puzzle.)

But if you want to be good at your job, learn your standard library. This is the piece of software that surrounds you, in which your software will execute.

I remember the time when I was a contractor and started at a new place in the early 2000s. Someone there had written their own connection pooling library. This seemed like madness because it’s a complicated piece of work and there were several free implementations available for the language (though, to be fair, none were part of the standard library). This code, which provided very little business value, would have to be maintained and bugfixed and upgraded.

Every modern language has a definition (this includes syntax and keywords) and a set of associated, standardized classes. These classes allow developers to “stand on the shoulders of giants” and should be used whenever possible. Why? These libraries evolve over time as the language is released and have a number of beneficial properties.

  • Using the standard library lets you accelerate development. You don’t have to roll your own hash or tree, you can just use the one that ships with your library.
  • It is far more likely to be correct. This can matter a little, if incorrectness introduces a bug. Or it can matter a lot, if incorrectness causes your system to be fundamentally insecure. Standard libraries have a lot of eyes on them, including specialists.
  • As a corollary, standard libraries handle edge cases. They are well used code, in the sense of this post (source code doesn’t rust). If the language is used by any number of people, it’s going to be more bullet proof and handle weird edge cases that your self rolled code won’t handle (until a bug is discovered and fixed).
  • It will be more performant. Again, this is because the effort to make operations faster is amortized across all of the users, so specialists and experts are more likely to be involved.
  • The standard library for any popular language will be maintained over years, likely by developers you won’t have to directly pay.

All of these benefits stem from the fact that more people work on standard libraries than work on code in your company.

But, you have to know the standard library to gain all of these advantages. Unlike the syntax of a language, you don’t need to know the standard library to get things done. While you can’t roll up your sleeves and start coding until you know the syntax of ruby, php, java, clojure, haskell, javascript, etc, you can code for quite some time without knowing all the ins and outs of the standard library.

This is because standard libraries can be large: Java has thousands of classes in their standard library, and even a language with a small library like Javascript has about seventy objects, many with tens of properties or methods. It takes a while to learn. Plus, new developer, it can feel like you’re not doing much when you are reading docs. Certainly such reading  isn’t as fun as banging out code. But, it will have two benefits for the codebase:

  • you will save time in developing because of all the benefits above. You may have to invest some time up front, but once you do, you’ll be faster for the rest of your career.
  • you will write idiomatic code. Other developers who are familiar with the standard library will know the characteristics and behavior of the code you write.

and one for you:

  • you will gain a transferable skillset. Learning how to write the particular kind of ruby that your company uses is not nearly so valuable to other companies as learning the standard library.

How can you learn the standard library? I’d recommend taking a high level overflight. Google for ‘<language> standard library overview’ and read whatever articles or listen to whatever presentations are available. I’d also recommend just scanning the docs and seeing what piques your interest.

After you have that, the best way to learn the standard library is to use it. When you are about to tackle a problem, make it a standard part of your process to ask “is there a standard library function for that”. It can be difficult, but as it becomes a habit your code will become better and you will learn to trust the authors of the standard library.

If you have time and inclination, you can also search for ‘<language> koans’ which are small exercises in languages. These will often exercise the standard library. See Ruby Koans for an example.

When there is more than one way to do it using the standard library, which happens more in some languages (like PHP) than in others (like golang), think about the options and then try to be consistent.

So, learn the standard library. Make the time.

Sincerely,

Dan