On debugging

Dear new developer,

Debugging systems is a key skill to have. Here are a few thoughts about it.

• Try to get the problem to be as simple as possible. Start with the problem and keep isolating and removing pieces and see if the problem persists. Modern systems are complex and the less you have to think about, the better.
• Keep notes about what you’ve tried. These can go into a chat system if the debugging is high priority (on a production system during an outage) or in a private text document if the debugging is not (a bug you’re trying to understand).
• Think about recent changes to the system if the bug is new. This isn’t always the cause, but it’s often part of it. Rarely does a system just degrade spontaneously.
• Write an automated test to illustrate the bug (if possible). This will
  • speed up your fix because it gives you a tight loop of run test, make change, run test, make change.
  • ensure that your change actually fixes the bug as opposed to something else.
  • provide a regression suite that will prevent this bug from popping up again.
• Start at one end of the system or the other when you are trying to isolate the issue and see where it appears. For example, for a user facing web application, start with either the browser or the data store.
• Minimize impact to users. If you are working on a production bug, ideally you can test on staging (right?) because this gives you the most latitude. However, if you do that, make sure that staging and production are exactly the same, otherwise you will chase your tail. Even if you have to test on production you can print debugging statements only for certain users (you or admin users) or to comments.
• Start with a hypothesis and work to continuously disprove or prove it, refining it as you know more information.

Sincerely,

Dan