Learn to look around corners

Dear new developer,

Sometimes you want to play out things two and three steps deep. Kinda like chess players, who think about many moves ahead, if you can consider the ramifications of your decisions, you’ll be well served.

I remember talking to someone about a software position at his company and he referred to the “original sin” of choosing a particular NoSQL database. This had ripple effects throughout the life of the company. Some were good, some were bad.

These kind of downstream effects are worth thinking about. The biggest ones are things like fundamental platform choices (Rails or Django, Java or golang, mysql or postgresql). But even smaller things can have ripple effects. I can’t count the times when I’ve said “I’ll just upgrade this one thing” and then the change just keeps cascading out.

Or if you are working on a feature definition, think through the ramifications. “We want to allow people to change their email address.” But if email address is the primary key of a table, or a primary means of connecting data across accounts, or some other invariant, what will the ripples in your system be.

And these are just the functional repercussions. There are also performance repercussions to changes, some of which are hard to test unless you are in production.

I’m not saying don’t accept changes. That way lies madness (and Fortran). Just be aware of how changes in complicated systems like web applications have a way of rippling out beyond their intended scope, and consider how to test, mitigate or handle such ripples.

Sincerely,

Dan

What if I have to make a technical decision and I don’t know the right answer?

Dear new developer,

Sometimes you are confronted with decisions for which you simply don’t know the correct answer. This has happened to me many times over the years. A recent example is that the client wanted to build an online quiz. They wanted to be able to edit quiz questions and answers. They wanted it to feel like an application (rather than a website). I had never built something like this before, but I was the person best situated to make the architectural decision which would underpin this application.

You may not have time or money to do all the research you feel you need. You may be doing something that has simply never been done before. You may be in an arena where you are being pressed for a decision (like a meeting).

If you are in a situation where you have to make a choice about a fundamental architectural decision (like a platform/library/framework), you probably are in a small team moving fast, or at a company where you are one of a very few number of technical folks. In this case you may feel over your head and incapable of making the correct decision.

And yet, the decision still needs to be made, and you are the one who has to do it. (The unfortunate fact is that the too late perfect decision is worse than the pretty good timely decision.)

Techniques that won’t work out well for large important decisions:

  • avoiding making the decision
  • googling for an answer and take the first option provided
  • deferring to someone else
  • doing a thorough full examination of all the possible solutions, draw up an excel spreadsheet and a powerpoint to make sure you haven’t missed anything
  • avoiding discussion among team members

Here’s how to proceed.

First make sure that you do need to make the decision, and that you have the proper business context. The number of choices that you should consider and amount of research you should do depend on the impact of the decision. Things to think about:

  • how much of the business does this touch? If it is contained or constrained, you can spend less time thinking about this. If you are implementing a small micro service, or if you are choosing a service that is only used by a portion of the organization (like github for your code), then a choice that needs to be unwound will be easier to do. If you are picking the development language that will be used for the core product, rewrites are almost guaranteed to be far in the future.
  • how irreversible is the decision? Most decisions aren’t irreversible given enough time and money, but some are easier than others. Swapping out one memecached provider for another is pretty trivial, but changing from mysql to postgresql is more complex. Changing from GCP to AWS can, depending on the amount of data you have, be person-years.
  • does your company have internal solutions that you might be able to leverage? The bigger the company, the more likely the problem has been solved before, and finding that solution will be quicker and better than rebuilding it.
  • can you defer the decision until later? Is there a manual process or a service you can buy that will avoid a technology investment, even for a few months?

If you can’t defer the decision, then make the best one you can with the information you have. Research based on the impact on the business. I want to acknowledge the uncomfortable tension between knowledge and action that are at the root of any decision made with incomplete knowledge (aka, all of them!). There’s a spectrum based on risk and commitment, and unfortunately, I’m not aware of any great heuristics for determining risk or commitment other than experience and reading.

My approach towards such a decision is:

  • learn what you can. This includes finding out as much as you can about the problem, including current solutions
  • bounce ideas off of others in the company, but make them specific (both in form and in who you ask). This will help avoid analysis paralysis
  • ask for recommendations. Start at your current company. Even if you can’t find someone who has direct experience with the problem, there may be people who have related experience. You can also google to see what other folks have said about the various solutions (make sure to limit the results to “the past year” or you’ll end up reading about old versions). It can also include searching for experts on twitter and linkedin to ask questions (or to look at their blogs/sites to see if they’ve answered it already). And finally, don’t forget to look for communities (slack, old fashioned forums, open source project email lists) and see what they say. It’s far better to read what you can rather than pop into these communities and ask–those kinds of requests can happen often enough to annoy folks. But if you don’t see any answers, do ask.
  • realize that no decision is perfect. In 20 years of development, I can tell you that I’ve never made a perfect decision, they all involved tradeoffs and lack of perfect knowledge.
  • communicate the risks with the business. This includes the risks of you feeling like you are over your head, as well as any other risks that your research has found.
  • come up with a recommendation and implement it.

Again, hopefully you won’t be confronted with a far reaching architectural decision for a number of years, but it happens. Hopefully this framework gives you a bit of guidance toward making that decision.

Oh, and if you wanted to know my quiz conundrum ended, some of the choices I made were implemented, and others got swapped out as the team changed. The app was a success.

Sincerely,

Dan

Strong convictions, loosely held

Dear new developer,

When building a system, you’re going to be confronted with lots of decisions. Unless you are operating in a total vacuum, you’ll have to reach agreement with other people. This will lead to discussions and arguments, as all the people involved will bring their viewpoints and experience.

You should definitely participate, and bring your own experience and viewpoints.

However, everyone should pay attention to two things:

  • the usefulness of listening to everyone’s experience
  • the overarching goal to be achieved

The first is important because different experiences build a stronger solution, the same way a forest is more resilient than a field of wheat. But it’s also important because even if someone’s experience has no bearing to the problem at hand, if it is dismissed the person will have a harder time both contributing to the current problem’s solution (they won’t be bought in) and to issues in the future (when their experience might be more relevant).

The second is the right way to guide the conversation and to steer it towards success. No project or problem ever exists in isolation. It’s worthwhile to examine all the big decisions, and many of the smaller ones, in the context of the overarching goal. (Heck, there are some times when it is worthwhile to make sure you are working toward the correct overarching goal.)

When you listen to everyone and focus on the higher goal, good things happen. One especially good thing is that you may change your mind. The right way to do that is to argue as passionately as you can for your point of view (have a strong conviction), but not lose sight of the fact that you may be incorrect (loosely held). And if you are, then everyone benefits when you acknowledge the better solution and move forward with it.

Sincerely,

Dan