Learn caching

Dear new developer,

Caching is a common architectural pattern that helps with performance and scalability. Spending some time learning about this will help you build better systems and understand existing architectures.

What is a Cache?

At the most fundamental level, a cache is a secondary store of a set of values which pulls values from a primary source. There are many reasons why you might want such a store, but a common one is that retrieving the data from the primary datastore is expensive. It could be costly because it is an expensive calculation, because it requires a network call or disk read, or for some other reason.

To avoid the cost of the retrieval, store the value in a cache, and then, when it is needed, retrieve it from there.

Cache Types

There are two main ways to categorize caches.

• How the data is stored
• What is stored

Cache data can be stored the same way any data can be stored. In practice, for most applications, you’ll be looking at a few options:

• Memory
• Disk
• Abstracted

Caches that are stored in memory are quick (yay!) but ephemeral (boo!). The cache can go away at any time (whenever the server maintaining the memory is shut down), which means that you can lose access to the cached items at inopportune times. In addition, a memory based cache competes for RAM with other elements of your system. In general, most systems have less memory than disk, which is an alternative storage mechanism.

Retrieving items from disk is slower than memory. Of course, this is true for all disk access, not just that of cached data. The upside of a disk based cache is that you can store far more information.

You can often switch between memory and disk easily and transparently to your application’s code; consult the cache’s documentation to learn more about this option.

The last type of cache is ‘abstracted’. This means you use a cache as a service and don’t really care about the underlying implementation. This could be part of a framework (rails caching, for example), a standalone program like memcached or redis, or a full fledged service like a CDN. Browsers are a cache too. You don’t have to care about the details of this abstraction to take advantage of it. However, you will need to have some understanding of the cache behavior and performance when you operate your software.

Cache Keys

Most caches have a key and a value. The value is the cached value. The key is something that can be constructed by a client to get to the value. There is an implicit communication between the process that puts the value in a cache and the processes that retrieve a value from the cache: they have to agree to how the key gets constructed. Otherwise the cache readers won’t be able to actually read the correct values from the cache.

If you have product data that you are caching, you might have a key of product-<productid>. For a product with the id 15, the data would have a key of product-15. The product prefix namespaces the id in the cache so you can have multiple different types of objects cached (categories, deals, etc). The product id (15) needs to be known by the client to get the data.

Evictions

At some point, your cache will run out of room to store new items. At that point, it will need to get rid of some old items. This process is called ‘eviction’, and there are multiple ways to configure a cache to evict old items:

• Least recently used (LRU): evict the items that were accessed furthest in the past.
• Least expensive. If you are caching calculations because they are difficult/expensive to perform, evict the least expensive calculations first.
• Oldest: evict items that are the oldest. This is also known as first in, first out (FIFO).

Sometimes it can be hard to judge which of these will be the right fit for your system, especially since you may not have a good grasp of usage patterns initially. Generally LRU is a safe choice to begin with.

You should also think about how to trigger evictions. This might occur if the item being cached is materially changed. An example is a new logo for a website. You might want that to be cached in a CDN for years, because it rarely changes and is used everywhere (so you might set the cache period to a long time). But when a new logo is included in a launch, you want to ensure it is used. One way to manually trigger a cache eviction is to use a new name for the logo file.

You might also want to evict a cache item when the underlying source of truth has changed. For instance, if you are displaying the price of an item, when the price changes in the underlying data, you are going to want to display the new cost right away. So when the price changes, you’ll want to force-evict any cache you have. You don’t want someone thinking they can get the product at $X, only to find they are charged $X+1. Not a fun conversation with the customer.

When Should You Use a Cache?

Caches optimize access but introduce complexity. A system is always easier to understand if you pull from a single source of truth.

Introduce a cache when the performance and scalability benefits are required. Otherwise you are simply prematurely optimizing. How do you know if they are required? Evaluate the performance of your code. You can do this by inspection and reasoning about the code. It can be easier to run tests with the cache both disabled and enabled if it is easy to integrate. If you don’t see big changes in performance, there is probably a different slow area of your system.

In addition, think about how expensive a retrieval from the primary datasource is. This will change over time based on your usage and the type of request. You’ll need some metrics to make a good decision as well as to determine if the cache actually helps. Yes, you can introduce a cache and have things be slower, especially if you are on a resource constrained system or you misconfigure it.

Also, consider how much effort it will be to introduce caching. If it is easily supported in your framework or you can add some headers to your HTTP response, caching can be simple to introduce. If it requires you to stand up a whole new system and refactor clients to use it, it can be difficult.

Finally, there are some caches that are built into systems already. Take some time to learn about them; there may be easy wins by tweaking the configuration. Linux has caching as do most databases.

Conclusion

As you might have gleaned from all the examples I mentioned, caches are everywhere. Learning about them and how they can be used is worth your time.

Sincerely,

Dan