What Are DynamoDB Hot Partitions And How To Avoid Them?
Hot partitions are a very common problem, here’s how to avoid them and make your database more scalable.
Imagine the following scenario.
You enter a very busy fast food restaurant at suppertime.
There are multiple cashiers and one self-help counter where customers can order food.
Now if every customer lines up at the self-help counter instead of spreading evenly across the other counters with cashiers, it’ll result in delaying everyone’s orders.
This issue, in the context of DynamoDB, is known as hot partitions.
What Are Hot Partitions?
DynamoDB Partitions
To understand what hot partitions are and how they affect your DynamoDB database, we have to understand the basics of DynamoDB first.
When you write an item to DynamoDB, your request passes through a request router. This router takes the partition key you define, hashes it, and stores it on a storage partition based on that hash value.
So essentially every item or collection of items has a different partition key and therefore is stored on a different storage node in the DynamoDB servers.
You can then define the capacity throughput to provision for incoming reads and writes.
Each partition will share in this capacity throughput. So if you provision 100 RCUs/WCUs for a table and have 4 partitions, each partition would have 25 RCUs/WCUs.
How Are Hot Partitions Created?
But what happens when your application starts running more than 25 requests — say 40 requests per second for a given item or item collection?
The requests will get throttled and DynamoDB will return a “ProvisionedThroughputExceededException”.
The issue here is that while your table can support up to 100 read requests per second if too many concurrent requests are made on items in the same partition, they will still get throttled even though they stay under the table capacity limit.
In other words, while one partition is getting overloaded with traffic, other partitions are being underutilized, wasting throughput capacity.
The result is that your application will experience latency issues despite having sufficient table throughput capacity.
In DynamoDB, this is known as a hot partition.
Common Causes
Poor partition design is often the cause of hot partitions.
Using low cardinality items causes hot partitions. Low cardinality happens when too many items share the same identifier — a partition key in this case.
Designing items for high cardinality instead, lets your data be spread across more partitions, lowering the risk of overloading a given partition with a high concurrency of requests.
Some good partition key design:
userID
productID
articleID
most ID type attributes
Some bad partition key design:
active (true for around 50% of items)
pending (all orders will start here)
monday/october (1/7 or 1/12 spikes)
electronics (category — some categories will be more popular)
The Impacts of Hot Partitions
The biggest impact of hot partitions is often throttling. This is always a negative impact to a system (and user experience).
Hot partitions also lead to other inefficiencies like increased latency, wasted capacity on other partitions and higher costs (by over-provisioning to diminish the hot partition).
How to avoid them
Hot partitions are most easily avoided with good partition key design.
There are a number of ways of improving this.
The first is to use high cardinality partition keys like IDs — userID, productID, articleID like mentioned earlier.
If you expect a certain partition to get more traffic, you can add a suffix to the key such as “active#5012” (random number) or “pending#2024–12–01” (date).
Adding information that makes the key less common (higher cardinality) will cause the requests to spread out more evenly and prevent hot partitioning.
It is important to note never to provision additional capacity on a hot partition as that will only delay the problem while necessarily increasing costs.
To learn good partition key design, I encourage you to read my article here.
Conclusion
Hot partitions in DynamoDB occur when too many requests target items sharing the same partition key, leading to throttling and performance issues even when overall table capacity is sufficient.
The primary cause of hot partitions is poor partition key design, particularly the use of low-cardinality keys that result in many items sharing the same partition.
To prevent hot partitions, focus on using high-cardinality partition keys (like unique IDs) and, when necessary, add suffixes to distribute traffic more evenly across partitions rather than attempting to solve the problem by increasing throughput capacity.
👋 My name is Uriel Bitton and I’m committed to helping you master Serverless, Cloud Computing, and AWS.
🚀 If you want to learn how to build serverless, scalable, and resilient applications, you can also follow me on Linkedin for valuable daily posts.
Thanks for reading and see you in the next one!