How Dropbox Saves Millions By Using Amazon DynamoDB & S3
Discover the story of how the popular storage company built a scalable and cost-efficient metadata store with DynamoDB and S3.
In2018, Dropbox faced a critical challenge: they’re om-premise metadata store was quickly running out of storage capacity.
With a constantly growing number of users, they needed a solution fast.
Either double their on-premise storage — which would cost them millions of dollars — or find a more scalable cloud-based solution that could help them grow without breaking the bank.
Dropbox turned towards AWS to build a scalable and cost-efficient solution.
Using a combination of Amazon DynamoDB and Amazon S3, Dropbox built a platform called Alki.
Alki is a metadata storage system that helped Dropbox avert its capacity crisis and save the company millions of dollars as well.
Migrating to the cloud
Since its founding in 2007, Dropbox has become a leading business in file storage and sharing with over 600 million users uploading billions of files.
Dropbox was managing an enormous amount of metadata through a system called Edgestore, an on-premise shared MySQL database.
However, as Dropbox’s user base grew, so did the cold metadata — data that is rarely accessed but must remain stored and accessible at any time.
By 2018, Dropbox realized it was less than two years away from hitting the limits of its on-premises storage capacity.
But expanding the storage of this system would mean investing millions of dollars. This would only delay the issue but wouldn’t fix it.
As well, storing rarely accessed data in the same environment as frequently accessed data wasn’t financially feasible.
To tackle this problem, Dropbox created a team of two engineers to create a new metadata store.
Their solution was to leverage services like DynamoDB and S3 to create a system that could store hot and cold data efficiently.
With AWS handling the complex tasks of data replication, scaling, and hardware provisioning, the Dropbox team was able to focus on building the solution without worrying about infrastructure.
“When building a storage system, you have to think about a lot of components, including replication, backups, and capacity management. Amazon DynamoDB and Amazon S3 fit that need well — they are industry standards.” — Jonathan Lee, Alki Team Tech Lead, Dropbox. [1]
Building Alki
Alki, Dropbox’s new metadata storage system was built with two tiers in mind:
Hot metadata is stored on DynamoDB
Cold metadata is offloaded to Amazon S3 for long-term, cost-effective storage.
Using Dynamo’s high throughput capabilities, Alki was able to process up to 6,000 writes per second per table and handle billions of transactions seamlessly.
At the end of each day, about 80GBs of data was moved to S3 to free up DynamoDB capacity.
The speed and scalability of DynamoDB and S3 allowed Dropbox to complete the migration of 300TB of audio data in just 2 weeks. This would have been impossible with on-premise systems.
By the end of 2019, Alki was fully operational and storing and retrieving metadata at a fraction of the cost it had previously cost them — the cost per gigabyte was reduced by a factor of 5.5.
Benefits Of Moving To AWS
The switch to AWS provided Dropbox with 2 major benefits:
Significant cost-savings
Virtually limitless scalability
By migrating to Alki, Dropbox was able to reduce the cost of storing metadata by 5.5 times compared to its on-premise systems.
And because DynamoDB and S3 scale seamlessly, Dropbox no longer had to plan years in advance and purchase costly hardware — they had all of that immediately with AWS.
Furthermore, by leveraging DynamoDB’s auto-scaling features, Dropbox could easily support spikes in demand without over-provisioning resources.
Improving for the future
In the future, Dropbox plans to continue improving Alki.
A potential improvement involves using Amazon EMR to simplify the offloading of data from DynamoDB to S3, which is currently handled by a custom system.
Dropbox also plans to expand Alki’s capabilities to serve as a general-purpose cold metadata store, which would result in moving more of its infrastructure off Edgestore and into the AWS cloud.
Some Impressive Stats
Dropbox launched its metadata storage system on AWS in 1 year only.
They saved millions of dollars in infrastructure costs by using the cloud.
They ingest 4k-6k queries per second.
They cut costs per user GB by a factor of 5.5.
They migrated 300TB of data in less than 2 weeks.
Conclusion
Dropbox’s migration to AWS and the building of Alki has been a game-changer.
By using DynamoDB and S3, Dropbox saved millions of dollars and avoided complex infrastructure expansions.
Alki’s success demonstrates the power of AWS-managed services and how they can solve complex challenges quickly and cost-efficiently.
Now, with its refinements, Alki looks to keep innovating to offer a faster and more reliable experience to its 600 million users worldwide.
👋 My name is Uriel Bitton and I’m committed to helping you master Serverless, Cloud Computing, and AWS.
🚀 If you want to learn how to build serverless, scalable, and resilient applications, you can also follow me on Linkedin for valuable daily posts.
Thanks for reading and see you in the next one!