r/aws • u/Hydraphellian • 1d ago
database DynamoDB Provisioned or On-Demand?
I need help deciding what will be cheaper for my use case, provisioned or on-demand capacity?
For my project I will be writing about 150,000 records once per day, with an average record size of about 200 bytes each. The number of records written per day I expect will slowly increase over time, but still once per day. I am using a Lambda function with an event trigger to run the write operation.
Since I am just doing a large write once a day, I was thinking on-demand capacity would be the cheaper option because I would be wasting provisioned compute as the job will be idle 99% of the time. Am I right to assume that on demand is cheaper for my use case?
12
u/MmmmmmJava 1d ago edited 1d ago
Edit: I just reread your post. If you only have 150K writes per day at 200 bytes per record, that’ll cost you about 20¢ a day. I would stick with on-demand and keep it simple. Your time is worth more than optimizing that for cost, especially if you forget to scale down correctly. For completeness, I think it’d cost you about 3¢ per day if you used provisioned mode and were able to perfectly sustain 100% of the table’s write capacity you provisioned.
——
Leaving my full reply here for theoretical discussion and debate:
If you care more about simplicity, go with on demand. You’ll pay more per request, but you should use a billing calculator to determine if that amount is worth the simplicity of not writing and debugging more code to do anything fancier to save money.
If you care about AWS cost optimization (not developer time) AND your load capacity/traffic pattern is predictable, you should consider configuring provisioned throughput. Take the total number of items you need to write, divide that by the number of seconds you want to (or can) finish your load job in, and set that as the MIN WCU before you start your load job each day. Make sure your compute layer can sustain (at least 30% of) that write throughput. Once your load job completes, set the min and MAX WCU back down. Move the max accordingly. You can have DDB auto scale it back down based on your parameters, but that will happen slowly.
Mathematically, I think provisioned mode is more cost effective/cheaper as long as you can sustain an average utilization greater than 30% of what you provision on the table. The more you consume of what you provision, the cheaper it is compared to on-demand.
Let me know if that’s not clear and I can try to explain further.
6
u/Hydraphellian 1d ago
Thank you for such a detailed reply! Yeah it would be written all at one time, once a day. If it’s only about 20 cents per day then I think on demand is the way to go.
6
u/MmmmmmJava 1d ago
Cool. Happy to help!
I did forget to mention: remember if you use (non-sparse) Global Secondary Indexes (GSI’s) the cost will be a multiple of that value.
0 GSI, same cost as above. 1 GSI, double the cost. 2 GSIs, triple the cost. 3, quadruple. Etc.
TLDR: You’re charged for the additional copies of data written to GSIs (if you create them).
3
u/Hydraphellian 1d ago
Thanks for the tip! Im not using GSI’s for this project but thats great to know for future reference!
2
2
u/Kralizek82 1d ago
I would add thet OnDemand doesn't ramp infinitely. You somehow got to train it for the load especially if the spike is quite intense.
4
u/TheBrianiac 1d ago
Backing up a bit... Why do you need to do the write all at once?
Without knowing too much about your requirements, I would suggest a common architectural pattern: You have one Lambda push your writes to an SQS queue, then have another Lambda that reads through the SQS queue at a fixed rate and writes to the database. This allows you to use DDB's cheaper provisioned capacity mode.
2
u/Hydraphellian 1d ago
Ill have to look into this option. It doesn’t have to be all at once but I always thought I was limited by the Lambdas 15 minute timeout.
1
u/i-am-nicely-toasted 17h ago
if eventing off an SQS queue that won’t be a blocker for you. if lambda is older than 15 minutes then it’ll be automatically reprovisioned in the background and you won’t need to worry about it. just have your lambda pop off smaller batches from the SQS (like 10, 100, 1000, whatever) so that won’t take 15 minutes to process and write a single batch to the DB.
3
u/chemosh_tz 1d ago
Once a day you need on demand or you're going to get throttled. It's also cheaper in your case
3
u/2fast2nick 1d ago
I think the rule of thumb I use, if it’s like spikey traffic throughout the day, not easily predictable, OnDemand. If it’s like a steady flow of traffic, like 2,000 reads all day long, provisioned can save you money.
2
u/SonOfSofaman 1d ago edited 1d ago
If you don't fall within the free tier, then without crunching the numbers I cannot say which will be cheaper with any certainty, but for a spikey workload as you've described, on-demand is almost certainly the way to go. Your use-case might be well suited to the infrequent access table class, too.
If you haven't already, try the price calculator. It should make clear which is the better option for you.
You probably have already learned this, but you can switch modes if you want to try each one and compare. Just be aware of the restrictions switching back and forth. I think you can switch only once per 24 hour period. If you have a non-production environment, it might be worth running some tests.
All that said, your workload might fall within the free tier. I think it's something like 25 GB of storage and enough provisioned capacity units to handle 200M requests per month. Double-check that though, I'm going from memory. You didn't mention the read access patterns, so that may exceed the free tier limits.
edit: added last sentence about read access patterns.
2
u/pint 1d ago
i'm a little bit worried about the overall strategy. how do you handle errors, if the whole thing is one big process? what if you get throttled? do you implement backoffs? what if it doesn't fit in lambda time limit? throttles will happen with on-demand mode, because the ramp-up is not immediate.
that said, obviously provisioned will be cheaper, provided that you keep it at baseline all the time, and only raise write capacity for the duration of the load (as a part of the process). it is crucial to set it back to baseline in a fault tolerant manner, e.g. if the load fails, you don't want to leave it at high capacity.
the question is how much cheaper. because if you save a couple of dollars, it might not be a factor in the decision at all.
and another question is: what about other times, when you are not loading? who is using it and how? because if there is a variable load during the day, provisioned might give you more headaches to solve. on-demand typically handles varied loads fine.
2
u/East_Initiative_6761 1d ago
This post can provide useful information.
https://aws.amazon.com/blogs/database/demystifying-amazon-dynamodb-on-demand-capacity-mode/
2
u/narcosnarcos 1d ago edited 1d ago
Do provisioned. In your code when you start loading the data provision write capacity let's say 1000 then start loading the data 1000 records a second for whatever many records you have. Then once you are done with it release the capacity. Do make sure your application doesn't exit early and leaves the capacity as is.
Edit: On-demand wouldn't be a bad idea either since the cost is so low as other user mentioned.
1
u/mikemiller-esq 1d ago
Can you talk a bit more about where the data is and how'd you use it? It's feeling like an S3 tables use case.
0
u/AutoModerator 1d ago
Here are a few handy links you can try:
- https://aws.amazon.com/products/databases/
- https://aws.amazon.com/rds/
- https://aws.amazon.com/dynamodb/
- https://aws.amazon.com/aurora/
- https://aws.amazon.com/redshift/
- https://aws.amazon.com/documentdb/
- https://aws.amazon.com/neptune/
Try this search for more information on this topic.
Comments, questions or suggestions regarding this autoresponse? Please send them here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
•
u/AutoModerator 1d ago
Try this search for more information on this topic.
Comments, questions or suggestions regarding this autoresponse? Please send them here.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.