The Problem
Your IoT sensors stream data to S3 via Kinesis Data Firehose, producing trillions of objects per year. The access pattern is well-defined:
- First 30 days: accessed daily for ML model retraining
- Day 31 to 1 year: accessed quarterly for analysis
- After 1 year: retained for long-term archival
You need a storage strategy that aligns cost with each phase.
The Solution
Use S3 Standard for the initial 30 days, then lifecycle policies to transition through S3 Standard-IA and finally S3 Glacier Deep Archive.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
aws s3api put-bucket-lifecycle-configuration \
--bucket iot-sensor-data \
--lifecycle-configuration '{
"Rules": [
{
"ID": "IoTDataLifecycle",
"Status": "Enabled",
"Filter": {},
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 365,
"StorageClass": "DEEP_ARCHIVE"
}
]
}
]
}'
How It Works
Why This Three-Tier Approach?
1
2
3
Day 0-30: S3 Standard — daily ML training reads (no retrieval fees)
Day 31-365: S3 Standard-IA — quarterly analysis (low retrieval fees acceptable)
Day 366+: Glacier Deep Archive — long-term archival at lowest cost
Why Not S3 Intelligent-Tiering?
Intelligent-Tiering charges a monitoring fee per object: $0.0025 per 1,000 objects/month.
With trillions of objects per year, that monitoring fee adds up fast:
1
1 trillion objects × $0.0025 / 1,000 = $2,500,000/month in monitoring fees alone
When the access pattern is predictable, lifecycle policies avoid this fee entirely. At scale, the difference is millions of dollars per month.
Why Not Start in Standard-IA?
Data is accessed daily for the first 30 days. S3 Standard-IA charges $0.01/GB per retrieval. Daily retrieval of the full dataset during ML training would cost more in retrieval fees than the storage savings justify. S3 Standard has no retrieval fees.
Key Takeaways
- For trillions of objects, Intelligent-Tiering monitoring fees can cost millions per month
- When access patterns are predictable, lifecycle policies are far cheaper than Intelligent-Tiering
- Three-tier lifecycle (Standard → Standard-IA → Deep Archive) aligns cost with access frequency
- Do not start in Standard-IA if data will be accessed frequently — retrieval fees add up
- S3 Glacier Deep Archive is the cheapest tier for long-term retention at ~$0.00099/GB/month
Never miss a story from us, subscribe to our newsletter