The Problem
Your IoT sensors stream data to S3 via Kinesis Data Firehose, producing trillions of objects per year. The access pattern is well-defined:
- First 30 days: accessed daily for ML model retraining
- Day 31 to 1 year: accessed quarterly for analysis
- After 1 year: retained for long-term archival
You need a storage strategy that aligns cost with each phase.
The Solution
Use S3 Standard for the initial 30 days, then lifecycle policies to transition through S3 Standard-IA and finally S3 Glacier Deep Archive.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
aws s3api put-bucket-lifecycle-configuration \
--bucket iot-sensor-data \
--lifecycle-configuration '{
"Rules": [
{
"ID": "IoTDataLifecycle",
"Status": "Enabled",
"Filter": {},
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 365,
"StorageClass": "DEEP_ARCHIVE"
}
]
}
]
}'
How It Works
Why This Three-Tier Approach?
1
2
3
Day 0-30: S3 Standard — daily ML training reads (no retrieval fees)
Day 31-365: S3 Standard-IA — quarterly analysis (low retrieval fees acceptable)
Day 366+: Glacier Deep Archive — long-term archival at lowest cost
Why Not S3 Intelligent-Tiering?
This is a critical decision point. Intelligent-Tiering charges a monitoring fee per object: $0.0025 per 1,000 objects/month.
With trillions of objects per year, that monitoring fee becomes massive:
1
1 trillion objects × $0.0025 / 1,000 = $2,500,000/month in monitoring fees alone
When the access pattern is predictable, lifecycle policies avoid this fee entirely. The cost difference at scale is enormous.
Why Not Start in Standard-IA?
Data is accessed daily for the first 30 days. S3 Standard-IA charges $0.01/GB per retrieval. Daily retrieval of the full dataset during ML training would cost more in retrieval fees than the storage savings justify. S3 Standard has no retrieval fees.
Key Takeaways
- For trillions of objects, Intelligent-Tiering monitoring fees can cost millions per month
- When access patterns are predictable, lifecycle policies are dramatically cheaper than Intelligent-Tiering
- Three-tier lifecycle (Standard → Standard-IA → Deep Archive) aligns cost with access frequency
- Do not start in Standard-IA if data will be accessed frequently — retrieval fees add up
- S3 Glacier Deep Archive is the cheapest tier for long-term retention at ~$0.00099/GB/month
Never miss a story from us, subscribe to our newsletter