The Problem
Thousands of edge devices generate 1 TB of status alerts daily (~2 KB each). You need to:
- Ingest the alerts at scale with no infrastructure to manage
- Keep 14 days available for immediate analysis
- Archive anything older than 14 days
- Minimize costs
The Solution
Use Amazon Kinesis Data Firehose to ingest and deliver alerts to Amazon S3, then apply an S3 lifecycle policy to transition data to S3 Glacier after 14 days.
How It Works
Architecture
1
2
3
Edge Devices → Kinesis Data Firehose → S3 (14 days hot) → S3 Glacier (archived)
↓
Amazon Athena (analysis)
Step 1: Create the Firehose Delivery Stream
1
2
3
4
5
6
7
8
9
10
11
12
aws firehose create-delivery-stream \
--delivery-stream-name iot-alerts-stream \
--s3-destination-configuration '{
"RoleARN": "arn:aws:iam::123456789012:role/FirehoseS3Role",
"BucketARN": "arn:aws:s3:::iot-alerts-data",
"Prefix": "alerts/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/",
"BufferingHints": {
"SizeInMBs": 128,
"IntervalInSeconds": 300
},
"CompressionFormat": "GZIP"
}'
Firehose automatically buffers, batches, compresses, and delivers data to S3. No servers to manage.
Step 2: Add S3 Lifecycle Policy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
aws s3api put-bucket-lifecycle-configuration \
--bucket iot-alerts-data \
--lifecycle-configuration '{
"Rules": [
{
"ID": "ArchiveAfter14Days",
"Status": "Enabled",
"Filter": {
"Prefix": "alerts/"
},
"Transitions": [
{
"Days": 14,
"StorageClass": "GLACIER"
}
]
}
]
}'
Step 3: Query Recent Data with Athena
1
2
3
4
5
SELECT device_id, alert_type, COUNT(*) as alert_count
FROM iot_alerts
WHERE year = '2026' AND month = '07' AND day >= '03'
GROUP BY device_id, alert_type
ORDER BY alert_count DESC;
Athena queries S3 directly — no database to provision or manage.
Why Kinesis Data Firehose?
| Feature | Firehose | EC2 Fleet | SQS + Lambda |
|---|---|---|---|
| Infrastructure management | None | Patching, scaling, monitoring | Lambda management |
| Auto-scaling | Built-in | Manual ASG config | Concurrency limits |
| Data batching | Built-in | Custom code | Custom code |
| Cost model | Pay per GB ingested | Per-instance hour | Per invocation |
At 1 TB/day with ~500 million 2KB messages, Firehose handles the volume without any infrastructure decisions.
Why Not the Alternatives?
EC2 instances behind ELB — Requires managing servers, patching, scaling, and monitoring. Contradicts the “no infrastructure management” requirement.
Kinesis Data Firehose to OpenSearch — OpenSearch is expensive for storing 1 TB of new data daily. S3 is orders of magnitude cheaper for bulk storage and analysis.
SQS with custom message processing — SQS maximum message retention is 14 days. The proposed workflow of checking message ages and copying to S3 is unnecessarily complex compared to a direct Firehose-to-S3 pipeline.
Key Takeaways
- Kinesis Data Firehose is the fully managed service for high-volume streaming data ingestion to S3
- Firehose handles buffering, batching, compression, and delivery with zero infrastructure
- S3 lifecycle policies automate the hot-to-archive transition (S3 Standard → Glacier)
- Use Amazon Athena for serverless SQL analysis of S3 data
- This pattern (Firehose → S3 → Lifecycle → Glacier) is the standard serverless ingestion-to-archive pipeline
Never miss a story from us, subscribe to our newsletter