The Problem
Your security team requires VPC flow logs for compliance. The logs will be frequently accessed for the first 90 days during active investigations, then only intermittently accessed for historical analysis. You need a cost-effective storage strategy that does not delete older logs.
The Solution
Publish VPC flow logs directly to Amazon S3 and configure a lifecycle policy to transition logs to S3 Standard-IA after 90 days.
How It Works
VPC flow logs support two destinations: CloudWatch Logs and Amazon S3. For large-scale, long-term log retention, S3 is the more cost-effective option.
Step 1: Create the Flow Log to S3
1
2
3
4
5
6
aws ec2 create-flow-logs \
--resource-type VPC \
--resource-ids vpc-0123456789abcdef0 \
--traffic-type ALL \
--log-destination-type s3 \
--log-destination arn:aws:s3:::my-flow-logs-bucket/vpc-logs/
Step 2: Add the Lifecycle Policy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
aws s3api put-bucket-lifecycle-configuration \
--bucket my-flow-logs-bucket \
--lifecycle-configuration '{
"Rules": [
{
"ID": "TierFlowLogsAfter90Days",
"Status": "Enabled",
"Filter": {
"Prefix": "vpc-logs/"
},
"Transitions": [
{
"Days": 90,
"StorageClass": "STANDARD_IA"
}
]
}
]
}'
Why S3 Over CloudWatch Logs?
| Feature | S3 | CloudWatch Logs |
|---|---|---|
| Storage cost | ~$0.023/GB (Standard) | ~$0.03/GB |
| Long-term archival | Lifecycle to IA/Glacier | Export to S3 (extra step) |
| Query capability | Athena (SQL) | CloudWatch Insights |
| Retention control | Lifecycle policies | Retention period (deletes data) |
The critical difference: CloudWatch Logs retention settings delete data after the configured period. S3 lifecycle policies transition data to cheaper storage. If you need logs retained indefinitely for compliance, S3 is the right choice.
Why Not the Alternatives?
CloudWatch Logs with 90-day expiration — Expiration means the logs are permanently deleted. The requirement is to retain logs for intermittent access, not to destroy them.
Amazon Kinesis as target — VPC flow logs cannot be published directly to Kinesis Data Streams. The valid destinations are CloudWatch Logs and Amazon S3.
AWS CloudTrail as target — CloudTrail logs API calls, not network traffic. VPC flow logs and CloudTrail serve different purposes entirely.
Key Takeaways
- VPC flow logs can write directly to S3 — no need to go through CloudWatch Logs first
- S3 lifecycle policies transition data to cheaper tiers; CloudWatch retention deletes data
- S3 Standard-IA is ideal for logs accessed intermittently — same durability, lower storage cost
- Use Amazon Athena to query flow logs stored in S3 with standard SQL
- VPC flow logs support only two destinations: CloudWatch Logs and S3
Never miss a story from us, subscribe to our newsletter