The Problem
Your bucket has 1.2 billion objects. Compliance asks you to prove that every object is encrypted, tagged correctly, and covered by a replication rule. Running aws s3api list-objects-v2 in a loop would take days and cost thousands of dollars in API calls.
The Solution
Enable S3 Inventory. S3 writes a scheduled report (daily or weekly) listing every object in the bucket along with the metadata fields you choose. Query it with Athena and answer the compliance question in seconds.
How It Works
What S3 Inventory Produces
S3 delivers a report on the schedule you pick (daily or weekly) into a destination bucket. Each report is a set of files plus a manifest.json that describes the schema. Formats: CSV, Apache ORC, or Apache Parquet. ORC and Parquet are far cheaper to query with Athena.
Standard fields include object key, size, last modified date, storage class, ETag, and version ID. Optional fields you can enable:
EncryptionStatus— SSE-S3, SSE-KMS, SSE-C, NOT-SSEReplicationStatus— PENDING, COMPLETED, FAILED, REPLICAChecksumAlgorithm— CRC32, CRC32C, SHA1, SHA256IsMultipartUploadedObjectLockRetainUntilDate,ObjectLockMode,ObjectLockLegalHoldStatusIntelligentTieringAccessTierBucketKeyStatus
Enabling Inventory
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
aws s3api put-bucket-inventory-configuration \
--bucket source-bucket-hadzimahmutovic \
--id compliance-inventory \
--inventory-configuration '{
"Id": "compliance-inventory",
"IsEnabled": true,
"Destination": {
"S3BucketDestination": {
"Bucket": "arn:aws:s3:::inventory-reports-hadzimahmutovic",
"Format": "Parquet",
"AccountId": "123456789012",
"Prefix": "compliance/"
}
},
"Schedule": { "Frequency": "Daily" },
"IncludedObjectVersions": "Current",
"OptionalFields": [
"Size", "LastModifiedDate", "StorageClass",
"EncryptionStatus", "ReplicationStatus",
"ObjectLockMode", "ObjectLockRetainUntilDate"
]
}'
The destination bucket needs a policy that allows s3:PutObject from the S3 service principal, scoped to your source bucket ARN.
The first report can take up to 48 hours to appear. After that, daily reports arrive on schedule.
Querying With Athena
Create an external table over the Parquet inventory:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
CREATE EXTERNAL TABLE inventory (
bucket string,
key string,
size bigint,
last_modified_date timestamp,
storage_class string,
encryption_status string,
replication_status string,
object_lock_mode string,
object_lock_retain_until_date timestamp
)
PARTITIONED BY (dt string)
STORED AS PARQUET
LOCATION 's3://inventory-reports-hadzimahmutovic/compliance/source-bucket-hadzimahmutovic/compliance-inventory/hive/';
Find every unencrypted object:
1
2
3
4
SELECT key, storage_class, last_modified_date
FROM inventory
WHERE dt = '2026-07-27-01-00'
AND encryption_status = 'NOT-SSE';
Count objects by replication status:
1
2
3
4
SELECT replication_status, count(*)
FROM inventory
WHERE dt = '2026-07-27-01-00'
GROUP BY replication_status;
Chaining Inventory Into Batch Operations
S3 Batch Operations accepts an inventory manifest directly. Use the report to drive:
- Encrypt-in-place jobs with
S3PutObjectCopy - Bulk tagging with
S3PutObjectTagging - Object Lock retention updates with
S3PutObjectRetention - Restores from Glacier with
S3InitiateRestoreObject
This is the AWS-endorsed pattern for anything you need to do to millions or billions of existing objects.
Cost Model
Inventory reports cost roughly $0.0025 per million objects listed per report. For a billion-object bucket, that is $2.50 per report. Compare to the cost of ListObjectsV2 calls at $0.005 per 1000 requests. Billions of API calls would run into the thousands of dollars.
Why Not the Alternatives?
aws s3 ls or ListObjectsV2 in a loop — Slow, expensive at scale, and only returns object keys plus a handful of fields. No encryption or replication status.
S3 Storage Lens — Great for aggregated metrics and dashboards across buckets. It does not give you per-object detail.
Amazon Macie — Scans object content for sensitive data. Very different job. Do not use Macie to check whether an object is encrypted; use Inventory.
CloudTrail data events — Log every API call, not the current state of the bucket. Wrong tool for a point-in-time audit.
Key Takeaways
- S3 Inventory delivers daily or weekly reports of every object plus rich metadata
- Choose Parquet or ORC for cheap Athena queries; CSV works but scans slowly
- Enable optional fields like
EncryptionStatusandReplicationStatusfor compliance work - Feed the inventory manifest into S3 Batch Operations to act on billions of objects
- First report can take up to 48 hours to appear, so plan for that lead time
Never miss a story from us, subscribe to our newsletter