The Problem
You are auditing an S3-based Lake Formation data lake to make sure it does not contain sensitive customer or employee data. You need to detect PII, passport numbers, credit card numbers, and other financial identifiers.
The Solution
Configure Amazon Macie to run a data discovery job using its managed data identifiers. Macie uses machine learning and pattern matching to find sensitive data types automatically.
How It Works
What Macie Detects
Macie ships with dozens of managed identifiers, including:
| Category | Examples |
|---|---|
| Personal Info | Names, addresses, phone numbers, driver’s license, passport |
| Financial | Credit card numbers, IBAN, SWIFT codes |
| Credentials | AWS access keys, private keys, OAuth tokens |
| Health | HIPAA identifiers, patient records |
| Custom | Regex patterns you define |
Step 1: Enable Macie
1
aws macie2 enable-macie
Step 2: Create a Discovery Job
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
aws macie2 create-classification-job \
--job-type ONE_TIME \
--name "audit-data-lake" \
--s3-job-definition '{
"bucketDefinitions": [{
"accountId": "123456789012",
"buckets": ["data-lake-bucket"]
}],
"scoping": {
"includes": {
"and": [{
"simpleScopeTerm": {
"comparator": "STARTS_WITH",
"key": "OBJECT_KEY",
"values": ["raw/", "curated/"]
}
}]
}
}
}' \
--managed-data-identifier-selector ALL
Macie scans the specified buckets, samples object content, and generates findings for any matches.
Step 3: Review Findings
1
2
3
4
5
6
aws macie2 list-findings \
--finding-criteria '{
"criterion": {
"severity.description": { "eq": ["High"] }
}
}'
Each finding includes:
- Bucket and object key
- Data type detected (e.g.,
CREDIT_CARD_NUMBER) - Count of matches
- Severity
Why Not Just Grep Through S3?
Macie provides:
- Managed identifiers for dozens of data types — no regex writing
- ML classification for structured and semi-structured formats (JSON, CSV, Parquet, Avro)
- Sampling logic that scales to petabyte-sized buckets without full-content scans
- Findings management with severity, remediation guidance, and Security Hub integration
Custom Lambda-based scanning misses edge cases and requires significant maintenance.
Why Not the Alternatives?
AWS Audit Manager with PCI DSS framework — Audit Manager tracks control evidence and compliance posture. It does not scan file contents for sensitive data.
S3 Inventory + Athena — Inventory lists objects and metadata (size, encryption, storage class). It does not read file content. Athena queries data lakes, but you would have to build your own PII detection queries.
S3 Select for reporting — S3 Select runs SQL against a single object. Not designed for cross-bucket discovery.
Key Takeaways
- Amazon Macie is the purpose-built AWS service for sensitive data discovery in S3
- Managed identifiers cover most common PII, financial, and credential patterns
- One-time discovery jobs for audits; scheduled jobs for ongoing compliance
- Integrates with EventBridge and Security Hub for automated alerting workflows
- Does not scan file content unless configured — sampling and cost controls prevent scanning entire petabyte buckets by accident
Never miss a story from us, subscribe to our newsletter