Data Masking
Overview
Data masking replaces real sensitive data with realistic but fake data, preserving format and structure. Essential for:
- Dev/staging environments: Use masked production data without exposing real PII
- Log sanitization: Prevent PII from appearing in log aggregation systems
- Analytics: Analyze behavioral patterns without raw PII
- Testing: Realistic test data that won't trigger real consequences
Instructions
- Inventory where PII lives (tables, log fields, API payloads, exports) and classify it as direct identifiers or quasi-identifiers.
- Pick the technique per use case from the table below; prefer irreversible masking for dev/staging and tokenization only where the real value must be recovered.
- Mask at the boundary (export job, log formatter, API serializer), never in the consuming app.
- Verify: scan the masked output with the patterns or Presidio and fail the pipeline on any hit.
Masking Techniques
| Technique | How | When to Use |
|---|
| Static masking | Replace data at rest permanently | Dev DB copy |
| Dynamic masking | Mask on-read, original preserved | Role-based views |
| Tokenization | Replace with token that maps to real value | Payment cards |
| Format-preserving | Keep format, change values (e.g., real-looking SSN) | Testing |
| Redaction | Replace with placeholder ([REDACTED]) | Logs |
| Generalization | Replace specific value with range (age 34 → 30-40) | Analytics |
PII Pattern Library
import re
PII_PATTERNS = {
"email": r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b',
"phone_us": r'\b(?:\+1[-.]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}\b',
"ssn": r'\b(?!000|666|9\d{2})\d{3}-(?!00)\d{2}-(?!0000)\d{4}\b',
"credit_card": r'\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13}|6(?:011|5[0-9]{2})[0-9]{12})\b',
"ip_address": r'\b(?:(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(?:25[0-5]|2[0-4]\d|[01]?\d\d?)\b',
"date_of_birth": r'\b(?:0[1-9]|1[0-2])[\/\-](?:0[1-9]|[12]\d|3[01])[\/\-](?:19|20)\d{2}\b',
"passport": r'\b[A-Z]{1,2}[0-9]{6,9}\b',
"zip_code": r'\b\d{5}(?:-\d{4})?\b',
}
These regexes are a fast first pass and over-match: passport and zip_code hit ordinary numbers and product codes, phone_us hits any 10-digit run. Use them for logs; for free text or unknown columns use Presidio below. Validate card candidates with a Luhn check before redacting.
Email and Credit Card Maskers
import re
from faker import Faker
fake = Faker()
Faker.seed(4471) # same seed -> same fake values on every run (stable fixtures)
def mask_email(email: str) -> str:
"""Mask email preserving domain structure."""
local, domain = email.split('@')
masked_local = local[0] + '*' * (len(local) - 2) + local[-1] if len(local) > 2 else '***'
return f"{masked_local}@{domain}"
def mask_credit_card(card_number: str) -> str:
"""Mask credit card — show only last 4 digits."""
cleaned = re.sub(r'[\s-]', '', card_number)
return '*' * (len(cleaned) - 4) + cleaned[-4:]
def mask_ssn(ssn: str) -> str:
"""Mask SSN — show only last 4."""
cleaned = ssn.replace('-', '').replace(' ', '')
return f"***-**-{cleaned[-4:]}"
def mask_phone(phone: str) -> str:
"""Mask phone — show only last 4 digits."""
digits = re.sub(r'\D', '', phone)
return f"***-***-{digits[-4:]}"
Log Sanitizer Middleware
// Node.js log scrubbing (winston)
const PII_PATTERNS = {
email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g,
creditCard: /\b(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|3[47][0-9]{13})\b/g,
ssn: /\b(?!000|666|9\d{2})\d{3}-(?!00)\d{2}-(?!0000)\d{4}\b/g,
phone: /\b(?:\+1[-.]?)?\(?[0-9]{3}\)?[-.\s]?[0-9]{3}[-.\s]?[0-9]{4}\b/g,
password: /"password"\s*:\s*"[^"]*"/g,
token: /"(?:token|api_key|secret|authorization)"\s*:\s*"[^"]*"/gi,
};
function sanitizeLog(data) {
let sanitized = typeof data === 'string' ? data : JSON.stringify(data);
sanitized = sanitized.replace(PII_PATTERNS.email, '[EMAIL]');
sanitized = sanitized.replace(PII_PATTERNS.creditCard, '[CREDIT_CARD]');
sanitized = sanitized.replace(PII_PATTERNS.ssn, '[SSN]');
sanitized = sanitized.replace(PII_PATTERNS.phone, '[PHONE]');
sanitized = sanitized.replace(PII_PATTERNS.password, '"password":"[REDACTED]"');
sanitized = sanitized.replace(PII_PATTERNS.token, (match) => {
const key = match.split(':')[0];
return `${key}:"[REDACTED]"`;
});
return sanitized;
}
// Wrap Winston logger to auto-sanitize
const winston = require('winston');
const logger = winston.createLogger({
transports: [new winston.transports.Console()],
format: winston.format.combine(
winston.format.printf(({ level, message, ...meta }) => {
return JSON.stringify({
level,
message: sanitizeLog(message),
...JSON.parse(sanitizeLog(JSON.stringify(meta)))
});
})
)
});
Database Masking (PostgreSQL)
-- Create masked view for dev access
CREATE OR REPLACE VIEW users_masked AS
SELECT
id,
-- Mask name: keep first letter + ***
LEFT(first_name, 1) || '***' AS first_name,
LEFT(last_name, 1) || '***' AS last_name,
-- Mask email: preserve domain
REGEXP_REPLACE(email, '^([^@])([^@]*)(@.+)$', '\1***\3') AS email,
-- Mask phone: show only last 4
'***-***-' || RIGHT(phone, 4) AS phone,
-- Mask SSN: show only last 4
'***-**-' || RIGHT(ssn, 4) AS ssn,
-- Keep non-sensitive fields as-is
created_at,
status,
country
FROM users;
-- Grant dev team access to masked view only (not base table)
GRANT SELECT ON users_masked TO dev_team;
REVOKE SELECT ON users FROM dev_team;
-- Dynamic, role-aware masking: use a SECURITY DEFINER function or the
-- PostgreSQL Anonymizer extension (anon.start_dynamic_masking) instead of ad-hoc views.
Microsoft Presidio — Auto-Detection
# pip install presidio-analyzer presidio-anonymizer && python -m spacy download en_core_web_lg
# (Python 3.10+; the default NLP engine needs the spaCy model)
import os
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
def mask_text_presidio(text: str, masking_style: str = "replace") -> str:
"""Auto-detect and mask PII using Presidio NLP."""
results = analyzer.analyze(text=text, language="en")
if masking_style == "replace":
# Replace with type label: [EMAIL_ADDRESS]
operators = {
"DEFAULT": OperatorConfig("replace", {"new_value": "[REDACTED]"}),
"EMAIL_ADDRESS": OperatorConfig("replace", {"new_value": "[EMAIL]"}),
"PHONE_NUMBER": OperatorConfig("replace", {"new_value": "[PHONE]"}),
"PERSON": OperatorConfig("replace", {"new_value": "[NAME]"}),
"US_SSN": OperatorConfig("replace", {"new_value": "[SSN]"}),
}
elif masking_style == "hash":
# Since 2.2.361 hashes use a random salt; pass a fixed salt (from a secret)
# if the same input must map to the same output across rows
operators = {"DEFAULT": OperatorConfig("hash", {"hash_type": "sha256", "salt": os.environ["MASKING_SALT"]})}
anonymized = anonymizer.anonymize(
text=text,
analyzer_results=results,
operators=operators
)
return anonymized.text
# Example
text = "Contact Dana Whitfield at dana.whitfield@northwind-mail.org or 555-123-4567"
print(mask_text_presidio(text))
# → "Contact [NAME] at [EMAIL] or [PHONE]" (exact labels depend on detected entities)
Production DB → Dev DB Pipeline
#!/bin/bash
# mask-db-for-dev.sh — Safe production → dev data pipeline
set -e
PROD_DB="${PROD_DATABASE_URL:?set PROD_DATABASE_URL}" # read-only replica credentials
DEV_DB="${DEV_DATABASE_URL:?set DEV_DATABASE_URL}"
echo "Dumping production schema..."
pg_dump --schema-only "$PROD_DB" > schema.sql
echo "Applying schema to dev..."
psql "$DEV_DB" < schema.sql
echo "Copying and masking data..."
psql "$PROD_DB" -c "\COPY (
SELECT
id,
LEFT(first_name, 1) || 'XXXX' AS first_name,
'User' AS last_name,
'user_' || id || '@masked.invalid' AS email,
'555-000-' || LPAD((ROW_NUMBER() OVER())::TEXT, 4, '0') AS phone,
created_at,
status
FROM users
) TO STDOUT WITH CSV" | psql "$DEV_DB" -c "\COPY users FROM STDIN WITH CSV"
echo "Done. Dev database ready with masked data."
Statistical Anonymization (GDPR)
Anonymization vs Pseudonymization (GDPR Article 4):
- Anonymization: Irreversible -- data can never be linked to an individual. Falls outside GDPR scope.
- Pseudonymization: Reversible -- data can be re-linked with additional info. Still personal data under GDPR.
Key techniques for true anonymization:
- k-Anonymity: Each record is indistinguishable from at least k-1 others on quasi-identifiers (age, ZIP, gender). Generalize values into ranges and suppress groups smaller than k.
- l-Diversity: Each equivalence class has at least l distinct sensitive attribute values, preventing attribute disclosure.
- Differential Privacy: Mathematical privacy guarantee controlled by epsilon -- add calibrated noise to query results. Use
diffprivlib (Python) or Google DP libraries.
k-anonymity alone is often insufficient for GDPR -- combine with l-diversity and/or differential privacy.
Examples
Example 1: Safe staging copy
User request: "Refresh staging from production but nobody on the dev team may see real emails or phone numbers."
Run the pipeline script above with PROD_DATABASE_URL and DEV_DATABASE_URL set, then verify:
psql "$DEV_DATABASE_URL" -tAc "SELECT count(*) FROM users WHERE email NOT LIKE '%@masked.invalid'"
Expected output: 0. Any other number means unmasked rows leaked and the refresh must be discarded.
Example 2: Scrub PII from application logs
User request: "Our request logs contain customer emails and card numbers; clean them before they reach Datadog."
Wrap the logger with sanitizeLog from the middleware section. A line such as {"msg":"refund for dana.whitfield@northwind-mail.org card 4111111111111111"} is emitted as {"msg":"refund for [EMAIL] card [CREDIT_CARD]"}. Add a unit test that feeds sample PII through the logger and asserts none survives.
Guidelines
- Mask irreversibly by default; a reversible mapping or key stored beside the data is pseudonymization, still personal data under GDPR.
- Regex finds formats, not names or addresses; use Presidio (and review its misses) for free text.
- Salts and encryption keys come from a secret manager or environment variable, never from the repository.
- Keep referential integrity: the same real value must map to the same fake value in every table (seeded Faker or salted hash).
- Test on a copy first, and scan the output before anyone gets access.
- Do not treat k-anonymity alone as GDPR-grade anonymization.