Face Recognition System Builder

Build production face recognition systems with InsightFace/ArcFace for enrollment, verification, identification, and clustering with privacy-compliant architecture.

Decision Points

Model Selection Decision Tree

Requirements Analysis:
├── Latency < 50ms per face?
│   ├── YES: Use buffalo_sc (lightweight, 0.7 accuracy)
│   └── NO: Continue to accuracy analysis
├── Accuracy > 95% required?
│   ├── YES: Use AdaFace (premium accuracy, 200ms latency)
│   └── NO: Use buffalo_l (balanced, 100ms latency)
└── Mobile/Edge deployment?
    ├── YES: buffalo_sc or custom quantized model
    └── NO: buffalo_l or AdaFace

Threshold Tuning Heuristic

Use Case	FAR Target	FRR Target	Threshold Range	Validation
Security/Access Control	<0.1%	1-5%	0.55-0.65	Test on impostor pairs
Photo Organization	2-5%	<1%	0.35-0.45	Test on family albums
Investigation/Forensics	<0.01%	10-20%	0.65-0.75	Test on mugshot datasets
Social Media Tagging	5-10%	<2%	0.30-0.40	Test on casual photos

Enrollment Quality Decision

Per photo assessment:
├── Face detected with confidence > 0.7?
│   ├── NO: Reject photo, request new one
│   └── YES: Continue
├── Face size > 80px AND < 50% of image?
│   ├── NO: Reject (too small or too close)
│   └── YES: Continue
├── Pose deviation < 30 degrees (yaw/pitch)?
│   ├── NO: Flag as "supplementary only"
│   └── YES: Mark as "primary reference"
└── Final decision:
    ├── 3+ primary references: Complete enrollment
    ├── 1-2 primary + 2+ supplementary: Proceed with warning
    └── <3 total valid: Require additional photos

Failure Modes

1. Pose/Lighting Degradation

Detection: Similarity scores dropping consistently below historical baseline (>10% decrease) Symptoms: Good enrollments start failing verification, complaints of "system doesn't recognize me" Diagnosis: Environmental conditions changed (new lighting, camera angle, seasonal changes) Fix:

• Re-tune threshold based on recent data
• Add current condition photos to existing enrollments
• Implement adaptive threshold based on lighting detection

2. Enrollment Contamination

Detection: New enrollment has unusually high similarity (>0.8) to existing different identity Symptoms: False positive identifications, "system thinks I'm someone else" Diagnosis: Multiple people enrolled under same identity, or identity mixup during enrollment Fix:

• Audit enrollment: manually review all photos for consistency
• Split contaminated enrollment into separate identities
• Re-enroll with verified identity documentation

3. Threshold Drift Over Time

Detection: FAR/FRR metrics shifting >2% from baseline over 30-day periods Symptoms: Gradual increase in false accepts OR false rejects system-wide Diagnosis: Population demographics changing, or model degradation on new data patterns Fix:

• Weekly threshold validation on held-out test set
• Retrain similarity calibration monthly
• Implement per-demographic threshold adjustments

4. Quality Gate Bypass

Detection: Average enrollment quality score dropping while new enrollments increase Symptoms: Poor recognition accuracy for recently enrolled users Diagnosis: Operators bypassing quality checks to meet enrollment quotas Fix:

• Implement hard technical blocks on low-quality enrollments
• Separate enrollment approval from quality assessment
• Add post-enrollment validation testing

5. Privacy Compliance Drift

Detection: Embedding storage growing faster than enrollment count, or missing consent records Symptoms: Legal compliance audit failures, data retention violations Diagnosis: Raw images stored without consent, or embeddings not properly deleted Fix:

• Automated compliance scanning: consent tracking, embedding lifecycle
• Purge process for embeddings without valid consent
• Convert raw image storage to embedding-only with consent re-confirmation

Worked Examples

Example 1: Masked Face at 0.3m Distance

Scenario: COVID-era access control, masks required, camera mounted at door (0.3m typical distance)

Decision Process:

1. Model Selection: buffalo_l chosen (balance accuracy vs speed for real-time)
2. Threshold Analysis: Security use case → target FAR <0.1%, initial threshold 0.55
3. Masked Face Testing:
   • Standard threshold: 40% FRR (too many false rejects)
   • Masked-specific threshold: 0.45 → 15% FRR, 0.2% FAR (acceptable)
4. Implementation: Mask detection → route to specialized threshold

Novice would miss: Using same threshold for masked/unmasked faces, not testing on representative masked dataset Expert catches: Separate calibration needed, eye region weighting more important, enrollment should include masked photos

Final Configuration:

def adaptive_verify(embedding1, embedding2, mask_detected=False):
    threshold = 0.45 if mask_detected else 0.55
    similarity = np.dot(embedding1, embedding2)
    return similarity > threshold

Example 2: Enrollment with 1 Poor-Quality Photo

Scenario: Employee self-enrollment via mobile app, submits 3 photos: 1 excellent frontal, 1 slight profile, 1 blurry/dark

Decision Process:

1. Quality Assessment:
   • Photo 1: 0.95 confidence, 150px face, frontal pose → PRIMARY
   • Photo 2: 0.85 confidence, 120px face, 20° yaw → SUPPLEMENTARY
   • Photo 3: 0.65 confidence, 90px face, poor lighting → REJECT
2. Enrollment Decision: Only 1 primary + 1 supplementary = insufficient for robust enrollment
3. User Feedback: "Please submit 1 more clear, well-lit frontal photo"
4. Fallback: Store partial enrollment as "pending", allow manual review override for urgent cases

Trade-off Decision: Reject enrollment (high quality standard) vs Accept with warning (user convenience) Chosen: Reject with guidance - better user experience long-term than poor recognition accuracy

Recovery Process:

• Clear photo guidelines with examples
• Real-time quality feedback during capture
• Manual review queue for edge cases
• Temporary PIN-based access while enrollment pending

Quality Gates

• Face detection confidence >0.7 AND face size >80px for all enrolled photos
• Minimum 3 photos per identity with pose variation <30° for primary references
• Similarity threshold validated on representative test set with documented FAR/FRR
• Demographic bias tested: <5% accuracy difference across age/gender/ethnicity groups
• Enrollment rejection rate <20% (too high indicates unrealistic quality standards)
• Average enrollment-to-first-successful-recognition time <24 hours
• Privacy audit: all stored embeddings have valid consent records
• Deletion capability tested: embedding removal within 24 hours of request
• Performance benchmark: <100ms average recognition latency at production scale
• Failure graceful degradation: uncertain matches flagged, not auto-decided

NOT-FOR Boundaries

This skill should NOT be used for:

• General object detection or image classification → Use computer-vision-pipeline
• Facial emotion, expression, or attribute analysis → Use ai-engineer
• Face generation, deepfakes, or synthetic face creation → Use specialized generative AI skills
• Real-time video streaming face recognition → Use video-frame-extraction-analysis first
• Large-scale batch processing (>10K faces) → Use data-pipeline-engineer for orchestration

Delegate when you encounter:

• Performance optimization beyond model selection → system-performance-optimizer
• Complex database design for vector storage → database-architect
• Legal/compliance requirements analysis → Domain expert consultation
• Custom model training or fine-tuning → machine-learning-engineer