Know your data: what “sensitive” means in practice
- PII: Under GDPR, personally identifiable information (PII) is any information relating to an identified or identifiable person, such as name, ID numbers, location data, online identifiers, and more. Data minimisation is a core principle: only process what’s necessary for the stated purpose.
- PHI: In the U.S., HIPAA protects protected health information (PHI) in healthcare contexts and permits two recognized de-identification methods: Expert Determination and Safe Harbor (removal of 18 identifier types), codified at 45 CFR §164.514.
- In California, the California Privacy Rights Act (CPRA) amended and significantly strengthened the California Consumer Privacy Act (CCPA), expanding consumer privacy rights and establishing the California Privacy Protection Agency as the enforcement authority.
- The U.S. federal Children’s Online Privacy Protection Act (COPPA), which is enforced by the Federal Trade Commission (FTC), applies to children under age 13 and opt-in consent must be provided by a parent or guardian.
- Takeaway: Start with a formal data inventory and classification policy. If you can’t clearly label PII/PHI and special-category data such as that applying to children, you can’t protect it.
Why this matters for LLMs: leakage is real
Modern models can memorize and later regurgitate rare or sensitive strings from training corpora. Research has demonstrated the extraction of training data from production LLMs via carefully crafted prompts, and a growing body of work on membership-inference risks.
The takeaway? Assume anything you train on could be exposed under adversarial querying. Plan privacy protections accordingly.
Privacy-by-design data prep: what to keep vs. strip
Adopt a layered approach before any model sees the data:
- Data minimisation & purpose binding. Keep only what supports the model objective; document the lawful basis/purpose.
- De-identification. For PHI, choose Safe Harbor (remove the 18 identifiers) or Expert Determination; for other sensitive corpora, use robust anonymisation in line with regulator guidance.
- Pseudonymisation & tokenisation. Replace direct identifiers with stable tokens so you can evaluate longitudinal behavior without exposing identities (store the mapping separately, under strict access control).
- Redaction & masking. Apply context-aware PII/PHI detectors to free-text, including names, addresses, emails, phone numbers, medical record numbers, device IDs, and timestamps—then have humans sample-audit the results.
- Synthetic data (careful). Synthetic corpora can reduce risk, but poor generation or linkage can re-identify individuals. Follow regulator guidance and document residual risk.
Pro tip: Build a gold-standard “PII canary” set to validate your pipeline — if detectors miss seeded identifiers, your process isn’t ready for scale.
Handling pipelines securely: controls that withstand audits
Auditors look for recognized frameworks and verifiable controls:
- Security certifications. ISO/IEC 27001 (ISMS) demonstrates an organization-wide, risk-based security program; SOC 2 (Type 2) attests to the operating effectiveness of controls over time.
- Access control & least privilege. SSO/MFA, role-based access, just-in-time elevation, and granular project scoping (annotators only see what they must).
- Data residency & clean rooms. Use secure facilities or virtual enclaves for highly restricted corpora; log all ingress/egress events.
- Encryption everywhere. Encrypt data in transit and at rest; consider format-preserving encryption for fields that must pass validation.
- Comprehensive logging. Immutable audit logs for who accessed which records, when, and for what purpose (tie to DPIAs where applicable).
- Retention & deletion. Define time-boxed retention aligned to purpose limitation; verify and log deletion downstream.
Preparing data for human annotation (and back again)
Human-in-the-loop (HITL) evaluation and labeling are essential for quality, but they add privacy considerations:
- Scoped views. Present only the minimum text/audio/video segments needed for a given task.
- On-screen redaction. Mask sensitive fields in the UI; never reveal full identifiers if they’re not directly required.
- Workflow separation. Split duties: one team redacts; a separate team labels.
- Continuous QA. Sample audits against the original source to ensure redaction quality stays high.
- Policy training. Require privacy/security training and attestations for all contributors; refresh periodically (and on policy changes).
A practical checklist you can adapt
- Establish a data inventory and apply GDPR-style data minimisation and purpose limitation.
- Select a de-identification standard (HIPAA methods for PHI; anonymisation or pseudonymisation elsewhere) and validate with human sampling.
- Enforce least-privilege access, encryption, and immutable logging; align your program to ISO 27001 and SOC 2 Type 2.
- Assume adversarial prompts — review training sets for rare strings and consider privacy-preserving training strategies.
- Document everything: DPIAs, redaction accuracy stats, vendor responsibilities, and retention/deletion SLAs (map to NIST AI RMF “Govern” and “Manage” functions for exec visibility).
Sensitive data can power better models, but only if you minimise, de-identify, and govern it end-to-end. Following recognized standards (ISO 27001, SOC 2), regulator guidance (HIPAA, GDPR), and privacy-by-design practices reduces legal and leakage risk while preserving learning value.
If you need a deeper dive into secure human-in-the-loop workflows for LLMs, our team of expert project managers is happy to share our approach to privacy protection.