
ai-privacy-governance-skills
15 AI privacy skills: AI DPIA, training data compliance, bias assessment, transparency, LLM output privacy, federated learning
- version
- 1.0.0
- GitHub stars
- 340
Install
/plugin install ai-privacy-governance-skills@privacy-data-protection-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add mukul975/Privacy-Data-Protection-Skills.
Contents
Discovered from sourceSkills 15
EU AI Act High-Risk AI System Documentation
Overview
The EU AI Act (Regulation 2024/1689, entered into force 1 August 2024, with high-risk obligations applicable from 2 August 2026) establishes a risk-based regulatory framework for artificial intelligence systems. High-risk AI systems — those listed in Annex III or used as safety components of products covered by Union harmonisation legislation in Annex I — must meet extensive documentation, transparency, and governance requirements before being placed on the EU market. Cerebrum AI Labs must prepare comprehensive technical documentation, implement a risk management system, ensure data governance, and undergo conformity assessment for each high-risk AI system.
High-Risk Classification
Annex III Categories Relevant to Cerebrum AI Labs
| Category | Annex III Reference | Cerebrum AI Labs System | Classification |
|---|---|---|---|
| Employment and workers management | Annex III, para. 4(a) | CV Screening AI — automated filtering of job applications | High-risk |
| Access to essential services | Annex III, para. 5(b) | Credit Scoring AI — creditworthiness assessment for financial products | High-risk |
| Law enforcement | Annex III, para. 6(a) | Not applicable | N/A |
| Biometric identification | Annex III, para. 1(a) | Facial Verification AI — identity verification at onboarding | High-risk |
| Education and vocational training | Annex III, para. 3(a) | Not applicable | N/A |
Classification Decision Tree
Is the AI system listed in Annex III?
├── Yes → High-risk (unless exception applies under Art. 6(3))
│ └── Does the system make decisions materially affecting natural persons?
│ ├── Yes → High-risk confirmed
│ └── No → May qualify for Art. 6(3) exception (narrow, profiling, preparatory)
│
├── No → Is it a safety component of a product under Annex I legislation?
│ ├── Yes → High-risk (subject to third-party conformity assessment)
│ └── No → Not high-risk under AI Act
│
└── Is it a general-purpose AI model with systemic risk? (Art. 51)
├── Yes → GPAI systemic risk obligations
└── No → GPAI transparency obligations only
Technical Documentation Requirements (Art. 11)
Annex IV — Required Documentation Content
Section 1: General Description
| Document Element | Content for Cerebrum AI Labs CV Screening AI |
|---|---|
| Intended purpose | Automated screening and ranking of job applications based on qualification match |
| Provider name and contact | Cerebrum AI Labs, 42 Innovation Drive, Dublin, Ireland |
| AI system version | v2.4.1 (deployed March 2026) |
| Hardware/software requirements | Cloud-hosted on EU infrastructure (AWS eu-west-1), Python 3.11, PyTorch 2.2 |
| Product integration | Integrated into Cerebrum TalentFlow ATS platform |
Section 2: Detailed Description of System Elements
| Element | Documentation Required |
|---|---|
| Development methodology | Model architecture, training approach, design choices and rationale |
| Computational resources | Training compute (GPU hours), energy consumption |
| Training data | Data sources, collection methods, size, labeling methodology, preprocessing |
| Validation and testing | Test datasets, metrics, results, known limitations |
| Input data specifications | Expected input format, quality requirements |
| Output description | Output format, confidence scores, decision thresholds |
Section 3: Monitoring, Functioning, and Control
| Element | Documentation Required |
|---|---|
| Human oversight measures | Art. 14 requirements: override capability, decision review process |
| Technical measures for accuracy | Accuracy metrics, drift detection, retraining triggers |
| Cybersecurity measures | Data encryption, access controls, adversarial robustness testing |
| Performance in edge cases | Known failure modes, boundary conditions, degradation behavior |
Section 4: Risk Management
| Element | Documentation Required |
|---|---|
| Risk management system | Art. 9 risk management process documentation |
| Known and foreseeable risks | Risk register with severity and likelihood |
| Mitigation measures | Controls for each identified risk |
| Residual risk assessment | Acceptable residual risk justification |
Risk Management System (Art. 9)
Continuous Risk Management Process for Cerebrum AI Labs
| Phase | Activity | Frequency |
|---|---|---|
| Identification | Identify risks to health, safety, and fundamental rights | Initial + quarterly |
| Analysis | Estimate risk severity and likelihood | Initial + quarterly |
| Evaluation | Compare risks against acceptance criteria | Initial + quarterly |
| Mitigation | Implement risk reduction measures | Ongoing |
| Monitoring | Track risk indicators in production | Continuous |
| Review | Review and update risk assessment | Quarterly |
Risk Register — CV Screening AI
| Risk ID | Risk Description | Severity | Likelihood | Mitigation | Residual Risk |
|---|---|---|---|---|---|
| R-001 | Gender bias in screening recommendations | High | Medium | Bias testing on protected attributes, debiasing training data | Low |
| R-002 | Discrimination against non-native language speakers | High | Medium | Multilingual evaluation, language-agnostic features | Medium |
| R-003 | Over-reliance on AI recommendations by recruiters | Medium | High | Mandatory human review, confidence thresholds | Low |
| R-004 | Inaccurate qualification matching for novel job roles | Medium | Medium | Fallback to keyword matching, human review flag | Low |
| R-005 | Privacy breach via training data memorization | High | Low | Differential privacy in training, memorization audit | Low |
Data Governance (Art. 10)
Training Data Requirements
| Requirement | Implementation at Cerebrum AI Labs |
|---|---|
| Relevance and representativeness | Training data sourced from 50,000 job applications across 12 EU countries, balanced by gender, age, nationality |
| Bias examination | Statistical parity analysis on protected attributes (gender, age, ethnicity, disability) before and after training |
| Gap identification | Identified underrepresentation of applicants with disabilities; augmented with synthetic examples |
| Data quality | Automated data quality checks: completeness >95%, label accuracy >98% (human-verified sample) |
| Personal data processing | DPIA completed (DPIA-AI-2026-001); lawful basis: Art. 6(1)(f) legitimate interest; special categories removed |
Conformity Assessment (Art. 43)
Assessment Procedure for Cerebrum AI Labs
| System | Assessment Type | Basis |
|---|---|---|
| CV Screening AI | Internal conformity assessment (Art. 43(2)) + quality management system | Annex III, para. 4 — not biometric, not critical infrastructure |
| Credit Scoring AI | Internal conformity assessment (Art. 43(2)) | Annex III, para. 5 |
| Facial Verification AI | Third-party conformity assessment (Art. 43(1)) via notified body | Annex III, para. 1 — biometric identification |
Internal Conformity Assessment Steps
- Verify quality management system is established (Art. 17)
- Prepare technical documentation per Annex IV
- Conduct risk management assessment per Art. 9
- Verify data governance per Art. 10
- Verify transparency and information provision per Art. 13
- Verify human oversight per Art. 14
- Verify accuracy, robustness, and cybersecurity per Art. 15
- Draw up EU Declaration of Conformity (Art. 47)
- Affix CE marking (Art. 48)
- Register in EU database (Art. 49)
Post-Market Monitoring (Art. 72)
| Activity | Frequency | Owner |
|---|---|---|
| Performance metric monitoring | Continuous | ML Engineering |
| Bias drift detection | Weekly | Responsible AI team |
| Incident reporting | As needed (within 15 days for serious incidents per Art. 73) | DPO + Legal |
| User feedback collection | Continuous | Product team |
| Risk register update | Quarterly | Risk Management |
| Technical documentation update | On material change | ML Engineering + Legal |
Key Legal References
- EU AI Act (Regulation 2024/1689) — Full text, entered into force 1 August 2024
- AI Act Art. 6 + Annex III — High-risk classification criteria
- AI Act Art. 9 — Risk management system requirements
- AI Act Art. 10 — Data and data governance requirements
- AI Act Art. 11 + Annex IV — Technical documentation requirements
- AI Act Art. 13 — Transparency and provision of information to deployers
- AI Act Art. 14 — Human oversight requirements
- AI Act Art. 43 — Conformity assessment procedures
- AI Act Art. 72 — Post-market monitoring obligations
- AI Act Art. 73 — Reporting of serious incidents
- GDPR Art. 22 — Automated individual decision-making (complementary to AI Act)
- EDPB-EDPS Joint Opinion 5/2021 on the AI Act Proposal — Data protection perspective on AI regulation
AI Automated Decision-Making and Human Oversight
Overview
GDPR Article 22 grants data subjects the right not to be subject to decisions based solely on automated processing, including profiling, which produce legal or similarly significant effects. The EU AI Act Art. 14 supplements this with specific human oversight design requirements for high-risk AI systems. Together, these provisions require organisations to identify when AI systems make consequential decisions, ensure meaningful human intervention where required, provide explainable decision logic, and offer effective contestation mechanisms. This skill provides the complete framework for Art. 22 compliance and AI Act human oversight implementation.
Art. 22 Scope and Applicability
Three Cumulative Conditions
Art. 22(1) is triggered only when all three conditions are met:
| Condition | Requirement | AI Application |
|---|---|---|
| 1. Decision | A decision is made (not merely a recommendation or input) | The AI output directly determines an outcome — no genuine human decision-making step between AI output and action |
| 2. Solely automated | Based solely on automated processing including profiling | No meaningful human intervention in the decision chain; rubber-stamping does not constitute human intervention |
| 3. Legal/significant effects | Produces legal effects or similarly significantly affects the data subject | Affects legal rights, contractual status, access to services, financial outcomes, or other significant life impacts |
"Solely Automated" — EDPB Interpretation
The EDPB Guidelines 06/2020 on automated decision-making clarify:
-
Solely automated means no meaningful human involvement in the decision process
-
A human who merely confirms an AI recommendation without genuine assessment is not providing meaningful intervention
-
Meaningful human intervention requires:
- The reviewer has authority and competence to change the decision
- The reviewer has access to all relevant information
- Sufficient time is allocated for genuine consideration
- The reviewer routinely exercises independent judgment (not just confirming AI output)
- Override capability is actually used in practice
-
Not solely automated when:
- A qualified human genuinely reviews the AI output as one input among several
- The human applies independent judgment and makes the final decision
- The human has the capability and actually exercises discretion to deviate from AI recommendations
"Legal or Similarly Significant Effects"
| Category | Examples | Significance |
|---|---|---|
| Legal effects | Contract formation/termination, legal obligation imposition, legal status determination | Directly affects legal rights |
| Access to services | Denial of credit, insurance, housing, education, employment | Significantly affects life circumstances |
| Financial impact | Pricing discrimination, benefit calculation, payment terms | Material financial consequences |
| Health and safety | Medical diagnosis prioritisation, emergency response triage | Potential physical harm |
| Freedom and autonomy | Surveillance scoring, movement restriction, content blocking | Affects fundamental freedoms |
Effects that are not similarly significant (per EDPB):
- Personalised advertising (unless it reinforces prejudices about vulnerable groups)
- Generic recommendation systems (unless they determine access to information)
- Spam filtering (unless it systematically blocks important communications)
Art. 22 Exceptions and Safeguards
Art. 22(2) — Permitted Automated Decisions
| Exception | Condition | Required Safeguards |
|---|---|---|
| Art. 22(2)(a) — Contract necessity | Decision is necessary for entering into or performance of a contract | Art. 22(3) safeguards required |
| Art. 22(2)(b) — Law authorisation | Authorised by Union or Member State law with suitable measures | Law must provide suitable safeguards |
| Art. 22(2)(c) — Explicit consent | Based on explicit consent | Art. 22(3) safeguards required |
Art. 22(3) — Mandatory Safeguards
When an Art. 22(2) exception is relied upon, the controller must implement at least:
- Right to obtain human intervention: A qualified person reviews the automated decision
- Right to express point of view: Data subject can present additional information or context
- Right to contest: Formal mechanism to challenge the automated decision with review by a different decision-maker
Art. 22(4) — Special Category Data Restriction
Automated decisions based on Art. 9 special category data are only permitted under:
- Art. 9(2)(a) — Explicit consent, OR
- Art. 9(2)(g) — Substantial public interest based on Union or Member State law
In both cases, suitable measures to safeguard data subject rights must be in place.
Human Oversight Design — AI Act Art. 14
Art. 14 Requirements for High-Risk AI
High-risk AI systems must be designed and developed so that they can be effectively overseen by natural persons during use:
| Requirement | Implementation |
|---|---|
| Understand capabilities and limitations | Documentation, training, model cards |
| Monitor operation | Real-time monitoring dashboards, alert systems |
| Detect anomalies and dysfunction | Drift detection, performance monitoring |
| Interpret outputs correctly | Confidence indicators, explanation tools |
| Override or reverse decisions | Override mechanism with authority chain |
| Intervene or stop the system | Emergency stop capability |
| Be aware of automation bias | Training on automation bias, countermeasures |
Human Oversight Levels
| Level | Description | Art. 22 Compliance | Appropriate When |
|---|---|---|---|
| Human-in-the-loop (HITL) | Human reviews every AI recommendation before decision | Fully compliant if review is meaningful | High-stakes individual decisions (hiring, credit, medical) |
| Human-on-the-loop (HOTL) | Human monitors AI decisions and can intervene | Compliant if intervention capability is genuine and exercised | Medium-risk decisions with effective monitoring |
| Human-in-command (HIC) | Human sets parameters and reviews outcomes periodically | May not satisfy Art. 22 — decision is solely automated | Low-risk bulk decisions with periodic audit |
| Fully autonomous | No human oversight of individual decisions | Art. 22 applies fully — exception needed | Only where Art. 22(2) exception applies with Art. 22(3) safeguards |
Meaningful Human Intervention Criteria
A human review qualifies as "meaningful intervention" when all criteria are met:
| Criterion | Test | Red Flag |
|---|---|---|
| Authority | Reviewer has formal authority to override AI | Reviewer can only escalate, not decide |
| Competence | Reviewer has domain expertise to evaluate the decision | Reviewer is a junior staff member without training |
| Information | Reviewer has access to all inputs, the AI output, and explanation | Reviewer sees only AI score with no context |
| Time | Sufficient time allocated for genuine consideration | Reviewer processes 200+ decisions per hour |
| Independence | Reviewer exercises genuine judgment | Override rate is < 1% suggesting rubber-stamping |
| Accountability | Reviewer is accountable for the decision | Accountability rests with the AI system owner, not reviewer |
Contestation and Appeal Mechanism
Design Requirements
| Element | Requirement |
|---|---|
| Accessibility | Contestation mechanism is easy to find, access, and use |
| Timeliness | Defined response timeframe (e.g., 30 days) |
| Qualified reviewer | Different from the original decision context; has authority to overturn |
| Information provision | Data subject receives explanation of decision factors and how to contest |
| Evidence consideration | Data subject can submit additional evidence and context |
| Written outcome | Decision on contestation is documented and communicated |
| Further appeal | If contestation is denied, path to DPA complaint or judicial remedy is indicated |
Contestation Workflow
- Data subject notified of automated decision with explanation
- Data subject submits contestation with reasons and any supporting evidence
- Contestation assigned to qualified human reviewer (different from original oversight)
- Reviewer assesses: AI inputs, AI output, explanation, data subject's arguments
- Reviewer makes independent decision: uphold, modify, or overturn
- Outcome communicated to data subject with reasons
- If upheld: data subject informed of further options (DPA complaint, judicial remedy)
- Record maintained in rights exercise register
AI-Specific Decision Categories
Credit and Financial Decisions
- AI application: Credit scoring, loan approval, insurance pricing, fraud detection
- Art. 22 trigger: Yes — determines access to financial services
- Required: Meaningful human review before denial; explanation of key scoring factors; contestation with independent review
- Enforcement: AEPD fined CaixaBank EUR 6M for automated credit decisions without adequate safeguards
Employment Decisions
- AI application: CV screening, candidate ranking, performance scoring, termination prediction
- Art. 22 trigger: Yes — significantly affects employment and livelihood
- Required: Human hiring manager makes final decision with genuine authority to deviate from AI ranking; applicants informed of AI use; rejected candidates can request explanation
- Enforcement: Italian DPA fined Deliveroo EUR 2.5M for algorithmic worker management without Art. 22 safeguards
Healthcare Decisions
- AI application: Diagnosis assistance, treatment recommendations, triage, risk scoring
- Art. 22 trigger: Yes if AI determines care pathway — usually mitigated by physician oversight
- Required: Physician makes final clinical decision; AI functions as decision support; patient informed of AI role
- Special category: Health data — Art. 22(4) applies
Public Administration
- AI application: Benefit eligibility, fraud detection, risk assessment, resource allocation
- Art. 22 trigger: Yes — determines access to public services and benefits
- Required: Art. 22(2)(b) legal basis required; suitable measures mandated by law; transparency about algorithmic criteria
- Enforcement: Dutch court struck down SyRI fraud detection system for lack of transparency and proportionality
Profiling Assessment
GDPR Definition of Profiling (Art. 4(4))
Any form of automated processing to evaluate personal aspects relating to a natural person, in particular to analyse or predict:
- Work performance
- Economic situation
- Health
- Personal preferences
- Interests
- Reliability
- Behaviour
- Location
- Movements
AI Profiling Risk Assessment
| Profiling Type | Risk Level | Art. 22 Trigger | Mitigation |
|---|---|---|---|
| Behavioural prediction (purchasing, browsing) | Medium | Only if decision with legal/significant effect | Opt-out, transparency |
| Credit scoring / financial risk | High | Yes — access to financial services | Human review, explanation, contestation |
| Health risk prediction | Very High | Yes — Art. 22(4) applies | Explicit consent, physician oversight |
| Criminal risk assessment | Very High | Yes — liberty and legal effects | Legal basis required, judicial oversight |
| Employment performance scoring | High | Yes — employment effects | HR human review, employee notification |
| Social scoring | Prohibited | N/A — AI Act Art. 5 prohibition | Do not implement |
Enforcement Precedents
- AEPD v. CaixaBank (PS/00421/2020, 2021): EUR 6M fine for automated credit decision-making without adequate Art. 22 safeguards, explanation, or contestation mechanism.
- Italian DPA v. Deliveroo (2021): EUR 2.5M fine for algorithmic management of delivery riders — Art. 22 applied to automated work allocation and performance scoring.
- Dutch Court v. SyRI (2020): Algorithmic fraud detection system struck down — automated profiling of citizens without proportionality, transparency, or adequate safeguards.
- Hungarian DPA v. Bank (2019): Fine for automated credit denial without providing meaningful information about decision logic or contestation mechanism.
- French Conseil d'Etat v. Parcoursup (2019): Court upheld use of algorithm for university admissions only because meaningful human review was genuinely conducted for each application.
- Austrian DPA v. CRIF (2023): Credit scoring company — violation of Art. 15(1)(h) for failing to provide meaningful information about automated scoring logic.
Integration Points
- ai-transparency-reqs: Explanation requirements feed into transparency framework
- ai-dpia: Human oversight assessment is DPIA Phase 5 component
- ai-data-subject-rights: Right to explanation and contestation are rights exercise procedures
- ai-deployment-checklist: Art. 22 compliance is pre-deployment validation item
- ai-bias-special-category: Bias in automated decisions creates Art. 22 harm
AI Bias Assessment for Special Category Data
Overview
AI systems can amplify, perpetuate, or introduce bias against protected groups defined by GDPR Art. 9 special categories (race, ethnicity, political opinion, religion, trade union membership, genetic data, biometric data, health, sexual orientation) and by EU equality law (gender, age, disability). The AI Act Art. 10 requires data governance practices for training data that address bias, while Art. 5 prohibits AI-based social scoring. This skill provides the methodology for detecting, measuring, and mitigating bias in AI systems that process or infer special category data, with documentation requirements meeting both GDPR and AI Act obligations.
Art. 9 Special Categories and AI Bias
Direct Processing of Special Category Data
When AI systems directly process Art. 9 data:
| Category | AI Bias Risk | Example |
|---|---|---|
| Racial or ethnic origin | Discrimination in hiring, credit, policing | CV screening penalising names associated with ethnic minorities |
| Political opinions | Political profiling, content suppression | News recommendation amplifying or suppressing political viewpoints |
| Religious beliefs | Service denial, discriminatory targeting | Insurance pricing varying by religious affiliation |
| Trade union membership | Employment discrimination | Performance scoring penalising union activity |
| Genetic data | Genetic discrimination in insurance/employment | Health insurance pricing based on genetic predisposition |
| Biometric data | Differential accuracy across demographics | Facial recognition with higher error rates for darker skin tones |
| Health data | Health-based discrimination | Hiring algorithms penalising disability or mental health history |
| Sexual orientation | Discrimination, outing | Content recommendation inadvertently revealing sexual orientation |
Proxy Inference of Special Categories
AI models frequently infer Art. 9 data from non-sensitive features:
| Proxy Feature | Inferred Category | Mechanism |
|---|---|---|
| Postcode/zip code | Race/ethnicity, income | Residential segregation patterns |
| First/last name | Race/ethnicity, religion | Name-ethnicity correlations |
| Browsing history | Political opinion, religion, health | Content consumption patterns |
| Purchase history | Health status, religion | Medication purchases, dietary products |
| Language patterns | National origin, education | Dialect, vocabulary, grammar patterns |
| Device/app usage | Age, income, disability | Accessibility features, device type |
EDPB position: inferring Art. 9 data from non-sensitive inputs constitutes processing of special category data — the same protections apply.
Fairness Metrics
Group Fairness Metrics
| Metric | Definition | When to Use |
|---|---|---|
| Demographic parity | P(positive outcome | group A) = P(positive outcome |
| Equalized odds | TPR and FPR equal across groups | When accuracy should be equal across groups |
| Equal opportunity | TPR equal across groups (relaxed equalized odds) | When true positive detection should be equal |
| Calibration | P(Y=1 | score=s, group=A) = P(Y=1 |
| Predictive parity | PPV equal across groups | When positive predictions should be equally reliable |
Individual Fairness Metrics
| Metric | Definition |
|---|---|
| Consistency | Similar individuals receive similar outcomes |
| Counterfactual fairness | Outcome would be the same if protected attribute were different |
| Causal fairness | No causal path from protected attribute to outcome |
Metric Selection Guidance
| Decision Context | Recommended Metric | Justification |
|---|---|---|
| Hiring/admissions | Equalized odds or equal opportunity | Equal detection of qualified candidates across groups |
| Credit scoring | Calibration | Score should mean the same probability regardless of group |
| Criminal risk | Equalized odds | Both FPR and TPR should be equal to avoid disproportionate impact |
| Healthcare | Equal opportunity + calibration | Equal detection of conditions; equal meaning of risk scores |
| Content moderation | Demographic parity | Content removal should not disproportionately affect groups |
Note: Mathematical impossibility results show that demographic parity, equalized odds, and calibration cannot all be satisfied simultaneously when base rates differ across groups. Document the trade-off explicitly.
Bias Detection Methodology
Phase 1: Data Audit
- Profile training data demographics against target population
- Identify underrepresented groups that may have lower model performance
- Check for historical bias in labelled data (e.g., biased hiring decisions used as ground truth)
- Identify label noise differential across groups
- Assess feature distributions across groups for proxy discrimination potential
Phase 2: Model Testing
- Evaluate model performance disaggregated by protected groups
- Compute selected fairness metrics for each protected group pair
- Run counterfactual testing: change protected attributes, observe output changes
- Test for intersectional bias (combinations of protected attributes)
- Evaluate model performance on edge cases and adversarial examples per group
Phase 3: Output Analysis
- Analyse decision distribution across groups
- Identify threshold effects that disproportionately impact specific groups
- Test for feedback loop amplification over time
- Assess output explanation fairness (are explanations equally informative across groups?)
Bias Mitigation Strategies
Pre-processing (Data-Level)
| Strategy | Description | Trade-off |
|---|---|---|
| Resampling | Over-sample underrepresented groups, under-sample overrepresented | May reduce data diversity or introduce duplicates |
| Reweighting | Assign higher weights to underrepresented group samples | Computationally simple; may not address structural bias |
| Relabelling | Correct historically biased labels | Requires domain expertise; may be subjective |
| Fair representation learning | Learn latent representation that removes protected attribute information | May lose legitimate correlations |
In-processing (Algorithm-Level)
| Strategy | Description | Trade-off |
|---|---|---|
| Adversarial debiasing | Train adversary to predict protected attribute from model; penalise success | Accuracy-fairness trade-off; requires protected attribute data |
| Fairness constraints | Add fairness metric as training constraint | May reduce overall accuracy; constraint satisfaction varies |
| Regularisation | Add fairness-related regularisation term to loss function | Balances accuracy and fairness; requires tuning |
| Causal modelling | Use causal graph to block discriminatory paths | Requires causal knowledge; complex to implement |
Post-processing (Output-Level)
| Strategy | Description | Trade-off |
|---|---|---|
| Threshold adjustment | Different decision thresholds per group to equalise metrics | May be perceived as unfair; legally complex |
| Score calibration | Calibrate scores per group | Requires sufficient group data; may reduce discrimination |
| Reject option | Abstain from decision for borderline cases across groups | Reduces coverage; requires human fallback |
AI Act Art. 10 Data Governance
Art. 10 requires for high-risk AI training data:
| Requirement | Implementation |
|---|---|
| Relevant data | Training data must be relevant to the intended purpose |
| Sufficiently representative | Data must represent the population the system will be deployed on |
| Free of errors | Data quality assessment and cleaning processes |
| Complete | Sufficient coverage of deployment scenarios |
| Appropriate statistical properties | Distribution analysis including demographic representation |
| Bias examination | Examine training data for possible biases, especially related to Art. 10(2)(f) |
Art. 10(5): Processing of special category data for bias detection is permitted for high-risk AI if:
- Strictly necessary for bias monitoring, detection, and correction
- Subject to appropriate safeguards (pseudonymisation, access controls)
- Data is not used for other purposes
- Deleted after bias assessment unless retention is required for compliance documentation
Documentation Requirements
Bias Assessment Report
| Section | Content |
|---|---|
| System description | Model, purpose, affected groups |
| Protected attributes assessed | Art. 9 categories + equality law characteristics |
| Fairness metrics selected | With justification for selection |
| Data audit results | Training data demographics, representation gaps |
| Model testing results | Per-group performance, fairness metrics, counterfactual results |
| Bias findings | Identified disparities with severity assessment |
| Mitigation measures | Applied strategies with effectiveness evidence |
| Residual bias | Remaining disparities after mitigation |
| Trade-off documentation | Accuracy-fairness trade-offs, metric impossibility acknowledgement |
| Ongoing monitoring plan | Post-deployment fairness monitoring |
Enforcement Precedents
- Dutch Tax Authority (SyRI, 2020): Court struck down algorithmic fraud detection for discriminatory profiling — system disproportionately targeted residents of low-income, immigrant-background neighbourhoods.
- Italian DPA v. Deliveroo (2021): Algorithmic worker management found to discriminate based on protected characteristics — Art. 22 and equality law violations.
- Austrian DPA v. AMS Algorithm (2020): Austrian employment service algorithm that scored job seekers lower based on gender, age, disability, and citizenship — DPA found processing unlawful.
- AEPD guidance (2021): Spanish DPA issued guidance on algorithmic discrimination, emphasising DPIA requirement for AI systems processing protected characteristics.
- CNIL (2024): AI bias assessment framework published requiring fairness testing for high-impact AI systems.
Integration Points
- ai-dpia: Bias assessment feeds into DPIA risk assessment for discrimination harms
- ai-automated-decisions: Biased automated decisions trigger Art. 22 and equality law violations
- ai-training-lawfulness: Art. 10(5) processing of special category data for bias detection requires documentation
- ai-transparency-reqs: Bias findings should be disclosed in model cards and transparency documentation
- ai-act-high-risk-docs: Art. 10 data governance documentation is part of conformity assessment
AI Model Retention and Unlearning
Overview
GDPR Art. 5(1)(e) storage limitation requires that personal data be kept no longer than necessary for the processing purpose. For AI systems, this creates complex retention challenges: training data used to build a model may no longer be needed once training is complete, but the model itself encodes information about the training data. Machine unlearning — the process of removing the influence of specific data from a trained model — is an emerging field that addresses the gap between deleting training data and eliminating its influence from model parameters. This skill provides retention policies, deletion verification methods, and machine unlearning techniques for AI compliance.
AI Data Retention Categories
| Data Category | Description | Retention Consideration |
|---|---|---|
| Raw training data | Original personal data used for model training | Delete after training unless retraining justifies retention |
| Processed training data | Cleaned, augmented, feature-engineered data | Same as raw — delete when training purpose exhausted |
| Validation/test data | Data used for model evaluation | Retain for model audit and comparison; pseudonymise |
| Model weights/parameters | Trained model artefacts encoding training data information | Retain while model is deployed; delete on decommission |
| Inference logs | Inputs and outputs of model predictions | Retention based on purpose (audit, debugging, rights exercise) |
| Model metadata | Training configuration, hyperparameters, provenance | Retain for compliance documentation; low privacy risk |
| Embedding vectors | Dense representations derived from personal data | May contain personal data — apply retention policy |
Retention Policy Framework
Training Data Retention Decision Tree
Training data category identified
│
├─ Is the data still needed for model retraining?
│ ├─ YES → Retain with documented justification and review date
│ └─ NO → Continue
│
├─ Is the data needed for model validation or audit?
│ ├─ YES → Retain in pseudonymised form with access controls
│ └─ NO → Continue
│
├─ Is the data needed for data subject rights exercise?
│ ├─ YES → Retain for rights exercise period, then delete
│ └─ NO → Continue
│
├─ Is there a legal obligation to retain?
│ ├─ YES → Retain per legal requirement
│ └─ NO → DELETE the training data
│
└─ After deletion: assess model for residual data encoding
Model Lifecycle Retention
| Phase | Retention Rule |
|---|---|
| Development | Training data retained during active development |
| Deployment | Training data deleted unless retraining is planned within defined period |
| Operation | Inference logs retained per purpose (30 days debug, 1 year audit) |
| Retraining | New training data collected; old data deleted post-training |
| Decommission | All model artefacts, training data, and logs deleted; retain only compliance documentation |
Machine Unlearning Techniques
Exact Unlearning
Full retraining: Retrain the model from scratch on the dataset minus deleted records.
| Property | Value |
|---|---|
| Guarantee | Complete — model has no knowledge of deleted data |
| Cost | Very high — full training cost for each deletion request |
| Feasibility | Impractical for large models or frequent deletion requests |
| When to use | Small models, infrequent requests, high-sensitivity data |
SISA (Sharded, Isolated, Sliced, Aggregated) Training
Train model on sharded data partitions. To unlearn, retrain only the affected shard.
| Property | Value |
|---|---|
| Guarantee | Exact within the affected shard |
| Cost | 1/k of full retraining (k = number of shards) |
| Feasibility | Requires SISA architecture from the start |
| Trade-off | Model accuracy may decrease with fewer shards contributing |
Approximate Unlearning
Gradient-Based Unlearning
Apply gradient ascent on the data to be forgotten, then fine-tune on remaining data.
| Property | Value |
|---|---|
| Guarantee | Approximate — statistically similar to retrained model |
| Cost | Low — few gradient steps |
| Feasibility | Works for most differentiable models |
| Verification | Requires membership inference testing to verify |
Influence Function-Based Unlearning
Use influence functions to estimate the effect of removing data and adjust model accordingly.
| Property | Value |
|---|---|
| Guarantee | Approximate — first-order approximation |
| Cost | Medium — requires Hessian computation |
| Feasibility | Best for smaller models or linear models |
Unlearning Verification
After applying unlearning, verify effectiveness:
- Membership inference test: Run MI attack on unlearned records — should classify as non-members
- Output comparison: Compare model outputs with and without the unlearned data
- Canary test: If canary records were included, verify they are no longer extractable
- Statistical test: Compare model to one retrained from scratch on the same data minus deleted records
Model Versioning for Compliance
Version Control Requirements
| Element | Documentation |
|---|---|
| Model version ID | Unique identifier (e.g., model-v2.3.1-20260314) |
| Training data snapshot | Hash of training dataset used for this version |
| Training date | When training was executed |
| Data deletions applied | Which data subject deletions are reflected in this version |
| Unlearning applied | Any approximate unlearning applied since last full retraining |
| Privacy properties | DP epsilon, MI test results for this version |
| Deployment dates | When deployed and when retired |
Retraining Triggers
| Trigger | Action |
|---|---|
| Accumulated deletion requests exceed threshold | Full retraining on updated dataset |
| Scheduled periodic retraining | Incorporate all pending deletions |
| Privacy audit reveals unacceptable leakage | Retrain with enhanced privacy measures |
| Model performance degradation | Retrain with current data (post-deletions) |
| Regulatory change | Assess if retraining needed for compliance |
Enforcement Relevance
- EDPB Guidelines 04/2025: Training data retention must be justified; deletion of training data does not automatically eliminate GDPR obligations for the model if it encodes personal data.
- Garante v. OpenAI (2023): Required mechanism for data deletion from training data; acknowledged technical challenges but expected good faith effort.
- EDPB ChatGPT Taskforce (2024): Controllers must demonstrate capability to address erasure requests affecting training data.
Integration Points
- ai-data-subject-rights: Erasure rights implementation requires unlearning
- ai-dpia: Retention and deletion capability assessed in DPIA
- ai-model-privacy-audit: Audit verifies deletion effectiveness
- ai-training-lawfulness: Retention justification part of lawful basis assessment
Data Subject Rights for AI Systems
Overview
AI systems create unique challenges for data subject rights exercise. Traditional rights mechanisms designed for structured databases do not map directly to ML model architectures where personal data is encoded in model weights, reproduced in model outputs, or used in opaque decision processes. This skill provides the framework for implementing each GDPR right (Arts. 15-22) and the AI Act Art. 86 right to explanation in the context of AI processing, addressing both training-time and inference-time rights.
Rights Framework for AI
Right of Access (Art. 15)
| AI Context | Obligation | Implementation |
|---|---|---|
| Training data contribution | Confirm whether data subject's data was in training set; provide copy if feasible | Training data catalogue indexed by data subject identifier; membership query |
| AI inference inputs | Provide data used as input to AI decision | Log inference inputs with data subject linkage |
| AI inference outputs | Provide AI decision/score/classification affecting data subject | Decision logging with data subject ID |
| Logic explanation | Art. 15(1)(h): meaningful information about logic of automated decisions | SHAP/LIME explanation on request or system-level explanation |
| Training data source | Art. 14(2)(f): source of data if not collected from data subject | Training data provenance documentation |
Technical Challenges:
- Identifying specific records in massive training datasets
- Determining if a data subject's data is in the training set without running membership inference
- Providing meaningful logic explanation for complex models
Right to Rectification (Art. 16)
| AI Context | Obligation | Implementation |
|---|---|---|
| Training data correction | Correct inaccurate data in training dataset | Update training data; assess if model retraining needed |
| Model output correction | Correct inaccurate AI outputs about a data subject | Output correction mechanism; flag in decision system |
| Inference input correction | Correct data used as inference input | Update input data; re-run inference |
Technical Challenge: Correcting training data may require model retraining to propagate the correction. For deployed models, correction may need model update or retraining pipeline.
Right to Erasure (Art. 17)
| AI Context | Obligation | Implementation |
|---|---|---|
| Training data deletion | Delete data subject's data from training dataset | Remove from training data; assess model impact |
| Model unlearning | Remove influence of deleted data from trained model | Machine unlearning technique or full retraining |
| Inference logs | Delete inference inputs and outputs linked to data subject | Purge from decision logs |
| Model outputs | Delete generated content about the data subject | Content removal mechanism |
Technical Challenge: True erasure from a trained model requires either verified machine unlearning or complete model retraining. The EDPB acknowledges this challenge but expects controllers to demonstrate good faith effort and use best available techniques.
Right to Restriction (Art. 18)
| AI Context | Obligation |
|---|---|
| Contested accuracy | Restrict processing while accuracy of training data or AI output is verified |
| Unlawful processing | Restrict rather than delete if data subject requests |
| During objection assessment | Restrict while controller assesses whether legitimate grounds override |
Implementation: Quarantine data subject's data from training pipeline and inference pipeline during restriction period.
Right to Data Portability (Art. 20)
| AI Context | Obligation |
|---|---|
| Training data contribution | Provide data subject's training data contribution in structured, machine-readable format |
| AI-generated profile | If AI has created a profile, provide in portable format |
| Inference history | Provide history of AI decisions affecting data subject |
Right to Object (Art. 21)
| AI Context | Obligation |
|---|---|
| AI training objection | If training is based on legitimate interest (Art. 6(1)(f)), data subject can object |
| Profiling objection | Object to AI profiling including inference of characteristics |
| Direct marketing | Absolute right to object to AI-driven direct marketing profiling |
Upon objection: Controller must cease processing unless compelling legitimate grounds override. For AI training: remove data from training pipeline and assess model impact.
Rights Relating to Automated Decisions (Art. 22)
| Right | Implementation |
|---|---|
| Right not to be subject | Opt-out from solely automated AI decisions with legal/significant effects |
| Right to human intervention | Qualified human reviewer with authority to override AI |
| Right to express views | Mechanism for data subject to provide additional context |
| Right to contest | Formal contestation with independent review |
| Right to explanation | AI Act Art. 86 + GDPR Recital 71: clear explanation of AI's role in the decision |
AI Act Art. 86 — Right to Explanation
For high-risk AI systems, affected persons have the right to:
- Clear and meaningful explanations of the role of the AI system in the decision-making procedure
- The main elements of the decision taken
- This right is without prejudice to GDPR data subject rights
Operational Implementation
Rights Request Triage for AI
Data subject rights request received
│
├─ Identify if AI processing is involved
│ ├─ Was AI used in any decision affecting the data subject?
│ ├─ Is the data subject's data in any training dataset?
│ └─ Has AI generated any content about the data subject?
│
├─ Determine applicable AI-specific rights
│ ├─ Access: training data, inference records, logic explanation
│ ├─ Rectification: training data or output correction
│ ├─ Erasure: training data deletion, model unlearning
│ ├─ Restriction: quarantine from AI pipeline
│ ├─ Objection: cease AI training on their data
│ └─ Automated decision: explanation, human review, contestation
│
├─ Route to appropriate team
│ ├─ Standard rights: data protection team
│ ├─ AI-specific: data protection + ML engineering team
│ └─ Contestation: independent review panel
│
└─ Process within Art. 12 timeframe (one month, extendable by two)
Response Timeframes
| Request Type | Standard | Extended | Justification for Extension |
|---|---|---|---|
| Access (standard) | 1 month | 3 months | Complex, voluminous |
| Access (AI logic) | 1 month | 3 months | Requires technical explanation generation |
| Rectification | 1 month | 3 months | May require model retraining |
| Erasure | 1 month | 3 months | Machine unlearning complexity |
| Explanation (Art. 86) | Without undue delay | — | Time-critical for affected persons |
| Contestation | 30 days | 90 days | Requires independent review |
Enforcement Precedents
- Austrian DPA v. CRIF (2023): Violation of Art. 15(1)(h) — credit scoring system failed to provide meaningful information about automated decision logic upon access request.
- Garante v. OpenAI (2023): Required mechanism for data subjects to request correction of inaccurate AI outputs and to object to AI training on their data.
- Dutch DPA guidance (2024): Right to erasure includes obligation to address erasure from AI training data, even if technically challenging.
- EDPB ChatGPT Taskforce (2024): Controllers must demonstrate capability to address rights requests affecting training data — technical difficulty does not exempt.
Integration Points
- ai-automated-decisions: Contestation and human review mechanisms
- ai-transparency-reqs: Logic explanation fulfils transparency obligations
- ai-data-retention: Erasure rights intersect with retention and unlearning
- ai-training-lawfulness: Objection right applies to legitimate interest-based training
AI System Pre-Deployment Privacy Checklist
Overview
Deploying an AI system that processes personal data requires verification of privacy compliance across multiple dimensions before the system goes live. This checklist serves as a compliance gate in the Cerebrum AI Labs ML deployment pipeline. No AI system may be deployed to production until all mandatory items are verified and signed off by the Data Protection Officer (DPO). The checklist is structured around GDPR requirements, the EU AI Act obligations (for high-risk systems), and internal governance standards.
Pre-Deployment Compliance Gate
Gate 1: Legal Basis and DPIA
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Lawful basis documented | Art. 6(1) basis identified and recorded for all personal data processing | Required | LIA or consent records |
| Special categories assessed | Art. 9 data identified; explicit consent or Art. 9(2) exception documented | Required | Data classification report |
| DPIA completed | Art. 35 DPIA completed for high-risk processing (profiling, systematic monitoring, large-scale special categories) | Required if applicable | DPIA document signed by DPO |
| DPIA risks mitigated | All high/critical risks from DPIA have documented mitigations | Required | Risk treatment plan |
| Prior consultation | Art. 36 consultation with supervisory authority if residual risk remains high | Required if applicable | Consultation record |
| Legitimate interest assessment | If relying on Art. 6(1)(f), LIA balancing test completed | Required if LI basis | LIA document |
Gate 2: Transparency and Information
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Privacy notice updated | Art. 13-14 information includes AI processing details | Required | Updated privacy notice |
| Logic described | "Meaningful information about the logic involved" documented for data subjects | Required for automated decisions | Explanation document |
| Significance disclosed | Envisaged consequences of AI processing disclosed | Required for automated decisions | Privacy notice section |
| Profiling disclosed | If system profiles individuals, this is disclosed in privacy notice | Required if profiling | Privacy notice section |
| AI Act transparency | Art. 52 transparency obligations met (if applicable): inform that they are interacting with AI | Required for AI Act | User interface disclosure |
Gate 3: Data Subject Rights
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Access process defined | Process for responding to access requests for AI data (inputs, outputs, profiles) | Required | Documented SOP |
| Explanation mechanism | Individual explanations can be generated on request | Required for Art. 22 | Technical capability verified |
| Human intervention available | Art. 22(3) human review process established | Required for solely automated decisions | Process document + trained staff |
| Contestation channel | Data subjects can contest AI decisions and have them reviewed | Required for Art. 22 | Appeal process document |
| Rectification process | Process for correcting AI input data and regenerating outputs | Required | Documented SOP |
| Erasure process | Process for deleting data from training sets, inference logs, embeddings | Required | Documented SOP |
Gate 4: Data Quality and Bias
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Training data documented | Data sources, size, collection method, preprocessing documented | Required | Data card / dataset documentation |
| Bias testing completed | Model tested for bias across protected attributes (gender, race, age, disability) | Required | Bias test report |
| Fairness metrics acceptable | Disparate impact ratio >0.8 (four-fifths rule) or equivalent metric within acceptable range | Required | Fairness metrics report |
| Data quality verified | Training data completeness, accuracy, representativeness verified | Required | Data quality report |
| Art. 9 data removed or justified | Special category data either removed or lawful basis documented | Required | Data classification report |
Gate 5: Security and Technical Controls
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Data encryption at rest | Training data and model weights encrypted (AES-256 or equivalent) | Required | Security configuration |
| Data encryption in transit | All API endpoints use TLS 1.2+ | Required | SSL certificate |
| Access controls | Role-based access to model, training data, and inference logs | Required | IAM policy |
| Audit logging | All model invocations logged with timestamp, input hash, output, user | Required | Logging configuration |
| Adversarial robustness | Model tested against common adversarial attacks relevant to its domain | Recommended | Security test report |
| Model versioning | Model versioned in registry with rollback capability | Required | MLflow / model registry |
Gate 6: Monitoring and Governance
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Performance monitoring | Dashboard tracking accuracy, latency, error rates in production | Required | Monitoring setup |
| Drift detection | Data drift and concept drift detection implemented | Required | Drift monitoring configuration |
| Bias monitoring | Post-deployment bias metrics tracked continuously | Required | Fairness monitoring dashboard |
| Incident response | Process for handling AI-related privacy incidents (e.g., discriminatory output, data leak) | Required | Incident response plan |
| Retraining schedule | Defined schedule for model retraining with fresh data | Required | Retraining plan |
| Retention enforcement | Automated deletion of inference logs and training data per retention schedule | Required | Retention policy + automation |
Gate 7: EU AI Act (High-Risk Systems Only)
| Check | Requirement | Status | Evidence |
|---|---|---|---|
| Risk classification | System classified per Annex III | Required | Classification document |
| Technical documentation | Annex IV documentation complete | Required | Tech doc package |
| Risk management system | Art. 9 continuous risk management implemented | Required | Risk register + process |
| Conformity assessment | Internal or third-party conformity assessment completed | Required | Assessment report |
| EU Declaration of Conformity | Art. 47 declaration prepared | Required | Signed declaration |
| EU database registration | Art. 49 registration completed | Required | Registration confirmation |
Sign-Off
| Role | Name | Approval | Date |
|---|---|---|---|
| ML Engineering Lead | [ ] Approved / [ ] Blocked | ||
| Data Protection Officer | [ ] Approved / [ ] Blocked | ||
| Information Security Officer | [ ] Approved / [ ] Blocked | ||
| Product Owner | [ ] Approved / [ ] Blocked | ||
| Legal Counsel | [ ] Approved / [ ] Blocked (high-risk only) |
Key Legal References
- GDPR Articles 5, 6, 9 — Data processing principles, lawful basis, special categories
- GDPR Article 22 — Automated individual decision-making safeguards
- GDPR Article 25 — Data protection by design and by default
- GDPR Article 35 — Data Protection Impact Assessment
- EU AI Act Articles 9-15 — High-risk AI system requirements
- EU AI Act Article 52 — Transparency obligations for certain AI systems
- EDPB Guidelines on Automated Decision-Making (WP 251 rev.01) — Art. 22 interpretation
- ISO/IEC 42001:2023 — AI management system standard
Data Protection Impact Assessment for AI/ML Systems
Overview
AI and ML systems present unique privacy challenges that traditional DPIA methodologies fail to adequately address. The EDPB Guidelines 04/2025 on processing personal data through AI systems establish a specialized framework that supplements the general DPIA requirements of GDPR Article 35 and WP248rev.01. AI-specific DPIAs must evaluate the entire ML pipeline — from training data collection through model deployment and inference — assessing risks that emerge from statistical learning, emergent model behaviours, and the opacity of algorithmic decision-making. This skill implements the EDPB's AI-specific DPIA methodology integrated with the EU AI Act risk classification framework.
AI-Specific DPIA Triggers
Mandatory DPIA Triggers for AI Systems
All AI processing that meets any of the following criteria requires a DPIA before deployment:
| Trigger | Legal Basis | Description |
|---|---|---|
| AI-based profiling with legal effects | Art. 35(3)(a) GDPR | ML models that produce decisions with legal or similarly significant effects on natural persons (credit scoring, hiring, insurance pricing) |
| Training on special category data | Art. 35(3)(b) GDPR | Models trained on health, biometric, genetic, racial, political, religious, sexual orientation, or trade union data at scale |
| AI-powered surveillance | Art. 35(3)(c) GDPR | Computer vision, facial recognition, behavioural analytics, or anomaly detection in public spaces |
| High-risk AI systems | Art. 6 EU AI Act | Systems listed in Annex III of the AI Act (biometric identification, critical infrastructure, employment, law enforcement, migration, justice) |
| Foundation models processing personal data | EDPB Guidelines 04/2025 | LLMs and foundation models trained on datasets containing personal data, regardless of downstream use |
| Automated inference of sensitive attributes | EDPB Guidelines 04/2025 | Models that infer Art. 9 special category data from non-sensitive inputs (inferring health status from purchasing patterns) |
EDPB WP248 Criteria Applied to AI
AI systems frequently trigger multiple WP248 criteria simultaneously:
- Evaluation/scoring: Inherent to classification and regression models
- Automated decision-making: Core function of deployed AI systems
- Innovative technology: Novel model architectures, training techniques
- Large-scale processing: Training datasets containing millions of records
- Matching/combining datasets: Multi-source training data aggregation
- Vulnerable data subjects: When AI is applied to children, employees, patients
When an AI system meets two or more criteria, a DPIA is presumptively required.
AI DPIA Methodology — EDPB Framework
Phase 1: AI System Description (Art. 35(7)(a) Extended)
The systematic description must cover the complete AI lifecycle:
1.1 Training Phase Documentation
- Training data sources: Origin, collection method, consent status, lawful basis for each dataset
- Data categories: All personal data categories present in training data, including data that may be inadvertently included (background individuals in images, metadata in text corpora)
- Data volume: Number of records, data subjects affected, geographic scope
- Data preprocessing: Cleaning, augmentation, labelling processes and any human review
- Feature engineering: Which personal data attributes are used as features, how derived features are computed
- Training infrastructure: Where training occurs (cloud provider, jurisdiction), data residency during training
1.2 Model Architecture Documentation
- Model type: Neural network architecture (transformer, CNN, RNN), ensemble methods, decision trees
- Model parameters: Number of parameters, model size, complexity indicators
- Explainability characteristics: Inherent interpretability level (white-box, grey-box, black-box)
- Memorization risk: Assessed propensity for the model to memorize training data (higher for large models with small datasets)
1.3 Deployment Phase Documentation
- Inference inputs: What personal data is processed at inference time
- Output types: Classifications, scores, recommendations, generated content
- Decision pipeline: How model outputs feed into decisions affecting data subjects
- Human oversight: Level and effectiveness of human review in the decision chain
- Monitoring: Drift detection, performance monitoring, feedback loops
Phase 2: AI-Specific Necessity and Proportionality
2.1 Purpose Limitation for AI
- Is the AI system necessary for the stated purpose, or could simpler processing achieve it?
- Has the controller evaluated non-AI alternatives and documented why AI is required?
- Are the training data processing purposes compatible with the original collection purposes (Art. 6(4) assessment)?
- For repurposed data: has a compatibility assessment been conducted per EDPB Guidelines 04/2025?
2.2 Data Minimisation for AI
- Has the minimum dataset required for acceptable model performance been determined through ablation studies?
- Can synthetic data, federated learning, or differential privacy reduce the personal data requirement?
- Are there personal data elements in the training data that do not contribute to model performance?
- Has the controller assessed whether anonymised or pseudonymised data could achieve adequate performance?
2.3 Training Data Lawfulness Assessment
For each training dataset, document:
| Assessment Element | Requirement |
|---|---|
| Original collection purpose | Was personal data collected for a purpose compatible with AI training? |
| Lawful basis | Art. 6(1) basis for the training processing — legitimate interest requires balancing test |
| Consent validity | If consent is the basis, was AI training specified as a purpose? Was consent freely given? |
| Special category conditions | If Art. 9 data is present, which Art. 9(2) exception applies? |
| Web-scraped data | EDPB position: web scraping for AI training generally cannot rely on legitimate interest without additional safeguards |
| Third-party datasets | Has the controller verified the upstream lawful basis chain? |
Phase 3: AI-Specific Risk Assessment
3.1 Privacy Risk Categories for AI
| Risk Category | Description | Likelihood Factors |
|---|---|---|
| Training data extraction | Adversary extracts verbatim training data from the model | Model size, training data repetition, overfitting degree |
| Membership inference | Adversary determines if specific data was in the training set | Model confidence distribution, overfitting, shadow model availability |
| Model inversion | Adversary reconstructs input features from model outputs | Output granularity, model type, auxiliary information available |
| Attribute inference | Model reveals sensitive attributes not provided as input | Correlations in training data, feature interactions |
| Emergent bias amplification | Model amplifies biases present in training data, producing discriminatory outcomes | Training data representativeness, debiasing measures applied |
| Concept drift discrimination | Model performance degrades unequally across demographic groups over time | Monitoring coverage, retraining frequency |
| Re-identification through AI output | Model outputs enable linking back to specific data subjects | Output specificity, population uniqueness, auxiliary data |
| Automated decision errors | Incorrect AI decisions causing material harm to data subjects | Model accuracy, error distribution across groups |
3.2 AI Risk Scoring Matrix
Combine likelihood and severity using the EDPB-recommended matrix:
Negligible Limited Significant Maximum
Almost Certain Medium High Very High Very High
Likely Medium High High Very High
Possible Low Medium High High
Remote Low Low Medium High
3.3 AI Act Risk Level Integration
Cross-reference GDPR risk assessment with AI Act classification:
- Unacceptable risk (Art. 5 AI Act): Processing must not proceed — social scoring, real-time biometric identification in public spaces (with limited exceptions)
- High risk (Art. 6 + Annex III): Enhanced DPIA obligations, conformity assessment required
- Limited risk (Art. 50): Transparency obligations — inform users they are interacting with AI
- Minimal risk: Standard DPIA process applies
Phase 4: AI-Specific Mitigation Measures
Technical Measures
| Measure | Risk Addressed | Implementation |
|---|---|---|
| Differential privacy | Training data extraction, membership inference | Apply DP-SGD during training with calibrated epsilon (ε ≤ 8 for moderate protection, ε ≤ 1 for strong) |
| Federated learning | Data centralisation risk | Distribute training across data holders without centralising personal data |
| Model output perturbation | Model inversion, attribute inference | Add calibrated noise to model outputs, round confidence scores |
| Training data deduplication | Memorization risk | Remove duplicate and near-duplicate records before training |
| Membership inference testing | Membership inference | Run MI attacks against the model pre-deployment; retrain if leakage exceeds threshold |
| Fairness constraints | Bias amplification | Apply demographic parity, equalised odds, or calibration constraints during training |
| Input/output filtering | PII leakage in generative models | Deploy PII detection on model inputs and outputs with automated redaction |
| Model pruning and distillation | Memorization, extraction | Compress the model to reduce capacity for memorizing individual records |
Organisational Measures
- Establish an AI Ethics Review Board with privacy representation
- Implement model cards documenting privacy properties for each deployed model
- Conduct regular model audits (minimum annually) testing for privacy leakage
- Maintain training data provenance documentation and deletion capability
- Define retraining triggers and ensure DPIA review accompanies each retraining cycle
- Implement incident response procedures specific to AI privacy incidents
Phase 5: Human Oversight Assessment
Per AI Act Art. 14 and GDPR Art. 22, assess the human oversight mechanism:
| Oversight Element | Assessment Question |
|---|---|
| Meaningful review | Can the human reviewer effectively evaluate the AI recommendation and override it? |
| Time and resources | Is sufficient time allocated for meaningful review, or is the human a rubber stamp? |
| Competence | Does the reviewer have the expertise to identify AI errors? |
| Authority | Does the reviewer have the authority and means to override the AI? |
| Feedback mechanism | Are overrides recorded and fed back into model improvement? |
| Automation bias | Are measures in place to mitigate the tendency to defer to the AI? |
Prior Consultation Triggers for AI
Art. 36 prior consultation with the supervisory authority is required when:
- The AI system produces high residual risk after all mitigation measures
- The AI system processes special category data at scale with novel techniques
- The supervisory authority's Art. 35(4) list specifically includes the AI use case
- The AI system is deployed for real-time biometric identification under AI Act Art. 5 exceptions
Enforcement Precedents
- Clearview AI (Multiple DPAs, 2021-2024): Fines totalling over EUR 90 million across Italy (EUR 20M), France (EUR 20M), UK (GBP 7.5M), Greece (EUR 20M) for facial recognition AI deployed without DPIA, lawful basis, or transparency
- CNIL v. Clearview AI (SAN-2022-019): Specific finding that no DPIA was conducted for biometric AI processing
- Italian DPA v. Replika (2023): Ordered cessation of AI chatbot processing due to inadequate age verification and failure to conduct DPIA for AI processing affecting minors
- Spanish DPA v. CaixaBank (PS/00421/2020): EUR 6 million fine for automated credit scoring without adequate DPIA addressing algorithmic decision-making risks
- Dutch DPA v. Tax Authority (2020): Finding that algorithmic fraud detection system (SyRI) lacked proportionality and adequate DPIA for AI-driven profiling
Integration Points
- ai-training-lawfulness: Detailed lawful basis analysis for training data feeds into Phase 2
- ai-automated-decisions: Art. 22 assessment integrates with Phase 5 human oversight
- ai-model-privacy-audit: Technical privacy testing results feed into Phase 3 risk assessment
- ai-act-high-risk-docs: AI Act conformity assessment aligns with Phase 3.3 risk classification
- ai-bias-special-category: Bias assessment results feed into risk scoring for discrimination harms
Federated Learning for GDPR Compliance
Overview
Federated learning (FL) is a distributed machine learning approach that trains models across multiple data holders without centralising personal data. Instead of collecting training data into a central repository, federated learning sends the model to the data, computes local updates on each participant's device or server, and aggregates only model updates (gradients or weights) at a central coordinator. This architecture directly addresses GDPR data minimisation (Art. 5(1)(c)) and data protection by design (Art. 25) principles by eliminating the need to transfer and centralise personal data for AI training. However, federated learning is not a privacy silver bullet — it introduces its own privacy risks that must be managed through complementary techniques.
Federated Learning Architecture Patterns
Pattern 1: Cross-Device Federated Learning
Use case: Training on data from millions of user devices (smartphones, tablets, IoT).
| Component | Description |
|---|---|
| Participants | End-user devices (smartphones, tablets, wearables) |
| Scale | Thousands to millions of participants |
| Data | Small per-device, large aggregate (e.g., keyboard predictions, health metrics) |
| Coordination | Central server selects participants per round, distributes model, aggregates updates |
| Communication | Compressed gradient updates over mobile networks |
| Privacy risk | Individual gradient updates may leak information about device data |
GDPR Analysis:
- Data minimisation: personal data never leaves the device — strong compliance
- Controller role: platform operator is controller; device owners are not processors
- Lawful basis: consent or legitimate interest for on-device processing
- International transfers: no personal data transfer if aggregation is privacy-preserving
- Right to erasure: device can be excluded from future rounds; model unlearning may be needed
Pattern 2: Cross-Silo Federated Learning
Use case: Training across organisational boundaries (hospitals, banks, subsidiaries).
| Component | Description |
|---|---|
| Participants | Organisational data silos (hospitals, branches, partner companies) |
| Scale | 2 to 100 participants |
| Data | Large per-silo, structured (e.g., medical records, financial transactions) |
| Coordination | Trusted aggregator or peer-to-peer protocol |
| Communication | Model updates over secure channels between organisations |
| Privacy risk | Gradient updates may reveal institutional data patterns |
GDPR Analysis:
- Joint controller determination: participants and aggregator may be joint controllers (Art. 26) — requires joint controller agreement
- Data processing agreements: if aggregator is a processor, Art. 28 DPA required
- Data minimisation: personal data stays within each silo — strong compliance
- International transfers: if silos are in different jurisdictions, gradient transfer legality depends on privacy guarantees
- Lawful basis: each silo needs independent lawful basis for local training; aggregation needs separate basis
Pattern 3: Vertical Federated Learning
Use case: Different organisations hold different features for the same individuals.
| Component | Description |
|---|---|
| Participants | Organisations with complementary data (bank + retailer sharing customer features) |
| Scale | 2 to 10 participants |
| Data | Different features for overlapping individuals |
| Coordination | Secure multi-party computation for feature combination |
| Communication | Encrypted intermediate representations |
| Privacy risk | Feature linkage may reveal individual attributes across parties |
GDPR Analysis:
- Joint controller: participants are likely joint controllers — each processes personal data for a common purpose
- Purpose limitation: combined training purpose must be compatible with each party's original collection purpose
- Data subject notification: data subjects must be informed about cross-organisation federated training
- Consent: may be required for combining data across organisations
Secure Aggregation Protocols
Protocol 1: Masking-Based Secure Aggregation
Participants mask their local updates with pairwise random masks that cancel out upon aggregation. The aggregator receives the sum without seeing individual updates.
| Property | Value |
|---|---|
| Privacy guarantee | Individual updates not visible to aggregator or other participants |
| Computational cost | O(n^2) pairwise key agreement, O(n) masking per round |
| Communication cost | 2x baseline (masks + masked updates) |
| Dropout tolerance | Handles participant dropout if sufficient participants remain |
| Collusion resistance | Secure against aggregator + up to t-1 participant collusion |
Protocol 2: Homomorphic Encryption Aggregation
Participants encrypt their updates with a homomorphic encryption scheme. The aggregator computes the sum on encrypted data without decryption.
| Property | Value |
|---|---|
| Privacy guarantee | Computationally secure — updates encrypted throughout |
| Computational cost | 100-1000x overhead for encryption/decryption operations |
| Communication cost | 2-10x baseline (ciphertext expansion) |
| Dropout tolerance | Excellent — encrypted updates can be summed independently |
| Collusion resistance | Secure against aggregator (does not hold decryption key) |
Protocol 3: Trusted Execution Environment (TEE)
Aggregation occurs within a hardware-protected enclave (Intel SGX, ARM TrustZone). Participants send updates to the TEE, which performs aggregation in isolated memory.
| Property | Value |
|---|---|
| Privacy guarantee | Hardware-based isolation — aggregator cannot inspect updates |
| Computational cost | Near-native (small overhead for enclave transitions) |
| Communication cost | Baseline (no encryption expansion for enclave-to-enclave) |
| Dropout tolerance | Excellent |
| Collusion resistance | Depends on hardware trust model; vulnerable to side-channel attacks |
Differential Privacy Integration
Local Differential Privacy (LDP)
Each participant adds noise to their gradient update before sending to the aggregator:
- Mechanism: Gaussian noise calibrated to sensitivity and privacy budget (epsilon, delta)
- Guarantee: Individual update is differentially private — aggregator cannot infer specific data points
- Trade-off: Higher noise reduces model accuracy; need more participants to compensate
- Recommended epsilon: 1-10 per round, with privacy accounting across rounds
Central Differential Privacy (CDP)
The aggregator adds noise to the aggregated update before applying to the global model:
- Mechanism: Gaussian noise added to the sum of clipped gradients
- Guarantee: Global model is differentially private with respect to any single participant's data
- Trade-off: Less noise needed than LDP for same accuracy (noise averages out); requires trusted aggregator
- Recommended epsilon: 1-8 total privacy budget
Privacy Budget Management
| Parameter | Description | Guidance |
|---|---|---|
| Epsilon (ε) | Privacy loss parameter — lower is more private | ε ≤ 1: strong privacy; ε ≤ 8: moderate; ε > 10: weak |
| Delta (δ) | Probability of privacy failure | δ < 1/N where N is dataset size |
| Rounds (T) | Number of federated training rounds | Privacy degrades with rounds — use composition theorems |
| Clip norm (C) | Maximum gradient norm per participant | Balance between privacy (lower C) and convergence (higher C) |
| Noise multiplier (σ) | Ratio of noise to sensitivity | Determined by ε, δ, C, and composition method |
GDPR Compliance Assessment for Federated Learning
Data Protection by Design (Art. 25) Alignment
| GDPR Principle | FL Implementation | Compliance Status |
|---|---|---|
| Data minimisation (Art. 5(1)(c)) | Personal data stays local — only model updates transmitted | Strong compliance |
| Purpose limitation (Art. 5(1)(b)) | Local processing for specified training purpose | Requires per-participant purpose documentation |
| Storage limitation (Art. 5(1)(e)) | No central training data repository — data retained locally per participant's policy | Compliance depends on participant retention |
| Integrity and confidentiality (Art. 5(1)(f)) | Secure aggregation protects update confidentiality | Strong with SA + DP |
| Accuracy (Art. 5(1)(d)) | Model accuracy may differ from centralised training | Monitor and document accuracy trade-offs |
Remaining GDPR Challenges
- Gradient leakage: Model updates can leak training data even without transmitting raw data — secure aggregation and DP mitigate but do not eliminate
- Model memorization: The aggregated model may still memorize individual records — privacy auditing required post-training
- Right to erasure: Removing a participant's contribution from the trained model is challenging — machine unlearning or retraining required
- Controller/processor determination: Complex multi-party FL architectures require clear role delineation
- Lawful basis: Each participant needs independent lawful basis; the aggregator needs basis for processing updates
- Transparency: Data subjects must be informed about federated learning participation
Implementation Considerations
Communication Efficiency
| Technique | Description | Privacy Impact |
|---|---|---|
| Gradient compression | Quantise or sparsify gradients before transmission | May interact with DP noise — careful calibration needed |
| Federated averaging (FedAvg) | Multiple local SGD steps before communication | Reduces communication rounds; may increase per-round privacy cost |
| Gradient selection | Send only top-k gradient components | Leaks which components are most significant — privacy concern |
Participant Selection
| Strategy | Description | Privacy Consideration |
|---|---|---|
| Random selection | Uniformly random participant sampling per round | Fair representation; privacy amplification through subsampling |
| Availability-based | Select participants with sufficient resources | May bias toward certain participant profiles |
| Contribution-based | Select participants whose data improves model most | Reveals information about data distribution — privacy risk |
Enforcement Relevance
While no enforcement action has specifically addressed federated learning, the technology is directly relevant to:
- CNIL AI recommendations (2024): Identified federated learning as a recommended privacy-enhancing technology for AI training
- EDPB Guidelines 04/2025: Mentioned distributed training as a data minimisation measure for AI systems
- Garante v. OpenAI (2023): Centralised data collection was a key compliance issue — federated alternatives could have mitigated
- AI Act Art. 10: Data governance for training data — FL enables data governance while maintaining data locality
Integration Points
- ai-dpia: FL architecture reduces data centralisation risk in DPIA assessment
- ai-training-lawfulness: FL may simplify lawful basis by keeping data local
- ai-model-privacy-audit: Federated models require distributed privacy auditing
- ai-privacy-inference: FL complements confidential computing for inference privacy
- ai-data-retention: Data locality in FL simplifies retention compliance
AI Model Privacy Audit
Overview
AI model privacy auditing is the systematic assessment of whether trained ML models leak information about their training data. Models can memorize individual training records, enabling adversaries to extract personal data, determine dataset membership, reconstruct input features, or infer sensitive attributes. This skill implements a comprehensive model privacy audit methodology using established attack techniques and tools (ML Privacy Meter, ART, Foolbox) to quantify privacy leakage before deployment and periodically during operation. The audit results feed directly into the AI DPIA risk assessment and inform mitigation measure selection.
Privacy Attack Taxonomy
1. Training Data Extraction
Objective: Extract verbatim or near-verbatim records from the model's training data.
| Attack Vector | Description | Target Models |
|---|---|---|
| Prompt-based extraction | Craft prompts that cause LLMs to regurgitate training data | Language models, generative models |
| Canary extraction | Insert known canary strings into training data and test if model reproduces them | Any model (testing methodology) |
| Gradient-based extraction | Use model gradients to reconstruct training inputs | Models with accessible gradients |
| Generative reconstruction | Use the model as an oracle to iteratively reconstruct training samples | GANs, VAEs, diffusion models |
Risk Factors Increasing Extraction Likelihood:
- Large model capacity relative to training data size (overfitting)
- Training data containing duplicated or near-duplicated records
- Longer training duration (more epochs)
- Lower regularisation
- Models with high output granularity (logits, probabilities)
Testing Methodology:
- Insert canary records with unique identifiers into training data
- Train the model
- Attempt extraction through various prompting strategies
- Measure extraction success rate (percentage of canaries recovered)
- Threshold: extraction rate should be below 0.1% for acceptable risk
2. Membership Inference
Objective: Determine whether a specific record was in the model's training set.
| Attack Type | Method | Computational Cost |
|---|---|---|
| Shadow model attack | Train shadow models on similar data, build a binary classifier on model outputs | High — requires training multiple shadow models |
| Metric-based attack | Use model confidence, loss, or entropy to distinguish members from non-members | Low — single model query per sample |
| Label-only attack | Use predicted labels (no confidence scores) to infer membership | Medium — requires multiple queries |
| Likelihood ratio attack (LiRA) | Compare per-sample loss to reference distributions | High — most accurate, requires multiple models |
ML Privacy Meter Implementation:
- Population metric-based attack: compares target model's loss on a sample against population loss distribution
- Reference metric-based attack: uses reference models to compute per-sample metrics
- Shadow model attack: trains shadow models and uses the attack model to classify members
Testing Methodology:
- Partition data: training set (members) and held-out set (non-members)
- Run membership inference attacks using ML Privacy Meter
- Measure attack success: true positive rate at low false positive rate (TPR@FPR=0.1%, 1%)
- Generate ROC curves per sample and aggregate
- Threshold: TPR@1%FPR should be below 5% for acceptable privacy
3. Model Inversion
Objective: Reconstruct input features from model outputs.
| Attack Type | Method | Target |
|---|---|---|
| Confidence-based inversion | Iteratively optimise input to maximise model confidence for a known label | Classification models |
| Gradient-based inversion | Use model gradients to reconstruct inputs from outputs | White-box models |
| GAN-based inversion | Train a GAN to invert model outputs to input space | Face recognition, image classifiers |
Testing Methodology:
- Select target classes or individuals
- Run inversion attacks with various initializations
- Measure reconstruction quality (SSIM, PSNR for images; cosine similarity for embeddings)
- Assess re-identification risk: can reconstructed data identify specific individuals?
- Threshold: reconstruction similarity should be below 0.3 (SSIM) for acceptable risk
4. Attribute Inference
Objective: Infer sensitive attributes not present in the model's output.
| Attack Type | Description |
|---|---|
| Correlation exploitation | Use correlated features to infer sensitive attributes from model behaviour |
| Partial knowledge attack | Attacker knows some attributes and uses model to infer remaining sensitive ones |
| Group inference | Determine statistical properties of training subgroups |
Testing Methodology:
- Identify sensitive attributes (Art. 9 categories) that may be correlated with model features
- Train attack models to predict sensitive attributes from model outputs
- Measure inference accuracy for each sensitive attribute
- Compare against random baseline
- Threshold: inference accuracy should not exceed random baseline + 10%
Audit Methodology
Phase 1: Audit Scoping (Days 1-3)
- Define audit scope: which models, what deployment context, what threat model
- Identify assets: training data, model artefacts, deployment infrastructure
- Define threat model: who are the adversaries, what access do they have?
- Black-box: API access only (queries and responses)
- Grey-box: API access plus model architecture knowledge
- White-box: Full access to model weights and architecture
- Select attacks based on threat model and model type
- Define success criteria (acceptable leakage thresholds)
- Obtain audit authorisation from model owner and legal
Phase 2: Environment Setup (Days 4-7)
- Set up isolated audit environment (no production data leakage)
- Install audit tools: ML Privacy Meter, ART (Adversarial Robustness Toolbox), custom scripts
- Obtain model access (API endpoint or model weights depending on threat model)
- Prepare member/non-member datasets for membership inference
- Prepare canary data for extraction testing
- Configure monitoring to log all audit queries
Phase 3: Attack Execution (Days 8-18)
For each selected attack:
- Configure attack parameters
- Execute attack against the target model
- Collect results (success rates, confidence intervals)
- Vary attack parameters to find worst-case leakage
- Document: attack configuration, results, computational cost
Phase 4: Analysis and Reporting (Days 19-25)
- Aggregate results across all attacks
- Calculate privacy risk scores per attack type
- Identify high-risk data subsets (records most vulnerable to extraction)
- Cross-reference with DPIA risk register
- Generate audit report with:
- Executive summary
- Attack results per category
- Risk assessment with GDPR alignment
- Recommended mitigations
- Residual risk after proposed mitigations
Phase 5: Remediation Validation (Days 26-30)
- If mitigations are applied (differential privacy, output perturbation, etc.)
- Re-run key attacks to validate mitigation effectiveness
- Document residual leakage post-mitigation
- Compare against acceptable thresholds
- Issue final audit certificate or remediation requirements
Privacy Leakage Thresholds
| Metric | Acceptable | Elevated | Unacceptable |
|---|---|---|---|
| Membership inference TPR@1%FPR | < 5% | 5-15% | > 15% |
| Training data extraction rate | < 0.1% | 0.1-1% | > 1% |
| Model inversion SSIM | < 0.3 | 0.3-0.6 | > 0.6 |
| Attribute inference accuracy above baseline | < 10% | 10-25% | > 25% |
Mitigation Measures
| Mitigation | Attacks Mitigated | Trade-off |
|---|---|---|
| Differential privacy (DP-SGD) | All — provides mathematical guarantee | Model accuracy reduction (calibrate epsilon) |
| Training data deduplication | Extraction, membership inference | One-time preprocessing cost |
| Regularisation (dropout, weight decay) | Membership inference, overfitting-related leakage | May affect model performance |
| Output perturbation | Model inversion, attribute inference | Reduces output precision |
| Confidence score rounding | Metric-based membership inference | Minor output precision loss |
| Model distillation | Extraction, membership inference | Requires additional training |
| Rate limiting | All query-based attacks | Affects legitimate use |
| Input/output PII filtering | Extraction of PII from generative models | May affect model utility |
Tools and Frameworks
| Tool | Purpose | Source |
|---|---|---|
| ML Privacy Meter | Membership inference auditing | github.com/privacytrustlab/ml_privacy_meter |
| IBM ART | Adversarial robustness and privacy testing | github.com/Trusted-AI/adversarial-robustness-toolbox |
| TensorFlow Privacy | Differential privacy training | github.com/tensorflow/privacy |
| Opacus | PyTorch differential privacy | github.com/pytorch/opacus |
| Google DP Library | Differential privacy algorithms | github.com/google/differential-privacy |
| Foolbox | Adversarial attack library | github.com/bethgelab/foolbox |
Enforcement Relevance
Model privacy auditing is not explicitly required by the GDPR or AI Act, but is effectively mandated through:
- Art. 35 DPIA: Risk assessment for AI systems must evaluate privacy leakage risks — auditing is the standard methodology
- Art. 32 Security: Appropriate technical measures to ensure security of processing — privacy auditing validates these measures
- AI Act Art. 9: Risk management for high-risk AI requires identification and mitigation of privacy risks
- AI Act Art. 15: Accuracy, robustness, and cybersecurity requirements — privacy attacks are a cybersecurity concern
- EDPB Guidelines 04/2025: Controllers must assess whether AI models have effectively anonymised training data — auditing tests this claim
Integration Points
- ai-dpia: Audit results feed into DPIA Phase 3 risk assessment
- ai-data-retention: Audit validates whether deletion from training data is effective
- ai-deployment-checklist: Pre-deployment privacy audit is a checklist requirement
- ai-federated-learning: Federated learning models require distributed privacy auditing
Combined DPIA and AI Act Conformity Assessment Template
Overview
High-risk AI systems under the EU AI Act must undergo both a GDPR Art. 35 DPIA and an AI Act conformity assessment. Rather than conducting these as separate exercises, this skill provides an integrated template that satisfies both frameworks simultaneously. The combined assessment ensures consistency between GDPR privacy risk analysis and AI Act safety and fundamental rights evaluation, reduces duplication, and provides a single risk scoring matrix covering both regulatory dimensions. Art. 26(9) AI Act explicitly requires deployers to use DPIA results when fulfilling AI Act obligations.
Combined Risk Scoring Matrix
Risk Dimensions
| Dimension | Source | Weight |
|---|---|---|
| Privacy risk to data subjects | GDPR Art. 35(7)(c) | 30% |
| Fundamental rights impact | EU AI Act Art. 9(2)(a) | 25% |
| Accuracy and reliability risk | EU AI Act Art. 15 | 20% |
| Transparency and explainability gap | GDPR Art. 13(2)(f) + AI Act Art. 13 | 15% |
| Human oversight adequacy | GDPR Art. 22 + AI Act Art. 14 | 10% |
Scoring Scale (Per Dimension)
| Score | Level | Description |
|---|---|---|
| 1 | Minimal | Risk negligible; controls effective |
| 2 | Low | Minor risk; standard controls sufficient |
| 3 | Medium | Moderate risk; enhanced controls needed |
| 4 | High | Significant risk; intensive mitigation required |
| 5 | Critical | Severe risk; may require processing suspension |
Overall Risk Classification
| Weighted Score | Classification | Action Required |
|---|---|---|
| 1.0-1.5 | Low | Standard monitoring |
| 1.6-2.5 | Medium | Enhanced monitoring and periodic review |
| 2.6-3.5 | High | Active mitigation and DPO/Board oversight |
| 3.6-4.5 | Very High | Art. 36 prior consultation; deployment hold pending mitigation |
| 4.6-5.0 | Critical | Do not deploy; fundamental redesign required |
GDPR DPIA Requirements (Art. 35(7))
| Element | Reference | Combined Assessment Section |
|---|---|---|
| Systematic description of processing | Art. 35(7)(a) | Section 2: AI System Description |
| Necessity and proportionality | Art. 35(7)(b) | Section 3: Necessity Assessment |
| Risk assessment | Art. 35(7)(c) | Section 5: Combined Risk Register |
| Mitigation measures | Art. 35(7)(d) | Section 6: Mitigation Measures |
AI Act Conformity Assessment Requirements
For high-risk AI systems (Annex III), the conformity assessment per Art. 43 requires:
| Element | Reference | Combined Assessment Section |
|---|---|---|
| Risk management system | Art. 9 | Section 5: Combined Risk Register |
| Data governance | Art. 10 | Section 2: Training Data Governance |
| Technical documentation | Art. 11 | Full combined assessment document |
| Record-keeping | Art. 12 | Section 7: Monitoring and Logging |
| Transparency | Art. 13 | Section 4: Transparency Assessment |
| Human oversight | Art. 14 | Section 4: Human Oversight |
| Accuracy, robustness, cybersecurity | Art. 15 | Section 5: Technical Risk Assessment |
| Quality management system | Art. 17 | Section 8: Quality Management |
Mitigation Measures Framework
Technical Measures
| Measure | GDPR Relevance | AI Act Relevance | Priority |
|---|---|---|---|
| Differential privacy | Training data protection | Robustness (Art. 15) | High for sensitive data |
| Model output perturbation | Model inversion protection | Accuracy trade-off (Art. 15) | Medium |
| Fairness constraints | Non-discrimination (Art. 5(1)(a)) | Bias prevention (Art. 10) | High for high-risk |
| Explainability tools (SHAP/LIME) | Art. 13(2)(f) logic explanation | Interpretability (Art. 13) | High |
| Input/output PII filtering | Data minimisation (Art. 5(1)(c)) | Accuracy (Art. 15) | High for generative AI |
| Encryption (rest/transit) | Security (Art. 32) | Cybersecurity (Art. 15) | Standard |
| Access controls (RBAC) | Security (Art. 32) | Cybersecurity (Art. 15) | Standard |
| Anomaly detection | Breach detection (Art. 33) | Robustness (Art. 15) | Medium |
Organisational Measures
| Measure | GDPR Relevance | AI Act Relevance |
|---|---|---|
| AI ethics review board | Accountability (Art. 5(2)) | Quality management (Art. 17) |
| Model cards | Transparency (Art. 13-14) | Technical documentation (Art. 11) |
| Regular bias audits | Fairness (Art. 5(1)(a)) | Bias monitoring (Art. 10) |
| Incident response for AI | Breach notification (Art. 33-34) | Post-market monitoring (Art. 72) |
| Staff training on AI risks | Accountability | Human oversight (Art. 14) |
| Documentation and record-keeping | Accountability (Art. 5(2)) | Record-keeping (Art. 12) |
Enforcement Relevance
- AI Act Art. 26(9): Deployers of high-risk AI shall use GDPR Art. 35 DPIA information when fulfilling AI Act obligations — the combined assessment implements this requirement.
- AI Act Art. 43: Conformity assessment procedures for high-risk AI systems.
- GDPR Art. 35: DPIA requirements — the combined template satisfies all Art. 35(7) elements.
Integration Points
- ai-dpia: Combined template extends the AI DPIA methodology
- ai-act-high-risk-docs: Conformity assessment documentation aligns with high-risk requirements
- ai-bias-special-category: Bias assessment results feed into combined risk register
- ai-automated-decisions: Human oversight assessment integrates Art. 22 and Art. 14 requirements
AI Privacy Inference and Derived Data
Overview
AI systems routinely generate inferences about individuals — predictions about creditworthiness, health risks, personality traits, political opinions, or behavioural patterns that were never directly provided by the data subject. These AI-derived inferences raise critical privacy questions: Are inferences personal data? When does inference become profiling under GDPR Article 22? What accuracy obligations apply to AI predictions? Can data subjects access, rectify, or object to inferences drawn about them? The CJEU, EDPB, and national DPAs have progressively clarified that inferences are personal data when they relate to an identified or identifiable person, and that GDPR rights extend to derived and inferred data. Cerebrum AI Labs must classify, govern, and provide transparency over all inferences its AI systems generate about individuals.
Legal Framework for AI Inferences
When Inferences Are Personal Data
| Criterion | Analysis | Example |
|---|---|---|
| Relates to an identified person | Inference is linked to a specific customer record or user profile | "Customer C-12345 has 78% churn probability" |
| Relates to an identifiable person | Inference can be linked to a person through combination with other data | "User with session token X-789 is likely aged 25-34" |
| Used to evaluate a person | Inference is used to assess, classify, or make decisions about someone | Credit score derived from transaction patterns |
| Has impact on a person | Inference affects how the person is treated or what options are available | Insurance premium adjusted based on predicted health risk |
CJEU C-434/16 (Nowak, 2017): Personal data includes "any information" relating to a data subject — this encompasses opinions, assessments, and inferences, not only factual data directly provided by the individual.
EDPB Guidelines 8/2020 on Targeting of Social Media Users: Inferred data (data created by the controller through observation or derivation) constitutes personal data and is subject to the full scope of GDPR rights.
GDPR Classification of Inference Types
| Inference Type | Classification | GDPR Implications |
|---|---|---|
| Observed data | Data collected through direct interaction (browsing history, purchase records) | Standard personal data — Art. 6 lawful basis required |
| Derived data | Data created by the controller through computation on existing data (credit score, risk rating) | Personal data — subject to access, rectification, objection rights |
| Inferred data | Probabilistic predictions about characteristics not directly observed (personality, health risk) | Personal data — potentially special category if predicting Art. 9 characteristics |
| Aggregated data | Statistical outputs at group level, not linked to individuals | Not personal data if truly anonymous (k-anonymity verified) |
When Inferences Become Special Category Data
| Predicted Characteristic | Art. 9 Category | Trigger |
|---|---|---|
| Ethnic origin from name/location patterns | Racial or ethnic origin | Any inference about ethnic background, even probabilistic |
| Political leaning from content engagement | Political opinions | Prediction used to classify or target based on politics |
| Religious affiliation from purchase patterns | Religious beliefs | Halal/kosher purchase scoring, prayer time activity patterns |
| Health condition from behavioural signals | Health data | Step count decline predicting depression, typing pattern analysis |
| Sexual orientation from browsing/social data | Sexual orientation | Any inference about sexual orientation regardless of accuracy |
| Pregnancy from purchase pattern shifts | Health data + gender | Purchase category analysis predicting pregnancy status |
Cerebrum AI Labs Policy: Any inference that predicts, estimates, or classifies an Art. 9 characteristic — even indirectly or probabilistically — must be treated as special category data and requires an Art. 9(2) condition for processing.
Profiling Under GDPR Article 22
Profiling Definition (Art. 4(4))
Profiling means any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning:
- Work performance
- Economic situation
- Health
- Personal preferences
- Interests
- Reliability
- Behaviour
- Location
- Movements
Three-Level Framework
| Level | Description | GDPR Requirement | Cerebrum AI Labs Example |
|---|---|---|---|
| Profiling only | Automated evaluation without decision | Art. 6 lawful basis + Art. 13-14 transparency | Customer segmentation for analytics dashboard |
| Profiling + human decision | Automated evaluation informing a human decision-maker | Art. 6 lawful basis + transparency + meaningful human involvement | Credit risk score reviewed by loan officer |
| Solely automated decision with legal/significant effects | Automated decision with no meaningful human involvement producing legal or similarly significant effects | Art. 22(1) prohibition applies — must fall within Art. 22(2) exceptions | Automated loan rejection based solely on AI credit score |
Art. 22(2) Exceptions Allowing Solely Automated Decisions
| Exception | Requirement | Cerebrum AI Labs Application |
|---|---|---|
| Art. 22(2)(a) — Contract | Decision necessary for entering into or performing a contract | Automated credit pre-approval for existing customers |
| Art. 22(2)(b) — Law | Authorised by EU or Member State law with suitable safeguards | Regulatory-mandated fraud screening |
| Art. 22(2)(c) — Explicit consent | Data subject's explicit consent obtained | Customer opts in to automated portfolio rebalancing |
Safeguards Required (Art. 22(3))
| Safeguard | Implementation at Cerebrum AI Labs |
|---|---|
| Right to obtain human intervention | Escalation button in customer portal routes to trained human reviewer within 2 business days |
| Right to express point of view | Customer can submit additional context through contestation form before human review |
| Right to contest the decision | Appeal process with independent review panel; decision reversed if AI error demonstrated |
| Right to explanation | Individual explanation generated using SHAP values showing top 5 factors influencing the AI decision |
Inference Accuracy and Quality Obligations
GDPR Article 5(1)(d) — Accuracy Principle
AI inferences must be accurate, and where necessary, kept up to date. For probabilistic predictions this means:
| Obligation | Implementation |
|---|---|
| Accuracy measurement | Track prediction accuracy metrics (precision, recall, F1) per demographic group |
| Confidence thresholds | Do not present inferences with confidence below 70% as actionable without human review |
| Staleness detection | Re-evaluate inferences when underlying data changes; flag inferences older than 90 days |
| Accuracy disclosure | Inform data subjects of the probabilistic nature and known accuracy of inferences |
| Rectification of inferences | Allow data subjects to challenge inferences; if input data is corrected, regenerate inference |
EDPB Position on Inference Accuracy
The EDPB Guidelines on Automated Decision-Making (WP 251 rev.01) state that controllers must:
- Use appropriate mathematical or statistical procedures for profiling
- Implement appropriate technical and organisational measures to minimise the risk of errors
- Correct inaccuracies and enable rectification
- Secure personal data to prevent discriminatory effects
Transparency Requirements for AI Inferences
Art. 13-14 Information Requirements
| Information | Requirement | Cerebrum AI Labs Implementation |
|---|---|---|
| Existence of profiling | Inform data subjects that profiling occurs | Privacy notice section: "How we use AI to analyse your data" |
| Logic involved | Meaningful information about the logic involved | Technical explainer: "Our AI analyses your transaction patterns, account tenure, and product usage to predict service needs" |
| Significance | Envisaged consequences of such processing | "This analysis may affect the products and offers shown to you, and may influence credit decisions" |
| Categories of data used | What data feeds the inference | "We use: transaction history, account tenure, product holdings, service interactions" |
| Inference outputs | What inferences are generated | "We generate: churn probability, product affinity scores, credit risk indicators" |
AI Act Transparency Requirements
| Obligation | AI Act Article | Cerebrum AI Labs Implementation |
|---|---|---|
| Inform users they are interacting with AI | Art. 52(1) | Chat interface disclosure: "You are interacting with an AI assistant" |
| Disclose AI-generated content | Art. 52(3) | Outputs marked: "This recommendation was generated by AI" |
| Disclose emotion recognition | Art. 52(2) | Not applicable — Cerebrum AI Labs does not use emotion recognition |
| High-risk system deployer transparency | Art. 13 | Technical documentation available to regulators on request |
Inference Governance Framework
Cerebrum AI Labs Inference Registry
All AI systems that generate inferences about individuals must be registered in the Cerebrum AI Labs Inference Registry:
| Registry Field | Description |
|---|---|
| System ID | Unique identifier for the AI system |
| Inference type | Category of inference generated (behavioural, demographic, financial, health) |
| Data subjects affected | Categories and approximate volume of individuals profiled |
| Input features | Data elements used to generate the inference |
| Output format | Inference output (score, category, probability, ranking) |
| Accuracy metrics | Latest precision, recall, F1 by demographic group |
| Retention period | How long inferences are stored before deletion |
| Art. 22 assessment | Whether the inference feeds a solely automated decision |
| Lawful basis | Art. 6 and (if applicable) Art. 9 basis for generating the inference |
| DPIA reference | Associated DPIA document ID |
Inference Lifecycle Controls
| Phase | Control |
|---|---|
| Generation | Log all inferences with timestamp, model version, confidence score, input data hash |
| Storage | Encrypt inference outputs; apply access controls limiting who can read individual-level inferences |
| Usage | Track downstream consumption of inferences; prevent scope creep beyond documented purposes |
| Disclosure | Make inferences available to data subjects on request (Art. 15 access right) |
| Rectification | If underlying data is corrected, flag dependent inferences for regeneration |
| Deletion | Delete inferences per retention schedule (90 days operational, 12 months audit) |
Enforcement and Regulatory Precedents
- CJEU C-634/21 (SCHUFA, 2023): The CJEU held that the generation of a credit score by a private entity constitutes "automated individual decision-making" under Art. 22 if the score plays a decisive role in a subsequent decision by a third party. This means AI-derived scores used in downstream decisions may trigger Art. 22 protections even if the AI provider is not the final decision-maker.
- Austrian DPA — Case DSB-D550.038 (2022): Found that inferred political opinions from social media activity constitute special category data under Art. 9, even when the inference is probabilistic and may be inaccurate.
- Norwegian DPA — Grindr Decision (2021): NOK 65 million fine for sharing data enabling inference of sexual orientation. Established that data enabling inference of Art. 9 characteristics is itself special category data.
- CNIL — Clearview AI (2022): EUR 20 million fine, finding that biometric inferences (face embeddings) from public photographs are personal data subject to full GDPR compliance.
Key Legal References
- GDPR Article 4(4) — Definition of profiling
- GDPR Article 5(1)(d) — Accuracy principle
- GDPR Article 13(2)(f) — Right to information about profiling logic and significance
- GDPR Article 15(1)(h) — Right of access to profiling information
- GDPR Article 22 — Automated individual decision-making including profiling
- GDPR Recital 71 — Safeguards for profiling and automated decisions
- GDPR Recital 72 — Guidelines on profiling
- CJEU C-434/16 (Nowak, 2017) — Broad interpretation of personal data including assessments
- CJEU C-634/21 (SCHUFA, 2023) — Credit scoring as automated decision-making
- EDPB Guidelines on Automated Decision-Making (WP 251 rev.01) — Art. 22 interpretation
- EDPB Guidelines 8/2020 on Targeting of Social Media Users — Inferred data as personal data
- EU AI Act Article 52 — Transparency obligations for AI systems
- EU AI Act Article 14 — Human oversight for high-risk AI systems
Lawful Basis for AI Training Data
Overview
The processing of personal data for AI model training constitutes a distinct processing operation requiring its own lawful basis under GDPR Art. 6(1). The EDPB Guidelines 04/2025 and the coordinated ChatGPT Taskforce findings establish that AI training creates unique lawful basis challenges: the scale of data collection, the difficulty of obtaining meaningful consent for open-ended AI training purposes, the tension between legitimate interest and data subject expectations, and the complexity of determining lawfulness for web-scraped and third-party datasets. This skill provides the comprehensive lawful basis assessment framework for AI training data processing, addressing each Art. 6(1) basis as applied to ML training contexts.
Fundamental Principles
AI Training as Personal Data Processing
The EDPB has confirmed that AI model training constitutes processing of personal data under Art. 4(2) GDPR when:
- Training datasets contain personal data (directly or indirectly identifiable natural persons)
- The model is trained on data that includes personal data, even if the intent is to learn general patterns
- The resulting model retains the capability to generate or reproduce personal data from training sets
- Personal data is used in any pipeline stage: collection, cleaning, annotation, augmentation, validation, testing
The controller cannot avoid GDPR obligations by claiming the model has "learned" rather than "stored" personal data. The processing occurs at the point of training, regardless of whether the model can later reproduce specific records.
Purpose Specification for AI Training
Art. 5(1)(b) requires that personal data be collected for specified, explicit, and legitimate purposes. For AI training, this means:
- "Training an AI model" is insufficiently specific — the controller must articulate the specific capability being developed
- "Improving our services" through AI training must be disaggregated into concrete purposes
- Each purpose must be documented before training begins, not retroactively justified
- The purpose must be communicated to data subjects in privacy notices per Arts. 13-14
Lawful Basis Analysis for AI Training
Art. 6(1)(a) — Consent
Requirements for Valid AI Training Consent
| Requirement | AI Training Application |
|---|---|
| Freely given | Data subjects must have genuine choice; consent cannot be bundled with service access unless AI training is necessary for the service |
| Specific | "AI training" alone is insufficient — must specify what type of model, for what purpose, what data elements are used |
| Informed | Must explain how personal data will be used in training, retention period for training data, risk of model memorization, inability to fully delete data from trained models |
| Unambiguous | Clear affirmative action; pre-ticked boxes or implied consent from terms of service are insufficient |
| Withdrawable | Controller must provide mechanism to withdraw consent; but model already trained on the data presents technical challenge |
Consent Challenges in AI Training
- Granularity problem: AI training often uses all available data — difficult to obtain specific consent for each data element's use in training
- Withdrawal complexity: Once a model is trained on personal data, true erasure requires model retraining or verified machine unlearning
- Purpose evolution: Foundation models and transfer learning mean the model may be repurposed — original consent may not cover downstream uses
- Scale impracticality: Obtaining consent from millions of data subjects whose data appears in web-scraped training corpora is practically impossible
- Power imbalance: When AI service use requires consent to training (e.g., "use our AI assistant and your conversations train our model"), consent may not be freely given
When Consent Works for AI Training
- Users explicitly opt into a research programme where AI model training is a primary purpose
- Users contribute data to a specific AI system with clear disclosure (e.g., "your feedback trains this recommendation engine")
- Fine-tuning on user-provided data where the user understands and consents to the training purpose
Art. 6(1)(b) — Contract Necessity
AI training can rely on contractual necessity only when:
- The AI model training is genuinely necessary for performing the contract with the data subject
- The data subject has entered into a contract that requires AI-powered features
- The training cannot be separated from the service delivery
Limitations per EDPB:
- General improvement of AI systems through aggregate training is not "necessary" for any individual contract
- Training a general-purpose model that benefits future users is not necessary for the current data subject's contract
- Fine-tuning based on individual user interactions may qualify if the personalised model is part of the contracted service
Art. 6(1)(f) — Legitimate Interest
This is the most commonly relied-upon basis for AI training. The EDPB requires a rigorous three-part assessment:
Part 1: Legitimate Interest Identification
The controller must identify a specific, real, and lawful interest:
| Interest Type | Example | EDPB Assessment |
|---|---|---|
| Commercial product improvement | Training a fraud detection model to protect customers | Generally legitimate — concrete benefit to data subjects |
| Research and development | Training models for medical imaging analysis | Legitimate if research purpose is genuine and specific |
| General AI capability | Training a foundation model for general-purpose use | Scrutinised — interest must be articulated with specificity |
| Competitive advantage | Training to match competitor AI capabilities | Legitimate commercial interest but weak in balancing |
Part 2: Necessity Assessment
| Question | Assessment Criteria |
|---|---|
| Is AI training necessary for the identified interest? | Could the interest be pursued without training on personal data? |
| Could anonymised data achieve the same result? | Has the controller tested model performance with anonymised data? |
| Could synthetic data supplement or replace personal data? | Has synthetic data generation been evaluated? |
| Is the volume of personal data proportionate? | Has the minimum effective dataset been determined? |
| Could federated learning avoid centralising personal data? | Has distributed training been assessed? |
Part 3: Balancing Test
Factors weighing in favour of the controller:
- Training data is publicly available (but this alone is not decisive)
- Model serves a beneficial purpose (fraud prevention, medical research)
- Strong technical safeguards applied (differential privacy, access controls)
- Data subjects can exercise opt-out rights effectively
- Training data is pseudonymised before use
Factors weighing in favour of data subjects:
- Data was not collected with AI training in mind — processing is far from original expectations
- Large-scale data collection from diverse sources without data subjects' awareness
- Special category data is present or can be inferred from training data
- Children's data is present in the training corpus
- No practical opt-out mechanism exists
- Model may memorize and regurgitate personal data
- Web scraping bypasses data subjects' choices about data sharing
EDPB Position on Legitimate Interest for AI Training
The EDPB Guidelines 04/2025 establish that:
- Legitimate interest for AI training is not automatically available — it requires case-by-case assessment
- Web scraping of personal data for AI training faces a particularly high bar
- The scale of data collection is a relevant factor — larger datasets require stronger justification
- The controller must demonstrate necessity: evidence that personal data is required rather than anonymised or synthetic alternatives
- Effective opt-out mechanisms are expected as a minimum safeguard
- The balancing test should consider the cumulative impact of multiple AI developers scraping and training on the same data subjects' data
Art. 6(1)(e) — Public Interest
Available to public bodies and organisations performing tasks in the public interest:
- Academic research institutions training AI models for publicly beneficial research
- Government agencies training AI for public service delivery
- Must have a basis in national or Union law
- Proportionality requirements apply
Special Situations
Web-Scraped Data
The EDPB has given specific guidance on web scraping for AI training:
- Robots.txt is not consent: Compliance with robots.txt does not establish lawful basis
- Public availability is not a lawful basis: Data being publicly accessible does not mean it can be freely used for AI training
- Reasonable expectations: Data subjects who post content online do not reasonably expect it to be used for AI training
- Children's data: Web-scraped data likely contains children's data — heightened protections apply
- Technical measures: Data subjects who implement privacy settings have expressed a preference against broad data use
Assessment framework for web-scraped training data:
| Factor | High Lawfulness Indicator | Low Lawfulness Indicator |
|---|---|---|
| Data source | Explicitly open-licence data (CC0, public domain) | Personal profiles, social media, private websites |
| Data type | Factual, non-personal content | Identifiable personal information, photos, opinions |
| Data subject expectations | Data published with intent for wide reuse | Data shared in specific context (social media, forums) |
| Safeguards | Differential privacy, PII filtering pre-training | No preprocessing to remove personal data |
| Opt-out | Effective and accessible opt-out mechanism | No opt-out or technically impractical opt-out |
| Transparency | Privacy notice covers AI training use | No notice to data subjects about AI training |
Third-Party Datasets
When using datasets obtained from third parties:
- Upstream lawful basis verification: The controller must verify that the data provider had a lawful basis to collect and share the data
- Contractual warranties: Obtain warranties from the provider regarding lawful collection, consent scope, and right to license for AI training
- Due diligence: Conduct reasonable due diligence on the provider's data collection practices
- Chain of accountability: The AI developer remains a controller responsible for lawful processing, even if the data was provided by a third party
Public Datasets
Academic and government datasets require assessment:
- Is personal data present? (Many datasets contain inadvertent personal data)
- What was the original purpose of the dataset? Is AI training compatible?
- Does the dataset licence permit commercial AI training?
- Has the dataset been ethically reviewed for consent and privacy?
- Are there known issues (bias, PII leakage, consent gaps)?
Training Data Retention
Art. 5(1)(e) storage limitation applies to AI training data:
- Training data must not be retained longer than necessary for the training purpose
- Once the model is trained, is continued retention of training data justified?
- If training data is retained for retraining, what is the maximum retention period?
- Model artefacts (weights, embeddings) that encode personal data are also subject to retention limits
- Deletion verification: can the controller demonstrate that training data has been effectively deleted?
Data Subject Rights for Training Data
| Right | AI Training Application | Technical Challenge |
|---|---|---|
| Access (Art. 15) | Data subject can request confirmation that their data was used in training and receive a copy | Identifying specific records in large training datasets |
| Rectification (Art. 16) | Inaccurate personal data in training sets must be corrected | Correction may require model retraining |
| Erasure (Art. 17) | Data subjects can request deletion of their data from training sets | Requires machine unlearning or model retraining |
| Objection (Art. 21) | Data subjects can object to processing based on legitimate interest | Controller must cease processing unless compelling grounds override |
| Restriction (Art. 18) | Processing must be restricted while accuracy or objection is contested | May require quarantining data from training pipeline |
Enforcement Precedents
- Garante v. OpenAI (2023): Temporary processing ban — no lawful basis identified for ChatGPT training data. Ordered OpenAI to identify Art. 6(1) basis for training data, implement age verification, and provide opt-out mechanism.
- CNIL v. Clearview AI (SAN-2022-019, 2022): EUR 20M fine — web scraping of biometric data without lawful basis. No consent, legitimate interest balancing test not conducted.
- Datatilsynet (Norway) v. Meta (2023): Temporary ban on using Norwegian user data for AI training — legitimate interest basis not sufficiently documented; balancing test inadequate.
- DPC (Ireland) v. Meta (2024): Investigation into use of public Facebook/Instagram posts for AI training under legitimate interest basis. Meta paused EU AI training following DPC engagement.
- EDPB Taskforce on ChatGPT (2024): Coordinated finding that legitimate interest for LLM training requires comprehensive balancing test, transparency, and effective opt-out — mere assertion of legitimate interest is insufficient.
Integration Points
- ai-dpia: Training data lawfulness feeds into DPIA Phase 2 assessment
- ai-data-subject-rights: Rights exercise mechanisms for training data
- ai-data-retention: Retention and deletion requirements for training datasets
- ai-transparency-reqs: Transparency obligations regarding training data use
AI Transparency Requirements
Overview
AI transparency operates at the intersection of two regulatory frameworks: the GDPR's data subject information rights (Arts. 13-14) and the EU AI Act's transparency obligations (Arts. 13-14, 50). Together they require controllers and deployers to provide meaningful, accessible information about AI system capabilities, limitations, decision logic, and personal data processing. This skill implements the combined transparency framework, addressing both the technical explainability challenge of complex ML models and the legal obligation to communicate AI processing in plain language to affected individuals.
GDPR Transparency for AI Systems
Art. 13-14 Information Requirements Applied to AI
When personal data is processed by AI systems, data subjects must receive:
| Information Element | GDPR Article | AI-Specific Application |
|---|---|---|
| Purposes of processing | Art. 13(1)(c) / 14(1)(c) | Specific AI use case, not generic "service improvement" |
| Lawful basis | Art. 13(1)(c) / 14(1)(c) | The basis for AI training and for AI inference separately |
| Legitimate interest | Art. 13(1)(d) / 14(2)(b) | The specific interest served by AI processing |
| Recipients | Art. 13(1)(e) / 14(1)(e) | AI infrastructure providers, model hosting services |
| International transfers | Art. 13(1)(f) / 14(1)(f) | Where AI processing occurs (training and inference locations) |
| Retention period | Art. 13(2)(a) / 14(2)(a) | Training data retention, inference log retention, model lifecycle |
| Data subject rights | Art. 13(2)(b) / 14(2)(c) | Including AI-specific rights: explanation, contestation, human review |
| Automated decision-making | Art. 13(2)(f) / 14(2)(g) | Meaningful information about logic, significance, and envisaged consequences |
| Source of data | Art. 14(2)(f) | Training data sources (categories, not necessarily individual sources) |
Art. 13(2)(f) / 14(2)(g) — Meaningful Information About Automated Logic
This is the most challenging transparency requirement for AI systems. The EDPB and Article 29 Working Party have clarified:
What "meaningful information about the logic involved" requires:
- System-level explanation: General description of how the AI system works — what factors are considered, what methodology is used, how the model was trained
- Purpose and context: Why the AI system is used and what role its output plays in decisions
- Key variables: The main data points or features that influence the AI output, without requiring disclosure of proprietary algorithms
- Decision criteria: How the model output translates into a decision (e.g., score thresholds, classification categories)
- Significance: What the decision means for the data subject in practical terms
- Consequences: The potential effects (both intended and foreseeable) of the AI-driven decision
What it does not require:
- Full disclosure of source code or model weights
- Mathematical description of the algorithm
- Proprietary trade secrets (but this does not exempt from meaningful explanation)
- Explanation of individual model predictions in real-time (though this may be required under Art. 22)
Layered Approach to AI Transparency
The EDPB recommends a layered transparency approach:
| Layer | Content | Delivery |
|---|---|---|
| Layer 1: Initial notice | AI is used in processing; general purpose; link to full information | At point of interaction (banner, tooltip, notification) |
| Layer 2: Summary | AI system description, key data used, decision logic summary, rights available | Privacy notice section, AI information page |
| Layer 3: Detailed information | Full technical description, training data categories, fairness measures, accuracy metrics, limitations | Supplementary documentation, upon request |
| Layer 4: Individual explanation | Specific factors influencing a particular decision, appeal mechanism | Upon request or automatically for significant decisions |
EU AI Act Transparency Obligations
Art. 13 — Transparency and Provision of Information to Deployers (High-Risk)
High-risk AI systems (Annex III) must be designed and developed to ensure:
| Requirement | Description |
|---|---|
| Interpretability | System design enables deployers to interpret outputs and use them appropriately |
| Instructions for use | Detailed documentation of capabilities, limitations, intended purpose, foreseeable misuse |
| Performance metrics | Accuracy levels, robustness metrics, known limitations for specific groups |
| Human oversight info | Description of human oversight measures and how to implement them |
| Input data specs | Description of input data the system was designed to process |
| Training data description | Relevant information about training data including provenance and preprocessing |
Art. 14 — Human Oversight (High-Risk)
High-risk AI systems must be designed to enable effective human oversight:
- Clear indication of AI system outputs and confidence levels
- Ability to correctly interpret AI outputs in context
- Ability to override or reverse AI decisions
- Ability to intervene or stop the system ("stop button")
- Awareness of automation bias risk
Art. 50 — Transparency for Specific AI Systems
| AI System Type | Transparency Obligation |
|---|---|
| AI interacting with persons | Inform that they are interacting with an AI system (unless obvious from context) |
| Emotion recognition / biometric categorisation | Inform about the system's operation and process personal data in compliance with GDPR |
| AI-generated or manipulated content (deepfakes) | Label content as AI-generated in a machine-readable format |
| AI-generated text on matters of public interest | Disclose that the text has been artificially generated or manipulated |
Art. 50(1) — AI Interaction Notification
Controllers must inform natural persons that they are interacting with an AI system. This applies to:
- Chatbots and virtual assistants
- AI-powered customer service systems
- Automated email or message generation
- AI-driven recommendation systems with direct user interface
- Voice-based AI assistants
Exceptions: where it is obvious from the circumstances and context that the person is interacting with AI (e.g., a robot in a factory setting).
AI Transparency Documentation Framework
Model Card Requirements
For each deployed AI model, maintain a model card containing:
| Section | Content |
|---|---|
| Model overview | Name, version, type, developer, deployment date |
| Intended use | Specific purpose, target users, deployment context |
| Out-of-scope use | Uses the model is not designed for; foreseeable misuse |
| Training data summary | Data sources (categories), volume, temporal range, geographic scope, known biases |
| Performance metrics | Accuracy, precision, recall, F1 by relevant subgroup; fairness metrics |
| Limitations | Known failure modes, demographic performance disparities, edge cases |
| Privacy properties | Differential privacy applied (epsilon), membership inference test results, training data extraction risk |
| Human oversight | Level of oversight required, reviewer qualifications, override procedures |
| Update history | Retraining dates, data updates, performance changes |
AI System Transparency Register
Organisations operating multiple AI systems should maintain a central register:
| Field | Description |
|---|---|
| System ID | Unique identifier |
| System name | Human-readable name |
| AI Act classification | Unacceptable / High / Limited / Minimal |
| Purpose | Specific processing purpose |
| Data subjects affected | Categories and estimated numbers |
| Personal data processed | At training and inference |
| Decision authority | AI decision-support vs. automated decision |
| Transparency measures | Notification, explanation, documentation |
| Deployer | Internal / External deployment |
| Registration date | EU AI Act database registration (if high-risk) |
Explainability Techniques for Compliance
Global Explainability (System-Level)
Techniques for providing Art. 13(2)(f) "meaningful information about the logic":
| Technique | Best For | Limitation |
|---|---|---|
| Feature importance (SHAP, LIME) | Identifying key variables | May oversimplify complex interactions |
| Decision rules extraction | Converting model logic to human-readable rules | Loss of accuracy for complex models |
| Partial dependence plots | Showing how features affect predictions | Assumes feature independence |
| Counterfactual explanations | Showing what change would lead to different outcome | Computationally expensive for many features |
| Attention visualisation | Transformer models — showing what the model focuses on | Attention does not always equal importance |
Local Explainability (Individual Decision)
For Art. 22 right to explanation of individual decisions:
| Technique | Description | Use Case |
|---|---|---|
| LIME | Local Interpretable Model-agnostic Explanations | Any model — approximate local behaviour with interpretable model |
| SHAP values | Shapley Additive Explanations for individual predictions | Feature contribution to specific prediction |
| Counterfactual | "You were denied because X; if X were Y, outcome would be different" | Credit, hiring, insurance decisions |
| Anchors | Sufficient conditions for a prediction | Rule-based explanation of individual case |
| Concept-based | High-level concepts that influenced the decision | When features are not directly interpretable |
Enforcement Precedents
- Garante v. OpenAI (2023): Required transparency about AI training data processing, model capabilities, and limitations — privacy notice deemed insufficient for AI system transparency.
- CNIL v. Clearview AI (SAN-2022-019, 2022): EUR 20M fine — complete absence of transparency about facial recognition AI processing; data subjects had no notice their images were scraped and processed.
- Austrian DPA v. CRIF (DSB-D213.636, 2023): Credit scoring AI — insufficient explanation of automated decision-making logic per Art. 13(2)(f); data subject received only a score without meaningful information about factors.
- Dutch DPA v. Tax Authority (SyRI, 2020): Court found algorithmic fraud detection lacked transparency — citizens could not understand how the system assessed them, violating right to private life.
- AEPD v. CaixaBank (PS/00421/2020, 2021): EUR 6M fine — automated credit decision-making without adequate transparency about the logic involved and significance for data subjects.
Integration Points
- ai-automated-decisions: Art. 22 explanation requirements integrate with transparency obligations
- ai-dpia: Transparency assessment is part of DPIA necessity and proportionality analysis
- ai-act-high-risk-docs: Art. 13 AI Act documentation requirements overlap with transparency framework
- ai-deployment-checklist: Pre-deployment transparency validation is a checklist item
AI Vendor Privacy Due Diligence
Overview
AI services create complex controller-processor relationships that differ significantly from traditional data processing arrangements. Whether an AI vendor is a processor, joint controller, or independent controller depends on the degree of autonomy the vendor has over personal data processing — particularly regarding model training on customer data, data retention for model improvement, and the vendor's independent purposes for the data. This skill provides the framework for determining controller-processor roles in AI service relationships, conducting privacy due diligence on AI vendors, and establishing appropriate contractual protections.
Controller-Processor Determination for AI
Decision Framework
| AI Service Model | Typical Role | Key Factors | GDPR Article |
|---|---|---|---|
| SaaS AI — Customer data processed per instructions | Vendor = Processor | Vendor processes data solely on controller's instructions; no independent use | Art. 28 DPA required |
| SaaS AI — Customer data used for model training | Vendor = Joint Controller or Independent Controller | Vendor uses customer data for own model improvement beyond contracted service | Art. 26 JCA or separate controller notice |
| Embedded AI — Pre-trained model in customer infrastructure | Customer = Controller; Vendor = may be processor for support | Model runs in customer environment; vendor may access data for support/updates | Art. 28 if vendor accesses data |
| API-based AI — Customer sends data for inference | Vendor = Processor (if no data retention) or Joint Controller (if training on inputs) | Depends on whether vendor retains, uses, or trains on input data | Assessment required |
| AI Platform — Customer builds models on vendor platform | Vendor = Processor for infrastructure; Controller for platform data | Vendor provides compute; customer controls data and model | Art. 28 DPA + audit rights |
| AI Marketplace — Pre-built models with customer data | Depends on data flow | If customer data enters vendor model → joint controller assessment | Case-by-case |
Key Determination Questions
- Who determines the purpose of processing? — The entity deciding why personal data is processed
- Who determines the means of processing? — The entity deciding how data is processed (but "non-essential means" may be delegated to processor)
- Does the vendor use data for its own purposes? — Model training, benchmarking, product improvement using customer data
- Does the vendor retain data beyond service delivery? — Data kept after inference, stored for training, retained in logs
- Does the vendor make independent decisions about the data? — Choosing to train models, sharing with sub-processors not instructed by customer
Common AI Vendor Patterns
Pattern 1: Pure Inference API (Vendor = Processor)
- Customer sends data; receives inference result
- Vendor does not retain input data beyond processing
- Vendor does not train on customer data
- Vendor acts solely on customer's instructions
- Contractual: Art. 28 DPA with clear scope
Pattern 2: AI with Model Improvement (Vendor = Joint Controller)
- Customer sends data; receives inference result
- Vendor retains data to improve its models
- Vendor determines that model training is a purpose
- Both parties benefit from the model improvement
- Contractual: Art. 26 Joint Controller Agreement
Pattern 3: AI with Anonymised Analytics (Assessment Required)
- Customer sends data; receives inference result
- Vendor claims to anonymise data and use aggregated analytics
- If anonymisation is effective: no personal data processing for analytics
- If anonymisation is not effective (re-identification possible): vendor is controller for analytics
- Assessment: Verify anonymisation effectiveness per WP216
Pattern 4: Embedded AI with Telemetry (Vendor = Processor + may be Controller)
- AI model runs in customer environment
- Vendor collects telemetry data including model performance
- If telemetry contains personal data: vendor may be controller for telemetry processing
- Contractual: Art. 28 DPA for model support + separate arrangement for telemetry
Due Diligence Assessment Framework
Phase 1: Vendor AI Processing Inventory
For each AI vendor, document:
| Element | Documentation Required |
|---|---|
| AI capabilities | What AI functions does the vendor provide? |
| Personal data inputs | What personal data is sent to the vendor? |
| Personal data outputs | What personal data does the vendor return? |
| Data retention | Does the vendor retain input data? For how long? |
| Model training | Does the vendor train on customer data? |
| Sub-processors | Does the vendor use sub-processors for AI processing? Where? |
| Data location | Where is AI processing performed? What jurisdictions? |
| Security measures | What security controls protect data during AI processing? |
| Human review | Does vendor personnel access customer data? |
Phase 2: Privacy Risk Assessment
| Risk Factor | Assessment | Risk Level |
|---|---|---|
| Data sensitivity | Special category data sent to AI vendor? | |
| Data volume | Volume of personal data processed by vendor | |
| Vendor data use | Vendor uses customer data for own purposes? | |
| International transfers | Data processed outside EU/EEA? | |
| Sub-processor chain | Number and location of sub-processors | |
| Security posture | Certifications (ISO 27001, SOC 2)? | |
| Incident history | Prior data breaches or enforcement actions? | |
| AI-specific risks | Model memorization, output leakage, bias? |
Phase 3: Contractual Assessment
| Contractual Element | Required? | Status |
|---|---|---|
| Art. 28 DPA or Art. 26 JCA | Yes | |
| Processing scope and purpose limitation | Yes | |
| Prohibition on data use beyond instructions (if processor) | Yes | |
| Model training opt-out | Yes (if vendor trains on data) | |
| Sub-processor notification and approval | Yes | |
| International transfer safeguards | If applicable | |
| Data deletion on termination | Yes | |
| Audit rights | Yes | |
| Breach notification obligations | Yes | |
| AI-specific: model privacy testing | Recommended | |
| AI-specific: bias assessment obligations | Recommended for high-risk | |
| AI-specific: output accuracy warranties | Recommended |
AI-Specific Contractual Clauses
Model Training Restrictions
The Processor shall not use Customer Data to train, improve, fine-tune, or
otherwise develop any machine learning model, algorithm, or AI system,
whether for the Customer's benefit or for any other purpose, without prior
written consent from the Customer. Any consent granted shall specify the
scope of permitted training, the data categories involved, and the privacy
safeguards to be applied.
AI Output Accuracy
The Provider acknowledges that AI system outputs about identifiable data
subjects must comply with the accuracy principle under Art. 5(1)(d) GDPR.
The Provider shall implement measures to minimise inaccurate outputs about
data subjects and shall promptly correct inaccurate outputs upon
notification.
Model Privacy and Bias Obligations
The Provider shall conduct periodic privacy audits of AI models processing
Customer Data, including membership inference testing and training data
extraction testing, and shall make results available to the Customer upon
request. The Provider shall monitor AI systems for discriminatory outcomes
and shall implement bias mitigation measures as required.
Enforcement Relevance
- EDPB Guidelines 07/2020 on Controller-Processor: Determination depends on factual circumstances, not contractual labels. A vendor labelled "processor" that uses data for its own purposes is factually a controller.
- Garante v. OpenAI (2023): OpenAI's role as controller for ChatGPT training data confirmed — processing personal data for model training is an independent controller purpose.
- CJEU C-40/17 (Fashion ID): Joint controller status can arise when a party has influence over the purpose and means of processing, even without access to the data.
- DPC v. Meta (WhatsApp, 2023): EUR 5.5M fine — processor-controller determination must reflect actual data use, not just contractual terms.
Integration Points
- ai-dpia: Vendor AI processing must be included in DPIA scope
- ai-training-lawfulness: Vendor training on customer data requires lawful basis assessment
- ai-transparency-reqs: Data subjects must be informed about AI vendor processing
- ai-deployment-checklist: Vendor due diligence is a pre-deployment checklist item
LLM Output Privacy Risk Assessment
Overview
Large language models (LLMs) present unique privacy risks that go beyond traditional ML systems. Because LLMs are trained on massive corpora that may contain personal data, they can memorise and reproduce verbatim training data — including names, email addresses, phone numbers, and other PII. Additionally, LLMs can hallucinate plausible but false personal data, creating defamation and accuracy risks. Prompt injection attacks can bypass safety guardrails to extract training data or system prompts containing confidential information. This skill provides a structured framework for assessing, mitigating, and monitoring privacy risks in LLM-generated outputs at Cerebrum AI Labs.
LLM Output Privacy Risk Categories
Risk 1: Training Data Memorisation
LLMs memorise training data, particularly sequences that appear multiple times or are distinctive. Extractable memorisation occurs when a model, given a prefix, completes the text with verbatim training data.
| Factor | Impact on Memorisation Risk |
|---|---|
| Model size | Larger models memorise more (Carlini et al., 2023) |
| Data duplication | Repeated sequences are memorised at higher rates |
| Training epochs | More passes over data increase memorisation |
| Data distinctiveness | Unique sequences (names, numbers) are more extractable |
| Temperature at inference | Lower temperature increases verbatim reproduction |
Regulatory concern: If a model reproduces personal data from training, this constitutes processing under GDPR Art. 4(2). The data subject has not consented to this output, and the controller must have a lawful basis for the disclosure.
Risk 2: PII Leakage in Generated Text
Even without verbatim memorisation, models can combine partial information to produce outputs containing personal data — email patterns, phone number formats with real area codes, or names associated with specific contexts.
| Leakage Type | Example | Detection Method |
|---|---|---|
| Verbatim reproduction | Model outputs exact email address from training data | Exact match against known training PII |
| Recombination | Model combines real first name + real surname from different records | Named entity recognition + cross-reference |
| Pattern completion | Model generates plausible phone number matching real format | Regex + validation against real registries |
| Contextual leakage | Model associates real person with correct employer when prompted | Entity relationship extraction |
Risk 3: Prompt Injection and Data Extraction
Adversarial prompts can cause models to bypass safety instructions and output training data, system prompts, or information about other users' conversations.
| Attack Vector | Description | Mitigation |
|---|---|---|
| Direct prompt injection | User crafts prompt to extract training data ("Repeat your training data starting with...") | Input filtering + output monitoring |
| Indirect prompt injection | Malicious content in retrieved documents instructs model to leak data | Sanitise retrieved content + instruction hierarchy |
| System prompt extraction | Prompts designed to reveal system-level instructions | System prompt isolation + extraction detection |
| Multi-turn extraction | Gradual extraction across conversation turns | Conversation-level monitoring |
| Jailbreaking | Bypassing safety alignment to disable PII filters | Robust alignment + secondary output filter |
Risk 4: Hallucinated Personal Data
LLMs can generate plausible but fabricated personal data — attributing false statements to real people, inventing credentials, or creating fictional but realistic-seeming personal records.
| Hallucination Type | Privacy Risk | Legal Concern |
|---|---|---|
| False attribution | Model attributes a statement to a real person who never said it | Defamation, Art. 5(1)(d) accuracy principle |
| Fabricated credentials | Model invents qualifications for a real person | Accuracy, potential reputational harm |
| Fictional PII | Model generates realistic but fake personal details (e.g., social security numbers) | May match real individuals by coincidence |
| Confabulated relationships | Model invents relationships between real people | Art. 5(1)(d) accuracy, potential harm |
EDPB position: EDPB Guidelines 04/2025 state that generating inaccurate personal data about identifiable individuals violates the accuracy principle (Art. 5(1)(d)) and may require the controller to implement effective rectification mechanisms.
Output Privacy Risk Assessment Framework
Assessment Dimensions
| Dimension | Weight | Scoring Criteria |
|---|---|---|
| Memorisation exposure | 25% | Model size, training data PII density, deduplication status |
| PII leakage likelihood | 25% | Output modality, user interaction pattern, PII filter coverage |
| Prompt injection resilience | 20% | Input validation, instruction hierarchy, extraction testing results |
| Hallucination risk | 15% | Factual grounding, attribution verification, entity linking |
| Monitoring coverage | 15% | Real-time PII detection, logging, alerting, human review |
Risk Scoring
| Score Range | Risk Level | Action Required |
|---|---|---|
| 0-20 | Low | Standard monitoring; annual review |
| 21-40 | Moderate | Enhanced PII filtering; quarterly review |
| 41-60 | Elevated | Mandatory output scanning; human review for sensitive queries |
| 61-80 | High | Restrict deployment scope; implement guardrails before production |
| 81-100 | Critical | Do not deploy; fundamental architecture changes required |
Mitigation Measures
Pre-Deployment Mitigations
| Measure | Description | Effectiveness |
|---|---|---|
| Training data deduplication | Remove duplicate sequences to reduce memorisation | High for verbatim memorisation |
| PII scrubbing of training data | Detect and remove/mask PII before training | High but incomplete (novel PII patterns missed) |
| Differential privacy training | Train with DP-SGD to bound memorisation | High theoretical guarantee; accuracy trade-off |
| RLHF safety alignment | Train model to refuse PII-generating requests | Moderate; can be bypassed by jailbreaking |
| Instruction tuning | Fine-tune model to follow safety instructions | Moderate; requires robust instruction hierarchy |
Runtime Mitigations
| Measure | Description | Effectiveness |
|---|---|---|
| Output PII scanner | NER-based scanner detecting PII in outputs before delivery | High for known PII patterns |
| Regex PII filter | Pattern matching for emails, phones, SSNs, credit cards | High for structured PII |
| Input prompt classifier | Classify incoming prompts for injection/extraction attempts | Moderate; adversarial prompts evolve |
| Retrieval grounding | Ground outputs in retrieved documents to reduce hallucination | Moderate; depends on retrieval quality |
| Rate limiting | Limit query volume to slow extraction attacks | Low-moderate; raises cost of attack |
| Conversation monitoring | Analyse multi-turn conversations for progressive extraction | Moderate; complex to implement |
Post-Deployment Monitoring
| Monitor | Metric | Alert Threshold |
|---|---|---|
| PII detection rate | % of outputs flagged by PII scanner | >0.1% of outputs in any 1-hour window |
| Prompt injection detection | % of inputs classified as injection attempts | >0.5% of inputs in any 1-hour window |
| User reports of PII exposure | Count of user-reported PII incidents | Any confirmed report |
| Extraction pattern detection | Repeated similar queries from same user/IP | >10 similar extraction-pattern queries per session |
| Hallucination complaints | Reports of false personal information | Any confirmed report about real individuals |
Cerebrum AI Labs — LLM Output Privacy Controls
Architecture
User Input → Input Filter → LLM → Output PII Scanner → Delivery
│ │
▼ ▼
Prompt Injection PII Detected?
Classifier │
│ YES → Redact + Log + Alert
Block/Allow NO → Deliver + Log
PII Detection Categories for Output Scanning
| PII Type | Detection Method | Action on Detection |
|---|---|---|
| Email addresses | Regex + validation | Redact and log |
| Phone numbers | Regex + country format validation | Redact and log |
| National ID numbers | Country-specific regex patterns | Block output + alert |
| Credit card numbers | Luhn algorithm + regex | Block output + alert |
| Physical addresses | NER + geocoding validation | Redact and log |
| Person names + context | NER + co-occurrence with other PII | Flag for review |
| Medical information | Medical NER + PHI patterns | Block output + alert |
Enforcement Relevance
- EDPB Guidelines 04/2025: Controllers deploying LLMs must assess memorisation risk and implement measures to prevent unlawful disclosure of personal data in outputs.
- Garante v. OpenAI (2023): Italian DPA ordered temporary ban partly due to lack of age verification and data accuracy concerns in ChatGPT outputs. Required implementation of output filtering.
- EDPB ChatGPT Taskforce Report (2024): Recommended that LLM providers implement technical measures to minimise inaccurate personal data in outputs and provide effective rectification mechanisms.
- CNIL AI Action Plan (2024): French DPA guidance emphasises that LLM outputs containing personal data constitute processing, requiring lawful basis and compliance with data quality principles.
Integration Points
- ai-model-privacy-audit: Memorisation testing is part of the model privacy audit
- ai-data-subject-rights: Rectification of hallucinated data, erasure requests
- ai-transparency-reqs: Disclosure that system is AI-generated, accuracy limitations
- ai-deployment-checklist: Output privacy controls verified before deployment
Sponsors
Details
- Marketplace
- mukul975/Privacy-Data-Protection-Skills
- Source
- plugins/ai-privacy-governance-skills
- Author
- Mahipal
- Category
- Compliance
- License
- Apache-2.0
- Keywords
- ai-privacy, ai-governance, ai-dpia, llm-privacy, ai-bias
More from mukul975/Privacy-Data-Protection-Skills
All 21- children-privacy-skillsv1.0.0
- consent-management-skillsv1.0.0
- cookie-consent-skillsv1.0.0
- cross-border-transfers-skillsv1.0.0
- data-breach-response-skillsv1.0.0
- data-classification-skillsv1.0.0
- data-retention-skillsv1.0.0
- data-subject-rights-skillsv1.0.0
- employee-privacy-skillsv1.0.0
- gdpr-compliance-skillsv1.0.0
- global-privacy-regulations-skillsv1.0.0
- healthcare-privacy-skillsv1.0.0
- privacy-audit-skillsv1.0.0
- privacy-by-design-skillsv1.0.0
- privacy-engineering-skillsv1.0.0
- privacy-impact-assessment-skillsv1.0.0
- privacy-skills-completev1.0.0
- records-of-processing-skillsv1.0.0
- us-state-privacy-skillsv1.0.0
- vendor-privacy-management-skillsv1.0.0