mukul975 avatar

ai-privacy-governance-skills

15 AI privacy skills: AI DPIA, training data compliance, bias assessment, transparency, LLM output privacy, federated learning

version
1.0.0
GitHub stars
340

Install

/plugin install ai-privacy-governance-skills@privacy-data-protection-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add mukul975/Privacy-Data-Protection-Skills.

Contents

Discovered from source

Skills 15

EU AI Act High-Risk AI System Documentation

Overview

The EU AI Act (Regulation 2024/1689, entered into force 1 August 2024, with high-risk obligations applicable from 2 August 2026) establishes a risk-based regulatory framework for artificial intelligence systems. High-risk AI systems — those listed in Annex III or used as safety components of products covered by Union harmonisation legislation in Annex I — must meet extensive documentation, transparency, and governance requirements before being placed on the EU market. Cerebrum AI Labs must prepare comprehensive technical documentation, implement a risk management system, ensure data governance, and undergo conformity assessment for each high-risk AI system.

High-Risk Classification

Annex III Categories Relevant to Cerebrum AI Labs

CategoryAnnex III ReferenceCerebrum AI Labs SystemClassification
Employment and workers managementAnnex III, para. 4(a)CV Screening AI — automated filtering of job applicationsHigh-risk
Access to essential servicesAnnex III, para. 5(b)Credit Scoring AI — creditworthiness assessment for financial productsHigh-risk
Law enforcementAnnex III, para. 6(a)Not applicableN/A
Biometric identificationAnnex III, para. 1(a)Facial Verification AI — identity verification at onboardingHigh-risk
Education and vocational trainingAnnex III, para. 3(a)Not applicableN/A

Classification Decision Tree

Is the AI system listed in Annex III?
├── Yes → High-risk (unless exception applies under Art. 6(3))
│   └── Does the system make decisions materially affecting natural persons?
│       ├── Yes → High-risk confirmed
│       └── No → May qualify for Art. 6(3) exception (narrow, profiling, preparatory)
│
├── No → Is it a safety component of a product under Annex I legislation?
│   ├── Yes → High-risk (subject to third-party conformity assessment)
│   └── No → Not high-risk under AI Act
│
└── Is it a general-purpose AI model with systemic risk? (Art. 51)
    ├── Yes → GPAI systemic risk obligations
    └── No → GPAI transparency obligations only

Technical Documentation Requirements (Art. 11)

Annex IV — Required Documentation Content

Section 1: General Description

Document ElementContent for Cerebrum AI Labs CV Screening AI
Intended purposeAutomated screening and ranking of job applications based on qualification match
Provider name and contactCerebrum AI Labs, 42 Innovation Drive, Dublin, Ireland
AI system versionv2.4.1 (deployed March 2026)
Hardware/software requirementsCloud-hosted on EU infrastructure (AWS eu-west-1), Python 3.11, PyTorch 2.2
Product integrationIntegrated into Cerebrum TalentFlow ATS platform

Section 2: Detailed Description of System Elements

ElementDocumentation Required
Development methodologyModel architecture, training approach, design choices and rationale
Computational resourcesTraining compute (GPU hours), energy consumption
Training dataData sources, collection methods, size, labeling methodology, preprocessing
Validation and testingTest datasets, metrics, results, known limitations
Input data specificationsExpected input format, quality requirements
Output descriptionOutput format, confidence scores, decision thresholds

Section 3: Monitoring, Functioning, and Control

ElementDocumentation Required
Human oversight measuresArt. 14 requirements: override capability, decision review process
Technical measures for accuracyAccuracy metrics, drift detection, retraining triggers
Cybersecurity measuresData encryption, access controls, adversarial robustness testing
Performance in edge casesKnown failure modes, boundary conditions, degradation behavior

Section 4: Risk Management

ElementDocumentation Required
Risk management systemArt. 9 risk management process documentation
Known and foreseeable risksRisk register with severity and likelihood
Mitigation measuresControls for each identified risk
Residual risk assessmentAcceptable residual risk justification

Risk Management System (Art. 9)

Continuous Risk Management Process for Cerebrum AI Labs

PhaseActivityFrequency
IdentificationIdentify risks to health, safety, and fundamental rightsInitial + quarterly
AnalysisEstimate risk severity and likelihoodInitial + quarterly
EvaluationCompare risks against acceptance criteriaInitial + quarterly
MitigationImplement risk reduction measuresOngoing
MonitoringTrack risk indicators in productionContinuous
ReviewReview and update risk assessmentQuarterly

Risk Register — CV Screening AI

Risk IDRisk DescriptionSeverityLikelihoodMitigationResidual Risk
R-001Gender bias in screening recommendationsHighMediumBias testing on protected attributes, debiasing training dataLow
R-002Discrimination against non-native language speakersHighMediumMultilingual evaluation, language-agnostic featuresMedium
R-003Over-reliance on AI recommendations by recruitersMediumHighMandatory human review, confidence thresholdsLow
R-004Inaccurate qualification matching for novel job rolesMediumMediumFallback to keyword matching, human review flagLow
R-005Privacy breach via training data memorizationHighLowDifferential privacy in training, memorization auditLow

Data Governance (Art. 10)

Training Data Requirements

RequirementImplementation at Cerebrum AI Labs
Relevance and representativenessTraining data sourced from 50,000 job applications across 12 EU countries, balanced by gender, age, nationality
Bias examinationStatistical parity analysis on protected attributes (gender, age, ethnicity, disability) before and after training
Gap identificationIdentified underrepresentation of applicants with disabilities; augmented with synthetic examples
Data qualityAutomated data quality checks: completeness >95%, label accuracy >98% (human-verified sample)
Personal data processingDPIA completed (DPIA-AI-2026-001); lawful basis: Art. 6(1)(f) legitimate interest; special categories removed

Conformity Assessment (Art. 43)

Assessment Procedure for Cerebrum AI Labs

SystemAssessment TypeBasis
CV Screening AIInternal conformity assessment (Art. 43(2)) + quality management systemAnnex III, para. 4 — not biometric, not critical infrastructure
Credit Scoring AIInternal conformity assessment (Art. 43(2))Annex III, para. 5
Facial Verification AIThird-party conformity assessment (Art. 43(1)) via notified bodyAnnex III, para. 1 — biometric identification

Internal Conformity Assessment Steps

  1. Verify quality management system is established (Art. 17)
  2. Prepare technical documentation per Annex IV
  3. Conduct risk management assessment per Art. 9
  4. Verify data governance per Art. 10
  5. Verify transparency and information provision per Art. 13
  6. Verify human oversight per Art. 14
  7. Verify accuracy, robustness, and cybersecurity per Art. 15
  8. Draw up EU Declaration of Conformity (Art. 47)
  9. Affix CE marking (Art. 48)
  10. Register in EU database (Art. 49)

Post-Market Monitoring (Art. 72)

ActivityFrequencyOwner
Performance metric monitoringContinuousML Engineering
Bias drift detectionWeeklyResponsible AI team
Incident reportingAs needed (within 15 days for serious incidents per Art. 73)DPO + Legal
User feedback collectionContinuousProduct team
Risk register updateQuarterlyRisk Management
Technical documentation updateOn material changeML Engineering + Legal

Key Legal References

  • EU AI Act (Regulation 2024/1689) — Full text, entered into force 1 August 2024
  • AI Act Art. 6 + Annex III — High-risk classification criteria
  • AI Act Art. 9 — Risk management system requirements
  • AI Act Art. 10 — Data and data governance requirements
  • AI Act Art. 11 + Annex IV — Technical documentation requirements
  • AI Act Art. 13 — Transparency and provision of information to deployers
  • AI Act Art. 14 — Human oversight requirements
  • AI Act Art. 43 — Conformity assessment procedures
  • AI Act Art. 72 — Post-market monitoring obligations
  • AI Act Art. 73 — Reporting of serious incidents
  • GDPR Art. 22 — Automated individual decision-making (complementary to AI Act)
  • EDPB-EDPS Joint Opinion 5/2021 on the AI Act Proposal — Data protection perspective on AI regulation

AI Automated Decision-Making and Human Oversight

Overview

GDPR Article 22 grants data subjects the right not to be subject to decisions based solely on automated processing, including profiling, which produce legal or similarly significant effects. The EU AI Act Art. 14 supplements this with specific human oversight design requirements for high-risk AI systems. Together, these provisions require organisations to identify when AI systems make consequential decisions, ensure meaningful human intervention where required, provide explainable decision logic, and offer effective contestation mechanisms. This skill provides the complete framework for Art. 22 compliance and AI Act human oversight implementation.

Art. 22 Scope and Applicability

Three Cumulative Conditions

Art. 22(1) is triggered only when all three conditions are met:

ConditionRequirementAI Application
1. DecisionA decision is made (not merely a recommendation or input)The AI output directly determines an outcome — no genuine human decision-making step between AI output and action
2. Solely automatedBased solely on automated processing including profilingNo meaningful human intervention in the decision chain; rubber-stamping does not constitute human intervention
3. Legal/significant effectsProduces legal effects or similarly significantly affects the data subjectAffects legal rights, contractual status, access to services, financial outcomes, or other significant life impacts

"Solely Automated" — EDPB Interpretation

The EDPB Guidelines 06/2020 on automated decision-making clarify:

  • Solely automated means no meaningful human involvement in the decision process

  • A human who merely confirms an AI recommendation without genuine assessment is not providing meaningful intervention

  • Meaningful human intervention requires:

    • The reviewer has authority and competence to change the decision
    • The reviewer has access to all relevant information
    • Sufficient time is allocated for genuine consideration
    • The reviewer routinely exercises independent judgment (not just confirming AI output)
    • Override capability is actually used in practice
  • Not solely automated when:

    • A qualified human genuinely reviews the AI output as one input among several
    • The human applies independent judgment and makes the final decision
    • The human has the capability and actually exercises discretion to deviate from AI recommendations

"Legal or Similarly Significant Effects"

CategoryExamplesSignificance
Legal effectsContract formation/termination, legal obligation imposition, legal status determinationDirectly affects legal rights
Access to servicesDenial of credit, insurance, housing, education, employmentSignificantly affects life circumstances
Financial impactPricing discrimination, benefit calculation, payment termsMaterial financial consequences
Health and safetyMedical diagnosis prioritisation, emergency response triagePotential physical harm
Freedom and autonomySurveillance scoring, movement restriction, content blockingAffects fundamental freedoms

Effects that are not similarly significant (per EDPB):

  • Personalised advertising (unless it reinforces prejudices about vulnerable groups)
  • Generic recommendation systems (unless they determine access to information)
  • Spam filtering (unless it systematically blocks important communications)

Art. 22 Exceptions and Safeguards

Art. 22(2) — Permitted Automated Decisions

ExceptionConditionRequired Safeguards
Art. 22(2)(a) — Contract necessityDecision is necessary for entering into or performance of a contractArt. 22(3) safeguards required
Art. 22(2)(b) — Law authorisationAuthorised by Union or Member State law with suitable measuresLaw must provide suitable safeguards
Art. 22(2)(c) — Explicit consentBased on explicit consentArt. 22(3) safeguards required

Art. 22(3) — Mandatory Safeguards

When an Art. 22(2) exception is relied upon, the controller must implement at least:

  1. Right to obtain human intervention: A qualified person reviews the automated decision
  2. Right to express point of view: Data subject can present additional information or context
  3. Right to contest: Formal mechanism to challenge the automated decision with review by a different decision-maker

Art. 22(4) — Special Category Data Restriction

Automated decisions based on Art. 9 special category data are only permitted under:

  • Art. 9(2)(a) — Explicit consent, OR
  • Art. 9(2)(g) — Substantial public interest based on Union or Member State law

In both cases, suitable measures to safeguard data subject rights must be in place.

Human Oversight Design — AI Act Art. 14

Art. 14 Requirements for High-Risk AI

High-risk AI systems must be designed and developed so that they can be effectively overseen by natural persons during use:

RequirementImplementation
Understand capabilities and limitationsDocumentation, training, model cards
Monitor operationReal-time monitoring dashboards, alert systems
Detect anomalies and dysfunctionDrift detection, performance monitoring
Interpret outputs correctlyConfidence indicators, explanation tools
Override or reverse decisionsOverride mechanism with authority chain
Intervene or stop the systemEmergency stop capability
Be aware of automation biasTraining on automation bias, countermeasures

Human Oversight Levels

LevelDescriptionArt. 22 ComplianceAppropriate When
Human-in-the-loop (HITL)Human reviews every AI recommendation before decisionFully compliant if review is meaningfulHigh-stakes individual decisions (hiring, credit, medical)
Human-on-the-loop (HOTL)Human monitors AI decisions and can interveneCompliant if intervention capability is genuine and exercisedMedium-risk decisions with effective monitoring
Human-in-command (HIC)Human sets parameters and reviews outcomes periodicallyMay not satisfy Art. 22 — decision is solely automatedLow-risk bulk decisions with periodic audit
Fully autonomousNo human oversight of individual decisionsArt. 22 applies fully — exception neededOnly where Art. 22(2) exception applies with Art. 22(3) safeguards

Meaningful Human Intervention Criteria

A human review qualifies as "meaningful intervention" when all criteria are met:

CriterionTestRed Flag
AuthorityReviewer has formal authority to override AIReviewer can only escalate, not decide
CompetenceReviewer has domain expertise to evaluate the decisionReviewer is a junior staff member without training
InformationReviewer has access to all inputs, the AI output, and explanationReviewer sees only AI score with no context
TimeSufficient time allocated for genuine considerationReviewer processes 200+ decisions per hour
IndependenceReviewer exercises genuine judgmentOverride rate is < 1% suggesting rubber-stamping
AccountabilityReviewer is accountable for the decisionAccountability rests with the AI system owner, not reviewer

Contestation and Appeal Mechanism

Design Requirements

ElementRequirement
AccessibilityContestation mechanism is easy to find, access, and use
TimelinessDefined response timeframe (e.g., 30 days)
Qualified reviewerDifferent from the original decision context; has authority to overturn
Information provisionData subject receives explanation of decision factors and how to contest
Evidence considerationData subject can submit additional evidence and context
Written outcomeDecision on contestation is documented and communicated
Further appealIf contestation is denied, path to DPA complaint or judicial remedy is indicated

Contestation Workflow

  1. Data subject notified of automated decision with explanation
  2. Data subject submits contestation with reasons and any supporting evidence
  3. Contestation assigned to qualified human reviewer (different from original oversight)
  4. Reviewer assesses: AI inputs, AI output, explanation, data subject's arguments
  5. Reviewer makes independent decision: uphold, modify, or overturn
  6. Outcome communicated to data subject with reasons
  7. If upheld: data subject informed of further options (DPA complaint, judicial remedy)
  8. Record maintained in rights exercise register

AI-Specific Decision Categories

Credit and Financial Decisions

  • AI application: Credit scoring, loan approval, insurance pricing, fraud detection
  • Art. 22 trigger: Yes — determines access to financial services
  • Required: Meaningful human review before denial; explanation of key scoring factors; contestation with independent review
  • Enforcement: AEPD fined CaixaBank EUR 6M for automated credit decisions without adequate safeguards

Employment Decisions

  • AI application: CV screening, candidate ranking, performance scoring, termination prediction
  • Art. 22 trigger: Yes — significantly affects employment and livelihood
  • Required: Human hiring manager makes final decision with genuine authority to deviate from AI ranking; applicants informed of AI use; rejected candidates can request explanation
  • Enforcement: Italian DPA fined Deliveroo EUR 2.5M for algorithmic worker management without Art. 22 safeguards

Healthcare Decisions

  • AI application: Diagnosis assistance, treatment recommendations, triage, risk scoring
  • Art. 22 trigger: Yes if AI determines care pathway — usually mitigated by physician oversight
  • Required: Physician makes final clinical decision; AI functions as decision support; patient informed of AI role
  • Special category: Health data — Art. 22(4) applies

Public Administration

  • AI application: Benefit eligibility, fraud detection, risk assessment, resource allocation
  • Art. 22 trigger: Yes — determines access to public services and benefits
  • Required: Art. 22(2)(b) legal basis required; suitable measures mandated by law; transparency about algorithmic criteria
  • Enforcement: Dutch court struck down SyRI fraud detection system for lack of transparency and proportionality

Profiling Assessment

GDPR Definition of Profiling (Art. 4(4))

Any form of automated processing to evaluate personal aspects relating to a natural person, in particular to analyse or predict:

  • Work performance
  • Economic situation
  • Health
  • Personal preferences
  • Interests
  • Reliability
  • Behaviour
  • Location
  • Movements

AI Profiling Risk Assessment

Profiling TypeRisk LevelArt. 22 TriggerMitigation
Behavioural prediction (purchasing, browsing)MediumOnly if decision with legal/significant effectOpt-out, transparency
Credit scoring / financial riskHighYes — access to financial servicesHuman review, explanation, contestation
Health risk predictionVery HighYes — Art. 22(4) appliesExplicit consent, physician oversight
Criminal risk assessmentVery HighYes — liberty and legal effectsLegal basis required, judicial oversight
Employment performance scoringHighYes — employment effectsHR human review, employee notification
Social scoringProhibitedN/A — AI Act Art. 5 prohibitionDo not implement

Enforcement Precedents

  • AEPD v. CaixaBank (PS/00421/2020, 2021): EUR 6M fine for automated credit decision-making without adequate Art. 22 safeguards, explanation, or contestation mechanism.
  • Italian DPA v. Deliveroo (2021): EUR 2.5M fine for algorithmic management of delivery riders — Art. 22 applied to automated work allocation and performance scoring.
  • Dutch Court v. SyRI (2020): Algorithmic fraud detection system struck down — automated profiling of citizens without proportionality, transparency, or adequate safeguards.
  • Hungarian DPA v. Bank (2019): Fine for automated credit denial without providing meaningful information about decision logic or contestation mechanism.
  • French Conseil d'Etat v. Parcoursup (2019): Court upheld use of algorithm for university admissions only because meaningful human review was genuinely conducted for each application.
  • Austrian DPA v. CRIF (2023): Credit scoring company — violation of Art. 15(1)(h) for failing to provide meaningful information about automated scoring logic.

Integration Points

  • ai-transparency-reqs: Explanation requirements feed into transparency framework
  • ai-dpia: Human oversight assessment is DPIA Phase 5 component
  • ai-data-subject-rights: Right to explanation and contestation are rights exercise procedures
  • ai-deployment-checklist: Art. 22 compliance is pre-deployment validation item
  • ai-bias-special-category: Bias in automated decisions creates Art. 22 harm

AI Bias Assessment for Special Category Data

Overview

AI systems can amplify, perpetuate, or introduce bias against protected groups defined by GDPR Art. 9 special categories (race, ethnicity, political opinion, religion, trade union membership, genetic data, biometric data, health, sexual orientation) and by EU equality law (gender, age, disability). The AI Act Art. 10 requires data governance practices for training data that address bias, while Art. 5 prohibits AI-based social scoring. This skill provides the methodology for detecting, measuring, and mitigating bias in AI systems that process or infer special category data, with documentation requirements meeting both GDPR and AI Act obligations.

Art. 9 Special Categories and AI Bias

Direct Processing of Special Category Data

When AI systems directly process Art. 9 data:

CategoryAI Bias RiskExample
Racial or ethnic originDiscrimination in hiring, credit, policingCV screening penalising names associated with ethnic minorities
Political opinionsPolitical profiling, content suppressionNews recommendation amplifying or suppressing political viewpoints
Religious beliefsService denial, discriminatory targetingInsurance pricing varying by religious affiliation
Trade union membershipEmployment discriminationPerformance scoring penalising union activity
Genetic dataGenetic discrimination in insurance/employmentHealth insurance pricing based on genetic predisposition
Biometric dataDifferential accuracy across demographicsFacial recognition with higher error rates for darker skin tones
Health dataHealth-based discriminationHiring algorithms penalising disability or mental health history
Sexual orientationDiscrimination, outingContent recommendation inadvertently revealing sexual orientation

Proxy Inference of Special Categories

AI models frequently infer Art. 9 data from non-sensitive features:

Proxy FeatureInferred CategoryMechanism
Postcode/zip codeRace/ethnicity, incomeResidential segregation patterns
First/last nameRace/ethnicity, religionName-ethnicity correlations
Browsing historyPolitical opinion, religion, healthContent consumption patterns
Purchase historyHealth status, religionMedication purchases, dietary products
Language patternsNational origin, educationDialect, vocabulary, grammar patterns
Device/app usageAge, income, disabilityAccessibility features, device type

EDPB position: inferring Art. 9 data from non-sensitive inputs constitutes processing of special category data — the same protections apply.

Fairness Metrics

Group Fairness Metrics

MetricDefinitionWhen to Use
Demographic parityP(positive outcomegroup A) = P(positive outcome
Equalized oddsTPR and FPR equal across groupsWhen accuracy should be equal across groups
Equal opportunityTPR equal across groups (relaxed equalized odds)When true positive detection should be equal
CalibrationP(Y=1score=s, group=A) = P(Y=1
Predictive parityPPV equal across groupsWhen positive predictions should be equally reliable

Individual Fairness Metrics

MetricDefinition
ConsistencySimilar individuals receive similar outcomes
Counterfactual fairnessOutcome would be the same if protected attribute were different
Causal fairnessNo causal path from protected attribute to outcome

Metric Selection Guidance

Decision ContextRecommended MetricJustification
Hiring/admissionsEqualized odds or equal opportunityEqual detection of qualified candidates across groups
Credit scoringCalibrationScore should mean the same probability regardless of group
Criminal riskEqualized oddsBoth FPR and TPR should be equal to avoid disproportionate impact
HealthcareEqual opportunity + calibrationEqual detection of conditions; equal meaning of risk scores
Content moderationDemographic parityContent removal should not disproportionately affect groups

Note: Mathematical impossibility results show that demographic parity, equalized odds, and calibration cannot all be satisfied simultaneously when base rates differ across groups. Document the trade-off explicitly.

Bias Detection Methodology

Phase 1: Data Audit

  1. Profile training data demographics against target population
  2. Identify underrepresented groups that may have lower model performance
  3. Check for historical bias in labelled data (e.g., biased hiring decisions used as ground truth)
  4. Identify label noise differential across groups
  5. Assess feature distributions across groups for proxy discrimination potential

Phase 2: Model Testing

  1. Evaluate model performance disaggregated by protected groups
  2. Compute selected fairness metrics for each protected group pair
  3. Run counterfactual testing: change protected attributes, observe output changes
  4. Test for intersectional bias (combinations of protected attributes)
  5. Evaluate model performance on edge cases and adversarial examples per group

Phase 3: Output Analysis

  1. Analyse decision distribution across groups
  2. Identify threshold effects that disproportionately impact specific groups
  3. Test for feedback loop amplification over time
  4. Assess output explanation fairness (are explanations equally informative across groups?)

Bias Mitigation Strategies

Pre-processing (Data-Level)

StrategyDescriptionTrade-off
ResamplingOver-sample underrepresented groups, under-sample overrepresentedMay reduce data diversity or introduce duplicates
ReweightingAssign higher weights to underrepresented group samplesComputationally simple; may not address structural bias
RelabellingCorrect historically biased labelsRequires domain expertise; may be subjective
Fair representation learningLearn latent representation that removes protected attribute informationMay lose legitimate correlations

In-processing (Algorithm-Level)

StrategyDescriptionTrade-off
Adversarial debiasingTrain adversary to predict protected attribute from model; penalise successAccuracy-fairness trade-off; requires protected attribute data
Fairness constraintsAdd fairness metric as training constraintMay reduce overall accuracy; constraint satisfaction varies
RegularisationAdd fairness-related regularisation term to loss functionBalances accuracy and fairness; requires tuning
Causal modellingUse causal graph to block discriminatory pathsRequires causal knowledge; complex to implement

Post-processing (Output-Level)

StrategyDescriptionTrade-off
Threshold adjustmentDifferent decision thresholds per group to equalise metricsMay be perceived as unfair; legally complex
Score calibrationCalibrate scores per groupRequires sufficient group data; may reduce discrimination
Reject optionAbstain from decision for borderline cases across groupsReduces coverage; requires human fallback

AI Act Art. 10 Data Governance

Art. 10 requires for high-risk AI training data:

RequirementImplementation
Relevant dataTraining data must be relevant to the intended purpose
Sufficiently representativeData must represent the population the system will be deployed on
Free of errorsData quality assessment and cleaning processes
CompleteSufficient coverage of deployment scenarios
Appropriate statistical propertiesDistribution analysis including demographic representation
Bias examinationExamine training data for possible biases, especially related to Art. 10(2)(f)

Art. 10(5): Processing of special category data for bias detection is permitted for high-risk AI if:

  • Strictly necessary for bias monitoring, detection, and correction
  • Subject to appropriate safeguards (pseudonymisation, access controls)
  • Data is not used for other purposes
  • Deleted after bias assessment unless retention is required for compliance documentation

Documentation Requirements

Bias Assessment Report

SectionContent
System descriptionModel, purpose, affected groups
Protected attributes assessedArt. 9 categories + equality law characteristics
Fairness metrics selectedWith justification for selection
Data audit resultsTraining data demographics, representation gaps
Model testing resultsPer-group performance, fairness metrics, counterfactual results
Bias findingsIdentified disparities with severity assessment
Mitigation measuresApplied strategies with effectiveness evidence
Residual biasRemaining disparities after mitigation
Trade-off documentationAccuracy-fairness trade-offs, metric impossibility acknowledgement
Ongoing monitoring planPost-deployment fairness monitoring

Enforcement Precedents

  • Dutch Tax Authority (SyRI, 2020): Court struck down algorithmic fraud detection for discriminatory profiling — system disproportionately targeted residents of low-income, immigrant-background neighbourhoods.
  • Italian DPA v. Deliveroo (2021): Algorithmic worker management found to discriminate based on protected characteristics — Art. 22 and equality law violations.
  • Austrian DPA v. AMS Algorithm (2020): Austrian employment service algorithm that scored job seekers lower based on gender, age, disability, and citizenship — DPA found processing unlawful.
  • AEPD guidance (2021): Spanish DPA issued guidance on algorithmic discrimination, emphasising DPIA requirement for AI systems processing protected characteristics.
  • CNIL (2024): AI bias assessment framework published requiring fairness testing for high-impact AI systems.

Integration Points

  • ai-dpia: Bias assessment feeds into DPIA risk assessment for discrimination harms
  • ai-automated-decisions: Biased automated decisions trigger Art. 22 and equality law violations
  • ai-training-lawfulness: Art. 10(5) processing of special category data for bias detection requires documentation
  • ai-transparency-reqs: Bias findings should be disclosed in model cards and transparency documentation
  • ai-act-high-risk-docs: Art. 10 data governance documentation is part of conformity assessment

AI Model Retention and Unlearning

Overview

GDPR Art. 5(1)(e) storage limitation requires that personal data be kept no longer than necessary for the processing purpose. For AI systems, this creates complex retention challenges: training data used to build a model may no longer be needed once training is complete, but the model itself encodes information about the training data. Machine unlearning — the process of removing the influence of specific data from a trained model — is an emerging field that addresses the gap between deleting training data and eliminating its influence from model parameters. This skill provides retention policies, deletion verification methods, and machine unlearning techniques for AI compliance.

AI Data Retention Categories

Data CategoryDescriptionRetention Consideration
Raw training dataOriginal personal data used for model trainingDelete after training unless retraining justifies retention
Processed training dataCleaned, augmented, feature-engineered dataSame as raw — delete when training purpose exhausted
Validation/test dataData used for model evaluationRetain for model audit and comparison; pseudonymise
Model weights/parametersTrained model artefacts encoding training data informationRetain while model is deployed; delete on decommission
Inference logsInputs and outputs of model predictionsRetention based on purpose (audit, debugging, rights exercise)
Model metadataTraining configuration, hyperparameters, provenanceRetain for compliance documentation; low privacy risk
Embedding vectorsDense representations derived from personal dataMay contain personal data — apply retention policy

Retention Policy Framework

Training Data Retention Decision Tree

Training data category identified
│
├─ Is the data still needed for model retraining?
│  ├─ YES → Retain with documented justification and review date
│  └─ NO → Continue
│
├─ Is the data needed for model validation or audit?
│  ├─ YES → Retain in pseudonymised form with access controls
│  └─ NO → Continue
│
├─ Is the data needed for data subject rights exercise?
│  ├─ YES → Retain for rights exercise period, then delete
│  └─ NO → Continue
│
├─ Is there a legal obligation to retain?
│  ├─ YES → Retain per legal requirement
│  └─ NO → DELETE the training data
│
└─ After deletion: assess model for residual data encoding

Model Lifecycle Retention

PhaseRetention Rule
DevelopmentTraining data retained during active development
DeploymentTraining data deleted unless retraining is planned within defined period
OperationInference logs retained per purpose (30 days debug, 1 year audit)
RetrainingNew training data collected; old data deleted post-training
DecommissionAll model artefacts, training data, and logs deleted; retain only compliance documentation

Machine Unlearning Techniques

Exact Unlearning

Full retraining: Retrain the model from scratch on the dataset minus deleted records.

PropertyValue
GuaranteeComplete — model has no knowledge of deleted data
CostVery high — full training cost for each deletion request
FeasibilityImpractical for large models or frequent deletion requests
When to useSmall models, infrequent requests, high-sensitivity data

SISA (Sharded, Isolated, Sliced, Aggregated) Training

Train model on sharded data partitions. To unlearn, retrain only the affected shard.

PropertyValue
GuaranteeExact within the affected shard
Cost1/k of full retraining (k = number of shards)
FeasibilityRequires SISA architecture from the start
Trade-offModel accuracy may decrease with fewer shards contributing

Approximate Unlearning

Gradient-Based Unlearning

Apply gradient ascent on the data to be forgotten, then fine-tune on remaining data.

PropertyValue
GuaranteeApproximate — statistically similar to retrained model
CostLow — few gradient steps
FeasibilityWorks for most differentiable models
VerificationRequires membership inference testing to verify
Influence Function-Based Unlearning

Use influence functions to estimate the effect of removing data and adjust model accordingly.

PropertyValue
GuaranteeApproximate — first-order approximation
CostMedium — requires Hessian computation
FeasibilityBest for smaller models or linear models

Unlearning Verification

After applying unlearning, verify effectiveness:

  1. Membership inference test: Run MI attack on unlearned records — should classify as non-members
  2. Output comparison: Compare model outputs with and without the unlearned data
  3. Canary test: If canary records were included, verify they are no longer extractable
  4. Statistical test: Compare model to one retrained from scratch on the same data minus deleted records

Model Versioning for Compliance

Version Control Requirements

ElementDocumentation
Model version IDUnique identifier (e.g., model-v2.3.1-20260314)
Training data snapshotHash of training dataset used for this version
Training dateWhen training was executed
Data deletions appliedWhich data subject deletions are reflected in this version
Unlearning appliedAny approximate unlearning applied since last full retraining
Privacy propertiesDP epsilon, MI test results for this version
Deployment datesWhen deployed and when retired

Retraining Triggers

TriggerAction
Accumulated deletion requests exceed thresholdFull retraining on updated dataset
Scheduled periodic retrainingIncorporate all pending deletions
Privacy audit reveals unacceptable leakageRetrain with enhanced privacy measures
Model performance degradationRetrain with current data (post-deletions)
Regulatory changeAssess if retraining needed for compliance

Enforcement Relevance

  • EDPB Guidelines 04/2025: Training data retention must be justified; deletion of training data does not automatically eliminate GDPR obligations for the model if it encodes personal data.
  • Garante v. OpenAI (2023): Required mechanism for data deletion from training data; acknowledged technical challenges but expected good faith effort.
  • EDPB ChatGPT Taskforce (2024): Controllers must demonstrate capability to address erasure requests affecting training data.

Integration Points

  • ai-data-subject-rights: Erasure rights implementation requires unlearning
  • ai-dpia: Retention and deletion capability assessed in DPIA
  • ai-model-privacy-audit: Audit verifies deletion effectiveness
  • ai-training-lawfulness: Retention justification part of lawful basis assessment

Data Subject Rights for AI Systems

Overview

AI systems create unique challenges for data subject rights exercise. Traditional rights mechanisms designed for structured databases do not map directly to ML model architectures where personal data is encoded in model weights, reproduced in model outputs, or used in opaque decision processes. This skill provides the framework for implementing each GDPR right (Arts. 15-22) and the AI Act Art. 86 right to explanation in the context of AI processing, addressing both training-time and inference-time rights.

Rights Framework for AI

Right of Access (Art. 15)

AI ContextObligationImplementation
Training data contributionConfirm whether data subject's data was in training set; provide copy if feasibleTraining data catalogue indexed by data subject identifier; membership query
AI inference inputsProvide data used as input to AI decisionLog inference inputs with data subject linkage
AI inference outputsProvide AI decision/score/classification affecting data subjectDecision logging with data subject ID
Logic explanationArt. 15(1)(h): meaningful information about logic of automated decisionsSHAP/LIME explanation on request or system-level explanation
Training data sourceArt. 14(2)(f): source of data if not collected from data subjectTraining data provenance documentation

Technical Challenges:

  • Identifying specific records in massive training datasets
  • Determining if a data subject's data is in the training set without running membership inference
  • Providing meaningful logic explanation for complex models

Right to Rectification (Art. 16)

AI ContextObligationImplementation
Training data correctionCorrect inaccurate data in training datasetUpdate training data; assess if model retraining needed
Model output correctionCorrect inaccurate AI outputs about a data subjectOutput correction mechanism; flag in decision system
Inference input correctionCorrect data used as inference inputUpdate input data; re-run inference

Technical Challenge: Correcting training data may require model retraining to propagate the correction. For deployed models, correction may need model update or retraining pipeline.

Right to Erasure (Art. 17)

AI ContextObligationImplementation
Training data deletionDelete data subject's data from training datasetRemove from training data; assess model impact
Model unlearningRemove influence of deleted data from trained modelMachine unlearning technique or full retraining
Inference logsDelete inference inputs and outputs linked to data subjectPurge from decision logs
Model outputsDelete generated content about the data subjectContent removal mechanism

Technical Challenge: True erasure from a trained model requires either verified machine unlearning or complete model retraining. The EDPB acknowledges this challenge but expects controllers to demonstrate good faith effort and use best available techniques.

Right to Restriction (Art. 18)

AI ContextObligation
Contested accuracyRestrict processing while accuracy of training data or AI output is verified
Unlawful processingRestrict rather than delete if data subject requests
During objection assessmentRestrict while controller assesses whether legitimate grounds override

Implementation: Quarantine data subject's data from training pipeline and inference pipeline during restriction period.

Right to Data Portability (Art. 20)

AI ContextObligation
Training data contributionProvide data subject's training data contribution in structured, machine-readable format
AI-generated profileIf AI has created a profile, provide in portable format
Inference historyProvide history of AI decisions affecting data subject

Right to Object (Art. 21)

AI ContextObligation
AI training objectionIf training is based on legitimate interest (Art. 6(1)(f)), data subject can object
Profiling objectionObject to AI profiling including inference of characteristics
Direct marketingAbsolute right to object to AI-driven direct marketing profiling

Upon objection: Controller must cease processing unless compelling legitimate grounds override. For AI training: remove data from training pipeline and assess model impact.

Rights Relating to Automated Decisions (Art. 22)

RightImplementation
Right not to be subjectOpt-out from solely automated AI decisions with legal/significant effects
Right to human interventionQualified human reviewer with authority to override AI
Right to express viewsMechanism for data subject to provide additional context
Right to contestFormal contestation with independent review
Right to explanationAI Act Art. 86 + GDPR Recital 71: clear explanation of AI's role in the decision

AI Act Art. 86 — Right to Explanation

For high-risk AI systems, affected persons have the right to:

  • Clear and meaningful explanations of the role of the AI system in the decision-making procedure
  • The main elements of the decision taken
  • This right is without prejudice to GDPR data subject rights

Operational Implementation

Rights Request Triage for AI

Data subject rights request received
│
├─ Identify if AI processing is involved
│  ├─ Was AI used in any decision affecting the data subject?
│  ├─ Is the data subject's data in any training dataset?
│  └─ Has AI generated any content about the data subject?
│
├─ Determine applicable AI-specific rights
│  ├─ Access: training data, inference records, logic explanation
│  ├─ Rectification: training data or output correction
│  ├─ Erasure: training data deletion, model unlearning
│  ├─ Restriction: quarantine from AI pipeline
│  ├─ Objection: cease AI training on their data
│  └─ Automated decision: explanation, human review, contestation
│
├─ Route to appropriate team
│  ├─ Standard rights: data protection team
│  ├─ AI-specific: data protection + ML engineering team
│  └─ Contestation: independent review panel
│
└─ Process within Art. 12 timeframe (one month, extendable by two)

Response Timeframes

Request TypeStandardExtendedJustification for Extension
Access (standard)1 month3 monthsComplex, voluminous
Access (AI logic)1 month3 monthsRequires technical explanation generation
Rectification1 month3 monthsMay require model retraining
Erasure1 month3 monthsMachine unlearning complexity
Explanation (Art. 86)Without undue delay—Time-critical for affected persons
Contestation30 days90 daysRequires independent review

Enforcement Precedents

  • Austrian DPA v. CRIF (2023): Violation of Art. 15(1)(h) — credit scoring system failed to provide meaningful information about automated decision logic upon access request.
  • Garante v. OpenAI (2023): Required mechanism for data subjects to request correction of inaccurate AI outputs and to object to AI training on their data.
  • Dutch DPA guidance (2024): Right to erasure includes obligation to address erasure from AI training data, even if technically challenging.
  • EDPB ChatGPT Taskforce (2024): Controllers must demonstrate capability to address rights requests affecting training data — technical difficulty does not exempt.

Integration Points

  • ai-automated-decisions: Contestation and human review mechanisms
  • ai-transparency-reqs: Logic explanation fulfils transparency obligations
  • ai-data-retention: Erasure rights intersect with retention and unlearning
  • ai-training-lawfulness: Objection right applies to legitimate interest-based training

AI System Pre-Deployment Privacy Checklist

Overview

Deploying an AI system that processes personal data requires verification of privacy compliance across multiple dimensions before the system goes live. This checklist serves as a compliance gate in the Cerebrum AI Labs ML deployment pipeline. No AI system may be deployed to production until all mandatory items are verified and signed off by the Data Protection Officer (DPO). The checklist is structured around GDPR requirements, the EU AI Act obligations (for high-risk systems), and internal governance standards.

Pre-Deployment Compliance Gate

Gate 1: Legal Basis and DPIA

CheckRequirementStatusEvidence
Lawful basis documentedArt. 6(1) basis identified and recorded for all personal data processingRequiredLIA or consent records
Special categories assessedArt. 9 data identified; explicit consent or Art. 9(2) exception documentedRequiredData classification report
DPIA completedArt. 35 DPIA completed for high-risk processing (profiling, systematic monitoring, large-scale special categories)Required if applicableDPIA document signed by DPO
DPIA risks mitigatedAll high/critical risks from DPIA have documented mitigationsRequiredRisk treatment plan
Prior consultationArt. 36 consultation with supervisory authority if residual risk remains highRequired if applicableConsultation record
Legitimate interest assessmentIf relying on Art. 6(1)(f), LIA balancing test completedRequired if LI basisLIA document

Gate 2: Transparency and Information

CheckRequirementStatusEvidence
Privacy notice updatedArt. 13-14 information includes AI processing detailsRequiredUpdated privacy notice
Logic described"Meaningful information about the logic involved" documented for data subjectsRequired for automated decisionsExplanation document
Significance disclosedEnvisaged consequences of AI processing disclosedRequired for automated decisionsPrivacy notice section
Profiling disclosedIf system profiles individuals, this is disclosed in privacy noticeRequired if profilingPrivacy notice section
AI Act transparencyArt. 52 transparency obligations met (if applicable): inform that they are interacting with AIRequired for AI ActUser interface disclosure

Gate 3: Data Subject Rights

CheckRequirementStatusEvidence
Access process definedProcess for responding to access requests for AI data (inputs, outputs, profiles)RequiredDocumented SOP
Explanation mechanismIndividual explanations can be generated on requestRequired for Art. 22Technical capability verified
Human intervention availableArt. 22(3) human review process establishedRequired for solely automated decisionsProcess document + trained staff
Contestation channelData subjects can contest AI decisions and have them reviewedRequired for Art. 22Appeal process document
Rectification processProcess for correcting AI input data and regenerating outputsRequiredDocumented SOP
Erasure processProcess for deleting data from training sets, inference logs, embeddingsRequiredDocumented SOP

Gate 4: Data Quality and Bias

CheckRequirementStatusEvidence
Training data documentedData sources, size, collection method, preprocessing documentedRequiredData card / dataset documentation
Bias testing completedModel tested for bias across protected attributes (gender, race, age, disability)RequiredBias test report
Fairness metrics acceptableDisparate impact ratio >0.8 (four-fifths rule) or equivalent metric within acceptable rangeRequiredFairness metrics report
Data quality verifiedTraining data completeness, accuracy, representativeness verifiedRequiredData quality report
Art. 9 data removed or justifiedSpecial category data either removed or lawful basis documentedRequiredData classification report

Gate 5: Security and Technical Controls

CheckRequirementStatusEvidence
Data encryption at restTraining data and model weights encrypted (AES-256 or equivalent)RequiredSecurity configuration
Data encryption in transitAll API endpoints use TLS 1.2+RequiredSSL certificate
Access controlsRole-based access to model, training data, and inference logsRequiredIAM policy
Audit loggingAll model invocations logged with timestamp, input hash, output, userRequiredLogging configuration
Adversarial robustnessModel tested against common adversarial attacks relevant to its domainRecommendedSecurity test report
Model versioningModel versioned in registry with rollback capabilityRequiredMLflow / model registry

Gate 6: Monitoring and Governance

CheckRequirementStatusEvidence
Performance monitoringDashboard tracking accuracy, latency, error rates in productionRequiredMonitoring setup
Drift detectionData drift and concept drift detection implementedRequiredDrift monitoring configuration
Bias monitoringPost-deployment bias metrics tracked continuouslyRequiredFairness monitoring dashboard
Incident responseProcess for handling AI-related privacy incidents (e.g., discriminatory output, data leak)RequiredIncident response plan
Retraining scheduleDefined schedule for model retraining with fresh dataRequiredRetraining plan
Retention enforcementAutomated deletion of inference logs and training data per retention scheduleRequiredRetention policy + automation

Gate 7: EU AI Act (High-Risk Systems Only)

CheckRequirementStatusEvidence
Risk classificationSystem classified per Annex IIIRequiredClassification document
Technical documentationAnnex IV documentation completeRequiredTech doc package
Risk management systemArt. 9 continuous risk management implementedRequiredRisk register + process
Conformity assessmentInternal or third-party conformity assessment completedRequiredAssessment report
EU Declaration of ConformityArt. 47 declaration preparedRequiredSigned declaration
EU database registrationArt. 49 registration completedRequiredRegistration confirmation

Sign-Off

RoleNameApprovalDate
ML Engineering Lead[ ] Approved / [ ] Blocked
Data Protection Officer[ ] Approved / [ ] Blocked
Information Security Officer[ ] Approved / [ ] Blocked
Product Owner[ ] Approved / [ ] Blocked
Legal Counsel[ ] Approved / [ ] Blocked (high-risk only)

Key Legal References

  • GDPR Articles 5, 6, 9 — Data processing principles, lawful basis, special categories
  • GDPR Article 22 — Automated individual decision-making safeguards
  • GDPR Article 25 — Data protection by design and by default
  • GDPR Article 35 — Data Protection Impact Assessment
  • EU AI Act Articles 9-15 — High-risk AI system requirements
  • EU AI Act Article 52 — Transparency obligations for certain AI systems
  • EDPB Guidelines on Automated Decision-Making (WP 251 rev.01) — Art. 22 interpretation
  • ISO/IEC 42001:2023 — AI management system standard

Data Protection Impact Assessment for AI/ML Systems

Overview

AI and ML systems present unique privacy challenges that traditional DPIA methodologies fail to adequately address. The EDPB Guidelines 04/2025 on processing personal data through AI systems establish a specialized framework that supplements the general DPIA requirements of GDPR Article 35 and WP248rev.01. AI-specific DPIAs must evaluate the entire ML pipeline — from training data collection through model deployment and inference — assessing risks that emerge from statistical learning, emergent model behaviours, and the opacity of algorithmic decision-making. This skill implements the EDPB's AI-specific DPIA methodology integrated with the EU AI Act risk classification framework.

AI-Specific DPIA Triggers

Mandatory DPIA Triggers for AI Systems

All AI processing that meets any of the following criteria requires a DPIA before deployment:

TriggerLegal BasisDescription
AI-based profiling with legal effectsArt. 35(3)(a) GDPRML models that produce decisions with legal or similarly significant effects on natural persons (credit scoring, hiring, insurance pricing)
Training on special category dataArt. 35(3)(b) GDPRModels trained on health, biometric, genetic, racial, political, religious, sexual orientation, or trade union data at scale
AI-powered surveillanceArt. 35(3)(c) GDPRComputer vision, facial recognition, behavioural analytics, or anomaly detection in public spaces
High-risk AI systemsArt. 6 EU AI ActSystems listed in Annex III of the AI Act (biometric identification, critical infrastructure, employment, law enforcement, migration, justice)
Foundation models processing personal dataEDPB Guidelines 04/2025LLMs and foundation models trained on datasets containing personal data, regardless of downstream use
Automated inference of sensitive attributesEDPB Guidelines 04/2025Models that infer Art. 9 special category data from non-sensitive inputs (inferring health status from purchasing patterns)

EDPB WP248 Criteria Applied to AI

AI systems frequently trigger multiple WP248 criteria simultaneously:

  • Evaluation/scoring: Inherent to classification and regression models
  • Automated decision-making: Core function of deployed AI systems
  • Innovative technology: Novel model architectures, training techniques
  • Large-scale processing: Training datasets containing millions of records
  • Matching/combining datasets: Multi-source training data aggregation
  • Vulnerable data subjects: When AI is applied to children, employees, patients

When an AI system meets two or more criteria, a DPIA is presumptively required.

AI DPIA Methodology — EDPB Framework

Phase 1: AI System Description (Art. 35(7)(a) Extended)

The systematic description must cover the complete AI lifecycle:

1.1 Training Phase Documentation
  • Training data sources: Origin, collection method, consent status, lawful basis for each dataset
  • Data categories: All personal data categories present in training data, including data that may be inadvertently included (background individuals in images, metadata in text corpora)
  • Data volume: Number of records, data subjects affected, geographic scope
  • Data preprocessing: Cleaning, augmentation, labelling processes and any human review
  • Feature engineering: Which personal data attributes are used as features, how derived features are computed
  • Training infrastructure: Where training occurs (cloud provider, jurisdiction), data residency during training
1.2 Model Architecture Documentation
  • Model type: Neural network architecture (transformer, CNN, RNN), ensemble methods, decision trees
  • Model parameters: Number of parameters, model size, complexity indicators
  • Explainability characteristics: Inherent interpretability level (white-box, grey-box, black-box)
  • Memorization risk: Assessed propensity for the model to memorize training data (higher for large models with small datasets)
1.3 Deployment Phase Documentation
  • Inference inputs: What personal data is processed at inference time
  • Output types: Classifications, scores, recommendations, generated content
  • Decision pipeline: How model outputs feed into decisions affecting data subjects
  • Human oversight: Level and effectiveness of human review in the decision chain
  • Monitoring: Drift detection, performance monitoring, feedback loops

Phase 2: AI-Specific Necessity and Proportionality

2.1 Purpose Limitation for AI
  • Is the AI system necessary for the stated purpose, or could simpler processing achieve it?
  • Has the controller evaluated non-AI alternatives and documented why AI is required?
  • Are the training data processing purposes compatible with the original collection purposes (Art. 6(4) assessment)?
  • For repurposed data: has a compatibility assessment been conducted per EDPB Guidelines 04/2025?
2.2 Data Minimisation for AI
  • Has the minimum dataset required for acceptable model performance been determined through ablation studies?
  • Can synthetic data, federated learning, or differential privacy reduce the personal data requirement?
  • Are there personal data elements in the training data that do not contribute to model performance?
  • Has the controller assessed whether anonymised or pseudonymised data could achieve adequate performance?
2.3 Training Data Lawfulness Assessment

For each training dataset, document:

Assessment ElementRequirement
Original collection purposeWas personal data collected for a purpose compatible with AI training?
Lawful basisArt. 6(1) basis for the training processing — legitimate interest requires balancing test
Consent validityIf consent is the basis, was AI training specified as a purpose? Was consent freely given?
Special category conditionsIf Art. 9 data is present, which Art. 9(2) exception applies?
Web-scraped dataEDPB position: web scraping for AI training generally cannot rely on legitimate interest without additional safeguards
Third-party datasetsHas the controller verified the upstream lawful basis chain?

Phase 3: AI-Specific Risk Assessment

3.1 Privacy Risk Categories for AI
Risk CategoryDescriptionLikelihood Factors
Training data extractionAdversary extracts verbatim training data from the modelModel size, training data repetition, overfitting degree
Membership inferenceAdversary determines if specific data was in the training setModel confidence distribution, overfitting, shadow model availability
Model inversionAdversary reconstructs input features from model outputsOutput granularity, model type, auxiliary information available
Attribute inferenceModel reveals sensitive attributes not provided as inputCorrelations in training data, feature interactions
Emergent bias amplificationModel amplifies biases present in training data, producing discriminatory outcomesTraining data representativeness, debiasing measures applied
Concept drift discriminationModel performance degrades unequally across demographic groups over timeMonitoring coverage, retraining frequency
Re-identification through AI outputModel outputs enable linking back to specific data subjectsOutput specificity, population uniqueness, auxiliary data
Automated decision errorsIncorrect AI decisions causing material harm to data subjectsModel accuracy, error distribution across groups
3.2 AI Risk Scoring Matrix

Combine likelihood and severity using the EDPB-recommended matrix:

                    Negligible    Limited    Significant    Maximum
Almost Certain      Medium        High       Very High      Very High
Likely              Medium        High       High           Very High
Possible            Low           Medium     High           High
Remote              Low           Low        Medium         High
3.3 AI Act Risk Level Integration

Cross-reference GDPR risk assessment with AI Act classification:

  • Unacceptable risk (Art. 5 AI Act): Processing must not proceed — social scoring, real-time biometric identification in public spaces (with limited exceptions)
  • High risk (Art. 6 + Annex III): Enhanced DPIA obligations, conformity assessment required
  • Limited risk (Art. 50): Transparency obligations — inform users they are interacting with AI
  • Minimal risk: Standard DPIA process applies

Phase 4: AI-Specific Mitigation Measures

Technical Measures
MeasureRisk AddressedImplementation
Differential privacyTraining data extraction, membership inferenceApply DP-SGD during training with calibrated epsilon (ε ≤ 8 for moderate protection, ε ≤ 1 for strong)
Federated learningData centralisation riskDistribute training across data holders without centralising personal data
Model output perturbationModel inversion, attribute inferenceAdd calibrated noise to model outputs, round confidence scores
Training data deduplicationMemorization riskRemove duplicate and near-duplicate records before training
Membership inference testingMembership inferenceRun MI attacks against the model pre-deployment; retrain if leakage exceeds threshold
Fairness constraintsBias amplificationApply demographic parity, equalised odds, or calibration constraints during training
Input/output filteringPII leakage in generative modelsDeploy PII detection on model inputs and outputs with automated redaction
Model pruning and distillationMemorization, extractionCompress the model to reduce capacity for memorizing individual records
Organisational Measures
  • Establish an AI Ethics Review Board with privacy representation
  • Implement model cards documenting privacy properties for each deployed model
  • Conduct regular model audits (minimum annually) testing for privacy leakage
  • Maintain training data provenance documentation and deletion capability
  • Define retraining triggers and ensure DPIA review accompanies each retraining cycle
  • Implement incident response procedures specific to AI privacy incidents

Phase 5: Human Oversight Assessment

Per AI Act Art. 14 and GDPR Art. 22, assess the human oversight mechanism:

Oversight ElementAssessment Question
Meaningful reviewCan the human reviewer effectively evaluate the AI recommendation and override it?
Time and resourcesIs sufficient time allocated for meaningful review, or is the human a rubber stamp?
CompetenceDoes the reviewer have the expertise to identify AI errors?
AuthorityDoes the reviewer have the authority and means to override the AI?
Feedback mechanismAre overrides recorded and fed back into model improvement?
Automation biasAre measures in place to mitigate the tendency to defer to the AI?

Prior Consultation Triggers for AI

Art. 36 prior consultation with the supervisory authority is required when:

  1. The AI system produces high residual risk after all mitigation measures
  2. The AI system processes special category data at scale with novel techniques
  3. The supervisory authority's Art. 35(4) list specifically includes the AI use case
  4. The AI system is deployed for real-time biometric identification under AI Act Art. 5 exceptions

Enforcement Precedents

  • Clearview AI (Multiple DPAs, 2021-2024): Fines totalling over EUR 90 million across Italy (EUR 20M), France (EUR 20M), UK (GBP 7.5M), Greece (EUR 20M) for facial recognition AI deployed without DPIA, lawful basis, or transparency
  • CNIL v. Clearview AI (SAN-2022-019): Specific finding that no DPIA was conducted for biometric AI processing
  • Italian DPA v. Replika (2023): Ordered cessation of AI chatbot processing due to inadequate age verification and failure to conduct DPIA for AI processing affecting minors
  • Spanish DPA v. CaixaBank (PS/00421/2020): EUR 6 million fine for automated credit scoring without adequate DPIA addressing algorithmic decision-making risks
  • Dutch DPA v. Tax Authority (2020): Finding that algorithmic fraud detection system (SyRI) lacked proportionality and adequate DPIA for AI-driven profiling

Integration Points

  • ai-training-lawfulness: Detailed lawful basis analysis for training data feeds into Phase 2
  • ai-automated-decisions: Art. 22 assessment integrates with Phase 5 human oversight
  • ai-model-privacy-audit: Technical privacy testing results feed into Phase 3 risk assessment
  • ai-act-high-risk-docs: AI Act conformity assessment aligns with Phase 3.3 risk classification
  • ai-bias-special-category: Bias assessment results feed into risk scoring for discrimination harms

Federated Learning for GDPR Compliance

Overview

Federated learning (FL) is a distributed machine learning approach that trains models across multiple data holders without centralising personal data. Instead of collecting training data into a central repository, federated learning sends the model to the data, computes local updates on each participant's device or server, and aggregates only model updates (gradients or weights) at a central coordinator. This architecture directly addresses GDPR data minimisation (Art. 5(1)(c)) and data protection by design (Art. 25) principles by eliminating the need to transfer and centralise personal data for AI training. However, federated learning is not a privacy silver bullet — it introduces its own privacy risks that must be managed through complementary techniques.

Federated Learning Architecture Patterns

Pattern 1: Cross-Device Federated Learning

Use case: Training on data from millions of user devices (smartphones, tablets, IoT).

ComponentDescription
ParticipantsEnd-user devices (smartphones, tablets, wearables)
ScaleThousands to millions of participants
DataSmall per-device, large aggregate (e.g., keyboard predictions, health metrics)
CoordinationCentral server selects participants per round, distributes model, aggregates updates
CommunicationCompressed gradient updates over mobile networks
Privacy riskIndividual gradient updates may leak information about device data

GDPR Analysis:

  • Data minimisation: personal data never leaves the device — strong compliance
  • Controller role: platform operator is controller; device owners are not processors
  • Lawful basis: consent or legitimate interest for on-device processing
  • International transfers: no personal data transfer if aggregation is privacy-preserving
  • Right to erasure: device can be excluded from future rounds; model unlearning may be needed

Pattern 2: Cross-Silo Federated Learning

Use case: Training across organisational boundaries (hospitals, banks, subsidiaries).

ComponentDescription
ParticipantsOrganisational data silos (hospitals, branches, partner companies)
Scale2 to 100 participants
DataLarge per-silo, structured (e.g., medical records, financial transactions)
CoordinationTrusted aggregator or peer-to-peer protocol
CommunicationModel updates over secure channels between organisations
Privacy riskGradient updates may reveal institutional data patterns

GDPR Analysis:

  • Joint controller determination: participants and aggregator may be joint controllers (Art. 26) — requires joint controller agreement
  • Data processing agreements: if aggregator is a processor, Art. 28 DPA required
  • Data minimisation: personal data stays within each silo — strong compliance
  • International transfers: if silos are in different jurisdictions, gradient transfer legality depends on privacy guarantees
  • Lawful basis: each silo needs independent lawful basis for local training; aggregation needs separate basis

Pattern 3: Vertical Federated Learning

Use case: Different organisations hold different features for the same individuals.

ComponentDescription
ParticipantsOrganisations with complementary data (bank + retailer sharing customer features)
Scale2 to 10 participants
DataDifferent features for overlapping individuals
CoordinationSecure multi-party computation for feature combination
CommunicationEncrypted intermediate representations
Privacy riskFeature linkage may reveal individual attributes across parties

GDPR Analysis:

  • Joint controller: participants are likely joint controllers — each processes personal data for a common purpose
  • Purpose limitation: combined training purpose must be compatible with each party's original collection purpose
  • Data subject notification: data subjects must be informed about cross-organisation federated training
  • Consent: may be required for combining data across organisations

Secure Aggregation Protocols

Protocol 1: Masking-Based Secure Aggregation

Participants mask their local updates with pairwise random masks that cancel out upon aggregation. The aggregator receives the sum without seeing individual updates.

PropertyValue
Privacy guaranteeIndividual updates not visible to aggregator or other participants
Computational costO(n^2) pairwise key agreement, O(n) masking per round
Communication cost2x baseline (masks + masked updates)
Dropout toleranceHandles participant dropout if sufficient participants remain
Collusion resistanceSecure against aggregator + up to t-1 participant collusion

Protocol 2: Homomorphic Encryption Aggregation

Participants encrypt their updates with a homomorphic encryption scheme. The aggregator computes the sum on encrypted data without decryption.

PropertyValue
Privacy guaranteeComputationally secure — updates encrypted throughout
Computational cost100-1000x overhead for encryption/decryption operations
Communication cost2-10x baseline (ciphertext expansion)
Dropout toleranceExcellent — encrypted updates can be summed independently
Collusion resistanceSecure against aggregator (does not hold decryption key)

Protocol 3: Trusted Execution Environment (TEE)

Aggregation occurs within a hardware-protected enclave (Intel SGX, ARM TrustZone). Participants send updates to the TEE, which performs aggregation in isolated memory.

PropertyValue
Privacy guaranteeHardware-based isolation — aggregator cannot inspect updates
Computational costNear-native (small overhead for enclave transitions)
Communication costBaseline (no encryption expansion for enclave-to-enclave)
Dropout toleranceExcellent
Collusion resistanceDepends on hardware trust model; vulnerable to side-channel attacks

Differential Privacy Integration

Local Differential Privacy (LDP)

Each participant adds noise to their gradient update before sending to the aggregator:

  • Mechanism: Gaussian noise calibrated to sensitivity and privacy budget (epsilon, delta)
  • Guarantee: Individual update is differentially private — aggregator cannot infer specific data points
  • Trade-off: Higher noise reduces model accuracy; need more participants to compensate
  • Recommended epsilon: 1-10 per round, with privacy accounting across rounds

Central Differential Privacy (CDP)

The aggregator adds noise to the aggregated update before applying to the global model:

  • Mechanism: Gaussian noise added to the sum of clipped gradients
  • Guarantee: Global model is differentially private with respect to any single participant's data
  • Trade-off: Less noise needed than LDP for same accuracy (noise averages out); requires trusted aggregator
  • Recommended epsilon: 1-8 total privacy budget

Privacy Budget Management

ParameterDescriptionGuidance
Epsilon (ε)Privacy loss parameter — lower is more privateε ≤ 1: strong privacy; ε ≤ 8: moderate; ε > 10: weak
Delta (δ)Probability of privacy failureδ < 1/N where N is dataset size
Rounds (T)Number of federated training roundsPrivacy degrades with rounds — use composition theorems
Clip norm (C)Maximum gradient norm per participantBalance between privacy (lower C) and convergence (higher C)
Noise multiplier (σ)Ratio of noise to sensitivityDetermined by ε, δ, C, and composition method

GDPR Compliance Assessment for Federated Learning

Data Protection by Design (Art. 25) Alignment

GDPR PrincipleFL ImplementationCompliance Status
Data minimisation (Art. 5(1)(c))Personal data stays local — only model updates transmittedStrong compliance
Purpose limitation (Art. 5(1)(b))Local processing for specified training purposeRequires per-participant purpose documentation
Storage limitation (Art. 5(1)(e))No central training data repository — data retained locally per participant's policyCompliance depends on participant retention
Integrity and confidentiality (Art. 5(1)(f))Secure aggregation protects update confidentialityStrong with SA + DP
Accuracy (Art. 5(1)(d))Model accuracy may differ from centralised trainingMonitor and document accuracy trade-offs

Remaining GDPR Challenges

  1. Gradient leakage: Model updates can leak training data even without transmitting raw data — secure aggregation and DP mitigate but do not eliminate
  2. Model memorization: The aggregated model may still memorize individual records — privacy auditing required post-training
  3. Right to erasure: Removing a participant's contribution from the trained model is challenging — machine unlearning or retraining required
  4. Controller/processor determination: Complex multi-party FL architectures require clear role delineation
  5. Lawful basis: Each participant needs independent lawful basis; the aggregator needs basis for processing updates
  6. Transparency: Data subjects must be informed about federated learning participation

Implementation Considerations

Communication Efficiency

TechniqueDescriptionPrivacy Impact
Gradient compressionQuantise or sparsify gradients before transmissionMay interact with DP noise — careful calibration needed
Federated averaging (FedAvg)Multiple local SGD steps before communicationReduces communication rounds; may increase per-round privacy cost
Gradient selectionSend only top-k gradient componentsLeaks which components are most significant — privacy concern

Participant Selection

StrategyDescriptionPrivacy Consideration
Random selectionUniformly random participant sampling per roundFair representation; privacy amplification through subsampling
Availability-basedSelect participants with sufficient resourcesMay bias toward certain participant profiles
Contribution-basedSelect participants whose data improves model mostReveals information about data distribution — privacy risk

Enforcement Relevance

While no enforcement action has specifically addressed federated learning, the technology is directly relevant to:

  • CNIL AI recommendations (2024): Identified federated learning as a recommended privacy-enhancing technology for AI training
  • EDPB Guidelines 04/2025: Mentioned distributed training as a data minimisation measure for AI systems
  • Garante v. OpenAI (2023): Centralised data collection was a key compliance issue — federated alternatives could have mitigated
  • AI Act Art. 10: Data governance for training data — FL enables data governance while maintaining data locality

Integration Points

  • ai-dpia: FL architecture reduces data centralisation risk in DPIA assessment
  • ai-training-lawfulness: FL may simplify lawful basis by keeping data local
  • ai-model-privacy-audit: Federated models require distributed privacy auditing
  • ai-privacy-inference: FL complements confidential computing for inference privacy
  • ai-data-retention: Data locality in FL simplifies retention compliance

AI Model Privacy Audit

Overview

AI model privacy auditing is the systematic assessment of whether trained ML models leak information about their training data. Models can memorize individual training records, enabling adversaries to extract personal data, determine dataset membership, reconstruct input features, or infer sensitive attributes. This skill implements a comprehensive model privacy audit methodology using established attack techniques and tools (ML Privacy Meter, ART, Foolbox) to quantify privacy leakage before deployment and periodically during operation. The audit results feed directly into the AI DPIA risk assessment and inform mitigation measure selection.

Privacy Attack Taxonomy

1. Training Data Extraction

Objective: Extract verbatim or near-verbatim records from the model's training data.

Attack VectorDescriptionTarget Models
Prompt-based extractionCraft prompts that cause LLMs to regurgitate training dataLanguage models, generative models
Canary extractionInsert known canary strings into training data and test if model reproduces themAny model (testing methodology)
Gradient-based extractionUse model gradients to reconstruct training inputsModels with accessible gradients
Generative reconstructionUse the model as an oracle to iteratively reconstruct training samplesGANs, VAEs, diffusion models

Risk Factors Increasing Extraction Likelihood:

  • Large model capacity relative to training data size (overfitting)
  • Training data containing duplicated or near-duplicated records
  • Longer training duration (more epochs)
  • Lower regularisation
  • Models with high output granularity (logits, probabilities)

Testing Methodology:

  1. Insert canary records with unique identifiers into training data
  2. Train the model
  3. Attempt extraction through various prompting strategies
  4. Measure extraction success rate (percentage of canaries recovered)
  5. Threshold: extraction rate should be below 0.1% for acceptable risk

2. Membership Inference

Objective: Determine whether a specific record was in the model's training set.

Attack TypeMethodComputational Cost
Shadow model attackTrain shadow models on similar data, build a binary classifier on model outputsHigh — requires training multiple shadow models
Metric-based attackUse model confidence, loss, or entropy to distinguish members from non-membersLow — single model query per sample
Label-only attackUse predicted labels (no confidence scores) to infer membershipMedium — requires multiple queries
Likelihood ratio attack (LiRA)Compare per-sample loss to reference distributionsHigh — most accurate, requires multiple models

ML Privacy Meter Implementation:

  • Population metric-based attack: compares target model's loss on a sample against population loss distribution
  • Reference metric-based attack: uses reference models to compute per-sample metrics
  • Shadow model attack: trains shadow models and uses the attack model to classify members

Testing Methodology:

  1. Partition data: training set (members) and held-out set (non-members)
  2. Run membership inference attacks using ML Privacy Meter
  3. Measure attack success: true positive rate at low false positive rate (TPR@FPR=0.1%, 1%)
  4. Generate ROC curves per sample and aggregate
  5. Threshold: TPR@1%FPR should be below 5% for acceptable privacy

3. Model Inversion

Objective: Reconstruct input features from model outputs.

Attack TypeMethodTarget
Confidence-based inversionIteratively optimise input to maximise model confidence for a known labelClassification models
Gradient-based inversionUse model gradients to reconstruct inputs from outputsWhite-box models
GAN-based inversionTrain a GAN to invert model outputs to input spaceFace recognition, image classifiers

Testing Methodology:

  1. Select target classes or individuals
  2. Run inversion attacks with various initializations
  3. Measure reconstruction quality (SSIM, PSNR for images; cosine similarity for embeddings)
  4. Assess re-identification risk: can reconstructed data identify specific individuals?
  5. Threshold: reconstruction similarity should be below 0.3 (SSIM) for acceptable risk

4. Attribute Inference

Objective: Infer sensitive attributes not present in the model's output.

Attack TypeDescription
Correlation exploitationUse correlated features to infer sensitive attributes from model behaviour
Partial knowledge attackAttacker knows some attributes and uses model to infer remaining sensitive ones
Group inferenceDetermine statistical properties of training subgroups

Testing Methodology:

  1. Identify sensitive attributes (Art. 9 categories) that may be correlated with model features
  2. Train attack models to predict sensitive attributes from model outputs
  3. Measure inference accuracy for each sensitive attribute
  4. Compare against random baseline
  5. Threshold: inference accuracy should not exceed random baseline + 10%

Audit Methodology

Phase 1: Audit Scoping (Days 1-3)

  1. Define audit scope: which models, what deployment context, what threat model
  2. Identify assets: training data, model artefacts, deployment infrastructure
  3. Define threat model: who are the adversaries, what access do they have?
    • Black-box: API access only (queries and responses)
    • Grey-box: API access plus model architecture knowledge
    • White-box: Full access to model weights and architecture
  4. Select attacks based on threat model and model type
  5. Define success criteria (acceptable leakage thresholds)
  6. Obtain audit authorisation from model owner and legal

Phase 2: Environment Setup (Days 4-7)

  1. Set up isolated audit environment (no production data leakage)
  2. Install audit tools: ML Privacy Meter, ART (Adversarial Robustness Toolbox), custom scripts
  3. Obtain model access (API endpoint or model weights depending on threat model)
  4. Prepare member/non-member datasets for membership inference
  5. Prepare canary data for extraction testing
  6. Configure monitoring to log all audit queries

Phase 3: Attack Execution (Days 8-18)

For each selected attack:

  1. Configure attack parameters
  2. Execute attack against the target model
  3. Collect results (success rates, confidence intervals)
  4. Vary attack parameters to find worst-case leakage
  5. Document: attack configuration, results, computational cost

Phase 4: Analysis and Reporting (Days 19-25)

  1. Aggregate results across all attacks
  2. Calculate privacy risk scores per attack type
  3. Identify high-risk data subsets (records most vulnerable to extraction)
  4. Cross-reference with DPIA risk register
  5. Generate audit report with:
    • Executive summary
    • Attack results per category
    • Risk assessment with GDPR alignment
    • Recommended mitigations
    • Residual risk after proposed mitigations

Phase 5: Remediation Validation (Days 26-30)

  1. If mitigations are applied (differential privacy, output perturbation, etc.)
  2. Re-run key attacks to validate mitigation effectiveness
  3. Document residual leakage post-mitigation
  4. Compare against acceptable thresholds
  5. Issue final audit certificate or remediation requirements

Privacy Leakage Thresholds

MetricAcceptableElevatedUnacceptable
Membership inference TPR@1%FPR< 5%5-15%> 15%
Training data extraction rate< 0.1%0.1-1%> 1%
Model inversion SSIM< 0.30.3-0.6> 0.6
Attribute inference accuracy above baseline< 10%10-25%> 25%

Mitigation Measures

MitigationAttacks MitigatedTrade-off
Differential privacy (DP-SGD)All — provides mathematical guaranteeModel accuracy reduction (calibrate epsilon)
Training data deduplicationExtraction, membership inferenceOne-time preprocessing cost
Regularisation (dropout, weight decay)Membership inference, overfitting-related leakageMay affect model performance
Output perturbationModel inversion, attribute inferenceReduces output precision
Confidence score roundingMetric-based membership inferenceMinor output precision loss
Model distillationExtraction, membership inferenceRequires additional training
Rate limitingAll query-based attacksAffects legitimate use
Input/output PII filteringExtraction of PII from generative modelsMay affect model utility

Tools and Frameworks

ToolPurposeSource
ML Privacy MeterMembership inference auditinggithub.com/privacytrustlab/ml_privacy_meter
IBM ARTAdversarial robustness and privacy testinggithub.com/Trusted-AI/adversarial-robustness-toolbox
TensorFlow PrivacyDifferential privacy traininggithub.com/tensorflow/privacy
OpacusPyTorch differential privacygithub.com/pytorch/opacus
Google DP LibraryDifferential privacy algorithmsgithub.com/google/differential-privacy
FoolboxAdversarial attack librarygithub.com/bethgelab/foolbox

Enforcement Relevance

Model privacy auditing is not explicitly required by the GDPR or AI Act, but is effectively mandated through:

  • Art. 35 DPIA: Risk assessment for AI systems must evaluate privacy leakage risks — auditing is the standard methodology
  • Art. 32 Security: Appropriate technical measures to ensure security of processing — privacy auditing validates these measures
  • AI Act Art. 9: Risk management for high-risk AI requires identification and mitigation of privacy risks
  • AI Act Art. 15: Accuracy, robustness, and cybersecurity requirements — privacy attacks are a cybersecurity concern
  • EDPB Guidelines 04/2025: Controllers must assess whether AI models have effectively anonymised training data — auditing tests this claim

Integration Points

  • ai-dpia: Audit results feed into DPIA Phase 3 risk assessment
  • ai-data-retention: Audit validates whether deletion from training data is effective
  • ai-deployment-checklist: Pre-deployment privacy audit is a checklist requirement
  • ai-federated-learning: Federated learning models require distributed privacy auditing

Combined DPIA and AI Act Conformity Assessment Template

Overview

High-risk AI systems under the EU AI Act must undergo both a GDPR Art. 35 DPIA and an AI Act conformity assessment. Rather than conducting these as separate exercises, this skill provides an integrated template that satisfies both frameworks simultaneously. The combined assessment ensures consistency between GDPR privacy risk analysis and AI Act safety and fundamental rights evaluation, reduces duplication, and provides a single risk scoring matrix covering both regulatory dimensions. Art. 26(9) AI Act explicitly requires deployers to use DPIA results when fulfilling AI Act obligations.

Combined Risk Scoring Matrix

Risk Dimensions

DimensionSourceWeight
Privacy risk to data subjectsGDPR Art. 35(7)(c)30%
Fundamental rights impactEU AI Act Art. 9(2)(a)25%
Accuracy and reliability riskEU AI Act Art. 1520%
Transparency and explainability gapGDPR Art. 13(2)(f) + AI Act Art. 1315%
Human oversight adequacyGDPR Art. 22 + AI Act Art. 1410%

Scoring Scale (Per Dimension)

ScoreLevelDescription
1MinimalRisk negligible; controls effective
2LowMinor risk; standard controls sufficient
3MediumModerate risk; enhanced controls needed
4HighSignificant risk; intensive mitigation required
5CriticalSevere risk; may require processing suspension

Overall Risk Classification

Weighted ScoreClassificationAction Required
1.0-1.5LowStandard monitoring
1.6-2.5MediumEnhanced monitoring and periodic review
2.6-3.5HighActive mitigation and DPO/Board oversight
3.6-4.5Very HighArt. 36 prior consultation; deployment hold pending mitigation
4.6-5.0CriticalDo not deploy; fundamental redesign required

GDPR DPIA Requirements (Art. 35(7))

ElementReferenceCombined Assessment Section
Systematic description of processingArt. 35(7)(a)Section 2: AI System Description
Necessity and proportionalityArt. 35(7)(b)Section 3: Necessity Assessment
Risk assessmentArt. 35(7)(c)Section 5: Combined Risk Register
Mitigation measuresArt. 35(7)(d)Section 6: Mitigation Measures

AI Act Conformity Assessment Requirements

For high-risk AI systems (Annex III), the conformity assessment per Art. 43 requires:

ElementReferenceCombined Assessment Section
Risk management systemArt. 9Section 5: Combined Risk Register
Data governanceArt. 10Section 2: Training Data Governance
Technical documentationArt. 11Full combined assessment document
Record-keepingArt. 12Section 7: Monitoring and Logging
TransparencyArt. 13Section 4: Transparency Assessment
Human oversightArt. 14Section 4: Human Oversight
Accuracy, robustness, cybersecurityArt. 15Section 5: Technical Risk Assessment
Quality management systemArt. 17Section 8: Quality Management

Mitigation Measures Framework

Technical Measures

MeasureGDPR RelevanceAI Act RelevancePriority
Differential privacyTraining data protectionRobustness (Art. 15)High for sensitive data
Model output perturbationModel inversion protectionAccuracy trade-off (Art. 15)Medium
Fairness constraintsNon-discrimination (Art. 5(1)(a))Bias prevention (Art. 10)High for high-risk
Explainability tools (SHAP/LIME)Art. 13(2)(f) logic explanationInterpretability (Art. 13)High
Input/output PII filteringData minimisation (Art. 5(1)(c))Accuracy (Art. 15)High for generative AI
Encryption (rest/transit)Security (Art. 32)Cybersecurity (Art. 15)Standard
Access controls (RBAC)Security (Art. 32)Cybersecurity (Art. 15)Standard
Anomaly detectionBreach detection (Art. 33)Robustness (Art. 15)Medium

Organisational Measures

MeasureGDPR RelevanceAI Act Relevance
AI ethics review boardAccountability (Art. 5(2))Quality management (Art. 17)
Model cardsTransparency (Art. 13-14)Technical documentation (Art. 11)
Regular bias auditsFairness (Art. 5(1)(a))Bias monitoring (Art. 10)
Incident response for AIBreach notification (Art. 33-34)Post-market monitoring (Art. 72)
Staff training on AI risksAccountabilityHuman oversight (Art. 14)
Documentation and record-keepingAccountability (Art. 5(2))Record-keeping (Art. 12)

Enforcement Relevance

  • AI Act Art. 26(9): Deployers of high-risk AI shall use GDPR Art. 35 DPIA information when fulfilling AI Act obligations — the combined assessment implements this requirement.
  • AI Act Art. 43: Conformity assessment procedures for high-risk AI systems.
  • GDPR Art. 35: DPIA requirements — the combined template satisfies all Art. 35(7) elements.

Integration Points

  • ai-dpia: Combined template extends the AI DPIA methodology
  • ai-act-high-risk-docs: Conformity assessment documentation aligns with high-risk requirements
  • ai-bias-special-category: Bias assessment results feed into combined risk register
  • ai-automated-decisions: Human oversight assessment integrates Art. 22 and Art. 14 requirements

AI Privacy Inference and Derived Data

Overview

AI systems routinely generate inferences about individuals — predictions about creditworthiness, health risks, personality traits, political opinions, or behavioural patterns that were never directly provided by the data subject. These AI-derived inferences raise critical privacy questions: Are inferences personal data? When does inference become profiling under GDPR Article 22? What accuracy obligations apply to AI predictions? Can data subjects access, rectify, or object to inferences drawn about them? The CJEU, EDPB, and national DPAs have progressively clarified that inferences are personal data when they relate to an identified or identifiable person, and that GDPR rights extend to derived and inferred data. Cerebrum AI Labs must classify, govern, and provide transparency over all inferences its AI systems generate about individuals.

Legal Framework for AI Inferences

When Inferences Are Personal Data

CriterionAnalysisExample
Relates to an identified personInference is linked to a specific customer record or user profile"Customer C-12345 has 78% churn probability"
Relates to an identifiable personInference can be linked to a person through combination with other data"User with session token X-789 is likely aged 25-34"
Used to evaluate a personInference is used to assess, classify, or make decisions about someoneCredit score derived from transaction patterns
Has impact on a personInference affects how the person is treated or what options are availableInsurance premium adjusted based on predicted health risk

CJEU C-434/16 (Nowak, 2017): Personal data includes "any information" relating to a data subject — this encompasses opinions, assessments, and inferences, not only factual data directly provided by the individual.

EDPB Guidelines 8/2020 on Targeting of Social Media Users: Inferred data (data created by the controller through observation or derivation) constitutes personal data and is subject to the full scope of GDPR rights.

GDPR Classification of Inference Types

Inference TypeClassificationGDPR Implications
Observed dataData collected through direct interaction (browsing history, purchase records)Standard personal data — Art. 6 lawful basis required
Derived dataData created by the controller through computation on existing data (credit score, risk rating)Personal data — subject to access, rectification, objection rights
Inferred dataProbabilistic predictions about characteristics not directly observed (personality, health risk)Personal data — potentially special category if predicting Art. 9 characteristics
Aggregated dataStatistical outputs at group level, not linked to individualsNot personal data if truly anonymous (k-anonymity verified)

When Inferences Become Special Category Data

Predicted CharacteristicArt. 9 CategoryTrigger
Ethnic origin from name/location patternsRacial or ethnic originAny inference about ethnic background, even probabilistic
Political leaning from content engagementPolitical opinionsPrediction used to classify or target based on politics
Religious affiliation from purchase patternsReligious beliefsHalal/kosher purchase scoring, prayer time activity patterns
Health condition from behavioural signalsHealth dataStep count decline predicting depression, typing pattern analysis
Sexual orientation from browsing/social dataSexual orientationAny inference about sexual orientation regardless of accuracy
Pregnancy from purchase pattern shiftsHealth data + genderPurchase category analysis predicting pregnancy status

Cerebrum AI Labs Policy: Any inference that predicts, estimates, or classifies an Art. 9 characteristic — even indirectly or probabilistically — must be treated as special category data and requires an Art. 9(2) condition for processing.

Profiling Under GDPR Article 22

Profiling Definition (Art. 4(4))

Profiling means any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning:

  • Work performance
  • Economic situation
  • Health
  • Personal preferences
  • Interests
  • Reliability
  • Behaviour
  • Location
  • Movements

Three-Level Framework

LevelDescriptionGDPR RequirementCerebrum AI Labs Example
Profiling onlyAutomated evaluation without decisionArt. 6 lawful basis + Art. 13-14 transparencyCustomer segmentation for analytics dashboard
Profiling + human decisionAutomated evaluation informing a human decision-makerArt. 6 lawful basis + transparency + meaningful human involvementCredit risk score reviewed by loan officer
Solely automated decision with legal/significant effectsAutomated decision with no meaningful human involvement producing legal or similarly significant effectsArt. 22(1) prohibition applies — must fall within Art. 22(2) exceptionsAutomated loan rejection based solely on AI credit score

Art. 22(2) Exceptions Allowing Solely Automated Decisions

ExceptionRequirementCerebrum AI Labs Application
Art. 22(2)(a) — ContractDecision necessary for entering into or performing a contractAutomated credit pre-approval for existing customers
Art. 22(2)(b) — LawAuthorised by EU or Member State law with suitable safeguardsRegulatory-mandated fraud screening
Art. 22(2)(c) — Explicit consentData subject's explicit consent obtainedCustomer opts in to automated portfolio rebalancing

Safeguards Required (Art. 22(3))

SafeguardImplementation at Cerebrum AI Labs
Right to obtain human interventionEscalation button in customer portal routes to trained human reviewer within 2 business days
Right to express point of viewCustomer can submit additional context through contestation form before human review
Right to contest the decisionAppeal process with independent review panel; decision reversed if AI error demonstrated
Right to explanationIndividual explanation generated using SHAP values showing top 5 factors influencing the AI decision

Inference Accuracy and Quality Obligations

GDPR Article 5(1)(d) — Accuracy Principle

AI inferences must be accurate, and where necessary, kept up to date. For probabilistic predictions this means:

ObligationImplementation
Accuracy measurementTrack prediction accuracy metrics (precision, recall, F1) per demographic group
Confidence thresholdsDo not present inferences with confidence below 70% as actionable without human review
Staleness detectionRe-evaluate inferences when underlying data changes; flag inferences older than 90 days
Accuracy disclosureInform data subjects of the probabilistic nature and known accuracy of inferences
Rectification of inferencesAllow data subjects to challenge inferences; if input data is corrected, regenerate inference

EDPB Position on Inference Accuracy

The EDPB Guidelines on Automated Decision-Making (WP 251 rev.01) state that controllers must:

  1. Use appropriate mathematical or statistical procedures for profiling
  2. Implement appropriate technical and organisational measures to minimise the risk of errors
  3. Correct inaccuracies and enable rectification
  4. Secure personal data to prevent discriminatory effects

Transparency Requirements for AI Inferences

Art. 13-14 Information Requirements

InformationRequirementCerebrum AI Labs Implementation
Existence of profilingInform data subjects that profiling occursPrivacy notice section: "How we use AI to analyse your data"
Logic involvedMeaningful information about the logic involvedTechnical explainer: "Our AI analyses your transaction patterns, account tenure, and product usage to predict service needs"
SignificanceEnvisaged consequences of such processing"This analysis may affect the products and offers shown to you, and may influence credit decisions"
Categories of data usedWhat data feeds the inference"We use: transaction history, account tenure, product holdings, service interactions"
Inference outputsWhat inferences are generated"We generate: churn probability, product affinity scores, credit risk indicators"

AI Act Transparency Requirements

ObligationAI Act ArticleCerebrum AI Labs Implementation
Inform users they are interacting with AIArt. 52(1)Chat interface disclosure: "You are interacting with an AI assistant"
Disclose AI-generated contentArt. 52(3)Outputs marked: "This recommendation was generated by AI"
Disclose emotion recognitionArt. 52(2)Not applicable — Cerebrum AI Labs does not use emotion recognition
High-risk system deployer transparencyArt. 13Technical documentation available to regulators on request

Inference Governance Framework

Cerebrum AI Labs Inference Registry

All AI systems that generate inferences about individuals must be registered in the Cerebrum AI Labs Inference Registry:

Registry FieldDescription
System IDUnique identifier for the AI system
Inference typeCategory of inference generated (behavioural, demographic, financial, health)
Data subjects affectedCategories and approximate volume of individuals profiled
Input featuresData elements used to generate the inference
Output formatInference output (score, category, probability, ranking)
Accuracy metricsLatest precision, recall, F1 by demographic group
Retention periodHow long inferences are stored before deletion
Art. 22 assessmentWhether the inference feeds a solely automated decision
Lawful basisArt. 6 and (if applicable) Art. 9 basis for generating the inference
DPIA referenceAssociated DPIA document ID

Inference Lifecycle Controls

PhaseControl
GenerationLog all inferences with timestamp, model version, confidence score, input data hash
StorageEncrypt inference outputs; apply access controls limiting who can read individual-level inferences
UsageTrack downstream consumption of inferences; prevent scope creep beyond documented purposes
DisclosureMake inferences available to data subjects on request (Art. 15 access right)
RectificationIf underlying data is corrected, flag dependent inferences for regeneration
DeletionDelete inferences per retention schedule (90 days operational, 12 months audit)

Enforcement and Regulatory Precedents

  • CJEU C-634/21 (SCHUFA, 2023): The CJEU held that the generation of a credit score by a private entity constitutes "automated individual decision-making" under Art. 22 if the score plays a decisive role in a subsequent decision by a third party. This means AI-derived scores used in downstream decisions may trigger Art. 22 protections even if the AI provider is not the final decision-maker.
  • Austrian DPA — Case DSB-D550.038 (2022): Found that inferred political opinions from social media activity constitute special category data under Art. 9, even when the inference is probabilistic and may be inaccurate.
  • Norwegian DPA — Grindr Decision (2021): NOK 65 million fine for sharing data enabling inference of sexual orientation. Established that data enabling inference of Art. 9 characteristics is itself special category data.
  • CNIL — Clearview AI (2022): EUR 20 million fine, finding that biometric inferences (face embeddings) from public photographs are personal data subject to full GDPR compliance.

Key Legal References

  • GDPR Article 4(4) — Definition of profiling
  • GDPR Article 5(1)(d) — Accuracy principle
  • GDPR Article 13(2)(f) — Right to information about profiling logic and significance
  • GDPR Article 15(1)(h) — Right of access to profiling information
  • GDPR Article 22 — Automated individual decision-making including profiling
  • GDPR Recital 71 — Safeguards for profiling and automated decisions
  • GDPR Recital 72 — Guidelines on profiling
  • CJEU C-434/16 (Nowak, 2017) — Broad interpretation of personal data including assessments
  • CJEU C-634/21 (SCHUFA, 2023) — Credit scoring as automated decision-making
  • EDPB Guidelines on Automated Decision-Making (WP 251 rev.01) — Art. 22 interpretation
  • EDPB Guidelines 8/2020 on Targeting of Social Media Users — Inferred data as personal data
  • EU AI Act Article 52 — Transparency obligations for AI systems
  • EU AI Act Article 14 — Human oversight for high-risk AI systems

Lawful Basis for AI Training Data

Overview

The processing of personal data for AI model training constitutes a distinct processing operation requiring its own lawful basis under GDPR Art. 6(1). The EDPB Guidelines 04/2025 and the coordinated ChatGPT Taskforce findings establish that AI training creates unique lawful basis challenges: the scale of data collection, the difficulty of obtaining meaningful consent for open-ended AI training purposes, the tension between legitimate interest and data subject expectations, and the complexity of determining lawfulness for web-scraped and third-party datasets. This skill provides the comprehensive lawful basis assessment framework for AI training data processing, addressing each Art. 6(1) basis as applied to ML training contexts.

Fundamental Principles

AI Training as Personal Data Processing

The EDPB has confirmed that AI model training constitutes processing of personal data under Art. 4(2) GDPR when:

  1. Training datasets contain personal data (directly or indirectly identifiable natural persons)
  2. The model is trained on data that includes personal data, even if the intent is to learn general patterns
  3. The resulting model retains the capability to generate or reproduce personal data from training sets
  4. Personal data is used in any pipeline stage: collection, cleaning, annotation, augmentation, validation, testing

The controller cannot avoid GDPR obligations by claiming the model has "learned" rather than "stored" personal data. The processing occurs at the point of training, regardless of whether the model can later reproduce specific records.

Purpose Specification for AI Training

Art. 5(1)(b) requires that personal data be collected for specified, explicit, and legitimate purposes. For AI training, this means:

  • "Training an AI model" is insufficiently specific — the controller must articulate the specific capability being developed
  • "Improving our services" through AI training must be disaggregated into concrete purposes
  • Each purpose must be documented before training begins, not retroactively justified
  • The purpose must be communicated to data subjects in privacy notices per Arts. 13-14

Lawful Basis Analysis for AI Training

Art. 6(1)(a) — Consent

Requirements for Valid AI Training Consent
RequirementAI Training Application
Freely givenData subjects must have genuine choice; consent cannot be bundled with service access unless AI training is necessary for the service
Specific"AI training" alone is insufficient — must specify what type of model, for what purpose, what data elements are used
InformedMust explain how personal data will be used in training, retention period for training data, risk of model memorization, inability to fully delete data from trained models
UnambiguousClear affirmative action; pre-ticked boxes or implied consent from terms of service are insufficient
WithdrawableController must provide mechanism to withdraw consent; but model already trained on the data presents technical challenge
Consent Challenges in AI Training
  1. Granularity problem: AI training often uses all available data — difficult to obtain specific consent for each data element's use in training
  2. Withdrawal complexity: Once a model is trained on personal data, true erasure requires model retraining or verified machine unlearning
  3. Purpose evolution: Foundation models and transfer learning mean the model may be repurposed — original consent may not cover downstream uses
  4. Scale impracticality: Obtaining consent from millions of data subjects whose data appears in web-scraped training corpora is practically impossible
  5. Power imbalance: When AI service use requires consent to training (e.g., "use our AI assistant and your conversations train our model"), consent may not be freely given
When Consent Works for AI Training
  • Users explicitly opt into a research programme where AI model training is a primary purpose
  • Users contribute data to a specific AI system with clear disclosure (e.g., "your feedback trains this recommendation engine")
  • Fine-tuning on user-provided data where the user understands and consents to the training purpose

Art. 6(1)(b) — Contract Necessity

AI training can rely on contractual necessity only when:

  • The AI model training is genuinely necessary for performing the contract with the data subject
  • The data subject has entered into a contract that requires AI-powered features
  • The training cannot be separated from the service delivery

Limitations per EDPB:

  • General improvement of AI systems through aggregate training is not "necessary" for any individual contract
  • Training a general-purpose model that benefits future users is not necessary for the current data subject's contract
  • Fine-tuning based on individual user interactions may qualify if the personalised model is part of the contracted service

Art. 6(1)(f) — Legitimate Interest

This is the most commonly relied-upon basis for AI training. The EDPB requires a rigorous three-part assessment:

Part 1: Legitimate Interest Identification

The controller must identify a specific, real, and lawful interest:

Interest TypeExampleEDPB Assessment
Commercial product improvementTraining a fraud detection model to protect customersGenerally legitimate — concrete benefit to data subjects
Research and developmentTraining models for medical imaging analysisLegitimate if research purpose is genuine and specific
General AI capabilityTraining a foundation model for general-purpose useScrutinised — interest must be articulated with specificity
Competitive advantageTraining to match competitor AI capabilitiesLegitimate commercial interest but weak in balancing
Part 2: Necessity Assessment
QuestionAssessment Criteria
Is AI training necessary for the identified interest?Could the interest be pursued without training on personal data?
Could anonymised data achieve the same result?Has the controller tested model performance with anonymised data?
Could synthetic data supplement or replace personal data?Has synthetic data generation been evaluated?
Is the volume of personal data proportionate?Has the minimum effective dataset been determined?
Could federated learning avoid centralising personal data?Has distributed training been assessed?
Part 3: Balancing Test

Factors weighing in favour of the controller:

  • Training data is publicly available (but this alone is not decisive)
  • Model serves a beneficial purpose (fraud prevention, medical research)
  • Strong technical safeguards applied (differential privacy, access controls)
  • Data subjects can exercise opt-out rights effectively
  • Training data is pseudonymised before use

Factors weighing in favour of data subjects:

  • Data was not collected with AI training in mind — processing is far from original expectations
  • Large-scale data collection from diverse sources without data subjects' awareness
  • Special category data is present or can be inferred from training data
  • Children's data is present in the training corpus
  • No practical opt-out mechanism exists
  • Model may memorize and regurgitate personal data
  • Web scraping bypasses data subjects' choices about data sharing
EDPB Position on Legitimate Interest for AI Training

The EDPB Guidelines 04/2025 establish that:

  1. Legitimate interest for AI training is not automatically available — it requires case-by-case assessment
  2. Web scraping of personal data for AI training faces a particularly high bar
  3. The scale of data collection is a relevant factor — larger datasets require stronger justification
  4. The controller must demonstrate necessity: evidence that personal data is required rather than anonymised or synthetic alternatives
  5. Effective opt-out mechanisms are expected as a minimum safeguard
  6. The balancing test should consider the cumulative impact of multiple AI developers scraping and training on the same data subjects' data

Art. 6(1)(e) — Public Interest

Available to public bodies and organisations performing tasks in the public interest:

  • Academic research institutions training AI models for publicly beneficial research
  • Government agencies training AI for public service delivery
  • Must have a basis in national or Union law
  • Proportionality requirements apply

Special Situations

Web-Scraped Data

The EDPB has given specific guidance on web scraping for AI training:

  1. Robots.txt is not consent: Compliance with robots.txt does not establish lawful basis
  2. Public availability is not a lawful basis: Data being publicly accessible does not mean it can be freely used for AI training
  3. Reasonable expectations: Data subjects who post content online do not reasonably expect it to be used for AI training
  4. Children's data: Web-scraped data likely contains children's data — heightened protections apply
  5. Technical measures: Data subjects who implement privacy settings have expressed a preference against broad data use

Assessment framework for web-scraped training data:

FactorHigh Lawfulness IndicatorLow Lawfulness Indicator
Data sourceExplicitly open-licence data (CC0, public domain)Personal profiles, social media, private websites
Data typeFactual, non-personal contentIdentifiable personal information, photos, opinions
Data subject expectationsData published with intent for wide reuseData shared in specific context (social media, forums)
SafeguardsDifferential privacy, PII filtering pre-trainingNo preprocessing to remove personal data
Opt-outEffective and accessible opt-out mechanismNo opt-out or technically impractical opt-out
TransparencyPrivacy notice covers AI training useNo notice to data subjects about AI training
Third-Party Datasets

When using datasets obtained from third parties:

  1. Upstream lawful basis verification: The controller must verify that the data provider had a lawful basis to collect and share the data
  2. Contractual warranties: Obtain warranties from the provider regarding lawful collection, consent scope, and right to license for AI training
  3. Due diligence: Conduct reasonable due diligence on the provider's data collection practices
  4. Chain of accountability: The AI developer remains a controller responsible for lawful processing, even if the data was provided by a third party
Public Datasets

Academic and government datasets require assessment:

  1. Is personal data present? (Many datasets contain inadvertent personal data)
  2. What was the original purpose of the dataset? Is AI training compatible?
  3. Does the dataset licence permit commercial AI training?
  4. Has the dataset been ethically reviewed for consent and privacy?
  5. Are there known issues (bias, PII leakage, consent gaps)?

Training Data Retention

Art. 5(1)(e) storage limitation applies to AI training data:

  • Training data must not be retained longer than necessary for the training purpose
  • Once the model is trained, is continued retention of training data justified?
  • If training data is retained for retraining, what is the maximum retention period?
  • Model artefacts (weights, embeddings) that encode personal data are also subject to retention limits
  • Deletion verification: can the controller demonstrate that training data has been effectively deleted?

Data Subject Rights for Training Data

RightAI Training ApplicationTechnical Challenge
Access (Art. 15)Data subject can request confirmation that their data was used in training and receive a copyIdentifying specific records in large training datasets
Rectification (Art. 16)Inaccurate personal data in training sets must be correctedCorrection may require model retraining
Erasure (Art. 17)Data subjects can request deletion of their data from training setsRequires machine unlearning or model retraining
Objection (Art. 21)Data subjects can object to processing based on legitimate interestController must cease processing unless compelling grounds override
Restriction (Art. 18)Processing must be restricted while accuracy or objection is contestedMay require quarantining data from training pipeline

Enforcement Precedents

  • Garante v. OpenAI (2023): Temporary processing ban — no lawful basis identified for ChatGPT training data. Ordered OpenAI to identify Art. 6(1) basis for training data, implement age verification, and provide opt-out mechanism.
  • CNIL v. Clearview AI (SAN-2022-019, 2022): EUR 20M fine — web scraping of biometric data without lawful basis. No consent, legitimate interest balancing test not conducted.
  • Datatilsynet (Norway) v. Meta (2023): Temporary ban on using Norwegian user data for AI training — legitimate interest basis not sufficiently documented; balancing test inadequate.
  • DPC (Ireland) v. Meta (2024): Investigation into use of public Facebook/Instagram posts for AI training under legitimate interest basis. Meta paused EU AI training following DPC engagement.
  • EDPB Taskforce on ChatGPT (2024): Coordinated finding that legitimate interest for LLM training requires comprehensive balancing test, transparency, and effective opt-out — mere assertion of legitimate interest is insufficient.

Integration Points

  • ai-dpia: Training data lawfulness feeds into DPIA Phase 2 assessment
  • ai-data-subject-rights: Rights exercise mechanisms for training data
  • ai-data-retention: Retention and deletion requirements for training datasets
  • ai-transparency-reqs: Transparency obligations regarding training data use

AI Transparency Requirements

Overview

AI transparency operates at the intersection of two regulatory frameworks: the GDPR's data subject information rights (Arts. 13-14) and the EU AI Act's transparency obligations (Arts. 13-14, 50). Together they require controllers and deployers to provide meaningful, accessible information about AI system capabilities, limitations, decision logic, and personal data processing. This skill implements the combined transparency framework, addressing both the technical explainability challenge of complex ML models and the legal obligation to communicate AI processing in plain language to affected individuals.

GDPR Transparency for AI Systems

Art. 13-14 Information Requirements Applied to AI

When personal data is processed by AI systems, data subjects must receive:

Information ElementGDPR ArticleAI-Specific Application
Purposes of processingArt. 13(1)(c) / 14(1)(c)Specific AI use case, not generic "service improvement"
Lawful basisArt. 13(1)(c) / 14(1)(c)The basis for AI training and for AI inference separately
Legitimate interestArt. 13(1)(d) / 14(2)(b)The specific interest served by AI processing
RecipientsArt. 13(1)(e) / 14(1)(e)AI infrastructure providers, model hosting services
International transfersArt. 13(1)(f) / 14(1)(f)Where AI processing occurs (training and inference locations)
Retention periodArt. 13(2)(a) / 14(2)(a)Training data retention, inference log retention, model lifecycle
Data subject rightsArt. 13(2)(b) / 14(2)(c)Including AI-specific rights: explanation, contestation, human review
Automated decision-makingArt. 13(2)(f) / 14(2)(g)Meaningful information about logic, significance, and envisaged consequences
Source of dataArt. 14(2)(f)Training data sources (categories, not necessarily individual sources)

Art. 13(2)(f) / 14(2)(g) — Meaningful Information About Automated Logic

This is the most challenging transparency requirement for AI systems. The EDPB and Article 29 Working Party have clarified:

What "meaningful information about the logic involved" requires:

  1. System-level explanation: General description of how the AI system works — what factors are considered, what methodology is used, how the model was trained
  2. Purpose and context: Why the AI system is used and what role its output plays in decisions
  3. Key variables: The main data points or features that influence the AI output, without requiring disclosure of proprietary algorithms
  4. Decision criteria: How the model output translates into a decision (e.g., score thresholds, classification categories)
  5. Significance: What the decision means for the data subject in practical terms
  6. Consequences: The potential effects (both intended and foreseeable) of the AI-driven decision

What it does not require:

  • Full disclosure of source code or model weights
  • Mathematical description of the algorithm
  • Proprietary trade secrets (but this does not exempt from meaningful explanation)
  • Explanation of individual model predictions in real-time (though this may be required under Art. 22)

Layered Approach to AI Transparency

The EDPB recommends a layered transparency approach:

LayerContentDelivery
Layer 1: Initial noticeAI is used in processing; general purpose; link to full informationAt point of interaction (banner, tooltip, notification)
Layer 2: SummaryAI system description, key data used, decision logic summary, rights availablePrivacy notice section, AI information page
Layer 3: Detailed informationFull technical description, training data categories, fairness measures, accuracy metrics, limitationsSupplementary documentation, upon request
Layer 4: Individual explanationSpecific factors influencing a particular decision, appeal mechanismUpon request or automatically for significant decisions

EU AI Act Transparency Obligations

Art. 13 — Transparency and Provision of Information to Deployers (High-Risk)

High-risk AI systems (Annex III) must be designed and developed to ensure:

RequirementDescription
InterpretabilitySystem design enables deployers to interpret outputs and use them appropriately
Instructions for useDetailed documentation of capabilities, limitations, intended purpose, foreseeable misuse
Performance metricsAccuracy levels, robustness metrics, known limitations for specific groups
Human oversight infoDescription of human oversight measures and how to implement them
Input data specsDescription of input data the system was designed to process
Training data descriptionRelevant information about training data including provenance and preprocessing

Art. 14 — Human Oversight (High-Risk)

High-risk AI systems must be designed to enable effective human oversight:

  • Clear indication of AI system outputs and confidence levels
  • Ability to correctly interpret AI outputs in context
  • Ability to override or reverse AI decisions
  • Ability to intervene or stop the system ("stop button")
  • Awareness of automation bias risk

Art. 50 — Transparency for Specific AI Systems

AI System TypeTransparency Obligation
AI interacting with personsInform that they are interacting with an AI system (unless obvious from context)
Emotion recognition / biometric categorisationInform about the system's operation and process personal data in compliance with GDPR
AI-generated or manipulated content (deepfakes)Label content as AI-generated in a machine-readable format
AI-generated text on matters of public interestDisclose that the text has been artificially generated or manipulated

Art. 50(1) — AI Interaction Notification

Controllers must inform natural persons that they are interacting with an AI system. This applies to:

  • Chatbots and virtual assistants
  • AI-powered customer service systems
  • Automated email or message generation
  • AI-driven recommendation systems with direct user interface
  • Voice-based AI assistants

Exceptions: where it is obvious from the circumstances and context that the person is interacting with AI (e.g., a robot in a factory setting).

AI Transparency Documentation Framework

Model Card Requirements

For each deployed AI model, maintain a model card containing:

SectionContent
Model overviewName, version, type, developer, deployment date
Intended useSpecific purpose, target users, deployment context
Out-of-scope useUses the model is not designed for; foreseeable misuse
Training data summaryData sources (categories), volume, temporal range, geographic scope, known biases
Performance metricsAccuracy, precision, recall, F1 by relevant subgroup; fairness metrics
LimitationsKnown failure modes, demographic performance disparities, edge cases
Privacy propertiesDifferential privacy applied (epsilon), membership inference test results, training data extraction risk
Human oversightLevel of oversight required, reviewer qualifications, override procedures
Update historyRetraining dates, data updates, performance changes

AI System Transparency Register

Organisations operating multiple AI systems should maintain a central register:

FieldDescription
System IDUnique identifier
System nameHuman-readable name
AI Act classificationUnacceptable / High / Limited / Minimal
PurposeSpecific processing purpose
Data subjects affectedCategories and estimated numbers
Personal data processedAt training and inference
Decision authorityAI decision-support vs. automated decision
Transparency measuresNotification, explanation, documentation
DeployerInternal / External deployment
Registration dateEU AI Act database registration (if high-risk)

Explainability Techniques for Compliance

Global Explainability (System-Level)

Techniques for providing Art. 13(2)(f) "meaningful information about the logic":

TechniqueBest ForLimitation
Feature importance (SHAP, LIME)Identifying key variablesMay oversimplify complex interactions
Decision rules extractionConverting model logic to human-readable rulesLoss of accuracy for complex models
Partial dependence plotsShowing how features affect predictionsAssumes feature independence
Counterfactual explanationsShowing what change would lead to different outcomeComputationally expensive for many features
Attention visualisationTransformer models — showing what the model focuses onAttention does not always equal importance

Local Explainability (Individual Decision)

For Art. 22 right to explanation of individual decisions:

TechniqueDescriptionUse Case
LIMELocal Interpretable Model-agnostic ExplanationsAny model — approximate local behaviour with interpretable model
SHAP valuesShapley Additive Explanations for individual predictionsFeature contribution to specific prediction
Counterfactual"You were denied because X; if X were Y, outcome would be different"Credit, hiring, insurance decisions
AnchorsSufficient conditions for a predictionRule-based explanation of individual case
Concept-basedHigh-level concepts that influenced the decisionWhen features are not directly interpretable

Enforcement Precedents

  • Garante v. OpenAI (2023): Required transparency about AI training data processing, model capabilities, and limitations — privacy notice deemed insufficient for AI system transparency.
  • CNIL v. Clearview AI (SAN-2022-019, 2022): EUR 20M fine — complete absence of transparency about facial recognition AI processing; data subjects had no notice their images were scraped and processed.
  • Austrian DPA v. CRIF (DSB-D213.636, 2023): Credit scoring AI — insufficient explanation of automated decision-making logic per Art. 13(2)(f); data subject received only a score without meaningful information about factors.
  • Dutch DPA v. Tax Authority (SyRI, 2020): Court found algorithmic fraud detection lacked transparency — citizens could not understand how the system assessed them, violating right to private life.
  • AEPD v. CaixaBank (PS/00421/2020, 2021): EUR 6M fine — automated credit decision-making without adequate transparency about the logic involved and significance for data subjects.

Integration Points

  • ai-automated-decisions: Art. 22 explanation requirements integrate with transparency obligations
  • ai-dpia: Transparency assessment is part of DPIA necessity and proportionality analysis
  • ai-act-high-risk-docs: Art. 13 AI Act documentation requirements overlap with transparency framework
  • ai-deployment-checklist: Pre-deployment transparency validation is a checklist item

AI Vendor Privacy Due Diligence

Overview

AI services create complex controller-processor relationships that differ significantly from traditional data processing arrangements. Whether an AI vendor is a processor, joint controller, or independent controller depends on the degree of autonomy the vendor has over personal data processing — particularly regarding model training on customer data, data retention for model improvement, and the vendor's independent purposes for the data. This skill provides the framework for determining controller-processor roles in AI service relationships, conducting privacy due diligence on AI vendors, and establishing appropriate contractual protections.

Controller-Processor Determination for AI

Decision Framework

AI Service ModelTypical RoleKey FactorsGDPR Article
SaaS AI — Customer data processed per instructionsVendor = ProcessorVendor processes data solely on controller's instructions; no independent useArt. 28 DPA required
SaaS AI — Customer data used for model trainingVendor = Joint Controller or Independent ControllerVendor uses customer data for own model improvement beyond contracted serviceArt. 26 JCA or separate controller notice
Embedded AI — Pre-trained model in customer infrastructureCustomer = Controller; Vendor = may be processor for supportModel runs in customer environment; vendor may access data for support/updatesArt. 28 if vendor accesses data
API-based AI — Customer sends data for inferenceVendor = Processor (if no data retention) or Joint Controller (if training on inputs)Depends on whether vendor retains, uses, or trains on input dataAssessment required
AI Platform — Customer builds models on vendor platformVendor = Processor for infrastructure; Controller for platform dataVendor provides compute; customer controls data and modelArt. 28 DPA + audit rights
AI Marketplace — Pre-built models with customer dataDepends on data flowIf customer data enters vendor model → joint controller assessmentCase-by-case

Key Determination Questions

  1. Who determines the purpose of processing? — The entity deciding why personal data is processed
  2. Who determines the means of processing? — The entity deciding how data is processed (but "non-essential means" may be delegated to processor)
  3. Does the vendor use data for its own purposes? — Model training, benchmarking, product improvement using customer data
  4. Does the vendor retain data beyond service delivery? — Data kept after inference, stored for training, retained in logs
  5. Does the vendor make independent decisions about the data? — Choosing to train models, sharing with sub-processors not instructed by customer

Common AI Vendor Patterns

Pattern 1: Pure Inference API (Vendor = Processor)
  • Customer sends data; receives inference result
  • Vendor does not retain input data beyond processing
  • Vendor does not train on customer data
  • Vendor acts solely on customer's instructions
  • Contractual: Art. 28 DPA with clear scope
Pattern 2: AI with Model Improvement (Vendor = Joint Controller)
  • Customer sends data; receives inference result
  • Vendor retains data to improve its models
  • Vendor determines that model training is a purpose
  • Both parties benefit from the model improvement
  • Contractual: Art. 26 Joint Controller Agreement
Pattern 3: AI with Anonymised Analytics (Assessment Required)
  • Customer sends data; receives inference result
  • Vendor claims to anonymise data and use aggregated analytics
  • If anonymisation is effective: no personal data processing for analytics
  • If anonymisation is not effective (re-identification possible): vendor is controller for analytics
  • Assessment: Verify anonymisation effectiveness per WP216
Pattern 4: Embedded AI with Telemetry (Vendor = Processor + may be Controller)
  • AI model runs in customer environment
  • Vendor collects telemetry data including model performance
  • If telemetry contains personal data: vendor may be controller for telemetry processing
  • Contractual: Art. 28 DPA for model support + separate arrangement for telemetry

Due Diligence Assessment Framework

Phase 1: Vendor AI Processing Inventory

For each AI vendor, document:

ElementDocumentation Required
AI capabilitiesWhat AI functions does the vendor provide?
Personal data inputsWhat personal data is sent to the vendor?
Personal data outputsWhat personal data does the vendor return?
Data retentionDoes the vendor retain input data? For how long?
Model trainingDoes the vendor train on customer data?
Sub-processorsDoes the vendor use sub-processors for AI processing? Where?
Data locationWhere is AI processing performed? What jurisdictions?
Security measuresWhat security controls protect data during AI processing?
Human reviewDoes vendor personnel access customer data?

Phase 2: Privacy Risk Assessment

Risk FactorAssessmentRisk Level
Data sensitivitySpecial category data sent to AI vendor?
Data volumeVolume of personal data processed by vendor
Vendor data useVendor uses customer data for own purposes?
International transfersData processed outside EU/EEA?
Sub-processor chainNumber and location of sub-processors
Security postureCertifications (ISO 27001, SOC 2)?
Incident historyPrior data breaches or enforcement actions?
AI-specific risksModel memorization, output leakage, bias?

Phase 3: Contractual Assessment

Contractual ElementRequired?Status
Art. 28 DPA or Art. 26 JCAYes
Processing scope and purpose limitationYes
Prohibition on data use beyond instructions (if processor)Yes
Model training opt-outYes (if vendor trains on data)
Sub-processor notification and approvalYes
International transfer safeguardsIf applicable
Data deletion on terminationYes
Audit rightsYes
Breach notification obligationsYes
AI-specific: model privacy testingRecommended
AI-specific: bias assessment obligationsRecommended for high-risk
AI-specific: output accuracy warrantiesRecommended

AI-Specific Contractual Clauses

Model Training Restrictions

The Processor shall not use Customer Data to train, improve, fine-tune, or
otherwise develop any machine learning model, algorithm, or AI system,
whether for the Customer's benefit or for any other purpose, without prior
written consent from the Customer. Any consent granted shall specify the
scope of permitted training, the data categories involved, and the privacy
safeguards to be applied.

AI Output Accuracy

The Provider acknowledges that AI system outputs about identifiable data
subjects must comply with the accuracy principle under Art. 5(1)(d) GDPR.
The Provider shall implement measures to minimise inaccurate outputs about
data subjects and shall promptly correct inaccurate outputs upon
notification.

Model Privacy and Bias Obligations

The Provider shall conduct periodic privacy audits of AI models processing
Customer Data, including membership inference testing and training data
extraction testing, and shall make results available to the Customer upon
request. The Provider shall monitor AI systems for discriminatory outcomes
and shall implement bias mitigation measures as required.

Enforcement Relevance

  • EDPB Guidelines 07/2020 on Controller-Processor: Determination depends on factual circumstances, not contractual labels. A vendor labelled "processor" that uses data for its own purposes is factually a controller.
  • Garante v. OpenAI (2023): OpenAI's role as controller for ChatGPT training data confirmed — processing personal data for model training is an independent controller purpose.
  • CJEU C-40/17 (Fashion ID): Joint controller status can arise when a party has influence over the purpose and means of processing, even without access to the data.
  • DPC v. Meta (WhatsApp, 2023): EUR 5.5M fine — processor-controller determination must reflect actual data use, not just contractual terms.

Integration Points

  • ai-dpia: Vendor AI processing must be included in DPIA scope
  • ai-training-lawfulness: Vendor training on customer data requires lawful basis assessment
  • ai-transparency-reqs: Data subjects must be informed about AI vendor processing
  • ai-deployment-checklist: Vendor due diligence is a pre-deployment checklist item

LLM Output Privacy Risk Assessment

Overview

Large language models (LLMs) present unique privacy risks that go beyond traditional ML systems. Because LLMs are trained on massive corpora that may contain personal data, they can memorise and reproduce verbatim training data — including names, email addresses, phone numbers, and other PII. Additionally, LLMs can hallucinate plausible but false personal data, creating defamation and accuracy risks. Prompt injection attacks can bypass safety guardrails to extract training data or system prompts containing confidential information. This skill provides a structured framework for assessing, mitigating, and monitoring privacy risks in LLM-generated outputs at Cerebrum AI Labs.

LLM Output Privacy Risk Categories

Risk 1: Training Data Memorisation

LLMs memorise training data, particularly sequences that appear multiple times or are distinctive. Extractable memorisation occurs when a model, given a prefix, completes the text with verbatim training data.

FactorImpact on Memorisation Risk
Model sizeLarger models memorise more (Carlini et al., 2023)
Data duplicationRepeated sequences are memorised at higher rates
Training epochsMore passes over data increase memorisation
Data distinctivenessUnique sequences (names, numbers) are more extractable
Temperature at inferenceLower temperature increases verbatim reproduction

Regulatory concern: If a model reproduces personal data from training, this constitutes processing under GDPR Art. 4(2). The data subject has not consented to this output, and the controller must have a lawful basis for the disclosure.

Risk 2: PII Leakage in Generated Text

Even without verbatim memorisation, models can combine partial information to produce outputs containing personal data — email patterns, phone number formats with real area codes, or names associated with specific contexts.

Leakage TypeExampleDetection Method
Verbatim reproductionModel outputs exact email address from training dataExact match against known training PII
RecombinationModel combines real first name + real surname from different recordsNamed entity recognition + cross-reference
Pattern completionModel generates plausible phone number matching real formatRegex + validation against real registries
Contextual leakageModel associates real person with correct employer when promptedEntity relationship extraction

Risk 3: Prompt Injection and Data Extraction

Adversarial prompts can cause models to bypass safety instructions and output training data, system prompts, or information about other users' conversations.

Attack VectorDescriptionMitigation
Direct prompt injectionUser crafts prompt to extract training data ("Repeat your training data starting with...")Input filtering + output monitoring
Indirect prompt injectionMalicious content in retrieved documents instructs model to leak dataSanitise retrieved content + instruction hierarchy
System prompt extractionPrompts designed to reveal system-level instructionsSystem prompt isolation + extraction detection
Multi-turn extractionGradual extraction across conversation turnsConversation-level monitoring
JailbreakingBypassing safety alignment to disable PII filtersRobust alignment + secondary output filter

Risk 4: Hallucinated Personal Data

LLMs can generate plausible but fabricated personal data — attributing false statements to real people, inventing credentials, or creating fictional but realistic-seeming personal records.

Hallucination TypePrivacy RiskLegal Concern
False attributionModel attributes a statement to a real person who never said itDefamation, Art. 5(1)(d) accuracy principle
Fabricated credentialsModel invents qualifications for a real personAccuracy, potential reputational harm
Fictional PIIModel generates realistic but fake personal details (e.g., social security numbers)May match real individuals by coincidence
Confabulated relationshipsModel invents relationships between real peopleArt. 5(1)(d) accuracy, potential harm

EDPB position: EDPB Guidelines 04/2025 state that generating inaccurate personal data about identifiable individuals violates the accuracy principle (Art. 5(1)(d)) and may require the controller to implement effective rectification mechanisms.

Output Privacy Risk Assessment Framework

Assessment Dimensions

DimensionWeightScoring Criteria
Memorisation exposure25%Model size, training data PII density, deduplication status
PII leakage likelihood25%Output modality, user interaction pattern, PII filter coverage
Prompt injection resilience20%Input validation, instruction hierarchy, extraction testing results
Hallucination risk15%Factual grounding, attribution verification, entity linking
Monitoring coverage15%Real-time PII detection, logging, alerting, human review

Risk Scoring

Score RangeRisk LevelAction Required
0-20LowStandard monitoring; annual review
21-40ModerateEnhanced PII filtering; quarterly review
41-60ElevatedMandatory output scanning; human review for sensitive queries
61-80HighRestrict deployment scope; implement guardrails before production
81-100CriticalDo not deploy; fundamental architecture changes required

Mitigation Measures

Pre-Deployment Mitigations

MeasureDescriptionEffectiveness
Training data deduplicationRemove duplicate sequences to reduce memorisationHigh for verbatim memorisation
PII scrubbing of training dataDetect and remove/mask PII before trainingHigh but incomplete (novel PII patterns missed)
Differential privacy trainingTrain with DP-SGD to bound memorisationHigh theoretical guarantee; accuracy trade-off
RLHF safety alignmentTrain model to refuse PII-generating requestsModerate; can be bypassed by jailbreaking
Instruction tuningFine-tune model to follow safety instructionsModerate; requires robust instruction hierarchy

Runtime Mitigations

MeasureDescriptionEffectiveness
Output PII scannerNER-based scanner detecting PII in outputs before deliveryHigh for known PII patterns
Regex PII filterPattern matching for emails, phones, SSNs, credit cardsHigh for structured PII
Input prompt classifierClassify incoming prompts for injection/extraction attemptsModerate; adversarial prompts evolve
Retrieval groundingGround outputs in retrieved documents to reduce hallucinationModerate; depends on retrieval quality
Rate limitingLimit query volume to slow extraction attacksLow-moderate; raises cost of attack
Conversation monitoringAnalyse multi-turn conversations for progressive extractionModerate; complex to implement

Post-Deployment Monitoring

MonitorMetricAlert Threshold
PII detection rate% of outputs flagged by PII scanner>0.1% of outputs in any 1-hour window
Prompt injection detection% of inputs classified as injection attempts>0.5% of inputs in any 1-hour window
User reports of PII exposureCount of user-reported PII incidentsAny confirmed report
Extraction pattern detectionRepeated similar queries from same user/IP>10 similar extraction-pattern queries per session
Hallucination complaintsReports of false personal informationAny confirmed report about real individuals

Cerebrum AI Labs — LLM Output Privacy Controls

Architecture

User Input → Input Filter → LLM → Output PII Scanner → Delivery
                │                         │
                ▼                         ▼
         Prompt Injection          PII Detected?
         Classifier                  │
                │                YES → Redact + Log + Alert
             Block/Allow         NO  → Deliver + Log

PII Detection Categories for Output Scanning

PII TypeDetection MethodAction on Detection
Email addressesRegex + validationRedact and log
Phone numbersRegex + country format validationRedact and log
National ID numbersCountry-specific regex patternsBlock output + alert
Credit card numbersLuhn algorithm + regexBlock output + alert
Physical addressesNER + geocoding validationRedact and log
Person names + contextNER + co-occurrence with other PIIFlag for review
Medical informationMedical NER + PHI patternsBlock output + alert

Enforcement Relevance

  • EDPB Guidelines 04/2025: Controllers deploying LLMs must assess memorisation risk and implement measures to prevent unlawful disclosure of personal data in outputs.
  • Garante v. OpenAI (2023): Italian DPA ordered temporary ban partly due to lack of age verification and data accuracy concerns in ChatGPT outputs. Required implementation of output filtering.
  • EDPB ChatGPT Taskforce Report (2024): Recommended that LLM providers implement technical measures to minimise inaccurate personal data in outputs and provide effective rectification mechanisms.
  • CNIL AI Action Plan (2024): French DPA guidance emphasises that LLM outputs containing personal data constitute processing, requiring lawful basis and compliance with data quality principles.

Integration Points

  • ai-model-privacy-audit: Memorisation testing is part of the model privacy audit
  • ai-data-subject-rights: Rectification of hallucinated data, erasure requests
  • ai-transparency-reqs: Disclosure that system is AI-generated, accuracy limitations
  • ai-deployment-checklist: Output privacy controls verified before deployment

Sponsors

Details

Marketplace
mukul975/Privacy-Data-Protection-Skills
Source
plugins/ai-privacy-governance-skills
Author
Mahipal
Category
Compliance
License
Apache-2.0
Keywords
ai-privacy, ai-governance, ai-dpia, llm-privacy, ai-bias

More from mukul975/Privacy-Data-Protection-Skills

All 21