
privacy-engineering-skills
14 privacy engineering skills: differential privacy, PII detection, NIST Privacy Framework, privacy APIs, data sharing, metrics
- version
- 1.0.0
- GitHub stars
- 340
Install
/plugin install privacy-engineering-skills@privacy-data-protection-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add mukul975/Privacy-Data-Protection-Skills.
Contents
Discovered from sourceSkills 14
Kantara Initiative Consent Receipt Specification
Overview
A consent receipt is a record of a consent transaction provided to the individual (data subject) as evidence that consent was given. The Kantara Initiative Consent Receipt Specification v1.1 defines a standard, machine-readable format that enables individuals to track and manage their consent across multiple organizations. This skill covers the implementation of consent receipts aligned with the Kantara specification and ISO/IEC 27560:2023.
Consent Receipt Structure
Core Fields (Kantara v1.1)
| Field | Type | Required | Description |
|---|---|---|---|
| version | String | Yes | Specification version (e.g., "KI-CR-v1.1.0") |
| jurisdiction | String | Yes | Legal jurisdiction (ISO 3166-1 alpha-2) |
| consentTimestamp | DateTime | Yes | UTC timestamp of consent grant |
| collectionMethod | String | Yes | How consent was collected (web form, verbal, paper) |
| consentReceiptID | UUID | Yes | Unique identifier for this receipt |
| publicKey | String | No | Public key for receipt verification |
| language | String | Yes | Language of the consent interaction (BCP 47) |
| piiPrincipalId | String | Yes | Identifier for the data subject |
| piiControllers | Array | Yes | List of data controllers |
| policyUrl | URL | Yes | Link to the privacy policy |
| services | Array | Yes | Services for which consent is given |
| sensitive | Boolean | Yes | Whether special category data is involved |
| spiCat | Array | No | Special categories of data processed |
PII Controller Object
| Field | Type | Required | Description |
|---|---|---|---|
| piiController | String | Yes | Name of the controller organization |
| onBehalf | Boolean | No | Whether acting on behalf of another controller |
| contact | String | Yes | Contact information for the controller |
| address | Object | Yes | Physical address of the controller |
| String | Yes | Contact email address | |
| phone | String | No | Contact phone number |
| piiControllerUrl | URL | No | URL of the controller's website |
Service Object
| Field | Type | Required | Description |
|---|---|---|---|
| service | String | Yes | Name of the service |
| purposes | Array | Yes | List of processing purposes |
Purpose Object
| Field | Type | Required | Description |
|---|---|---|---|
| purpose | String | Yes | Description of the processing purpose |
| purposeCategory | Array | Yes | Category codes for the purpose |
| consentType | String | Yes | "explicit" or "implicit" |
| piiCategory | Array | Yes | Categories of PII processed |
| primaryPurpose | Boolean | Yes | Whether this is the primary purpose |
| termination | String | Yes | How consent can be withdrawn |
| thirdPartyDisclosure | Boolean | Yes | Whether data is shared with third parties |
| thirdPartyName | String | No | Name of third party (if applicable) |
Machine-Readable Receipt Format
JSON Consent Receipt
{
"version": "KI-CR-v1.1.0",
"jurisdiction": "EU",
"consentTimestamp": "2026-03-14T10:30:00.000Z",
"collectionMethod": "web_form",
"consentReceiptID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
"language": "en",
"piiPrincipalId": "user-98765",
"piiControllers": [
{
"piiController": "Cipher Engineering Labs",
"onBehalf": false,
"contact": "Data Protection Officer",
"address": {
"streetAddress": "100 Technology Drive",
"locality": "London",
"region": "Greater London",
"postalCode": "EC2A 1NT",
"country": "GB"
},
"email": "dpo@cipherengineeringlabs.com",
"phone": "+44-20-7946-0958",
"piiControllerUrl": "https://www.cipherengineeringlabs.com"
}
],
"policyUrl": "https://www.cipherengineeringlabs.com/privacy-policy",
"services": [
{
"service": "Privacy Analytics Platform",
"purposes": [
{
"purpose": "Provide personalized privacy compliance recommendations",
"purposeCategory": ["core_service"],
"consentType": "explicit",
"piiCategory": ["contact_information", "professional_information"],
"primaryPurpose": true,
"termination": "Account settings > Privacy > Withdraw consent",
"thirdPartyDisclosure": false
},
{
"purpose": "Send product updates and feature announcements",
"purposeCategory": ["marketing"],
"consentType": "explicit",
"piiCategory": ["contact_information"],
"primaryPurpose": false,
"termination": "Unsubscribe link in email or Account settings",
"thirdPartyDisclosure": false
},
{
"purpose": "Aggregate usage analytics to improve platform features",
"purposeCategory": ["analytics"],
"consentType": "explicit",
"piiCategory": ["usage_data", "device_information"],
"primaryPurpose": false,
"termination": "Account settings > Privacy > Analytics opt-out",
"thirdPartyDisclosure": true,
"thirdPartyName": "Cipher Analytics Processing Ltd"
}
]
}
],
"sensitive": false,
"spiCat": []
}
JWT-Based Verification Mechanism
Signed Consent Receipt Implementation
"""
Consent receipt generation and verification using JWT (JSON Web Tokens).
Implements Kantara Initiative Consent Receipt Specification v1.1
with cryptographic verification.
"""
import json
import uuid
from datetime import datetime, timezone
from typing import Optional
import jwt # PyJWT
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.backends import default_backend
class ConsentReceiptIssuer:
"""
Issue and sign consent receipts as JWTs.
Uses RS256 (RSA with SHA-256) for signing.
"""
def __init__(self, private_key_pem: bytes, issuer_name: str):
"""
Args:
private_key_pem: PEM-encoded RSA private key
issuer_name: Name of the issuing organization
"""
self.private_key = serialization.load_pem_private_key(
private_key_pem, password=None, backend=default_backend()
)
self.public_key = self.private_key.public_key()
self.issuer_name = issuer_name
def issue_receipt(
self,
principal_id: str,
services: list[dict],
jurisdiction: str,
collection_method: str,
policy_url: str,
sensitive: bool = False,
language: str = "en",
controller_info: dict = None
) -> tuple[str, str]:
"""
Generate a signed consent receipt.
Args:
principal_id: Data subject identifier
services: List of service/purpose objects
jurisdiction: Legal jurisdiction code
collection_method: How consent was collected
policy_url: URL of the privacy policy
sensitive: Whether special categories are involved
language: Language of consent interaction
controller_info: PII controller details
Returns:
Tuple of (receipt_id, signed_jwt_string)
"""
receipt_id = str(uuid.uuid4())
if controller_info is None:
controller_info = {
"piiController": self.issuer_name,
"onBehalf": False,
"contact": "Data Protection Officer",
"email": f"dpo@{self.issuer_name.lower().replace(' ', '')}.com"
}
receipt_payload = {
"version": "KI-CR-v1.1.0",
"jurisdiction": jurisdiction,
"consentTimestamp": datetime.now(timezone.utc).isoformat(),
"collectionMethod": collection_method,
"consentReceiptID": receipt_id,
"language": language,
"piiPrincipalId": principal_id,
"piiControllers": [controller_info],
"policyUrl": policy_url,
"services": services,
"sensitive": sensitive,
"spiCat": [],
# JWT standard claims
"iss": self.issuer_name,
"sub": principal_id,
"iat": int(datetime.now(timezone.utc).timestamp()),
"jti": receipt_id,
}
# Sign the receipt as a JWT
signed_token = jwt.encode(
receipt_payload,
self.private_key,
algorithm="RS256",
headers={"kid": f"{self.issuer_name}-consent-key-1"}
)
return receipt_id, signed_token
def get_public_key_pem(self) -> str:
"""Export the public key for verification by data subjects."""
return self.public_key.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo
).decode("utf-8")
class ConsentReceiptVerifier:
"""
Verify signed consent receipts.
Used by data subjects or auditors to confirm receipt authenticity.
"""
def __init__(self):
self.trusted_keys: dict[str, bytes] = {}
def register_issuer_key(self, issuer_name: str, public_key_pem: str):
"""Register a trusted issuer's public key."""
self.trusted_keys[issuer_name] = public_key_pem.encode("utf-8")
def verify_receipt(self, token: str) -> tuple[bool, Optional[dict], Optional[str]]:
"""
Verify a signed consent receipt JWT.
Returns:
Tuple of (is_valid, decoded_payload_or_None, error_message_or_None)
"""
try:
# Decode without verification first to get the issuer
unverified = jwt.decode(token, options={"verify_signature": False})
issuer = unverified.get("iss")
if issuer not in self.trusted_keys:
return (False, None, f"Unknown issuer: {issuer}")
public_key = serialization.load_pem_public_key(
self.trusted_keys[issuer],
backend=default_backend()
)
# Verify the signature
decoded = jwt.decode(
token,
public_key,
algorithms=["RS256"],
issuer=issuer
)
# Validate required Kantara fields
required_fields = [
"version", "jurisdiction", "consentTimestamp",
"collectionMethod", "consentReceiptID", "piiPrincipalId",
"piiControllers", "policyUrl", "services", "sensitive"
]
missing = [f for f in required_fields if f not in decoded]
if missing:
return (False, decoded, f"Missing required fields: {missing}")
return (True, decoded, None)
except jwt.ExpiredSignatureError:
return (False, None, "Receipt JWT has expired")
except jwt.InvalidSignatureError:
return (False, None, "Invalid signature — receipt may have been tampered with")
except jwt.DecodeError as e:
return (False, None, f"Failed to decode JWT: {str(e)}")
Receipt Lifecycle Management
Lifecycle States
ISSUED --> ACTIVE --> WITHDRAWN
| | |
| v v
| UPDATED ARCHIVED
| |
v v
EXPIRED ACTIVE (new version)
Lifecycle Manager
"""
Manage the lifecycle of consent receipts including
updates, withdrawals, and expiration tracking.
"""
from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum
class ReceiptStatus(Enum):
ISSUED = "issued"
ACTIVE = "active"
UPDATED = "updated"
WITHDRAWN = "withdrawn"
EXPIRED = "expired"
ARCHIVED = "archived"
@dataclass
class ReceiptRecord:
receipt_id: str
principal_id: str
status: ReceiptStatus
issued_at: datetime
last_updated: datetime
withdrawn_at: datetime | None
jwt_token: str
version: int
superseded_by: str | None
purposes: list[str]
class ConsentReceiptLifecycle:
"""
Manage consent receipt state transitions and history.
"""
def __init__(self, receipt_store, issuer: ConsentReceiptIssuer):
self.store = receipt_store
self.issuer = issuer
def activate_receipt(self, receipt_id: str) -> bool:
"""Transition a receipt from ISSUED to ACTIVE."""
record = self.store.get(receipt_id)
if record and record.status == ReceiptStatus.ISSUED:
record.status = ReceiptStatus.ACTIVE
record.last_updated = datetime.now(timezone.utc)
self.store.update(record)
return True
return False
def update_receipt(
self,
receipt_id: str,
updated_services: list[dict],
jurisdiction: str,
policy_url: str
) -> str | None:
"""
Update an existing receipt by issuing a new version.
The old receipt is marked as UPDATED and linked to the new one.
Returns the new receipt ID or None if update failed.
"""
old_record = self.store.get(receipt_id)
if not old_record or old_record.status not in [
ReceiptStatus.ACTIVE, ReceiptStatus.ISSUED
]:
return None
# Issue new receipt
new_receipt_id, new_jwt = self.issuer.issue_receipt(
principal_id=old_record.principal_id,
services=updated_services,
jurisdiction=jurisdiction,
collection_method="consent_update",
policy_url=policy_url
)
# Create new record
new_record = ReceiptRecord(
receipt_id=new_receipt_id,
principal_id=old_record.principal_id,
status=ReceiptStatus.ACTIVE,
issued_at=datetime.now(timezone.utc),
last_updated=datetime.now(timezone.utc),
withdrawn_at=None,
jwt_token=new_jwt,
version=old_record.version + 1,
superseded_by=None,
purposes=[p["purpose"] for s in updated_services for p in s.get("purposes", [])]
)
self.store.save(new_record)
# Mark old receipt as updated
old_record.status = ReceiptStatus.UPDATED
old_record.superseded_by = new_receipt_id
old_record.last_updated = datetime.now(timezone.utc)
self.store.update(old_record)
return new_receipt_id
def withdraw_consent(self, receipt_id: str, reason: str = "") -> bool:
"""
Withdraw consent and mark the receipt accordingly.
Returns True if withdrawal was successful.
"""
record = self.store.get(receipt_id)
if not record or record.status not in [
ReceiptStatus.ACTIVE, ReceiptStatus.ISSUED
]:
return False
record.status = ReceiptStatus.WITHDRAWN
record.withdrawn_at = datetime.now(timezone.utc)
record.last_updated = datetime.now(timezone.utc)
self.store.update(record)
return True
def get_active_receipts(self, principal_id: str) -> list[ReceiptRecord]:
"""Get all active consent receipts for a data subject."""
return self.store.find_by_principal(
principal_id, status=ReceiptStatus.ACTIVE
)
def get_receipt_history(self, principal_id: str) -> list[ReceiptRecord]:
"""Get the complete consent receipt history for a data subject."""
return self.store.find_by_principal(principal_id)
ISO/IEC 27560:2023 Alignment
| Kantara CR Field | ISO 27560 Equivalent | Notes |
|---|---|---|
| consentReceiptID | Consent Record Identifier | Unique identifier for the record |
| consentTimestamp | Date and time of consent | When consent was collected |
| piiPrincipalId | PII Principal Identifier | Data subject identifier |
| piiControllers | PII Controller | Organization processing data |
| services.purposes | Purpose(s) | Processing purposes |
| services.purposes.piiCategory | PII Categories | Types of personal data |
| sensitive | Sensitive PII indicator | Special category flag |
| jurisdiction | Jurisdiction | Applicable legal framework |
| collectionMethod | Mechanism of consent | How consent was obtained |
| policyUrl | Privacy notice reference | Link to privacy notice |
References
- Kantara Initiative Consent Receipt Specification v1.1.0 (2018)
- ISO/IEC 27560:2023 — Privacy Technologies — Consent Record Information Structure
- ISO/IEC 29184:2020 — Guidelines for Online Privacy Notices and Consent
- RFC 7519 — JSON Web Token (JWT)
- W3C Data Privacy Vocabularies and Controls Community Group
- IEEE P7012 — Standard for Machine Readable Personal Privacy Terms
Differential Privacy in Production
Overview
Differential privacy is a mathematical framework for quantifying and bounding the privacy loss incurred when publishing statistical information about a dataset. It provides a provable guarantee that the output of a computation does not significantly depend on whether any single individual's data is included. This skill covers the practical engineering of differential privacy systems for production deployment.
Core Definitions
(epsilon, delta)-Differential Privacy
A randomized mechanism M satisfies (epsilon, delta)-differential privacy if for all neighboring datasets D and D' (differing in at most one record) and for all possible outputs S:
P[M(D) in S] <= e^epsilon * P[M(D') in S] + delta
Where:
- epsilon (privacy loss budget): Quantifies the maximum information leakage. Lower epsilon = stronger privacy.
- delta: Probability of an additional privacy breach beyond the epsilon guarantee. Should be cryptographically small (< 1/n^2 where n is the dataset size).
Sensitivity
The sensitivity of a function f measures how much one individual can affect the output:
- Global sensitivity: max over all neighboring D,D' of |f(D) - f(D')|
- Local sensitivity: for a specific D, max over all neighboring D' of |f(D) - f(D')|
Epsilon Selection Strategy
Epsilon Guidelines by Use Case
| Use Case | Epsilon Range | Rationale |
|---|---|---|
| Census/government statistics | 0.1 - 1.0 | Maximum protection for mandatory participation |
| Healthcare analytics | 0.5 - 2.0 | High sensitivity, regulatory requirements |
| Location analytics | 1.0 - 3.0 | Moderate sensitivity, aggregate insights |
| Product analytics | 1.0 - 5.0 | Lower sensitivity, business utility needs |
| A/B testing | 2.0 - 8.0 | Statistical significance requirements |
| Aggregate reporting | 0.5 - 3.0 | Public-facing outputs need stronger guarantees |
Factors Influencing Epsilon Choice
- Data sensitivity: More sensitive data requires lower epsilon
- Dataset size: Larger datasets tolerate lower epsilon with acceptable utility
- Query frequency: More queries consume more budget (see composition)
- Regulatory requirements: Some regulations imply specific privacy levels
- Utility requirements: Business needs may set a floor on acceptable accuracy
- Audience: Public release needs stronger guarantees than internal analytics
Noise Mechanisms
Laplace Mechanism
For numeric queries with bounded global sensitivity. Provides pure epsilon-differential privacy.
import numpy as np
def laplace_mechanism(true_value: float, sensitivity: float, epsilon: float) -> float:
"""
Apply Laplace noise for epsilon-differential privacy.
Args:
true_value: The true query result
sensitivity: Global sensitivity of the query (L1)
epsilon: Privacy parameter
Returns:
Noisy result satisfying epsilon-differential privacy
"""
scale = sensitivity / epsilon
noise = np.random.laplace(loc=0, scale=scale)
return true_value + noise
def laplace_mechanism_vector(true_values: np.ndarray, sensitivity: float, epsilon: float) -> np.ndarray:
"""Apply Laplace noise to a vector of values."""
scale = sensitivity / epsilon
noise = np.random.laplace(loc=0, scale=scale, size=true_values.shape)
return true_values + noise
Gaussian Mechanism
For numeric queries. Provides (epsilon, delta)-differential privacy with tighter noise for high-dimensional outputs.
import numpy as np
import math
def gaussian_mechanism(true_value: float, sensitivity: float, epsilon: float, delta: float) -> float:
"""
Apply Gaussian noise for (epsilon, delta)-differential privacy.
Uses the analytic Gaussian mechanism calibration.
Args:
true_value: The true query result
sensitivity: Global sensitivity of the query (L2)
epsilon: Privacy parameter
delta: Failure probability parameter
Returns:
Noisy result satisfying (epsilon, delta)-differential privacy
"""
sigma = sensitivity * math.sqrt(2 * math.log(1.25 / delta)) / epsilon
noise = np.random.normal(loc=0, scale=sigma)
return true_value + noise
Exponential Mechanism
For non-numeric outputs (categorical selection) where adding noise directly is not meaningful.
import numpy as np
def exponential_mechanism(
candidates: list,
utility_scores: np.ndarray,
sensitivity: float,
epsilon: float
) -> object:
"""
Select an output using the exponential mechanism.
Args:
candidates: List of possible outputs
utility_scores: Utility score for each candidate
sensitivity: Global sensitivity of the utility function
epsilon: Privacy parameter
Returns:
Selected candidate satisfying epsilon-differential privacy
"""
# Calculate selection probabilities
probabilities = np.exp(epsilon * utility_scores / (2 * sensitivity))
probabilities = probabilities / probabilities.sum()
# Sample according to probabilities
index = np.random.choice(len(candidates), p=probabilities)
return candidates[index]
Randomized Response (Local DP)
For collecting individual data points with local differential privacy.
import random
import math
def randomized_response(true_bit: bool, epsilon: float) -> bool:
"""
Apply randomized response for local differential privacy.
Args:
true_bit: The individual's true binary response
epsilon: Privacy parameter
Returns:
Randomized response satisfying epsilon-local-DP
"""
p = math.exp(epsilon) / (math.exp(epsilon) + 1)
if random.random() < p:
return true_bit # Report truthfully
else:
return not true_bit # Flip the answer
def estimate_from_randomized_responses(
responses: list,
epsilon: float
) -> float:
"""
Estimate true proportion from randomized responses.
Args:
responses: List of randomized boolean responses
epsilon: The epsilon used during collection
Returns:
Estimated true proportion
"""
n = len(responses)
observed_proportion = sum(responses) / n
p = math.exp(epsilon) / (math.exp(epsilon) + 1)
# Correct for randomization bias
estimated_proportion = (observed_proportion - (1 - p)) / (2 * p - 1)
return max(0.0, min(1.0, estimated_proportion))
Privacy Budget Tracking
Budget Manager Implementation
import threading
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
import math
@dataclass
class BudgetAllocation:
query_id: str
epsilon_spent: float
delta_spent: float
timestamp: datetime
query_description: str
analyst_id: str
class PrivacyBudgetManager:
"""
Track and enforce differential privacy budget across queries.
Supports both basic and advanced composition theorems.
"""
def __init__(
self,
total_epsilon: float,
total_delta: float,
composition_method: str = "advanced"
):
self.total_epsilon = total_epsilon
self.total_delta = total_delta
self.composition_method = composition_method
self.allocations: list[BudgetAllocation] = []
self._lock = threading.Lock()
def remaining_budget(self) -> tuple[float, float]:
"""Calculate remaining (epsilon, delta) budget."""
if self.composition_method == "basic":
return self._basic_composition_remaining()
elif self.composition_method == "advanced":
return self._advanced_composition_remaining()
elif self.composition_method == "rdp":
return self._rdp_composition_remaining()
else:
raise ValueError(f"Unknown composition method: {self.composition_method}")
def _basic_composition_remaining(self) -> tuple[float, float]:
"""Basic sequential composition: epsilons and deltas sum."""
spent_epsilon = sum(a.epsilon_spent for a in self.allocations)
spent_delta = sum(a.delta_spent for a in self.allocations)
return (self.total_epsilon - spent_epsilon, self.total_delta - spent_delta)
def _advanced_composition_remaining(self) -> tuple[float, float]:
"""
Advanced composition theorem:
k queries each with epsilon_i satisfy
(sqrt(2k * ln(1/delta')) * max(epsilon_i) + k * epsilon_i * (e^epsilon_i - 1),
k * delta_i + delta')-DP
"""
k = len(self.allocations)
if k == 0:
return (self.total_epsilon, self.total_delta)
epsilons = [a.epsilon_spent for a in self.allocations]
deltas = [a.delta_spent for a in self.allocations]
max_eps = max(epsilons)
sum_delta = sum(deltas)
# Reserve delta' for composition overhead
delta_prime = (self.total_delta - sum_delta) / 2
if delta_prime <= 0:
return (0.0, 0.0)
composed_epsilon = (
math.sqrt(2 * k * math.log(1 / delta_prime)) * max_eps
+ k * max_eps * (math.exp(max_eps) - 1)
)
composed_delta = sum_delta + delta_prime
return (
max(0.0, self.total_epsilon - composed_epsilon),
max(0.0, self.total_delta - composed_delta)
)
def _rdp_composition_remaining(self) -> tuple[float, float]:
"""Renyi DP composition (simplified)."""
# RDP provides tighter bounds through Renyi divergence
spent_epsilon = sum(a.epsilon_spent for a in self.allocations)
spent_delta = sum(a.delta_spent for a in self.allocations)
return (self.total_epsilon - spent_epsilon, self.total_delta - spent_delta)
def request_budget(
self,
query_id: str,
epsilon_requested: float,
delta_requested: float,
query_description: str,
analyst_id: str
) -> bool:
"""
Request budget allocation for a query.
Returns True if budget is available and allocated, False otherwise.
"""
with self._lock:
remaining_eps, remaining_delta = self.remaining_budget()
if epsilon_requested > remaining_eps or delta_requested > remaining_delta:
return False
allocation = BudgetAllocation(
query_id=query_id,
epsilon_spent=epsilon_requested,
delta_spent=delta_requested,
timestamp=datetime.utcnow(),
query_description=query_description,
analyst_id=analyst_id
)
self.allocations.append(allocation)
return True
def get_usage_report(self) -> dict:
"""Generate budget usage report."""
remaining_eps, remaining_delta = self.remaining_budget()
return {
"total_epsilon": self.total_epsilon,
"total_delta": self.total_delta,
"remaining_epsilon": remaining_eps,
"remaining_delta": remaining_delta,
"utilization_pct": (1 - remaining_eps / self.total_epsilon) * 100,
"num_queries": len(self.allocations),
"composition_method": self.composition_method,
"allocations": [
{
"query_id": a.query_id,
"epsilon": a.epsilon_spent,
"delta": a.delta_spent,
"timestamp": a.timestamp.isoformat(),
"analyst": a.analyst_id,
}
for a in self.allocations
],
}
Composition Theorems
Basic Composition
If M1 satisfies (e1, d1)-DP and M2 satisfies (e2, d2)-DP, then releasing both M1(D) and M2(D) satisfies (e1 + e2, d1 + d2)-DP.
Advanced Composition
For k mechanisms each satisfying (epsilon, delta)-DP, the composed mechanism satisfies (epsilon', k*delta + delta')-DP where:
epsilon' = sqrt(2k * ln(1/delta')) * epsilon + k * epsilon * (e^epsilon - 1)
Renyi Differential Privacy (RDP) Composition
RDP provides tighter composition bounds by tracking privacy loss through Renyi divergence of order alpha. Convert to (epsilon, delta)-DP at the end:
For alpha > 1: epsilon(delta) = RDP_alpha - ln(delta) / (alpha - 1)
Moments Accountant
Used by TensorFlow Privacy and Opacus. Tracks the log of the moment-generating function of the privacy loss variable. Provides the tightest known bounds for iterative mechanisms (e.g., DP-SGD).
Production Deployment Patterns
Pattern 1: Central DP Analytics Pipeline
Raw Data Store --> Sensitivity Calibration --> DP Mechanism --> Result Cache
| |
v v
Budget Manager <------------- Audit Log
Pattern 2: Local DP Collection
Client Device --> Local Randomizer --> Aggregation Server --> Estimator
(epsilon-LDP) | |
v v
Budget Tracker Utility Monitor
Pattern 3: DP Model Training (DP-SGD)
Training Data --> Mini-batch Sampling --> Per-example Gradient
(Poisson sampling) |
v
Gradient Clipping (norm bound C)
|
v
Gaussian Noise Addition (sigma * C)
|
v
Model Update --> Privacy Accountant
Utility Measurement
| Metric | Description | Formula |
|---|---|---|
| Mean Absolute Error | Average absolute difference from true value | MAE = (1/n) * sum( |
| Relative Error | Error as fraction of true value | RE = |
| Coverage | Fraction of true values within confidence interval | Count(true in CI) / n |
| Utility Ratio | Ratio of noisy to true signal-to-noise ratio | SNR_noisy / SNR_true |
References
- Dwork, C. and Roth, A. "The Algorithmic Foundations of Differential Privacy." Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407, 2014.
- Mironov, I. "Renyi Differential Privacy." IEEE CSF, 2017.
- Abadi, M. et al. "Deep Learning with Differential Privacy." ACM CCS, 2016.
- Google Differential Privacy Library: github.com/google/differential-privacy
- OpenDP Project: opendp.org
- Apple Differential Privacy Technical Overview (2017)
- U.S. Census Bureau Disclosure Avoidance System (2020 Census)
LINDDUN Privacy Threat Modeling
Overview
LINDDUN is a systematic privacy threat modeling methodology developed by the DistriNet research group at KU Leuven. It provides a structured approach to identify and mitigate privacy threats in software systems. The acronym represents seven privacy threat categories that map to violations of privacy properties defined in ISO/IEC 29100.
The Seven LINDDUN Threat Categories
L — Linking
Definition: The ability to associate two or more data items or actions with an individual or group, beyond what is intended by the data subject.
Privacy Property Violated: Unlinkability
Threat Scenarios:
- Cross-referencing anonymized datasets to re-identify individuals
- Correlating browsing behavior across websites using fingerprinting
- Linking social media profiles to real identities through metadata
- Combining location data points to infer home/work addresses
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Data minimization | Collect only necessary attributes | Review each data field against stated purpose |
| Pseudonymization | Replace identifiers with tokens | Use cryptographic pseudonymization with key separation |
| Mix networks | Obscure communication patterns | Route messages through anonymity networks |
| Aggregation | Present only group-level data | Enforce minimum group size (k>=5) for any query result |
| Session unlinkability | Prevent cross-session tracking | Rotate session tokens, avoid persistent identifiers |
I — Identifying
Definition: The ability to identify a data subject from a set of data items, connecting them to a known individual.
Privacy Property Violated: Anonymity
Threat Scenarios:
- Direct identification from unmasked PII in logs
- Quasi-identifier attacks (combining age, ZIP code, gender)
- Facial recognition from images in datasets
- Voice identification from audio recordings
- Writing style analysis (stylometry) in anonymous forums
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Anonymization | Remove direct identifiers | Strip names, emails, SSNs before processing |
| k-Anonymity | Generalize quasi-identifiers | Ensure each record shares attributes with k-1 others |
| Differential privacy | Add calibrated noise | Apply epsilon-differential privacy to query responses |
| Data masking | Obscure identifying fields | Replace with realistic synthetic values |
| Access control | Restrict who can see raw data | Implement need-to-know access with purpose verification |
N — Non-repudiation
Definition: The inability of a data subject to deny having performed an action, even when such denial would be desirable for privacy.
Privacy Property Violated: Plausible deniability
Threat Scenarios:
- Immutable audit logs linking users to actions
- Digital signatures proving authorship of documents
- Non-repudiable transaction records
- Email delivery confirmations and read receipts
- Blockchain-based records that cannot be denied or deleted
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Deniable encryption | Enable plausible deniability | Use deniable encryption schemes for sensitive data |
| Group signatures | Hide individual identity in group | Implement group signature schemes for authenticated actions |
| Minimal logging | Log only what is legally required | Review and minimize audit trail scope |
| Aggregate reporting | Report actions at group level | Aggregate activity reports rather than individual-level |
| Configurable receipts | Let users control acknowledgments | Allow opt-out of read receipts and delivery confirmations |
D — Detecting
Definition: The ability to determine whether a data subject has been involved in an action or is present in a dataset, even without identifying them specifically.
Privacy Property Violated: Undetectability
Threat Scenarios:
- Traffic analysis revealing communication patterns
- Database membership inference attacks
- Timing attacks revealing user activity
- Side-channel attacks revealing data access patterns
- Presence detection through network metadata
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Traffic padding | Mask communication patterns | Generate dummy traffic to obscure real patterns |
| Steganography | Hide data within other data | Embed sensitive communications in innocuous content |
| Constant-time operations | Prevent timing analysis | Implement constant-time algorithms for sensitive operations |
| Oblivious RAM | Hide access patterns | Use ORAM protocols for privacy-critical data access |
| Differential privacy | Provide membership privacy | Apply differential privacy to prevent membership inference |
D — Data Disclosure
Definition: Unauthorized exposure of personal data to parties who should not have access.
Privacy Property Violated: Confidentiality
Threat Scenarios:
- SQL injection exposing database contents
- Misconfigured cloud storage buckets
- Insider threats with excessive access privileges
- Man-in-the-middle attacks on unencrypted channels
- API responses returning excessive data fields
- Backup media loss or theft
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Encryption | Protect data at rest and in transit | AES-256 at rest, TLS 1.3 in transit |
| Access control | Enforce least privilege | RBAC with regular access reviews |
| Input validation | Prevent injection attacks | Parameterized queries, input sanitization |
| API field filtering | Return only requested fields | Implement field-level access control in APIs |
| DLP | Detect and prevent data exfiltration | Deploy DLP at network egress and endpoints |
U — Unawareness
Definition: Data subjects being insufficiently aware of data processing activities, their rights, or the consequences of providing or withholding data.
Privacy Property Violated: Transparency, Intervenability
Threat Scenarios:
- Hidden data collection through tracking pixels
- Opaque algorithmic decision-making
- Buried privacy policies with complex legal language
- Undisclosed data sharing with third parties
- Lack of notification when processing purposes change
- No mechanism for data subjects to exercise rights
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Layered notices | Progressive disclosure | Short notice + full policy + just-in-time |
| Privacy dashboards | Centralized visibility | User-facing dashboard showing all data held |
| Consent management | Granular, informed consent | Purpose-specific consent with clear descriptions |
| Explainable AI | Algorithmic transparency | Provide meaningful explanations of automated decisions |
| Right facilitation | Easy rights exercise | Self-service portal for access, correction, deletion |
N — Non-compliance
Definition: Processing personal data in ways that violate applicable laws, regulations, standards, or organizational policies.
Privacy Property Violated: Policy and consent compliance
Threat Scenarios:
- Processing without valid legal basis
- Failing to honor data subject rights within deadlines
- Cross-border transfers without adequate safeguards
- Retaining data beyond the stated retention period
- Processing children's data without parental consent
- Failing to conduct required impact assessments
Mitigation Strategies:
| Strategy | Technique | Implementation |
|---|---|---|
| Compliance mapping | Map processing to legal bases | Document legal basis per processing activity per jurisdiction |
| Automated enforcement | Technical compliance controls | Automated retention enforcement, consent verification |
| DPIA process | Impact assessment for high-risk processing | Mandatory DPIA before deploying new high-risk processing |
| Regulatory monitoring | Track legal developments | Subscribe to regulatory updates, conduct periodic gap analysis |
| Audit program | Verify ongoing compliance | Annual compliance audits with corrective action tracking |
DFD-Based Threat Analysis Process
Step 1: Create Privacy-Annotated Data Flow Diagram
[Data Subject] ---(personal data)--> [Web Application]
^ |
| v
[Notice] [Application Server]
|
+-------------+-------------+
| | |
v v v
[User Database] [Analytics DB] [Third-Party API]
(encrypted) (pseudonymized) (data sharing)
Annotate each element with:
- Data types flowing through (PII categories)
- Trust boundaries crossed
- Storage locations and durations
- Processing purposes
- Access controls in place
Step 2: Map Threats to DFD Elements
| DFD Element | L | I | N | D(etect) | D(isclose) | U | N(on-comply) |
|---|---|---|---|---|---|---|---|
| Data flows | X | X | X | X | |||
| Data stores | X | X | X | X | |||
| Processes | X | X | X | X | X | X | X |
| External entities | X | X | X | X | X |
Step 3: Build Threat Trees
For each applicable threat category per DFD element, construct a threat tree:
Identifying Threat to User Database
├── Direct identifier exposure
│ ├── SQL injection reveals raw PII
│ ├── Backup media contains unencrypted PII
│ └── Admin access to production database
├── Quasi-identifier attack
│ ├── Combination of age + ZIP + gender
│ └── Temporal correlation of records
└── Inference attack
├── Aggregate query with small group size
└── Differential attack across query results
Step 4: Prioritize and Mitigate
Use a risk matrix to prioritize identified threats:
| Likelihood / Impact | Negligible | Limited | Significant | Maximum |
|---|---|---|---|---|
| Very Likely | Medium | High | Critical | Critical |
| Likely | Low | Medium | High | Critical |
| Possible | Low | Medium | Medium | High |
| Unlikely | Low | Low | Medium | Medium |
| Rare | Low | Low | Low | Medium |
LINDDUN and STRIDE Integration
| LINDDUN Category | Related STRIDE Category | Overlap Area |
|---|---|---|
| Data Disclosure | Information Disclosure | Both address unauthorized data exposure |
| Non-compliance | Tampering | Integrity of consent records |
| Detecting | Information Disclosure | Metadata leakage |
| Identifying | Information Disclosure | PII exposure |
| Non-repudiation | Repudiation | Opposing perspectives on the same property |
Practical Application at Cipher Engineering Labs
Threat Modeling Workshop Agenda (4 hours)
- System Overview (30 min): Present the system architecture and data flows
- DFD Construction (45 min): Collaboratively build privacy-annotated DFD
- Threat Identification (90 min): Walk through each LINDDUN category per DFD element
- Prioritization (30 min): Score threats on risk matrix
- Mitigation Planning (45 min): Identify controls for high and critical risks
Deliverables
- Privacy-annotated Data Flow Diagram
- LINDDUN Threat Register (all identified threats with risk scores)
- Mitigation Plan (prioritized controls for high/critical threats)
- Residual Risk Statement
References
- Deng, M., Wuyts, K., Scandariato, R., Preneel, B., and Joosen, W. "A Privacy Threat Analysis Framework: Supporting the Elicitation and Fulfillment of Privacy Requirements." Requirements Engineering, 16(1):3-32, 2011.
- Wuyts, K. and Joosen, W. "LINDDUN Privacy Threat Modeling: A Tutorial." CW Reports, KU Leuven, 2015.
- LINDDUN GO — Lightweight approach: linddun.org
- ISO/IEC 29100:2011 — Information Technology — Security Techniques — Privacy Framework
- OWASP Privacy Risks Project
NIST Privacy Framework — COMMUNICATE Function
Overview
The COMMUNICATE function develops and implements appropriate activities to enable organizations and individuals to have a reliable understanding and engage in a dialogue about how data are processed and associated privacy risks. This function is essential for building trust and ensuring transparency with data subjects, regulators, and business partners.
COMMUNICATE Function Subcategories
CM.AW — Awareness
Raising awareness of data processing practices and privacy risks among stakeholders.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| CM.AW-P1 | Mechanisms (e.g., notices, internal policies) for communicating data processing purposes, practices, and associated privacy risks are established and in service | Publish layered privacy notices. Maintain internal data processing documentation. Establish communication channels for privacy inquiries. |
| CM.AW-P2 | Mechanisms for obtaining feedback from individuals about data processing and associated privacy risks are established and in service | Deploy feedback forms on privacy pages. Monitor privacy-related inquiries through support channels. Conduct periodic privacy perception surveys. |
| CM.AW-P3 | System/product/service design enables data processing visibility | Implement privacy dashboards for data subjects. Provide real-time data processing status. Enable granular consent visibility. |
| CM.AW-P4 | Records of data disclosures and sharing are maintained and can be accessed for review or transmission | Maintain disclosure registers. Enable data subjects to view sharing history. Automate disclosure logging. |
| CM.AW-P5 | Data corrections or deletions can be communicated to individuals or organizations that have received the data | Implement downstream notification workflows. Track data recipients for correction propagation. Automate deletion verification across partners. |
| CM.AW-P6 | Data provenance and lineage are maintained and can be accessed for review or transmission | Document data origins and transformation history. Provide lineage visualization. Enable audit trail access for authorized parties. |
| CM.AW-P7 | Impacted individuals and organizations are notified about a privacy breach or event | Develop breach notification templates. Establish notification timelines per jurisdiction. Maintain contact information for notification delivery. |
| CM.AW-P8 | Individuals are provided with mitigation mechanisms for observed privacy risks | Offer identity protection services post-breach. Provide self-service privacy controls. Communicate available remediation options. |
CM.PO — Communication Policies
Policies governing how privacy-related information is communicated.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| CM.PO-P1 | Transparency policies, processes, and procedures for communicating data processing purposes, practices, associated privacy risks, and options for enabling data subjects' data processing preferences are established and in service | Develop layered transparency framework. Define communication standards for each audience. |
| CM.PO-P2 | Roles and responsibilities for communicating privacy-related information are established | Assign communications ownership. Define approval workflows for external privacy communications. |
Transparency Framework
Layered Privacy Notice Architecture
Layer 1: Short Notice (Banner/Summary)
├── Who we are
├── What data we collect
├── Key purposes
├── Core rights
└── Link to full notice
Layer 2: Full Privacy Notice
├── Detailed purposes and legal bases
├── Data categories and sources
├── Recipients and transfers
├── Retention periods
├── Complete rights information
├── Contact information
└── Links to supplementary notices
Layer 3: Supplementary Notices
├── Cookie Policy
├── Children's Privacy Notice
├── Employee Privacy Notice
├── Specific Service Notices
└── Jurisdiction-Specific Addenda
Layer 4: Just-in-Time Notices
├── Collection-point notifications
├── Consent requests with context
├── Purpose-specific explanations
└── Processing change notifications
Communication Channel Matrix
| Audience | Channel | Frequency | Content Type | Owner |
|---|---|---|---|---|
| Data subjects (general) | Website privacy center | Continuous | Notices, policies, FAQs | Privacy Team |
| Data subjects (customers) | Email, in-app notifications | Event-driven | Consent changes, updates | Marketing + Privacy |
| Data subjects (employees) | Intranet, HR portal | Semi-annual + event | Employee privacy info | HR + Privacy |
| Regulators | Formal correspondence | As required | Compliance reports, notifications | Legal + DPO |
| Business partners | Contract addenda, portal | Annual + event | Processing terms, audit rights | Legal + Procurement |
| Board/executives | Dashboards, presentations | Quarterly | Risk reports, KPIs | CPO |
| Internal teams | Confluence, training | Monthly | Guidelines, best practices | Privacy Champions |
Privacy Notice Content Requirements
Mandatory Elements by Regulation
| Element | GDPR Art. 13/14 | CCPA/CPRA | LGPD | PIPEDA |
|---|---|---|---|---|
| Controller identity | Required | Required | Required | Required |
| DPO contact | Required | N/A | Required (DPO equivalent) | N/A |
| Processing purposes | Required | Required | Required | Required |
| Legal basis | Required | N/A | Required | N/A |
| Data categories | Required | Required | Required | Required |
| Recipients | Required | Required | Required | Required |
| International transfers | Required | N/A | Required | Required |
| Retention periods | Required | Disclosed on request | Required | Required |
| Data subject rights | Required | Required | Required | Required |
| Right to complain | Required | N/A | Required | Required |
| Automated decision-making | Required | CPRA: Required | Required | N/A |
| Data source (indirect collection) | Required | Required | Required | Required |
| Sale/sharing opt-out | N/A | Required | N/A | N/A |
| Financial incentive terms | N/A | Required | N/A | N/A |
Stakeholder Engagement Workflows
Data Subject Inquiry Handling
Inquiry Received
|
v
Classify Inquiry Type
├── General Privacy Question → FAQ/Knowledge Base → Respond within 5 business days
├── Data Subject Request → Route to DSR Workflow → Respond within regulatory deadline
├── Complaint → Escalate to DPO → Investigate → Respond within 15 business days
├── Breach Inquiry → Route to Incident Team → Coordinate response
└── Consent Change → Process immediately → Confirm within 24 hours
Privacy Communication Approval Workflow
| Communication Type | Drafting | Review | Approval | Distribution |
|---|---|---|---|---|
| Privacy policy update | Privacy Team | Legal | CPO + General Counsel | Web team |
| Breach notification | Incident Lead | Legal + DPO | CEO + General Counsel | Communications |
| Regulatory response | DPO | Legal | General Counsel | DPO |
| Marketing consent request | Marketing | Privacy Team | CPO | Marketing |
| Internal policy update | Privacy Team | Stakeholders | CPO | Internal comms |
Feedback Mechanisms
Privacy Perception Survey Template (Annual)
- How aware are you of our privacy practices? (1-5 scale)
- How easy is it to find information about how we use your data? (1-5 scale)
- How confident are you that we protect your personal data? (1-5 scale)
- Have you exercised any privacy rights in the past 12 months? (Yes/No, which ones)
- How satisfied were you with the response? (1-5 scale, if applicable)
- What privacy improvements would you like to see? (Open text)
- How likely are you to recommend us based on our privacy practices? (0-10 NPS)
Control Mapping
| NIST PF COMMUNICATE | ISO 27701 | GDPR Article | APEC CBPR |
|---|---|---|---|
| CM.AW-P1 | A.7.3.2 | Art. 12, 13, 14 | Principle 5 |
| CM.AW-P2 | A.7.3.8 | Art. 77 | Principle 9 |
| CM.AW-P3 | A.7.2.5 | Art. 12(1) | Principle 5 |
| CM.AW-P4 | A.7.5.3 | Art. 19 | Principle 7 |
| CM.AW-P7 | A.7.3.9 | Art. 34 | Principle 8 |
| CM.PO-P1 | A.7.3.2 | Art. 12 | Principle 5 |
| CM.PO-P2 | 5.3 | Art. 37-39 | Principle 9 |
References
- NIST Privacy Framework Version 1.0 (January 16, 2020)
- Article 29 Working Party WP260 — Guidelines on Transparency
- EDPB Guidelines 04/2022 on Calculation of Administrative Fines
- ISO/IEC 29100:2011 — Privacy Framework (Privacy Principles)
NIST Privacy Framework — CONTROL Function
Overview
The CONTROL function develops and implements appropriate activities to enable organizations and individuals to manage data with sufficient granularity to manage privacy risks. It addresses data management, data processing governance, and disassociated processing techniques.
CONTROL Function Subcategories
CT.DM — Data Management
Policies and procedures for managing data throughout its lifecycle.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| CT.DM-P1 | Data elements can be accessed for review | Implement data subject access portals. Provide export in machine-readable formats (JSON, CSV). Response within regulatory timelines. |
| CT.DM-P2 | Data elements can be accessed for transmission or disclosure | Support data portability requests. Implement secure transfer mechanisms. Maintain disclosure logs. |
| CT.DM-P3 | Data elements can be accessed for alteration | Enable data correction workflows. Propagate corrections across all systems holding the data. |
| CT.DM-P4 | Data elements can be accessed for deletion | Implement cascading deletion across primary and backup systems. Maintain deletion verification logs. Address data in analytics pipelines. |
| CT.DM-P5 | Data processing activities are configured to meet the organization's data processing principles | Default to most privacy-protective settings. Implement purpose limitation at the system level. |
| CT.DM-P6 | Retention schedules are established and in place | Define retention periods per data category. Implement automated purging. Document legal hold exceptions. |
| CT.DM-P7 | Data are processed in accordance with established retention schedules | Automated enforcement of retention policies. Regular audits of data stores. Reporting on retention compliance. |
| CT.DM-P8 | Integrity of data elements is maintained | Implement checksums and validation rules. Detect and resolve data quality issues. Track data lineage. |
| CT.DM-P9 | Audit/log records are determined, documented, implemented, and reviewed | Privacy-specific audit logging for all data access and modifications. Log retention per regulatory requirements. |
| CT.DM-P10 | Mechanisms for data subjects to authorize processing exist | Consent collection and management infrastructure. Granular consent options. Consent withdrawal capability. |
CT.DP — Data Processing Policies and Procedures
Governance controls for data processing activities.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| CT.DP-P1 | Policies, processes, and procedures for authorizing data processing are maintained and enforced | Document lawful basis for each processing activity. Implement access controls aligned with authorized purposes. |
| CT.DP-P2 | Data processing environment is identified, managed, and monitored | Maintain environment inventory (production, staging, development). Monitor for unauthorized processing. Separate test data from production data. |
| CT.DP-P3 | Data processing procedures are maintained and monitored to meet established terms | Standard operating procedures for each processing type. Automated compliance checking. Exception management workflow. |
| CT.DP-P4 | System or device configurations are set to authorized specifications | Secure configuration baselines for systems processing personal data. Configuration drift detection. Change management for privacy-relevant settings. |
| CT.DP-P5 | Data processing activities are monitored and logged | Real-time monitoring of data processing operations. Anomaly detection for unusual processing patterns. Alerting for policy violations. |
CT.PO — Disassociated Processing
Techniques to process data while minimizing identifiability.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| CT.PO-P1 | De-identification is applied to data elements to limit observability or linkability | Apply pseudonymization, anonymization, or generalization based on use case. Document de-identification methodology. |
| CT.PO-P2 | Technical measures for disassociated processing are implemented | Deploy differential privacy, secure multi-party computation, or homomorphic encryption as appropriate. Select technique based on data utility requirements. |
Data Management Architecture
Lifecycle Control Points
Collection --> Storage --> Processing --> Sharing --> Retention --> Deletion
| | | | | |
v v v v v v
Consent Encryption Purpose Access Schedule Cascading
Capture at Rest Limitation Controls Enforcement Purge
| | | | | |
v v v v v v
Audit Log Access Log Process Log Transfer Log Retention Log Deletion Log
Data Element Access Control Matrix
| Operation | Data Subject | Data Steward | Processor | Analyst | Admin |
|---|---|---|---|---|---|
| View own data | Yes | Yes | No | No | Yes |
| Export own data | Yes | Yes | No | No | Yes |
| Correct own data | Request | Yes | No | No | Yes |
| Delete own data | Request | Yes | No | No | Yes |
| View aggregate | No | Yes | Yes | Yes | Yes |
| Process data | No | No | Yes | No | Yes |
| Analyze data | No | No | No | Yes | Yes |
| Configure system | No | No | No | No | Yes |
Disassociated Processing Techniques
Technique Selection Guide
| Technique | Identifiability Reduction | Data Utility | Computational Cost | Reversible |
|---|---|---|---|---|
| Pseudonymization | Medium | High | Low | Yes (with key) |
| k-Anonymity | Medium-High | Medium | Medium | No |
| l-Diversity | High | Medium | Medium | No |
| t-Closeness | High | Medium-Low | High | No |
| Differential Privacy | Very High | Medium | Medium | No |
| Data Masking | Low-Medium | Medium | Low | Depends |
| Generalization | Medium | Medium | Low | No |
| Suppression | High | Low | Low | No |
| Synthetic Data | Very High | Variable | High | No |
De-identification Decision Framework
Is the data being shared externally?
├── Yes → Is re-identification risk acceptable?
│ ├── No → Apply formal anonymization (differential privacy, k-anonymity with k>=5)
│ └── Yes → Apply pseudonymization with contractual controls
└── No → Is the data used for analytics?
├── Yes → Apply aggregation or differential privacy
└── No → Apply pseudonymization with access controls
Retention Schedule Template
| Data Category | Retention Period | Legal Basis | Deletion Method | Verification |
|---|---|---|---|---|
| Customer transaction records | 7 years from transaction | Tax/accounting regulations | Secure overwrite | Automated audit |
| Marketing consent records | Duration of consent + 3 years | Legitimate interest (proof) | Logical deletion | Annual review |
| Employee personnel files | Employment + 7 years | Employment law | Secure destruction | HR verification |
| Website analytics | 26 months | Consent | Automated purge | Monthly check |
| Support tickets | 3 years from resolution | Contract performance | Secure overwrite | Quarterly audit |
| CCTV footage | 30 days | Legitimate interest | Automated overwrite | Weekly check |
| Backup tapes | 90 days rolling | Business continuity | Tape degaussing | Quarterly audit |
Audit Logging Requirements
Privacy-Specific Events to Log
| Event Category | Specific Events | Required Fields |
|---|---|---|
| Data Access | View, export, download | User ID, timestamp, data subject ID, data elements accessed, purpose |
| Data Modification | Create, update, correct | User ID, timestamp, data subject ID, before/after values, purpose |
| Data Deletion | Delete, purge, anonymize | User ID, timestamp, data subject ID, deletion scope, verification |
| Consent Changes | Grant, withdraw, modify | Data subject ID, timestamp, consent scope, collection channel |
| Data Transfer | Internal transfer, external disclosure | User ID, timestamp, recipient, data elements, legal basis |
| Configuration | Access control changes, retention policy changes | Admin ID, timestamp, before/after settings, approval reference |
Control Mapping
| NIST PF CONTROL | ISO 27701 | GDPR Article | NIST 800-53 |
|---|---|---|---|
| CT.DM-P1 | A.7.3.3 | Art. 15 | PM-25 |
| CT.DM-P2 | A.7.3.7 | Art. 20 | PM-25 |
| CT.DM-P3 | A.7.3.4 | Art. 16 | PM-25 |
| CT.DM-P4 | A.7.3.5 | Art. 17 | PM-25 |
| CT.DM-P6 | A.7.4.7 | Art. 5(1)(e) | SI-12 |
| CT.DM-P9 | A.7.2.8 | Art. 30 | AU-2 |
| CT.DM-P10 | A.7.2.3 | Art. 6, 7 | IP-1 |
| CT.DP-P1 | A.7.2.2 | Art. 6(1) | PT-3 |
| CT.DP-P5 | A.7.2.8 | Art. 30 | AU-6 |
| CT.PO-P1 | A.7.4.5 | Art. 25(1) | PM-24 |
| CT.PO-P2 | A.7.4.5 | Art. 25(1) | PM-24 |
References
- NIST Privacy Framework Version 1.0 (January 16, 2020)
- NIST SP 800-188 — De-Identifying Government Datasets
- NIST IR 8053 — De-Identification of Personal Information
- ISO/IEC 20889:2018 — Privacy Enhancing Data De-identification Techniques
NIST Privacy Framework — GOVERN Function
Overview
The GOVERN function establishes and monitors the organization's privacy governance structure, including policies, processes, and procedures for managing privacy risk. It is the foundational function that enables all other NIST Privacy Framework functions.
GOVERN Function Subcategories
GV.AT — Awareness and Training
Building organizational awareness and capability for privacy risk management.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| GV.AT-P1 | The workforce is informed and trained on its roles and responsibilities | Develop role-based privacy training curricula. New hire training within 30 days, annual refresher for all staff. |
| GV.AT-P2 | Senior executives understand their roles and responsibilities | Executive briefings quarterly. Board-level privacy risk reporting semi-annually. |
| GV.AT-P3 | Privacy personnel understand their roles and responsibilities | Specialized training for DPOs, privacy engineers, and legal counsel. Professional certification support (CIPP, CIPM, CIPT). |
| GV.AT-P4 | Third parties (service providers, customers, partners) understand their roles and responsibilities | Include privacy requirements in contracts. Provide vendor privacy guidelines. Verify third-party training compliance. |
GV.MT — Monitoring and Review
Ongoing assessment of privacy governance effectiveness.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| GV.MT-P1 | Privacy risk is re-assessed on an ongoing basis and as key factors change | Trigger-based reassessment: new processing activities, regulatory changes, incidents, organizational changes. |
| GV.MT-P2 | Privacy risk management activities are periodically reviewed | Conduct quarterly operational reviews and annual strategic reviews. Benchmark against industry peers. |
| GV.MT-P3 | Privacy policies, processes, and procedures are maintained and used to manage privacy risk | Version control for all privacy documentation. Annual policy review cycle with defined approval workflow. |
| GV.MT-P4 | Privacy risk assessment results are shared with stakeholders | Distribute risk dashboards to executive leadership monthly. Share relevant findings with operational teams weekly. |
GV.PO — Policy
Privacy policies governing organizational approach to privacy risk management.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| GV.PO-P1 | Organizational privacy values and policies are established and communicated | Publish a privacy mission statement. Develop comprehensive privacy policy suite aligned with organizational values. |
| GV.PO-P2 | Processes to instill organizational privacy values within data processing activities are established | Embed privacy requirements into SDLC. Require privacy review at design stage for all new data processing. |
| GV.PO-P3 | Roles and responsibilities for privacy workforce are established | Define RACI matrices for privacy functions. Document authority levels and escalation paths. |
| GV.PO-P4 | Privacy is integrated into organizational risk management | Include privacy in enterprise risk register. Align privacy risk appetite with organizational risk tolerance. |
| GV.PO-P5 | Legal, regulatory, and contractual requirements regarding privacy are understood and managed | Maintain regulatory tracking register. Map obligations to controls. Conduct annual compliance assessments. |
| GV.PO-P6 | Governance and risk management policies address privacy risks | Privacy risk addressed in board risk committee charter. Executive-level accountability for privacy. |
GV.RR — Roles and Responsibilities
Establishing accountability structures for privacy risk management.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| GV.RR-P1 | Organizational leadership is responsible and accountable for privacy risk | C-suite privacy accountability documented. Chief Privacy Officer reports to CEO or General Counsel. |
| GV.RR-P2 | Adequate resources for privacy programs are allocated | Annual privacy budget planning. Headcount planning for privacy team. Technology investment for privacy tools. |
| GV.RR-P3 | Processes are in place to determine, assess, and manage privacy risks | Formal privacy risk management methodology adopted. Standard operating procedures for risk assessment. |
Governance Structure Template
Organizational Hierarchy
Board of Directors / Audit Committee
|
Chief Executive Officer
|
Chief Privacy Officer (CPO) / Data Protection Officer (DPO)
|
+-- Privacy Legal Counsel
+-- Privacy Engineering Lead
+-- Privacy Operations Manager
+-- Privacy Compliance Analyst
|
Business Unit Privacy Champions (distributed model)
RACI Matrix for Key Privacy Activities
| Activity | CPO/DPO | Legal | Engineering | Business Units | Executive |
|---|---|---|---|---|---|
| Privacy strategy | A | C | C | I | R |
| Policy development | R | R | C | C | A |
| DPIA execution | A | C | R | R | I |
| Incident response | R | R | C | C | I |
| Training delivery | A | C | I | R | I |
| Regulatory monitoring | C | R | I | I | A |
| Vendor assessment | A | R | C | R | I |
| Metrics reporting | R | I | C | C | A |
R = Responsible, A = Accountable, C = Consulted, I = Informed
Training Program Framework
Tier 1 — General Awareness (All Staff)
- Frequency: Annual, plus new hire onboarding
- Duration: 45 minutes
- Topics: Privacy principles, data handling basics, incident reporting, individual rights
- Assessment: Quiz with 80% pass threshold
Tier 2 — Role-Specific (Data Handlers)
- Frequency: Semi-annual
- Duration: 2 hours
- Topics: Data classification, access controls, secure processing, retention schedules
- Assessment: Scenario-based exercises
Tier 3 — Specialist (Privacy Team)
- Frequency: Quarterly
- Duration: 4 hours
- Topics: Regulatory updates, advanced risk assessment, PET implementation, incident management
- Assessment: Practical exercises and case studies
Tier 4 — Executive (Leadership)
- Frequency: Semi-annual
- Duration: 1 hour
- Topics: Privacy risk landscape, regulatory exposure, strategic alignment, board reporting
- Assessment: N/A (discussion-based)
Policy Suite Inventory
| Policy | Owner | Review Cycle | Approval Authority |
|---|---|---|---|
| Enterprise Privacy Policy | CPO | Annual | CEO |
| Data Classification Policy | CISO/CPO | Annual | CTO |
| Data Retention Policy | CPO | Annual | General Counsel |
| Data Subject Rights Policy | CPO | Annual | General Counsel |
| Privacy Incident Response Policy | CPO | Semi-Annual | CISO |
| Third-Party Data Processing Policy | CPO | Annual | Procurement Lead |
| Cross-Border Transfer Policy | Legal | Annual | General Counsel |
| Cookie and Tracking Policy | CPO | Semi-Annual | CMO |
| Employee Privacy Policy | CPO/HR | Annual | CHRO |
| Privacy-by-Design Standards | Privacy Engineering | Annual | CTO |
Control Mapping
| NIST PF GOVERN | ISO 27701 | GDPR Article | SOC 2 TSC |
|---|---|---|---|
| GV.AT-P1 | 7.2.2 | Art. 39(1)(b) | CC1.4 |
| GV.AT-P2 | 5.1 | Art. 38(3) | CC1.2 |
| GV.MT-P1 | 9.1 | Art. 24(1) | CC4.1 |
| GV.MT-P3 | 5.2 | Art. 24(2) | CC1.3 |
| GV.PO-P1 | 5.2 | Art. 5(1) | CC1.1 |
| GV.PO-P4 | 5.3.2 | Art. 24(1) | CC3.1 |
| GV.PO-P5 | 5.2.1 | Art. 5(1)(a) | CC2.2 |
| GV.RR-P1 | 5.3 | Art. 37-39 | CC1.2 |
| GV.RR-P2 | 7.1 | Art. 38(2) | CC1.1 |
References
- NIST Privacy Framework Version 1.0 (January 16, 2020)
- NIST SP 800-53 Rev. 5 — Security and Privacy Controls (PM family)
- ISO/IEC 27701:2019 — Clause 5 (PIMS-specific requirements)
- IAPP CIPM Body of Knowledge — Privacy Program Management
NIST Privacy Framework — IDENTIFY Function
Overview
The IDENTIFY function in the NIST Privacy Framework (Version 1.0, January 2020) enables organizations to develop organizational understanding of privacy risk arising from data processing. This skill covers all four subcategories: Business Environment (ID.BE), Data Actions (ID.DA), Improvement (ID.IM), and Risk Assessment (ID.RA).
IDENTIFY Function Subcategories
ID.BE — Business Environment
Understanding the organization's mission, objectives, stakeholders, and activities to prioritize privacy risk management decisions.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| ID.BE-P1 | The organization's role(s) in the data processing ecosystem are identified and communicated | Document whether the organization acts as data controller, processor, or both. Map all data flows identifying organizational role at each stage. |
| ID.BE-P2 | Priorities for organizational mission, objectives, and activities are established and communicated | Align privacy objectives with business strategy. Ensure executive leadership endorses privacy as a business priority. |
| ID.BE-P3 | Systems/products/services that process data are identified and prioritized | Maintain an inventory of all systems processing personal data. Classify by risk tier based on data sensitivity and volume. |
ID.DA — Data Actions
Understanding the data actions the organization performs and associated privacy risks.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| ID.DA-P1 | A data processing ecosystem inventory is created and maintained | Catalog all data processing activities including collection points, storage locations, sharing partners, and retention periods. |
| ID.DA-P2 | Owners of data actions are identified | Assign clear ownership for each data processing activity. Document accountability chains from operational to executive level. |
| ID.DA-P3 | Problematic data actions are identified and prioritized for management | Use the NIST problematic data actions catalog to assess risk. Score based on likelihood and impact to individuals. |
ID.IM — Improvement
Continuous improvement of privacy risk management.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| ID.IM-P1 | A process for continuous improvement of the privacy risk assessment approach is established | Schedule quarterly reviews of risk methodology. Incorporate lessons from incidents and regulatory developments. |
| ID.IM-P2 | Privacy risk assessment findings are incorporated into improvement plans | Track remediation actions in a centralized register. Assign deadlines and owners for each improvement item. |
ID.RA — Risk Assessment
Understanding and evaluating privacy risks.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| ID.RA-P1 | Data actions and their expected problematic data actions are identified | Map each data action to potential problematic outcomes using NIST's catalog of problematic data actions. |
| ID.RA-P2 | Organizational systems, products, and services are monitored for problematic data actions | Deploy continuous monitoring for unauthorized access, unexpected data flows, and policy violations. |
| ID.RA-P3 | Risk responses are identified and prioritized | Define response strategies: mitigate, transfer, avoid, or accept. Document rationale for each risk response decision. |
Implementation Workflow
Phase 1: Inventory and Discovery (Weeks 1-4)
- Stakeholder Identification: Identify all internal and external stakeholders involved in data processing
- System Inventory: Catalog all systems, applications, and services that process personal data
- Data Flow Mapping: Document how data moves through the organization from collection to deletion
- Role Classification: Determine the organization's role (controller/processor) for each data processing activity
Phase 2: Data Action Analysis (Weeks 5-8)
- Data Action Catalog: Document all data actions (collection, retention, logging, generation, disclosure, transfer)
- Ownership Assignment: Assign data stewards for each data action category
- Problematic Data Action Screening: Evaluate each data action against NIST's problematic data actions list
- Impact Assessment: Score each problematic data action on likelihood and severity scales (1-5)
Phase 3: Risk Assessment (Weeks 9-12)
- Risk Scoring: Calculate risk scores as Likelihood x Impact for each identified problematic data action
- Risk Prioritization: Rank risks and identify those exceeding organizational risk tolerance
- Response Planning: Select and document risk response strategies for each high-priority risk
- Control Mapping: Map existing and planned controls to identified risks
Phase 4: Continuous Improvement (Ongoing)
- Quarterly Reviews: Reassess risk landscape and update risk register
- Incident Integration: Incorporate findings from privacy incidents into risk assessments
- Regulatory Monitoring: Track changes in applicable privacy regulations
- Metrics Tracking: Monitor key risk indicators and report to governance bodies
Control Mapping to Other Frameworks
| NIST PF IDENTIFY | ISO 27701 | GDPR Article | CCPA Section |
|---|---|---|---|
| ID.BE-P1 | 5.2.1 | Art. 24, 26, 28 | 1798.140(d),(v) |
| ID.BE-P2 | 5.2.2 | Art. 5(2) | 1798.100 |
| ID.BE-P3 | A.7.2.1 | Art. 30 | 1798.110 |
| ID.DA-P1 | A.7.2.8 | Art. 30(1) | 1798.110(c) |
| ID.DA-P2 | 5.3 | Art. 37-39 | 1798.130 |
| ID.DA-P3 | A.7.2.5 | Art. 35 | 1798.185 |
| ID.RA-P1 | 6.1.2 | Art. 35(7) | 1798.185(a)(15) |
| ID.RA-P2 | 9.1 | Art. 32(1)(d) | 1798.150 |
| ID.RA-P3 | 6.1.3 | Art. 35(7)(d) | 1798.185(a)(15) |
Maturity Assessment
Level 1 — Initial
- Ad hoc privacy risk identification
- No formal inventory of data processing activities
- Reactive approach to privacy issues
Level 2 — Managed
- Basic data inventory exists
- Some data actions documented
- Risk assessments conducted periodically
Level 3 — Defined
- Comprehensive data processing inventory maintained
- All data actions cataloged with owners
- Structured risk assessment process in place
Level 4 — Quantitatively Managed
- Metrics-driven risk assessment
- Automated monitoring of data actions
- Quantitative risk scoring methodology
Level 5 — Optimizing
- Continuous improvement process embedded
- Predictive risk analytics
- Industry-leading privacy risk management practices
References
- NIST Privacy Framework Version 1.0 (January 16, 2020)
- NIST SP 800-37 Rev. 2 — Risk Management Framework
- NIST IR 8062 — An Introduction to Privacy Engineering and Risk Management in Federal Systems
- ISO/IEC 27701:2019 — Privacy Information Management
NIST Privacy Framework — PROTECT Function
Overview
The PROTECT function develops and implements appropriate data processing safeguards to prevent cybersecurity-related privacy events. This function directly bridges the NIST Cybersecurity Framework and Privacy Framework, addressing the overlap where security controls serve privacy objectives. It covers access control, data security, and protective policies.
PROTECT Function Subcategories
PR.AC — Access Control
Managing access to data and systems to protect privacy.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| PR.AC-P1 | Identities and credentials are issued, managed, verified, revoked, and audited for authorized individuals, processes, and devices | Implement identity lifecycle management. Enforce MFA for systems processing personal data. Conduct quarterly access reviews. |
| PR.AC-P2 | Physical access to data and devices is managed | Restrict physical access to data centers and server rooms. Implement visitor logging. Secure endpoint devices with encryption and remote wipe capability. |
| PR.AC-P3 | Remote access is managed | Enforce VPN for remote access to personal data systems. Implement zero-trust network architecture. Monitor and log all remote sessions. |
| PR.AC-P4 | Access permissions and authorizations are managed with least privilege and separation of duties | Implement role-based access control (RBAC). Enforce separation of duties for sensitive operations. Regular privileged access reviews. |
| PR.AC-P5 | Network integrity is protected | Network segmentation for systems processing personal data. Intrusion detection/prevention systems. Encrypted network communications. |
| PR.AC-P6 | Individuals and devices are proofed and bound to credentials | Implement strong identity verification for data subject account creation. Device certificate management. Biometric authentication where appropriate. |
PR.DS — Data Security
Protecting data at rest, in transit, and during processing.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| PR.DS-P1 | Data-at-rest are protected | AES-256 encryption for stored personal data. Key management with HSM. Database-level encryption (TDE). |
| PR.DS-P2 | Data-in-transit are protected | TLS 1.3 for all data transmissions. Certificate pinning for mobile applications. End-to-end encryption for sensitive communications. |
| PR.DS-P3 | Systems/products/services and associated data are formally managed throughout removal, transfers, and disposition | Secure media sanitization (NIST SP 800-88). Certificate of destruction for physical media. Verified data deletion from cloud services. |
| PR.DS-P4 | Adequate capacity to ensure availability is maintained | Capacity planning for privacy-critical systems. DDoS protection. Redundancy for data subject rights portals. |
| PR.DS-P5 | Protections against data leaks are implemented | DLP solutions monitoring egress points. Email DLP for personal data. Cloud access security broker (CASB) deployment. |
| PR.DS-P6 | Integrity checking mechanisms are used to verify software, firmware, and information integrity | File integrity monitoring for privacy-critical systems. Code signing for privacy-related applications. Checksum verification for data transfers. |
| PR.DS-P7 | The development and testing environment(s) are separate from the production environment | Separate environments with no production personal data in dev/test. Synthetic data generation for testing. Access controls between environments. |
| PR.DS-P8 | Data disclosures are consistent with the purposes for which the data were collected | Technical enforcement of purpose limitation. API-level access controls tied to documented purposes. Automated purpose verification at query time. |
PR.PO — Protective Policies
Policies governing protective measures for privacy.
| Subcategory | Description | Implementation Guidance |
|---|---|---|
| PR.PO-P1 | A baseline configuration of information technology/industrial control systems is created and maintained | Establish secure baselines for all systems processing personal data. Configuration management database. Automated compliance scanning. |
| PR.PO-P2 | Configuration change control processes are in place | Change advisory board review for privacy-impacting changes. Impact assessment for configuration changes. Rollback procedures documented. |
| PR.PO-P3 | Backups of information are conducted, maintained, and tested | Encrypted backups of personal data. Regular restoration testing. Backup retention aligned with data retention policies. |
| PR.PO-P4 | Policy and regulations regarding the physical operating environment are met | Environmental controls for data processing facilities. Compliance with data residency requirements. Physical security audits. |
| PR.PO-P5 | Data destruction processes are defined and managed | Documented destruction procedures per data classification. Verification and certification of destruction. Compliant disposal of hardware containing personal data. |
| PR.PO-P6 | Maintenance and repair of organizational assets are performed and logged | Vendor supervision during maintenance. Data sanitization before equipment service. Maintenance activity logging. |
| PR.PO-P7 | Response plans (Incident Response and Business Continuity) and recovery plans (Incident Recovery and Disaster Recovery) are in place and managed | Privacy-integrated incident response plan. Business continuity for privacy-critical services. Regular tabletop exercises. |
| PR.PO-P8 | Response and recovery plans are tested | Annual incident response exercises including privacy scenarios. DR testing with privacy recovery objectives. Post-exercise improvement plans. |
| PR.PO-P9 | Privacy procedures are included in human resources practices | Background checks for privacy-sensitive roles. Confidentiality agreements. Exit procedures including access revocation. |
| PR.PO-P10 | A vulnerability management plan is developed and implemented | Vulnerability scanning for privacy-critical systems. Patch management prioritizing privacy-impacting vulnerabilities. Penetration testing including privacy controls. |
Technical Control Architecture
Access Control Architecture
Identity Provider (IdP)
|
v
Multi-Factor Authentication
|
v
Authorization Engine (RBAC + ABAC)
|
+-- Role-Based Access Control
| ├── Privacy Administrator
| ├── Data Steward
| ├── Data Processor
| ├── Data Analyst (aggregated only)
| └── Data Subject (own data only)
|
+-- Attribute-Based Access Control
| ├── Purpose attribute (why accessing)
| ├── Data classification (sensitivity level)
| ├── Time-based restrictions
| └── Location-based restrictions
|
v
Audit Logging Engine
Encryption Standards
| Data State | Minimum Standard | Recommended Standard | Key Management |
|---|---|---|---|
| At Rest | AES-128 | AES-256 | HSM-managed keys, annual rotation |
| In Transit | TLS 1.2 | TLS 1.3 | Certificate management with auto-renewal |
| In Processing | Application-level encryption | Envelope encryption | Per-session keys |
| Backup | AES-256 | AES-256 + separate key | Offline key escrow |
| Archive | AES-256 | AES-256 + integrity check | Long-term key management |
Data Loss Prevention Configuration
| Channel | DLP Control | Detection Method | Action |
|---|---|---|---|
| Email gateway DLP | Pattern matching (SSN, CC, IBAN) | Block + notify | |
| Web upload | Web proxy DLP | Content inspection | Block + notify |
| Cloud storage | CASB | Classification labels | Encrypt + restrict sharing |
| Endpoint | Endpoint DLP | File content scanning | Prevent copy to removable media |
| Database | Database activity monitoring | Query analysis | Alert on bulk extraction |
| API | API gateway | Payload inspection | Rate limit + log |
Network Security for Privacy
Segmentation Architecture
Internet
|
v
[DMZ / WAF]
|
v
[Application Tier] ── TLS 1.3 ── [API Gateway]
| |
v v
[Business Logic Tier] [Privacy Services Tier]
| ├── Consent Manager
v ├── DSR Processor
[Data Tier] ├── Anonymization Engine
├── PII Database (encrypted) └── Audit Logger
├── Analytics Database (aggregated)
└── Backup Storage (encrypted, isolated)
Control Mapping
| NIST PF PROTECT | NIST CSF | ISO 27701 | GDPR Article |
|---|---|---|---|
| PR.AC-P1 | PR.AC-1 | A.7.2.2 | Art. 32(1)(b) |
| PR.AC-P4 | PR.AC-4 | A.7.2.2 | Art. 25(2) |
| PR.DS-P1 | PR.DS-1 | A.7.4.9 | Art. 32(1)(a) |
| PR.DS-P2 | PR.DS-2 | A.7.4.9 | Art. 32(1)(a) |
| PR.DS-P5 | PR.DS-5 | A.7.4.9 | Art. 32(1)(b) |
| PR.DS-P7 | PR.DS-7 | A.7.4.9 | Art. 32(1)(a) |
| PR.DS-P8 | N/A | A.7.2.2 | Art. 5(1)(b) |
| PR.PO-P3 | PR.IP-4 | A.7.4.9 | Art. 32(1)(c) |
| PR.PO-P5 | PR.IP-6 | A.7.4.7 | Art. 17(1) |
| PR.PO-P7 | PR.IP-9 | A.7.2.8 | Art. 33, 34 |
| PR.PO-P10 | PR.IP-12 | A.7.4.9 | Art. 32(1)(d) |
References
- NIST Privacy Framework Version 1.0 (January 16, 2020)
- NIST Cybersecurity Framework Version 1.1 (April 16, 2018)
- NIST SP 800-53 Rev. 5 — Security and Privacy Controls for Information Systems
- NIST SP 800-88 Rev. 1 — Guidelines for Media Sanitization
- NIST SP 800-175B — Guideline for Using Cryptographic Standards
Automated PII Detection and Redaction Pipeline
Overview
Automated PII detection is a foundational capability for privacy engineering, enabling organizations to discover, classify, and protect personal data at scale. This skill covers building production-grade PII detection pipelines that combine rule-based pattern matching, machine learning-based Named Entity Recognition (NER), and cloud-native discovery services.
PII Entity Types Catalog
Direct Identifiers
| Entity Type | Examples | Detection Method | Risk Level |
|---|---|---|---|
| PERSON_NAME | "John Smith", "Maria Garcia" | NER model | High |
| EMAIL_ADDRESS | "j.smith@cipherengineeringlabs.com" | Regex pattern | High |
| PHONE_NUMBER | "+1-555-0123", "(555) 012-3456" | Regex + validation | High |
| SSN | "123-45-6789" | Regex + checksum | Critical |
| PASSPORT_NUMBER | "AB1234567" | Regex per country format | Critical |
| DRIVER_LICENSE | "D123-4567-8901" | Regex per state/country | Critical |
| CREDIT_CARD | "4111-1111-1111-1111" | Regex + Luhn checksum | Critical |
| IBAN | "GB82 WEST 1234 5698 7654 32" | Regex + modulo-97 check | High |
| IP_ADDRESS | "192.168.1.1", "2001:db8::1" | Regex (IPv4/IPv6) | Medium |
| MAC_ADDRESS | "00:1A:2B:3C:4D:5E" | Regex pattern | Medium |
Quasi-Identifiers
| Entity Type | Examples | Detection Method | Risk Level |
|---|---|---|---|
| DATE_OF_BIRTH | "1990-01-15", "January 15, 1990" | NER + date parsing | Medium |
| POSTAL_CODE | "10001", "SW1A 1AA" | Regex per country | Medium |
| AGE | "35 years old", "age: 42" | NER + context | Low-Medium |
| GENDER | "male", "female", "non-binary" | Dictionary + context | Low |
| NATIONALITY | "British", "Japanese" | Dictionary + NER | Low-Medium |
| LOCATION | "123 Main St", "New York" | NER model | Medium |
Sensitive Categories
| Entity Type | Examples | Detection Method | Risk Level |
|---|---|---|---|
| MEDICAL_RECORD | "MRN: 12345678" | Regex + context | Critical |
| HEALTH_CONDITION | "diabetes", "HIV positive" | Medical NER + dictionary | Critical |
| RELIGIOUS_BELIEF | "Muslim", "Catholic" | Dictionary + context | High |
| POLITICAL_OPINION | "Democratic Party member" | Dictionary + context | High |
| SEXUAL_ORIENTATION | "gay", "bisexual" | Dictionary + context | High |
| BIOMETRIC_DATA | "fingerprint hash: ..." | Context + pattern | Critical |
| GENETIC_DATA | "BRCA1 positive" | Medical dictionary | Critical |
Microsoft Presidio Implementation
Pipeline Architecture
Input Data --> Presidio Analyzer --> Detected Entities --> Presidio Anonymizer --> Redacted Output
| | |
v v v
+------------+ +-------------+ +---------------+
| Recognizers| | Score Filter | | Operators |
| - Pattern | | (threshold) | | - Replace |
| - NER | | | | - Redact |
| - Custom | | | | - Hash |
+------------+ +-------------+ | - Mask |
| - Encrypt |
+---------------+
Core Implementation
"""
PII detection and redaction pipeline using Microsoft Presidio.
Supports structured and unstructured text with configurable
entity types, confidence thresholds, and redaction strategies.
"""
from presidio_analyzer import AnalyzerEngine, PatternRecognizer, Pattern
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig
import json
from dataclasses import dataclass, field
@dataclass
class DetectionResult:
entity_type: str
text: str
start: int
end: int
score: float
source: str
@dataclass
class PipelineConfig:
language: str = "en"
score_threshold: float = 0.5
entities_to_detect: list[str] = field(default_factory=lambda: [
"PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD",
"US_SSN", "US_DRIVER_LICENSE", "IBAN_CODE", "IP_ADDRESS",
"LOCATION", "DATE_TIME", "NRP", "MEDICAL_LICENSE",
"US_PASSPORT", "US_BANK_NUMBER", "UK_NHS"
])
redaction_strategy: str = "replace" # replace, redact, hash, mask, encrypt
class PIIDetectionPipeline:
"""
Production PII detection pipeline built on Microsoft Presidio.
Supports custom entity types and configurable redaction strategies.
"""
def __init__(self, config: PipelineConfig):
self.config = config
# Initialize NLP engine with spaCy
nlp_config = {
"nlp_engine_name": "spacy",
"models": [{"lang_code": config.language, "model_name": "en_core_web_lg"}]
}
nlp_engine = NlpEngineProvider(nlp_configuration=nlp_config).create_engine()
# Initialize analyzer
self.analyzer = AnalyzerEngine(nlp_engine=nlp_engine)
# Register custom recognizers
self._register_custom_recognizers()
# Initialize anonymizer
self.anonymizer = AnonymizerEngine()
def _register_custom_recognizers(self):
"""Register custom PII recognizers beyond built-in types."""
# UK National Insurance Number
nino_pattern = Pattern(
name="uk_nino",
regex=r"\b[A-CEGHJ-PR-TW-Z]{2}\s?\d{2}\s?\d{2}\s?\d{2}\s?[A-D]\b",
score=0.85
)
nino_recognizer = PatternRecognizer(
supported_entity="UK_NINO",
patterns=[nino_pattern],
supported_language="en"
)
self.analyzer.registry.add_recognizer(nino_recognizer)
# Vehicle Registration Number (UK format)
vrn_pattern = Pattern(
name="uk_vrn",
regex=r"\b[A-Z]{2}\d{2}\s?[A-Z]{3}\b",
score=0.6
)
vrn_recognizer = PatternRecognizer(
supported_entity="UK_VRN",
patterns=[vrn_pattern],
supported_language="en"
)
self.analyzer.registry.add_recognizer(vrn_recognizer)
# Employee ID (Cipher Engineering Labs format: CEL-XXXXX)
emp_id_pattern = Pattern(
name="employee_id",
regex=r"\bCEL-\d{5}\b",
score=0.95
)
emp_id_recognizer = PatternRecognizer(
supported_entity="EMPLOYEE_ID",
patterns=[emp_id_pattern],
supported_language="en"
)
self.analyzer.registry.add_recognizer(emp_id_recognizer)
def detect(self, text: str) -> list[DetectionResult]:
"""
Detect PII entities in text.
Args:
text: Input text to scan
Returns:
List of detected PII entities with confidence scores
"""
results = self.analyzer.analyze(
text=text,
entities=self.config.entities_to_detect,
language=self.config.language,
score_threshold=self.config.score_threshold
)
return [
DetectionResult(
entity_type=r.entity_type,
text=text[r.start:r.end],
start=r.start,
end=r.end,
score=r.score,
source=r.analysis_explanation.recognizer if r.analysis_explanation else "unknown"
)
for r in results
]
def redact(self, text: str) -> tuple[str, list[DetectionResult]]:
"""
Detect and redact PII from text.
Returns:
Tuple of (redacted_text, list_of_detections)
"""
# Detect entities
analyzer_results = self.analyzer.analyze(
text=text,
entities=self.config.entities_to_detect,
language=self.config.language,
score_threshold=self.config.score_threshold
)
# Configure redaction operator
operators = self._get_operators()
# Apply redaction
anonymized = self.anonymizer.anonymize(
text=text,
analyzer_results=analyzer_results,
operators=operators
)
detections = [
DetectionResult(
entity_type=r.entity_type,
text=text[r.start:r.end],
start=r.start,
end=r.end,
score=r.score,
source="presidio"
)
for r in analyzer_results
]
return anonymized.text, detections
def _get_operators(self) -> dict:
"""Configure anonymization operators based on strategy."""
if self.config.redaction_strategy == "replace":
return {"DEFAULT": OperatorConfig("replace", {"new_value": "<REDACTED>"})}
elif self.config.redaction_strategy == "hash":
return {"DEFAULT": OperatorConfig("hash", {"hash_type": "sha256"})}
elif self.config.redaction_strategy == "mask":
return {"DEFAULT": OperatorConfig("mask", {
"type": "mask",
"masking_char": "*",
"chars_to_mask": 100,
"from_end": False
})}
elif self.config.redaction_strategy == "redact":
return {"DEFAULT": OperatorConfig("redact", {})}
else:
return {"DEFAULT": OperatorConfig("replace", {"new_value": "<REDACTED>"})}
class BatchPIIProcessor:
"""
Process large volumes of documents for PII detection.
Supports structured (CSV/JSON) and unstructured (text) formats.
"""
def __init__(self, pipeline: PIIDetectionPipeline):
self.pipeline = pipeline
def process_csv(
self,
input_path: str,
output_path: str,
columns_to_scan: list[str] = None
) -> dict:
"""
Scan and redact PII in a CSV file.
Args:
input_path: Path to input CSV
output_path: Path to write redacted CSV
columns_to_scan: Columns to scan (all if None)
Returns:
Summary statistics of detections
"""
import pandas as pd
df = pd.read_csv(input_path)
stats = {"total_rows": len(df), "detections": {}, "columns_scanned": []}
scan_columns = columns_to_scan or df.columns.tolist()
for col in scan_columns:
if col not in df.columns:
continue
stats["columns_scanned"].append(col)
col_detections = []
for idx, value in df[col].items():
if pd.isna(value):
continue
text = str(value)
redacted, detections = self.pipeline.redact(text)
df.at[idx, col] = redacted
for d in detections:
col_detections.append(d.entity_type)
# Count detection types for this column
for entity_type in set(col_detections):
count = col_detections.count(entity_type)
key = f"{col}:{entity_type}"
stats["detections"][key] = count
df.to_csv(output_path, index=False)
stats["total_detections"] = sum(stats["detections"].values())
return stats
def process_text_files(
self,
file_paths: list[str],
output_dir: str
) -> dict:
"""
Batch process text files for PII detection and redaction.
Returns summary statistics.
"""
import os
stats = {
"files_processed": 0,
"total_detections": 0,
"entity_counts": {},
"high_risk_files": []
}
for file_path in file_paths:
with open(file_path, "r", encoding="utf-8") as f:
text = f.read()
redacted, detections = self.pipeline.redact(text)
# Write redacted output
output_path = os.path.join(output_dir, os.path.basename(file_path))
with open(output_path, "w", encoding="utf-8") as f:
f.write(redacted)
# Update statistics
stats["files_processed"] += 1
stats["total_detections"] += len(detections)
for d in detections:
stats["entity_counts"][d.entity_type] = (
stats["entity_counts"].get(d.entity_type, 0) + 1
)
# Flag high-risk files (contain critical PII)
critical_types = {"US_SSN", "CREDIT_CARD", "MEDICAL_LICENSE", "UK_NHS"}
if any(d.entity_type in critical_types for d in detections):
stats["high_risk_files"].append(file_path)
return stats
spaCy NER Custom Training
Training Custom Entity Types
"""
Train custom spaCy NER model for domain-specific PII entities.
"""
import spacy
from spacy.training import Example
import random
def create_training_data() -> list[tuple[str, dict]]:
"""
Create training examples for custom PII entity types.
Returns list of (text, annotations) tuples.
"""
training_data = [
(
"Patient MRN 12345678 was admitted on 2024-01-15",
{"entities": [(12, 20, "MEDICAL_RECORD_NUMBER")]}
),
(
"Employee CEL-00142 reported the incident",
{"entities": [(9, 18, "EMPLOYEE_ID")]}
),
(
"Policy holder number PLH-2024-99887 filed a claim",
{"entities": [(22, 35, "POLICY_NUMBER")]}
),
(
"The customer with loyalty ID LYL-A1B2C3 requested data export",
{"entities": [(29, 39, "LOYALTY_ID")]}
),
]
return training_data
def train_custom_ner(
base_model: str = "en_core_web_lg",
training_data: list = None,
n_iter: int = 30,
output_dir: str = "./custom_ner_model"
):
"""
Fine-tune spaCy NER model with custom PII entity types.
Args:
base_model: Base spaCy model to fine-tune
training_data: List of (text, annotations) tuples
n_iter: Number of training iterations
output_dir: Directory to save trained model
"""
if training_data is None:
training_data = create_training_data()
nlp = spacy.load(base_model)
# Get or create NER pipe
if "ner" not in nlp.pipe_names:
ner = nlp.add_pipe("ner", last=True)
else:
ner = nlp.get_pipe("ner")
# Add custom entity labels
custom_labels = set()
for _, annotations in training_data:
for ent in annotations.get("entities", []):
custom_labels.add(ent[2])
for label in custom_labels:
ner.add_label(label)
# Train
optimizer = nlp.resume_training()
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
with nlp.disable_pipes(*other_pipes):
for iteration in range(n_iter):
random.shuffle(training_data)
losses = {}
for text, annotations in training_data:
doc = nlp.make_doc(text)
example = Example.from_dict(doc, annotations)
nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)
if iteration % 10 == 0:
print(f"Iteration {iteration}, Losses: {losses}")
nlp.to_disk(output_dir)
return nlp
AWS Macie Integration
Architecture
S3 Buckets --> Macie Classification Jobs --> Findings --> EventBridge --> Lambda
| |
v v
Security Hub Remediation
(centralized) - Tag sensitive
- Encrypt
- Notify owner
- Quarantine
Macie Job Configuration
"""
AWS Macie integration for cloud-native PII detection in S3 buckets.
"""
import boto3
from datetime import datetime
class MacieIntegration:
"""
Configure and manage AWS Macie classification jobs for
automated PII detection across S3 data stores.
"""
def __init__(self, region: str = "us-east-1"):
self.macie_client = boto3.client("macie2", region_name=region)
def create_classification_job(
self,
bucket_name: str,
job_name: str,
custom_data_identifiers: list[str] = None,
schedule: str = "ONE_TIME"
) -> str:
"""
Create a Macie classification job for an S3 bucket.
Args:
bucket_name: Target S3 bucket
job_name: Descriptive job name
custom_data_identifiers: IDs of custom data identifier resources
schedule: ONE_TIME or SCHEDULED
Returns:
Job ID
"""
job_config = {
"name": job_name,
"description": f"PII detection scan for {bucket_name}",
"jobType": schedule,
"s3JobDefinition": {
"bucketDefinitions": [
{
"accountId": self._get_account_id(),
"buckets": [bucket_name]
}
],
"scoping": {
"includes": {
"and": [
{
"simpleScopeTerm": {
"comparator": "STARTS_WITH",
"key": "OBJECT_EXTENSION",
"values": ["csv", "json", "txt", "pdf", "docx", "xlsx", "parquet"]
}
}
]
}
}
},
"managedDataIdentifierSelector": "ALL",
"tags": {
"Team": "Privacy Engineering",
"Purpose": "PII Detection"
}
}
if custom_data_identifiers:
job_config["customDataIdentifierIds"] = custom_data_identifiers
response = self.macie_client.create_classification_job(**job_config)
return response["jobId"]
def create_custom_data_identifier(
self,
name: str,
description: str,
regex: str,
keywords: list[str] = None,
maximum_match_distance: int = 50
) -> str:
"""
Create a custom data identifier for organization-specific PII patterns.
Returns:
Custom data identifier ID
"""
params = {
"name": name,
"description": description,
"regex": regex,
"maximumMatchDistance": maximum_match_distance
}
if keywords:
params["keywords"] = keywords
response = self.macie_client.create_custom_data_identifier(**params)
return response["customDataIdentifierId"]
def _get_account_id(self) -> str:
"""Get the current AWS account ID."""
sts = boto3.client("sts")
return sts.get_caller_identity()["Account"]
def get_findings_summary(self, job_id: str) -> dict:
"""Get summary of findings from a classification job."""
response = self.macie_client.list_findings(
findingCriteria={
"criterion": {
"classificationDetails.jobId": {
"eq": [job_id]
}
}
}
)
findings = []
if response["findingIds"]:
details = self.macie_client.get_findings(findingIds=response["findingIds"])
findings = details["findings"]
summary = {
"total_findings": len(findings),
"severity_counts": {},
"entity_type_counts": {},
"affected_objects": []
}
for finding in findings:
severity = finding.get("severity", {}).get("description", "unknown")
summary["severity_counts"][severity] = (
summary["severity_counts"].get(severity, 0) + 1
)
sensitive_data = finding.get("classificationDetails", {}).get(
"result", {}
).get("sensitiveData", [])
for sd in sensitive_data:
category = sd.get("category", "unknown")
summary["entity_type_counts"][category] = (
summary["entity_type_counts"].get(category, 0)
+ sd.get("totalCount", 0)
)
resource = finding.get("resourcesAffected", {}).get("s3Object", {})
if resource:
summary["affected_objects"].append(resource.get("key", "unknown"))
return summary
Confidence Scoring Framework
| Score Range | Confidence Level | Recommended Action |
|---|---|---|
| 0.95 - 1.00 | Very High | Auto-redact |
| 0.80 - 0.94 | High | Auto-redact with logging |
| 0.60 - 0.79 | Medium | Flag for human review |
| 0.40 - 0.59 | Low | Log only, no action |
| 0.00 - 0.39 | Very Low | Ignore |
References
- Microsoft Presidio Documentation: microsoft.github.io/presidio
- spaCy NER Documentation: spacy.io/usage/linguistic-features#named-entities
- AWS Macie Documentation: docs.aws.amazon.com/macie
- Google Cloud DLP API Documentation
- NIST SP 800-188 — De-Identifying Government Datasets
- Article 29 WP Opinion 05/2014 on Anonymisation Techniques
Privacy API Design Patterns
Overview
Privacy APIs provide programmatic interfaces for data subject rights exercise, consent management, data deletion, and audit logging. Well-designed privacy APIs enable organizations to automate compliance workflows, integrate with consent management platforms, and provide data subjects with self-service privacy controls. This skill covers API design patterns, OpenAPI specifications, authentication, and error handling for privacy-critical endpoints.
API Architecture Overview
External Consumers Privacy API Gateway Backend Services
+------------------+ +--------------------+ +------------------+
| Data Subject App |----HTTPS/TLS--->| Authentication | | DSAR Service |
+------------------+ | Rate Limiting |----internal---->| Consent Service |
| Request Validation | | Deletion Service |
+------------------+ | Audit Logging | | Audit Service |
| Partner Portal |----HTTPS/TLS--->| Versioning | | Identity Service |
+------------------+ +--------------------+ +------------------+
|
+------------------+ v
| Internal Systems |----mTLS-------->+--------------------+
+------------------+ | Privacy Event Bus |
| (async processing) |
+--------------------+
1. Data Subject API (DSAR Endpoints)
OpenAPI Specification
openapi: 3.1.0
info:
title: Cipher Engineering Labs - Data Subject Rights API
version: 1.0.0
description: >
API for data subjects to exercise their privacy rights under GDPR,
CCPA/CPRA, and other applicable privacy regulations.
contact:
name: Privacy Engineering Team
email: privacy-engineering@cipherengineeringlabs.com
servers:
- url: https://api.cipherengineeringlabs.com/privacy/v1
description: Production
security:
- BearerAuth: []
- OAuth2: [dsar:read, dsar:write]
paths:
/dsar/requests:
post:
operationId: createDSARRequest
summary: Submit a new data subject access request
tags: [DSAR]
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/DSARCreateRequest'
responses:
'202':
description: Request accepted for processing
content:
application/json:
schema:
$ref: '#/components/schemas/DSARResponse'
headers:
X-Request-ID:
schema:
type: string
format: uuid
Location:
schema:
type: string
format: uri
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
'429':
$ref: '#/components/responses/RateLimited'
get:
operationId: listDSARRequests
summary: List data subject's own requests
tags: [DSAR]
parameters:
- name: status
in: query
schema:
type: string
enum: [pending, in_progress, completed, rejected]
- name: type
in: query
schema:
type: string
enum: [access, deletion, portability, rectification, restriction, objection]
- name: page
in: query
schema:
type: integer
minimum: 1
default: 1
- name: per_page
in: query
schema:
type: integer
minimum: 1
maximum: 100
default: 20
responses:
'200':
description: List of DSAR requests
content:
application/json:
schema:
$ref: '#/components/schemas/DSARListResponse'
/dsar/requests/{requestId}:
get:
operationId: getDSARRequest
summary: Get status and details of a specific DSAR
tags: [DSAR]
parameters:
- name: requestId
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: DSAR details
content:
application/json:
schema:
$ref: '#/components/schemas/DSARDetailResponse'
'404':
$ref: '#/components/responses/NotFound'
/dsar/requests/{requestId}/download:
get:
operationId: downloadDSARData
summary: Download the data package for a completed access request
tags: [DSAR]
parameters:
- name: requestId
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: Data package download
content:
application/zip:
schema:
type: string
format: binary
headers:
Content-Disposition:
schema:
type: string
'404':
$ref: '#/components/responses/NotFound'
'410':
description: Download link has expired
components:
schemas:
DSARCreateRequest:
type: object
required:
- requestType
- email
- identityVerification
properties:
requestType:
type: string
enum: [access, deletion, portability, rectification, restriction, objection]
description: Type of data subject right being exercised
email:
type: string
format: email
description: Email address associated with the account
identityVerification:
type: object
properties:
method:
type: string
enum: [account_login, email_verification, document_upload]
verificationToken:
type: string
required: [method]
details:
type: string
maxLength: 2000
description: Additional details about the request
dataCategories:
type: array
items:
type: string
enum: [profile, transactions, communications, analytics, marketing, all]
description: Specific data categories (for access/portability)
rectificationData:
type: object
additionalProperties: true
description: Corrected data values (for rectification requests)
preferredFormat:
type: string
enum: [json, csv, pdf]
default: json
description: Preferred data export format (for access/portability)
DSARResponse:
type: object
properties:
requestId:
type: string
format: uuid
status:
type: string
enum: [pending, identity_verification, in_progress, completed, rejected]
requestType:
type: string
submittedAt:
type: string
format: date-time
estimatedCompletionDate:
type: string
format: date-time
regulatoryDeadline:
type: string
format: date-time
description: Maximum date by which response must be provided
DSARDetailResponse:
allOf:
- $ref: '#/components/schemas/DSARResponse'
- type: object
properties:
statusHistory:
type: array
items:
type: object
properties:
status:
type: string
timestamp:
type: string
format: date-time
note:
type: string
downloadAvailable:
type: boolean
downloadExpiresAt:
type: string
format: date-time
DSARListResponse:
type: object
properties:
data:
type: array
items:
$ref: '#/components/schemas/DSARResponse'
pagination:
$ref: '#/components/schemas/Pagination'
Pagination:
type: object
properties:
page:
type: integer
perPage:
type: integer
totalPages:
type: integer
totalItems:
type: integer
responses:
BadRequest:
description: Invalid request parameters
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
Unauthorized:
description: Authentication required
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
NotFound:
description: Resource not found
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
RateLimited:
description: Rate limit exceeded
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponse'
headers:
Retry-After:
schema:
type: integer
X-RateLimit-Limit:
schema:
type: integer
X-RateLimit-Remaining:
schema:
type: integer
securitySchemes:
BearerAuth:
type: http
scheme: bearer
bearerFormat: JWT
OAuth2:
type: oauth2
flows:
authorizationCode:
authorizationUrl: https://auth.cipherengineeringlabs.com/oauth2/authorize
tokenUrl: https://auth.cipherengineeringlabs.com/oauth2/token
scopes:
dsar:read: Read DSAR request status
dsar:write: Submit DSAR requests
consent:read: Read consent preferences
consent:write: Update consent preferences
audit:read: Read audit logs
2. Consent API
Consent Preference Management
paths:
/consent/preferences:
get:
operationId: getConsentPreferences
summary: Get current consent preferences for the authenticated user
tags: [Consent]
responses:
'200':
description: Current consent preferences
content:
application/json:
schema:
type: object
properties:
subjectId:
type: string
preferences:
type: array
items:
$ref: '#/components/schemas/ConsentPreference'
lastUpdated:
type: string
format: date-time
put:
operationId: updateConsentPreferences
summary: Update consent preferences (bulk update)
tags: [Consent]
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [preferences]
properties:
preferences:
type: array
items:
type: object
required: [purposeId, granted]
properties:
purposeId:
type: string
granted:
type: boolean
channels:
type: array
items:
type: string
enum: [email, sms, push, phone, postal]
responses:
'200':
description: Preferences updated
content:
application/json:
schema:
type: object
properties:
updated:
type: array
items:
$ref: '#/components/schemas/ConsentPreference'
consentReceiptId:
type: string
format: uuid
description: ID of the issued consent receipt
/consent/purposes:
get:
operationId: listConsentPurposes
summary: List all available consent purposes
tags: [Consent]
responses:
'200':
description: List of consent purposes
content:
application/json:
schema:
type: object
properties:
purposes:
type: array
items:
type: object
properties:
purposeId:
type: string
name:
type: string
description:
type: string
category:
type: string
enum: [essential, functional, analytics, marketing]
mandatory:
type: boolean
defaultState:
type: boolean
dataCategories:
type: array
items:
type: string
/consent/receipts/{receiptId}:
get:
operationId: getConsentReceipt
summary: Retrieve a specific consent receipt
tags: [Consent]
parameters:
- name: receiptId
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: Consent receipt
content:
application/json:
schema:
$ref: '#/components/schemas/ConsentReceipt'
3. Deletion API
Cascading Delete Orchestration
"""
Privacy deletion API with cascading delete orchestration.
Handles deletion requests across multiple backend services
with verification and audit logging.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Optional
import uuid
class DeletionStatus(Enum):
PENDING = "pending"
IN_PROGRESS = "in_progress"
PARTIALLY_COMPLETED = "partially_completed"
COMPLETED = "completed"
FAILED = "failed"
VERIFIED = "verified"
class ServiceDeletionStatus(Enum):
PENDING = "pending"
IN_PROGRESS = "in_progress"
COMPLETED = "completed"
FAILED = "failed"
SKIPPED = "skipped" # For legally exempt data
@dataclass
class DeletionTarget:
service_name: str
data_categories: list[str]
priority: int # Lower number = delete first
legal_hold_check: bool = True
retention_exempt: bool = False # True if data must be retained (e.g., tax records)
@dataclass
class ServiceDeletionResult:
service_name: str
status: ServiceDeletionStatus
records_deleted: int
records_retained: int
retention_reason: Optional[str]
completed_at: Optional[datetime]
error_message: Optional[str]
@dataclass
class DeletionRequest:
request_id: str
subject_id: str
status: DeletionStatus
created_at: datetime
scope: list[str] # Data categories to delete
exclude: list[str] # Data categories exempt from deletion
service_results: list[ServiceDeletionResult] = field(default_factory=list)
verified_at: Optional[datetime] = None
verification_method: Optional[str] = None
class CascadingDeletionOrchestrator:
"""
Orchestrate data deletion across multiple backend services
in the correct order, handling dependencies, legal holds,
and retention requirements.
"""
def __init__(self, service_registry: list[DeletionTarget], audit_logger):
self.targets = sorted(service_registry, key=lambda t: t.priority)
self.audit = audit_logger
def execute_deletion(
self,
subject_id: str,
scope: list[str],
exclude: list[str] = None
) -> DeletionRequest:
"""
Execute cascading deletion across all registered services.
Args:
subject_id: The data subject whose data should be deleted
scope: Data categories to delete (or ["all"])
exclude: Data categories to exclude from deletion
Returns:
DeletionRequest with results from each service
"""
exclude = exclude or []
request = DeletionRequest(
request_id=str(uuid.uuid4()),
subject_id=subject_id,
status=DeletionStatus.IN_PROGRESS,
created_at=datetime.now(timezone.utc),
scope=scope,
exclude=exclude
)
self.audit.log_deletion_start(request)
for target in self.targets:
# Skip if data category not in scope
if "all" not in scope:
relevant_categories = set(target.data_categories) & set(scope)
if not relevant_categories:
continue
# Skip if explicitly excluded
if set(target.data_categories) & set(exclude):
result = ServiceDeletionResult(
service_name=target.service_name,
status=ServiceDeletionStatus.SKIPPED,
records_deleted=0,
records_retained=0,
retention_reason="Excluded by request",
completed_at=datetime.now(timezone.utc),
error_message=None
)
request.service_results.append(result)
continue
# Check retention exemptions
if target.retention_exempt:
result = ServiceDeletionResult(
service_name=target.service_name,
status=ServiceDeletionStatus.SKIPPED,
records_deleted=0,
records_retained=0,
retention_reason="Legal retention requirement",
completed_at=datetime.now(timezone.utc),
error_message=None
)
request.service_results.append(result)
continue
# Check legal holds
if target.legal_hold_check:
hold = self._check_legal_hold(subject_id, target.service_name)
if hold:
result = ServiceDeletionResult(
service_name=target.service_name,
status=ServiceDeletionStatus.SKIPPED,
records_deleted=0,
records_retained=0,
retention_reason=f"Legal hold: {hold}",
completed_at=datetime.now(timezone.utc),
error_message=None
)
request.service_results.append(result)
continue
# Execute deletion for this service
result = self._delete_from_service(subject_id, target)
request.service_results.append(result)
# Determine overall status
statuses = [r.status for r in request.service_results]
if all(s in (ServiceDeletionStatus.COMPLETED, ServiceDeletionStatus.SKIPPED) for s in statuses):
request.status = DeletionStatus.COMPLETED
elif any(s == ServiceDeletionStatus.FAILED for s in statuses):
request.status = DeletionStatus.PARTIALLY_COMPLETED
else:
request.status = DeletionStatus.COMPLETED
self.audit.log_deletion_complete(request)
return request
def verify_deletion(self, request: DeletionRequest) -> DeletionRequest:
"""
Verify that data was actually deleted from all services.
Run after deletion to confirm no data remnants exist.
"""
for result in request.service_results:
if result.status == ServiceDeletionStatus.COMPLETED:
# Attempt to query for the subject's data
remaining = self._check_data_exists(
request.subject_id, result.service_name
)
if remaining > 0:
result.status = ServiceDeletionStatus.FAILED
result.error_message = f"Verification failed: {remaining} records still exist"
request.verified_at = datetime.now(timezone.utc)
request.verification_method = "post_deletion_query"
all_verified = all(
r.status in (ServiceDeletionStatus.COMPLETED, ServiceDeletionStatus.SKIPPED)
for r in request.service_results
)
request.status = DeletionStatus.VERIFIED if all_verified else DeletionStatus.PARTIALLY_COMPLETED
self.audit.log_deletion_verification(request)
return request
def _check_legal_hold(self, subject_id: str, service: str) -> Optional[str]:
"""Check if any legal hold applies to this subject's data."""
# Integration point with legal hold management system
return None # Override in production with actual legal hold check
def _delete_from_service(
self, subject_id: str, target: DeletionTarget
) -> ServiceDeletionResult:
"""Execute deletion against a specific backend service."""
# Integration point with actual backend service deletion API
return ServiceDeletionResult(
service_name=target.service_name,
status=ServiceDeletionStatus.COMPLETED,
records_deleted=0,
records_retained=0,
retention_reason=None,
completed_at=datetime.now(timezone.utc),
error_message=None
)
def _check_data_exists(self, subject_id: str, service: str) -> int:
"""Check if data still exists for a subject in a service."""
# Integration point with service data existence check
return 0
4. Audit API
Audit Log Query Endpoints
paths:
/audit/events:
get:
operationId: queryAuditEvents
summary: Query privacy audit events
tags: [Audit]
security:
- OAuth2: [audit:read]
parameters:
- name: subjectId
in: query
schema:
type: string
- name: eventType
in: query
schema:
type: string
enum: [data_access, consent_change, deletion, dsar_submitted, dsar_completed, breach_detected]
- name: startDate
in: query
required: true
schema:
type: string
format: date-time
- name: endDate
in: query
required: true
schema:
type: string
format: date-time
- name: actorId
in: query
schema:
type: string
- name: page
in: query
schema:
type: integer
default: 1
- name: perPage
in: query
schema:
type: integer
default: 50
maximum: 200
responses:
'200':
description: Audit events
content:
application/json:
schema:
type: object
properties:
events:
type: array
items:
type: object
properties:
eventId:
type: string
format: uuid
eventType:
type: string
timestamp:
type: string
format: date-time
actorId:
type: string
actorType:
type: string
enum: [user, system, admin, api_client]
subjectId:
type: string
resource:
type: string
action:
type: string
purpose:
type: string
outcome:
type: string
enum: [success, failure, denied]
details:
type: object
pagination:
$ref: '#/components/schemas/Pagination'
/audit/reports/compliance:
get:
operationId: getComplianceReport
summary: Generate compliance audit report for a date range
tags: [Audit]
parameters:
- name: startDate
in: query
required: true
schema:
type: string
format: date
- name: endDate
in: query
required: true
schema:
type: string
format: date
- name: format
in: query
schema:
type: string
enum: [json, pdf, csv]
default: json
responses:
'200':
description: Compliance report
content:
application/json:
schema:
type: object
properties:
reportId:
type: string
generatedAt:
type: string
format: date-time
period:
type: object
properties:
start:
type: string
end:
type: string
summary:
type: object
properties:
totalDataAccesses:
type: integer
totalConsentChanges:
type: integer
totalDeletions:
type: integer
totalDSARs:
type: integer
accessDenials:
type: integer
policyViolations:
type: integer
API Security Patterns
Authentication and Authorization
| Endpoint Category | Auth Method | Scopes Required | Rate Limit |
|---|---|---|---|
| DSAR (data subject) | OAuth2 PKCE / JWT | dsar:read, dsar:write | 10 req/min |
| DSAR (admin) | OAuth2 Client Credentials | dsar:admin | 100 req/min |
| Consent (data subject) | OAuth2 PKCE / JWT | consent:read, consent:write | 30 req/min |
| Deletion (internal) | mTLS + API key | deletion:execute | 5 req/min |
| Audit (compliance) | OAuth2 Client Credentials | audit:read | 50 req/min |
Error Response Format
{
"error": {
"code": "PRIVACY_001",
"type": "identity_verification_required",
"message": "Identity verification is required before processing this request",
"details": {
"verificationMethods": ["email_verification", "document_upload"],
"supportUrl": "https://support.cipherengineeringlabs.com/privacy"
},
"requestId": "req_a1b2c3d4",
"timestamp": "2026-03-14T10:30:00.000Z"
}
}
References
- OpenAPI Specification 3.1.0
- GDPR Articles 12-22 (Data Subject Rights)
- CCPA/CPRA Sections 1798.100-1798.135
- RFC 6749 — OAuth 2.0 Authorization Framework
- RFC 7519 — JSON Web Token (JWT)
- OWASP API Security Top 10 (2023)
- Google Privacy Sandbox API Design Guidelines
- Apple App Tracking Transparency API Design Patterns
Privacy-Preserving Data Sharing Platform
Overview
Privacy-preserving data sharing enables organizations to derive analytical value from combined datasets without exposing raw personal data. This skill covers four primary approaches: synthetic data generation, data clean rooms, secure enclaves, and federated analytics, along with utility measurement frameworks to ensure shared data remains useful.
Approach Selection Framework
| Approach | Privacy Guarantee | Data Utility | Computational Cost | Trust Model |
|---|---|---|---|---|
| Synthetic Data | Statistical (configurable) | High for distributions, lower for edge cases | Medium (training) | No trust required |
| Data Clean Rooms | Contractual + technical | High (real data, restricted queries) | Low-Medium | Trusted operator |
| Secure Enclaves (TEE) | Hardware-backed isolation | Very high (real data) | Medium | Trust hardware vendor |
| Federated Analytics | Cryptographic/DP | Medium-High | High (communication) | Minimal trust |
| Homomorphic Encryption | Cryptographic | High | Very High | No trust required |
| Secure Multi-Party Computation | Cryptographic | High | High | Honest majority |
Synthetic Data Generation with SDV
Architecture
Source Data --> Statistical Profiling --> Model Training --> Synthetic Generation
| | |
v v v
Metadata Analysis Model Selection Quality Assessment
- Column types - GaussianCopula - Statistical tests
- Distributions - CTGAN - Privacy metrics
- Correlations - CopulaGAN - Utility metrics
- Constraints - TVAE - Visual comparison
SDV Implementation
"""
Synthetic data generation using the Synthetic Data Vault (SDV) library.
Generates privacy-preserving synthetic datasets that maintain statistical
properties of the original data.
"""
import pandas as pd
import numpy as np
from sdv.metadata import SingleTableMetadata
from sdv.single_table import GaussianCopulaSynthesizer, CTGANSynthesizer, TVAESynthesizer
from sdv.evaluation.single_table import run_diagnostic, evaluate_quality
from sdmetrics.reports.single_table import QualityReport
def create_metadata(df: pd.DataFrame) -> SingleTableMetadata:
"""Auto-detect and create metadata for a DataFrame."""
metadata = SingleTableMetadata()
metadata.detect_from_dataframe(df)
return metadata
def train_gaussian_copula(
df: pd.DataFrame,
metadata: SingleTableMetadata,
enforce_min_max: bool = True
) -> GaussianCopulaSynthesizer:
"""
Train a Gaussian Copula model for synthetic data generation.
Best for: Datasets with mostly numerical data and linear correlations.
"""
synthesizer = GaussianCopulaSynthesizer(
metadata,
enforce_min_max_values=enforce_min_max,
numerical_distributions={
"norm": "beta", # Fit beta distributions for bounded numerical data
}
)
synthesizer.fit(df)
return synthesizer
def train_ctgan(
df: pd.DataFrame,
metadata: SingleTableMetadata,
epochs: int = 300,
batch_size: int = 500
) -> CTGANSynthesizer:
"""
Train a CTGAN model for synthetic data generation.
Best for: Complex distributions, mixed data types, mode-specific patterns.
"""
synthesizer = CTGANSynthesizer(
metadata,
epochs=epochs,
batch_size=batch_size,
verbose=True
)
synthesizer.fit(df)
return synthesizer
def train_tvae(
df: pd.DataFrame,
metadata: SingleTableMetadata,
epochs: int = 300
) -> TVAESynthesizer:
"""
Train a TVAE model for synthetic data generation.
Best for: Datasets where CTGAN struggles, faster training than CTGAN.
"""
synthesizer = TVAESynthesizer(
metadata,
epochs=epochs
)
synthesizer.fit(df)
return synthesizer
def generate_synthetic_data(
synthesizer,
num_rows: int
) -> pd.DataFrame:
"""Generate synthetic data from a trained synthesizer."""
return synthesizer.sample(num_rows=num_rows)
def evaluate_synthetic_quality(
real_data: pd.DataFrame,
synthetic_data: pd.DataFrame,
metadata: SingleTableMetadata
) -> dict:
"""
Evaluate the quality of synthetic data against real data.
Returns diagnostic and quality scores.
"""
# Run diagnostic checks
diagnostic = run_diagnostic(
real_data=real_data,
synthetic_data=synthetic_data,
metadata=metadata
)
# Run quality evaluation
quality = evaluate_quality(
real_data=real_data,
synthetic_data=synthetic_data,
metadata=metadata
)
return {
"diagnostic_score": diagnostic.get_score(),
"quality_score": quality.get_score(),
}
def measure_privacy_risk(
real_data: pd.DataFrame,
synthetic_data: pd.DataFrame,
metadata: SingleTableMetadata,
key_fields: list[str]
) -> dict:
"""
Measure re-identification risk in synthetic data.
Checks for exact matches and nearest-neighbor distances
between real and synthetic records.
"""
# Check for exact record matches
merged = real_data.merge(synthetic_data, how="inner")
exact_match_rate = len(merged) / len(real_data)
# Check key field matches
if key_fields:
key_merged = real_data[key_fields].merge(
synthetic_data[key_fields], how="inner"
)
key_match_rate = len(key_merged) / len(real_data)
else:
key_match_rate = 0.0
return {
"exact_match_rate": exact_match_rate,
"key_match_rate": key_match_rate,
"privacy_safe": exact_match_rate < 0.01 and key_match_rate < 0.05,
}
Model Selection Guide
| Factor | GaussianCopula | CTGAN | TVAE |
|---|---|---|---|
| Training speed | Fast (minutes) | Slow (hours) | Medium (30-60 min) |
| Small datasets (<1K rows) | Good | Poor | Fair |
| Large datasets (>100K rows) | Good | Good | Good |
| Numerical data | Excellent | Good | Good |
| Categorical data (high cardinality) | Fair | Good | Good |
| Complex correlations | Fair | Good | Good |
| Constraint handling | Good | Fair | Fair |
| Reproducibility | Excellent | Fair (seed-dependent) | Fair |
Data Clean Room Architecture
Components
Organization A Clean Room Organization B
+-------------+ encrypted +------------------+ encrypted +-------------+
| Source Data | -----------> | Ingestion Layer | <----------- | Source Data |
+-------------+ +------------------+ +-------------+
|
v
+------------------+
| Data Preparation |
| - Schema mapping |
| - Normalization |
| - Deduplication |
+------------------+
|
v
+------------------+
| Approved Queries |
| - Pre-approved |
| query templates|
| - Aggregate only |
| - Min group size |
+------------------+
|
v
+------------------+
| Output Validation|
| - k-anonymity |
| - DP noise |
| - Disclosure risk|
+------------------+
|
+-----------+-----------+
| |
v v
Results for Org A Results for Org B
Clean Room Policy Engine
"""
Policy engine for data clean room query validation.
Enforces privacy rules on all queries before execution.
"""
from dataclasses import dataclass
@dataclass
class CleanRoomPolicy:
min_group_size: int = 50
allowed_operations: list[str] = None
blocked_columns: list[str] = None
max_output_rows: int = 1000
require_aggregation: bool = True
dp_epsilon: float = 1.0
def __post_init__(self):
if self.allowed_operations is None:
self.allowed_operations = ["COUNT", "SUM", "AVG", "MEDIAN", "PERCENTILE"]
if self.blocked_columns is None:
self.blocked_columns = ["ssn", "email", "phone", "full_name", "address"]
class QueryValidator:
"""Validate clean room queries against privacy policies."""
def __init__(self, policy: CleanRoomPolicy):
self.policy = policy
def validate(self, query_ast: dict) -> tuple[bool, list[str]]:
"""
Validate a parsed query against the policy.
Returns (is_valid, list_of_violations).
"""
violations = []
# Check for blocked columns
referenced_columns = query_ast.get("columns", [])
for col in referenced_columns:
if col.lower() in self.policy.blocked_columns:
violations.append(f"Column '{col}' is blocked by policy")
# Check aggregation requirement
if self.policy.require_aggregation:
if not query_ast.get("has_aggregation", False):
violations.append("Query must include aggregation (no raw record output)")
# Check operations
operations = query_ast.get("operations", [])
for op in operations:
if op.upper() not in self.policy.allowed_operations:
violations.append(f"Operation '{op}' is not in allowed operations list")
# Check output size
if query_ast.get("limit", float("inf")) > self.policy.max_output_rows:
violations.append(
f"Output exceeds max rows ({self.policy.max_output_rows})"
)
return (len(violations) == 0, violations)
Secure Enclave Integration
Intel SGX / Azure Confidential Computing
Data Owner A Confidential Computing Data Owner B
+----------------------+
Data (encrypted) ----> | Enclave Environment | <---- Data (encrypted)
| - Decryption in TEE |
| - Join/Analysis |
| - Re-encrypt results |
+----------------------+
|
Encrypted Results
(only to authorized parties)
Key Properties
- Confidentiality: Data is encrypted outside the enclave; only decrypted within the TEE
- Integrity: Enclave code is measured and attested; tampering is detectable
- Attestation: Remote parties can verify the enclave is running approved code
- Isolation: Even the cloud provider cannot access data inside the enclave
Utility Measurement Framework
Statistical Utility Metrics
| Metric | Description | Target |
|---|---|---|
| Column Shapes | Distribution similarity per column (KS test) | > 0.85 |
| Column Pair Trends | Correlation preservation between column pairs | > 0.80 |
| Boundary Adherence | Values within real data min/max ranges | > 0.95 |
| Category Coverage | All categories in real data appear in synthetic | > 0.90 |
| Range Coverage | Numeric ranges adequately represented | > 0.85 |
Privacy Metrics
| Metric | Description | Target |
|---|---|---|
| Exact Match Rate | % of synthetic records identical to real records | < 1% |
| Nearest Neighbor Distance | Minimum distance from synthetic to nearest real record | > threshold |
| Membership Inference AUC | Ability of attack model to determine membership | < 0.55 |
| Attribute Inference Accuracy | Ability to infer sensitive attributes | < random + 5% |
| k-Anonymity of output | Minimum equivalence class size | k >= 5 |
References
- Patki, N., Wedge, R., and Veeramachaneni, K. "The Synthetic Data Vault." IEEE DSAA, 2016.
- SDV Documentation: docs.sdv.dev
- Xu, L. et al. "Modeling Tabular Data Using Conditional GAN." NeurIPS, 2019.
- Google BigQuery Clean Rooms Documentation
- AWS Clean Rooms Service Documentation
- Microsoft Azure Confidential Computing Documentation
- Stadler, T. et al. "Synthetic Data — Anonymisation Groundhog Day." USENIX Security, 2022.
Privacy Metrics Dashboard
Overview
A privacy metrics dashboard provides quantitative visibility into the effectiveness and maturity of an organization's privacy program. This skill defines the key privacy performance indicators (KPIs), data collection methods, visualization patterns, and reporting cadences needed to measure, monitor, and continuously improve privacy operations.
Core Privacy KPI Categories
1. Data Subject Access Requests (DSARs)
| Metric | Definition | Target | Frequency |
|---|---|---|---|
| DSAR Volume | Total DSARs received per period | Trend monitoring | Monthly |
| DSAR by Type | Breakdown by right exercised (access, deletion, portability, rectification) | N/A (distribution) | Monthly |
| Average Response Time | Mean calendar days from receipt to completion | < 25 days (GDPR: 30 day limit) | Monthly |
| Median Response Time | Median calendar days from receipt to completion | < 20 days | Monthly |
| P95 Response Time | 95th percentile response time | < 28 days | Monthly |
| On-Time Completion Rate | % completed within regulatory deadline | > 98% | Monthly |
| DSAR Backlog | Open DSARs at period end | < 10% of monthly volume | Weekly |
| Automated Fulfillment Rate | % of DSARs fulfilled without manual intervention | > 60% | Quarterly |
| Identity Verification Failure Rate | % of DSARs where identity could not be verified | < 5% | Monthly |
| Cost per DSAR | Average cost to process one DSAR | Trend reduction | Quarterly |
2. Data Breaches
| Metric | Definition | Target | Frequency |
|---|---|---|---|
| Breach Count | Total confirmed breaches per period | 0 (minimize) | Monthly |
| Breach Severity Distribution | Count by severity level (low/medium/high/critical) | No critical breaches | Monthly |
| Mean Time to Detect (MTTD) | Average hours from breach occurrence to detection | < 24 hours | Per incident |
| Mean Time to Notify (MTTN) | Average hours from detection to regulatory notification | < 48 hours (GDPR: 72 hours) | Per incident |
| Mean Time to Contain (MTTC) | Average hours from detection to containment | < 12 hours | Per incident |
| Records Affected | Total data subject records affected | Minimize | Per incident |
| Repeat Breach Rate | % of breaches with same root cause as prior breach | 0% | Quarterly |
| Regulatory Fines | Total fines imposed due to breaches | $0 | Annual |
| Breach Simulation Score | Score from most recent tabletop exercise | > 85% | Semi-annual |
3. Privacy Impact Assessments (DPIAs)
| Metric | Definition | Target | Frequency |
|---|---|---|---|
| DPIA Completion Rate | % of high-risk processing with completed DPIA | 100% | Quarterly |
| DPIA Backlog | DPIAs required but not yet started | 0 | Monthly |
| Average DPIA Duration | Mean days from initiation to sign-off | < 30 days | Quarterly |
| DPIA Findings Remediated | % of identified risks with implemented mitigations | > 90% | Quarterly |
| DPIA Review Rate | % of existing DPIAs reviewed within their review cycle | > 95% | Quarterly |
| New Processing Without DPIA | Count of new high-risk processing launched without DPIA | 0 | Monthly |
4. Training and Awareness
| Metric | Definition | Target | Frequency |
|---|---|---|---|
| Training Completion Rate | % of staff who completed required privacy training | > 95% | Quarterly |
| Training Pass Rate | % passing the assessment on first attempt | > 85% | Quarterly |
| New Hire Training Time | Average days from start date to training completion | < 14 days | Monthly |
| Specialist Certification Rate | % of privacy team with professional certification | > 80% | Annual |
| Privacy Champion Coverage | % of business units with designated privacy champion | 100% | Quarterly |
| Phishing Simulation Pass Rate | % of staff correctly identifying privacy-related phishing | > 90% | Quarterly |
5. Consent Management
| Metric | Definition | Target | Frequency |
|---|---|---|---|
| Consent Rate | % of users providing consent (by purpose) | Trend monitoring | Monthly |
| Opt-in Rate | % choosing to opt in to optional processing | Trend monitoring | Monthly |
| Opt-out/Withdrawal Rate | % withdrawing previously given consent | < 5% monthly | Monthly |
| Consent Freshness | % of active consents collected within last 12 months | > 70% | Quarterly |
| Consent Collection Coverage | % of data processing with documented consent/legal basis | 100% | Quarterly |
| Cookie Consent Rate | % of website visitors accepting cookies (by category) | Trend monitoring | Monthly |
| Consent Record Accuracy | % of consent records matching actual processing | > 99% | Quarterly |
6. Operational Metrics
| Metric | Definition | Target | Frequency |
|---|---|---|---|
| Privacy Incidents Reported | Internal privacy incidents reported (not just breaches) | Trend monitoring | Monthly |
| Vendor Privacy Assessments Completed | % of high-risk vendors assessed | 100% | Quarterly |
| Data Inventory Completeness | % of processing activities documented in ROPA | > 95% | Quarterly |
| Policy Acknowledgment Rate | % of staff acknowledging updated privacy policies | > 98% | Per update |
| Privacy-by-Design Gate Pass Rate | % of projects passing privacy review at design gate | > 85% | Quarterly |
| Regulatory Inquiry Response Time | Average days to respond to regulatory inquiries | < 5 days | Per event |
Data Collection Architecture
Data Sources Collection Layer Dashboard Layer
+------------------+
| DSAR Platform |---+
+------------------+ |
| +-------------------+ +-----------------------+
+------------------+ +--->| | | |
| Breach Tracker |------->| Data Integration |--->| Visualization Engine |
+------------------+ +--->| (ETL Pipeline) | | (Grafana / Tableau / |
| | | | Power BI / Metabase) |
+------------------+ | +-------------------+ +-----------------------+
| DPIA Register |---+ | |
+------------------+ | v v
| +-------------------+ +-----------------------+
+------------------+ | | Metrics Database | | Alerting System |
| LMS (Training) |---+ | (Time-series / | | - Threshold breaches |
+------------------+ | | PostgreSQL) | | - Trend deviations |
| +-------------------+ | - SLA warnings |
+------------------+ | +-----------------------+
| Consent Platform |---+
+------------------+
Metric Collection Implementation
"""
Privacy metrics collection and aggregation engine.
Collects KPI data from multiple privacy operational systems
and computes dashboard-ready metrics.
"""
from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional
import statistics
@dataclass
class DSARMetrics:
period_start: datetime
period_end: datetime
total_received: int
total_completed: int
total_overdue: int
avg_response_days: float
median_response_days: float
p95_response_days: float
on_time_rate: float
by_type: dict[str, int]
backlog: int
automated_rate: float
@dataclass
class BreachMetrics:
period_start: datetime
period_end: datetime
total_breaches: int
by_severity: dict[str, int]
avg_detection_hours: float
avg_notification_hours: float
avg_containment_hours: float
total_records_affected: int
repeat_breach_count: int
@dataclass
class ConsentMetrics:
period_start: datetime
period_end: datetime
total_consent_events: int
opt_in_rate: float
withdrawal_rate: float
consent_rate_by_purpose: dict[str, float]
cookie_acceptance_rate: float
consent_freshness_pct: float
class PrivacyMetricsCollector:
"""
Collect and compute privacy KPIs from operational data sources.
"""
def __init__(self, dsar_repo, breach_repo, dpia_repo, training_repo, consent_repo):
self.dsar_repo = dsar_repo
self.breach_repo = breach_repo
self.dpia_repo = dpia_repo
self.training_repo = training_repo
self.consent_repo = consent_repo
def compute_dsar_metrics(
self,
start_date: datetime,
end_date: datetime,
regulatory_deadline_days: int = 30
) -> DSARMetrics:
"""Compute DSAR KPIs for a given period."""
dsars = self.dsar_repo.get_by_period(start_date, end_date)
completed = [d for d in dsars if d.status == "completed"]
response_times = [
(d.completed_at - d.received_at).days
for d in completed
if d.completed_at and d.received_at
]
on_time = [
d for d in completed
if (d.completed_at - d.received_at).days <= regulatory_deadline_days
]
automated = [d for d in completed if d.automated]
open_dsars = [d for d in dsars if d.status in ("open", "in_progress")]
by_type = {}
for d in dsars:
by_type[d.request_type] = by_type.get(d.request_type, 0) + 1
return DSARMetrics(
period_start=start_date,
period_end=end_date,
total_received=len(dsars),
total_completed=len(completed),
total_overdue=len(dsars) - len(on_time) - len(open_dsars),
avg_response_days=statistics.mean(response_times) if response_times else 0,
median_response_days=statistics.median(response_times) if response_times else 0,
p95_response_days=(
sorted(response_times)[int(len(response_times) * 0.95)]
if len(response_times) >= 20 else
max(response_times) if response_times else 0
),
on_time_rate=len(on_time) / len(completed) if completed else 1.0,
by_type=by_type,
backlog=len(open_dsars),
automated_rate=len(automated) / len(completed) if completed else 0,
)
def compute_breach_metrics(
self,
start_date: datetime,
end_date: datetime
) -> BreachMetrics:
"""Compute breach KPIs for a given period."""
breaches = self.breach_repo.get_by_period(start_date, end_date)
by_severity = {}
detection_hours = []
notification_hours = []
containment_hours = []
total_records = 0
for b in breaches:
by_severity[b.severity] = by_severity.get(b.severity, 0) + 1
total_records += b.records_affected or 0
if b.detected_at and b.occurred_at:
detection_hours.append(
(b.detected_at - b.occurred_at).total_seconds() / 3600
)
if b.notified_at and b.detected_at:
notification_hours.append(
(b.notified_at - b.detected_at).total_seconds() / 3600
)
if b.contained_at and b.detected_at:
containment_hours.append(
(b.contained_at - b.detected_at).total_seconds() / 3600
)
# Count repeat breaches (same root cause as previous)
root_causes = [b.root_cause for b in breaches if b.root_cause]
repeat_count = len(root_causes) - len(set(root_causes))
return BreachMetrics(
period_start=start_date,
period_end=end_date,
total_breaches=len(breaches),
by_severity=by_severity,
avg_detection_hours=statistics.mean(detection_hours) if detection_hours else 0,
avg_notification_hours=statistics.mean(notification_hours) if notification_hours else 0,
avg_containment_hours=statistics.mean(containment_hours) if containment_hours else 0,
total_records_affected=total_records,
repeat_breach_count=repeat_count,
)
def generate_executive_report(
self,
period_start: datetime,
period_end: datetime
) -> dict:
"""
Generate a comprehensive executive privacy report.
Combines all KPI categories into a single report structure.
"""
dsar = self.compute_dsar_metrics(period_start, period_end)
breach = self.compute_breach_metrics(period_start, period_end)
# Calculate overall privacy health score (0-100)
health_components = []
# DSAR component (25% weight)
dsar_score = min(100, dsar.on_time_rate * 100)
health_components.append(("DSAR Compliance", dsar_score, 0.25))
# Breach component (30% weight)
breach_score = 100 if breach.total_breaches == 0 else max(0, 100 - breach.total_breaches * 20)
health_components.append(("Breach Prevention", breach_score, 0.30))
# Response time component (20% weight)
response_score = min(100, max(0, (30 - dsar.avg_response_days) / 30 * 100))
health_components.append(("Response Timeliness", response_score, 0.20))
# Detection speed component (25% weight)
detection_score = min(100, max(0, (72 - breach.avg_detection_hours) / 72 * 100)) if breach.total_breaches > 0 else 100
health_components.append(("Detection Speed", detection_score, 0.25))
overall_score = sum(score * weight for _, score, weight in health_components)
return {
"period": {
"start": period_start.isoformat(),
"end": period_end.isoformat()
},
"overall_privacy_health_score": round(overall_score, 1),
"health_rating": (
"Excellent" if overall_score >= 90 else
"Good" if overall_score >= 75 else
"Needs Improvement" if overall_score >= 60 else
"Critical"
),
"components": {
name: {"score": round(score, 1), "weight": f"{weight*100:.0f}%"}
for name, score, weight in health_components
},
"dsar_summary": {
"volume": dsar.total_received,
"on_time_rate": f"{dsar.on_time_rate*100:.1f}%",
"avg_response_days": round(dsar.avg_response_days, 1),
"backlog": dsar.backlog
},
"breach_summary": {
"count": breach.total_breaches,
"severity_distribution": breach.by_severity,
"avg_detection_hours": round(breach.avg_detection_hours, 1),
"records_affected": breach.total_records_affected
},
}
Dashboard Visualization Patterns
Executive Summary View
| Panel | Chart Type | Metrics Displayed | Refresh Rate |
|---|---|---|---|
| Privacy Health Score | Gauge (0-100) | Overall weighted score | Daily |
| DSAR Volume Trend | Line chart | Monthly DSAR count (12-month rolling) | Daily |
| Breach Timeline | Event timeline | Breaches by severity over time | Real-time |
| On-Time Rate | Single stat with spark line | DSAR on-time completion % | Daily |
| Training Coverage | Horizontal bar | Completion rate by department | Weekly |
| Consent Rate Trend | Multi-line chart | Opt-in rate by purpose (6-month) | Daily |
Operational View
| Panel | Chart Type | Metrics Displayed | Refresh Rate |
|---|---|---|---|
| Active DSARs | Table with status | Open requests with deadlines | Hourly |
| DSAR Aging | Histogram | Distribution of open DSAR ages | Daily |
| Response Time Box Plot | Box plot | Response time distribution by month | Daily |
| Breach Detection Funnel | Funnel chart | Detection > Assessment > Notification > Resolution | Per event |
| DPIA Pipeline | Kanban | DPIAs by status (draft, review, approved) | Daily |
| Vendor Risk Heat Map | Heat map | Vendors by risk score and last assessment date | Weekly |
Alerting Rules
| Alert | Condition | Severity | Recipient |
|---|---|---|---|
| DSAR SLA Warning | Any DSAR open > 21 days | Warning | DSAR team |
| DSAR SLA Breach | Any DSAR open > 28 days | Critical | CPO + Legal |
| Breach Detected | New breach record created | Critical | Incident team + CPO |
| Training Compliance Drop | Department completion < 90% | Warning | HR + Privacy |
| Consent Rate Anomaly | Opt-in rate drops > 20% in a week | Warning | Privacy + Marketing |
| DPIA Overdue | DPIA review past scheduled date | Warning | DPIA owner + CPO |
| Vendor Assessment Due | High-risk vendor assessment > 11 months old | Warning | Procurement + Privacy |
Executive Reporting Template
Monthly Privacy Board Report Structure
- Privacy Health Score: Overall score with trend arrow
- Key Highlights: 3-5 bullet points on notable events
- DSAR Performance: Volume, on-time rate, top request types
- Breach Status: Count, severity, response times, lessons learned
- Compliance Posture: DPIA coverage, training rates, policy updates
- Risk Outlook: Emerging risks, regulatory developments, planned changes
- Resource Needs: Budget requests, headcount, tooling
References
- IAPP Privacy Program Management Guide (CIPM Body of Knowledge)
- NIST Privacy Framework — GV.MT Monitoring and Review
- ISO/IEC 27701:2019 — Clause 9 (Performance Evaluation)
- EDPB Guidelines on Data Breach Notification (WP250 rev.01)
- Gartner Privacy Program Metrics Framework
- ISACA Privacy Audit and Assurance Program
Privacy-Preserving Record Linkage
Overview
Privacy-Preserving Record Linkage (PPRL) enables two or more organizations to identify matching records across their datasets without revealing the underlying personal data to each other. This is critical for healthcare research, fraud detection, national statistics, and cross-organizational analytics where direct data sharing is prohibited by privacy regulations.
Approach Comparison
| Approach | Privacy Level | Accuracy | Scalability | Communication Cost |
|---|---|---|---|---|
| Bloom Filter Encoding | High | Good (>95% F1) | Very High | Low |
| Secure Hash Matching | Very High | High (exact match only) | Very High | Very Low |
| Secure Multi-Party Computation | Cryptographic | Very High | Medium | High |
| Trusted Third Party | Depends on TTP | Very High | High | Medium |
| Differential Privacy Linkage | Formally private | Moderate | High | Low |
Bloom Filter-Based PPRL
How It Works
- Each organization encodes their quasi-identifiers (name, date of birth, address) into Bloom filters
- The Bloom filter encoding uses cryptographic hash functions with a shared secret key
- Encoded Bloom filters are compared using similarity metrics (Dice coefficient, Jaccard)
- Matching pairs above a threshold are identified as linked records
- Raw data is never exchanged — only Bloom filter bit arrays
Bloom Filter Encoding Implementation
"""
Privacy-preserving record linkage using Bloom filter encoding.
Implements the approach described by Schnell, Bachteler, and Reiher (2009).
"""
import hashlib
import hmac
import math
from typing import Optional
import numpy as np
class BloomFilterEncoder:
"""
Encode string attributes into Bloom filters for privacy-preserving
record linkage using cryptographic keyed hashing.
"""
def __init__(
self,
filter_size: int = 1024,
num_hash_functions: int = 30,
ngram_size: int = 2,
secret_key: bytes = b""
):
"""
Args:
filter_size: Number of bits in the Bloom filter
num_hash_functions: Number of hash functions (k)
ngram_size: Size of character n-grams (bigrams = 2)
secret_key: Shared secret key for HMAC hashing
"""
self.filter_size = filter_size
self.num_hash_functions = num_hash_functions
self.ngram_size = ngram_size
self.secret_key = secret_key
def _generate_ngrams(self, value: str) -> list[str]:
"""Generate character n-grams from a string value."""
# Pad the string to handle edge characters
padded = f"_{value}_"
return [
padded[i:i + self.ngram_size]
for i in range(len(padded) - self.ngram_size + 1)
]
def _hash_ngram(self, ngram: str, hash_index: int) -> int:
"""
Hash an n-gram using HMAC with the shared key and hash index.
Returns a bit position in the Bloom filter.
"""
message = f"{hash_index}:{ngram}".encode("utf-8")
digest = hmac.new(self.secret_key, message, hashlib.sha256).digest()
# Convert first 4 bytes to integer and modulo by filter size
position = int.from_bytes(digest[:4], byteorder="big") % self.filter_size
return position
def encode_value(self, value: str) -> np.ndarray:
"""
Encode a single attribute value into a Bloom filter.
Args:
value: The string value to encode (e.g., a name)
Returns:
Numpy array of bits (0/1) representing the Bloom filter
"""
bloom_filter = np.zeros(self.filter_size, dtype=np.uint8)
# Normalize the input
normalized = value.strip().lower()
# Generate n-grams
ngrams = self._generate_ngrams(normalized)
# Hash each n-gram with each hash function
for ngram in ngrams:
for h in range(self.num_hash_functions):
position = self._hash_ngram(ngram, h)
bloom_filter[position] = 1
return bloom_filter
def encode_record(self, attributes: dict[str, str]) -> np.ndarray:
"""
Encode multiple attributes into a single composite Bloom filter
using Cryptographic Longterm Key (CLK) approach.
Args:
attributes: Dict mapping attribute names to values
e.g., {"first_name": "John", "last_name": "Smith", "dob": "1990-01-15"}
Returns:
Composite Bloom filter as numpy array
"""
composite = np.zeros(self.filter_size, dtype=np.uint8)
for attr_name, attr_value in attributes.items():
if attr_value:
# Use attribute name as additional salt
salted_key = self.secret_key + attr_name.encode("utf-8")
encoder = BloomFilterEncoder(
filter_size=self.filter_size,
num_hash_functions=self.num_hash_functions,
ngram_size=self.ngram_size,
secret_key=salted_key
)
attr_bf = encoder.encode_value(attr_value)
composite = np.bitwise_or(composite, attr_bf)
return composite
class BloomFilterMatcher:
"""
Compare Bloom filter-encoded records to find matching pairs.
"""
@staticmethod
def dice_coefficient(bf1: np.ndarray, bf2: np.ndarray) -> float:
"""
Calculate the Dice coefficient between two Bloom filters.
Dice = 2 * |bf1 AND bf2| / (|bf1| + |bf2|)
Returns a similarity score between 0 and 1.
"""
intersection = np.sum(np.bitwise_and(bf1, bf2))
cardinality_sum = np.sum(bf1) + np.sum(bf2)
if cardinality_sum == 0:
return 0.0
return 2.0 * intersection / cardinality_sum
@staticmethod
def jaccard_similarity(bf1: np.ndarray, bf2: np.ndarray) -> float:
"""
Calculate the Jaccard similarity between two Bloom filters.
Jaccard = |bf1 AND bf2| / |bf1 OR bf2|
"""
intersection = np.sum(np.bitwise_and(bf1, bf2))
union = np.sum(np.bitwise_or(bf1, bf2))
if union == 0:
return 0.0
return intersection / union
def find_matches(
self,
encodings_a: list[tuple[str, np.ndarray]],
encodings_b: list[tuple[str, np.ndarray]],
threshold: float = 0.8,
similarity_metric: str = "dice"
) -> list[tuple[str, str, float]]:
"""
Find matching record pairs between two encoded datasets.
Args:
encodings_a: List of (record_id, bloom_filter) from organization A
encodings_b: List of (record_id, bloom_filter) from organization B
threshold: Minimum similarity score for a match
similarity_metric: "dice" or "jaccard"
Returns:
List of (id_a, id_b, similarity_score) for matching pairs
"""
metric_fn = (
self.dice_coefficient if similarity_metric == "dice"
else self.jaccard_similarity
)
matches = []
for id_a, bf_a in encodings_a:
best_score = 0.0
best_id_b = None
for id_b, bf_b in encodings_b:
score = metric_fn(bf_a, bf_b)
if score > best_score:
best_score = score
best_id_b = id_b
if best_score >= threshold and best_id_b is not None:
matches.append((id_a, best_id_b, best_score))
return matches
Secure Hash Matching
For exact matching scenarios where approximate matching is not needed.
"""
Secure hash-based record linkage for exact matching.
Uses keyed HMAC to prevent rainbow table attacks.
"""
import hashlib
import hmac
class SecureHashLinker:
"""
Link records across organizations using keyed hash matching.
Suitable for exact match on standardized identifiers.
"""
def __init__(self, shared_key: bytes):
self.shared_key = shared_key
def hash_identifier(self, *fields: str) -> str:
"""
Create a keyed hash of concatenated identifier fields.
Args:
fields: Identifier fields in standardized order
e.g., ("john", "smith", "19900115")
Returns:
Hex-encoded HMAC-SHA256 hash
"""
# Normalize and concatenate fields
normalized = "|".join(f.strip().lower() for f in fields)
# Generate keyed hash
digest = hmac.new(
self.shared_key,
normalized.encode("utf-8"),
hashlib.sha256
).hexdigest()
return digest
def hash_dataset(
self,
records: list[dict],
id_field: str,
linkage_fields: list[str]
) -> dict[str, str]:
"""
Hash all records in a dataset for linkage.
Returns mapping of hash -> record_id.
"""
hash_map = {}
for record in records:
fields = [str(record.get(f, "")) for f in linkage_fields]
record_hash = self.hash_identifier(*fields)
hash_map[record_hash] = record[id_field]
return hash_map
@staticmethod
def find_exact_matches(
hashes_a: dict[str, str],
hashes_b: dict[str, str]
) -> list[tuple[str, str]]:
"""
Find exact matches between two hash maps.
Returns list of (id_a, id_b) pairs.
"""
common_hashes = set(hashes_a.keys()) & set(hashes_b.keys())
return [(hashes_a[h], hashes_b[h]) for h in common_hashes]
Threshold Tuning
Methodology
| Threshold Range | Precision | Recall | Use Case |
|---|---|---|---|
| 0.90 - 1.00 | Very High | Low | High-stakes decisions (medical records) |
| 0.80 - 0.90 | High | Medium | Standard record linkage |
| 0.70 - 0.80 | Medium | High | Exploratory analysis, broad matching |
| 0.60 - 0.70 | Low | Very High | Candidate generation (with manual review) |
Optimal Threshold Selection Process
- Generate labeled pairs: Create a sample of known matches and non-matches
- Compute similarity scores: Calculate Dice/Jaccard for all pairs in the sample
- Plot precision-recall curve: Sweep threshold from 0 to 1
- Select threshold: Choose based on acceptable false positive rate for the use case
- Validate: Test on held-out labeled data
False Positive Management
"""
Post-linkage false positive reduction through multi-stage verification.
"""
class FalsePositiveManager:
"""
Reduce false positive matches through additional verification stages
without revealing raw data between parties.
"""
def __init__(self, primary_threshold: float = 0.8, verification_threshold: float = 0.9):
self.primary_threshold = primary_threshold
self.verification_threshold = verification_threshold
def multi_field_verification(
self,
candidate_pairs: list[tuple[str, str, float]],
secondary_encodings_a: dict[str, dict[str, np.ndarray]],
secondary_encodings_b: dict[str, dict[str, np.ndarray]],
matcher: BloomFilterMatcher
) -> list[tuple[str, str, float, bool]]:
"""
Verify candidate matches using additional encoded fields.
Args:
candidate_pairs: (id_a, id_b, primary_score) from initial matching
secondary_encodings_a: {record_id: {field: bloom_filter}} from org A
secondary_encodings_b: {record_id: {field: bloom_filter}} from org B
Returns:
(id_a, id_b, composite_score, verified) for each candidate
"""
verified_pairs = []
for id_a, id_b, primary_score in candidate_pairs:
secondary_scores = []
fields_a = secondary_encodings_a.get(id_a, {})
fields_b = secondary_encodings_b.get(id_b, {})
common_fields = set(fields_a.keys()) & set(fields_b.keys())
for field_name in common_fields:
score = matcher.dice_coefficient(
fields_a[field_name],
fields_b[field_name]
)
secondary_scores.append(score)
if secondary_scores:
avg_secondary = sum(secondary_scores) / len(secondary_scores)
composite = 0.6 * primary_score + 0.4 * avg_secondary
verified = composite >= self.verification_threshold
else:
composite = primary_score
verified = primary_score >= self.verification_threshold
verified_pairs.append((id_a, id_b, composite, verified))
return verified_pairs
Security Considerations
| Attack | Description | Mitigation |
|---|---|---|
| Frequency analysis | Analyzing bit patterns to infer common values | Use composite Bloom filters (CLK), add noise bits |
| Dictionary attack | Pre-computing Bloom filters for known values | Use strong shared secret keys, rotate keys periodically |
| Bit pattern cryptanalysis | Exploiting structure in Bloom filter bit patterns | Sufficient filter size (>= 1024), adequate hash functions (>= 20) |
| Collision exploitation | Deliberately crafting records to match target hashes | HMAC-based hashing, input validation |
References
- Schnell, R., Bachteler, T., and Reiher, J. "Privacy-Preserving Record Linkage Using Bloom Filters." BMC Medical Informatics and Decision Making, 9(1):41, 2009.
- Vatsalan, D., Christen, P., and Verykios, V.S. "A Taxonomy of Privacy-Preserving Record Linkage Techniques." Information Systems, 38(6):946-969, 2013.
- Randall, S.M. et al. "Privacy-Preserving Record Linkage on Large Real World Datasets." Journal of Biomedical Informatics, 50:205-212, 2014.
- AIHW (Australian Institute of Health and Welfare) PPRL Implementation Guide
- Christen, P. "Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection." Springer, 2012.
Purpose-Based Access Control (PBAC) Architecture
Overview
Purpose-Based Access Control (PBAC) extends traditional access control models (RBAC, ABAC) by adding purpose as a mandatory dimension in every access decision. Under PBAC, data access is granted only when the requester can demonstrate a valid, pre-authorized purpose that aligns with the basis under which the data was collected. This directly implements GDPR Article 5(1)(b) purpose limitation, CCPA purpose restrictions, and similar requirements across global privacy regulations.
PBAC vs Traditional Access Control
| Dimension | RBAC | ABAC | PBAC |
|---|---|---|---|
| Access decision based on | Role membership | Attributes (user, resource, environment) | Purpose + attributes |
| Answers the question | "Who can access?" | "Under what conditions?" | "Why is this access needed?" |
| Purpose enforcement | None (implicit) | Possible as attribute | Core requirement |
| Audit trail | Who accessed what | Who, what, when, where | Who, what, when, where, WHY |
| Privacy alignment | Low | Medium | High |
| Consent integration | None | Possible | Native |
Purpose Ontology
Purpose Hierarchy
Root Purpose
├── Service Delivery
│ ├── Order Fulfillment
│ │ ├── Payment Processing
│ │ ├── Shipping Logistics
│ │ └── Order Confirmation
│ ├── Account Management
│ │ ├── Account Creation
│ │ ├── Account Maintenance
│ │ └── Account Recovery
│ └── Customer Support
│ ├── Ticket Resolution
│ └── Escalation Handling
├── Legal Compliance
│ ├── Regulatory Reporting
│ │ ├── Tax Reporting
│ │ ├── AML Compliance
│ │ └── Regulatory Audit
│ ├── Data Subject Rights
│ │ ├── Access Request
│ │ ├── Deletion Request
│ │ ├── Portability Request
│ │ └── Rectification Request
│ └── Litigation Hold
├── Analytics
│ ├── Product Analytics
│ │ ├── Feature Usage Analysis
│ │ └── UX Research
│ ├── Business Intelligence
│ │ ├── Revenue Reporting
│ │ └── Forecasting
│ └── Aggregate Reporting
│ ├── Board Reporting
│ └── Investor Reporting
├── Marketing
│ ├── Direct Marketing
│ │ ├── Email Campaigns
│ │ └── Personalized Offers
│ ├── Market Research
│ │ └── Survey Analysis
│ └── Advertising
│ ├── Audience Segmentation
│ └── Campaign Measurement
└── Security
├── Fraud Detection
├── Incident Investigation
└── Access Auditing
Purpose Definition Schema
from dataclasses import dataclass, field
from enum import Enum
from datetime import datetime
class LegalBasis(Enum):
CONSENT = "consent"
CONTRACT = "contract"
LEGAL_OBLIGATION = "legal_obligation"
VITAL_INTEREST = "vital_interest"
PUBLIC_INTEREST = "public_interest"
LEGITIMATE_INTEREST = "legitimate_interest"
class PurposeStatus(Enum):
ACTIVE = "active"
DEPRECATED = "deprecated"
PENDING_APPROVAL = "pending_approval"
@dataclass
class Purpose:
purpose_id: str
name: str
description: str
parent_purpose_id: str | None
legal_bases: list[LegalBasis]
data_categories_allowed: list[str]
retention_period_days: int
requires_consent: bool
status: PurposeStatus
owner: str
created_date: datetime
review_date: datetime
jurisdictions: list[str] = field(default_factory=lambda: ["global"])
compatible_purposes: list[str] = field(default_factory=list)
incompatible_purposes: list[str] = field(default_factory=list)
Policy Engine Architecture
System Design
Access Request
|
v
+-------------------+
| Request Parser |
| - Identity (who) |
| - Resource (what) |
| - Action (how) |
| - Purpose (why) |
| - Context (when/ |
| where) |
+-------------------+
|
v
+-------------------+ +--------------------+
| Purpose Validator |---->| Purpose Registry |
| - Purpose exists | | - Ontology tree |
| - Purpose active | | - Legal bases |
| - Purpose applies | | - Data categories |
| to data category| +--------------------+
+-------------------+
|
v
+-------------------+ +--------------------+
| Consent Checker |---->| Consent Store |
| - Consent exists | | - Data subject |
| - Consent active | | preferences |
| - Scope matches | | - Consent records |
+-------------------+ +--------------------+
|
v
+-------------------+ +--------------------+
| ABAC Policy |---->| Policy Repository |
| Evaluator | | - XACML policies |
| - Role check | | - Purpose-data |
| - Attribute check | | mappings |
| - Environment check| +--------------------+
+-------------------+
|
v
+-------------------+
| Decision Engine |
| - PERMIT / DENY |
| - Obligations |
| - Advice |
+-------------------+
|
+------+-------+
| |
v v
PERMIT DENY
+ Audit Log + Audit Log
+ Obligations + Denial Reason
Policy Engine Implementation
"""
Purpose-Based Access Control policy engine.
Evaluates access requests against purpose definitions,
consent records, and attribute-based policies.
"""
from dataclasses import dataclass
from datetime import datetime
from enum import Enum
from typing import Optional
class Decision(Enum):
PERMIT = "PERMIT"
DENY = "DENY"
@dataclass
class AccessRequest:
requester_id: str
requester_roles: list[str]
resource_id: str
data_category: str
action: str # read, write, delete, export
purpose_id: str
data_subject_id: Optional[str]
timestamp: datetime
source_ip: str
justification: str
@dataclass
class AccessDecision:
decision: Decision
reason: str
obligations: list[str]
request: AccessRequest
evaluated_at: datetime
policy_version: str
class PurposeBasedAccessController:
"""
Core PBAC engine that evaluates access requests against
purpose definitions, consent records, and organizational policies.
"""
def __init__(
self,
purpose_registry,
consent_store,
policy_repository,
audit_logger
):
self.purpose_registry = purpose_registry
self.consent_store = consent_store
self.policy_repository = policy_repository
self.audit_logger = audit_logger
def evaluate(self, request: AccessRequest) -> AccessDecision:
"""
Evaluate an access request through the PBAC pipeline.
Pipeline stages:
1. Purpose validation
2. Consent verification
3. Attribute-based policy evaluation
4. Decision with obligations
"""
now = datetime.utcnow()
# Stage 1: Validate purpose
purpose = self.purpose_registry.get_purpose(request.purpose_id)
if purpose is None:
return self._deny(request, "Purpose not found in registry", now)
if purpose.status != PurposeStatus.ACTIVE:
return self._deny(request, f"Purpose is {purpose.status.value}", now)
if request.data_category not in purpose.data_categories_allowed:
return self._deny(
request,
f"Data category '{request.data_category}' not allowed for purpose '{purpose.name}'",
now
)
# Stage 2: Verify consent (if required)
if purpose.requires_consent and request.data_subject_id:
consent = self.consent_store.get_active_consent(
data_subject_id=request.data_subject_id,
purpose_id=request.purpose_id
)
if consent is None:
return self._deny(
request,
f"No active consent for purpose '{purpose.name}' from data subject",
now
)
# Stage 3: Evaluate attribute-based policies
policy_result = self.policy_repository.evaluate(
roles=request.requester_roles,
resource=request.resource_id,
action=request.action,
purpose=request.purpose_id,
data_category=request.data_category,
context={"source_ip": request.source_ip, "time": now}
)
if not policy_result.permitted:
return self._deny(request, policy_result.denial_reason, now)
# Stage 4: Permit with obligations
obligations = self._determine_obligations(purpose, request)
decision = AccessDecision(
decision=Decision.PERMIT,
reason=f"Access permitted for purpose '{purpose.name}'",
obligations=obligations,
request=request,
evaluated_at=now,
policy_version=self.policy_repository.version
)
self.audit_logger.log_access_decision(decision)
return decision
def _deny(
self, request: AccessRequest, reason: str, timestamp: datetime
) -> AccessDecision:
"""Create a DENY decision and log it."""
decision = AccessDecision(
decision=Decision.DENY,
reason=reason,
obligations=[],
request=request,
evaluated_at=timestamp,
policy_version=self.policy_repository.version
)
self.audit_logger.log_access_decision(decision)
return decision
def _determine_obligations(self, purpose, request: AccessRequest) -> list[str]:
"""Determine post-access obligations based on purpose and context."""
obligations = []
if request.action == "export":
obligations.append("LOG_DATA_EXPORT")
obligations.append("APPLY_ENCRYPTION_TO_EXPORT")
if request.action == "read" and request.data_category in [
"health_data", "financial_data", "biometric_data"
]:
obligations.append("MASK_SENSITIVE_FIELDS")
obligations.append(f"ENFORCE_RETENTION_{purpose.retention_period_days}_DAYS")
obligations.append("LOG_PURPOSE_USAGE")
return obligations
Purpose Verification at Query Time
SQL Proxy with Purpose Enforcement
"""
SQL proxy that intercepts database queries and enforces
purpose-based access control at the query level.
"""
import re
from datetime import datetime
class PurposeAwareSQLProxy:
"""
Intercepts SQL queries, validates purpose, and rewrites
queries to enforce data category restrictions.
"""
def __init__(self, pbac_controller, column_category_map: dict):
self.pbac = pbac_controller
self.column_category_map = column_category_map
def execute_query(
self,
sql: str,
requester_id: str,
requester_roles: list[str],
purpose_id: str,
justification: str
) -> tuple[bool, object]:
"""
Execute a SQL query with purpose verification.
Returns (success, result_or_error).
"""
# Parse referenced columns/tables from SQL
referenced_columns = self._extract_columns(sql)
referenced_categories = set()
for col in referenced_columns:
category = self.column_category_map.get(col, "general")
referenced_categories.add(category)
# Validate access for each data category
denied_categories = []
for category in referenced_categories:
request = AccessRequest(
requester_id=requester_id,
requester_roles=requester_roles,
resource_id="database",
data_category=category,
action="read",
purpose_id=purpose_id,
data_subject_id=None,
timestamp=datetime.utcnow(),
source_ip="internal",
justification=justification
)
decision = self.pbac.evaluate(request)
if decision.decision == Decision.DENY:
denied_categories.append((category, decision.reason))
if denied_categories:
reasons = "; ".join(
f"{cat}: {reason}" for cat, reason in denied_categories
)
return (False, f"Access denied for categories: {reasons}")
# Rewrite query to mask unauthorized columns if needed
rewritten_sql = self._apply_column_masking(sql, purpose_id)
# Execute the rewritten query (actual DB execution would go here)
return (True, rewritten_sql)
def _extract_columns(self, sql: str) -> list[str]:
"""Extract column references from SQL (simplified parser)."""
select_pattern = re.compile(
r"SELECT\s+(.+?)\s+FROM", re.IGNORECASE | re.DOTALL
)
match = select_pattern.search(sql)
if not match:
return []
columns_str = match.group(1)
if columns_str.strip() == "*":
return list(self.column_category_map.keys())
columns = [
col.strip().split(".")[-1].split(" ")[0]
for col in columns_str.split(",")
]
return columns
def _apply_column_masking(self, sql: str, purpose_id: str) -> str:
"""Rewrite query to mask columns not needed for the given purpose."""
return sql # Production implementation would rewrite SELECT columns
Audit Logging for PBAC
Audit Record Schema
| Field | Type | Description |
|---|---|---|
| audit_id | UUID | Unique audit record identifier |
| timestamp | DateTime | When the access decision was made |
| requester_id | String | Identity of the requester |
| requester_roles | Array[String] | Roles held by requester |
| resource_id | String | Resource being accessed |
| data_category | String | Category of data being accessed |
| action | String | Type of access (read/write/delete/export) |
| purpose_id | String | Stated purpose for access |
| purpose_name | String | Human-readable purpose name |
| decision | Enum | PERMIT or DENY |
| denial_reason | String | Reason for denial (if denied) |
| obligations | Array[String] | Post-access obligations applied |
| justification | String | Free-text justification from requester |
| policy_version | String | Version of policy used for decision |
Integration with Existing IAM
PBAC Overlay Architecture
Existing IAM (RBAC/ABAC)
|
v
+---------------------------+
| PBAC Middleware Layer |
| (intercepts access calls) |
+---------------------------+
|
+-- Purpose verification
+-- Consent validation
+-- Data category mapping
+-- Obligation enforcement
+-- Enhanced audit logging
|
v
Resource (Database / API / File System)
References
- Byun, J. and Li, N. "Purpose Based Access Control of Complex Data for Privacy Protection." ACM SACMAT, 2008.
- GDPR Article 5(1)(b) — Purpose Limitation Principle
- ISO/IEC 29101:2018 — Privacy Architecture Framework
- NIST SP 800-162 — Guide to Attribute Based Access Control Definition and Considerations
- Ni, Q. et al. "Privacy-Aware Role-Based Access Control." ACM TOPS, 2010.
Sponsors
Details
- Marketplace
- mukul975/Privacy-Data-Protection-Skills
- Source
- plugins/privacy-engineering-skills
- Author
- Mahipal
- Category
- Engineering
- License
- Apache-2.0
- Keywords
- privacy-engineering, differential-privacy, pii-detection, nist-privacy-framework
More from mukul975/Privacy-Data-Protection-Skills
All 21- privacy-impact-assessment-skillsv1.0.0
- privacy-skills-completev1.0.0
- records-of-processing-skillsv1.0.0
- us-state-privacy-skillsv1.0.0
- vendor-privacy-management-skillsv1.0.0
- ai-privacy-governance-skillsv1.0.0
- children-privacy-skillsv1.0.0
- consent-management-skillsv1.0.0
- cookie-consent-skillsv1.0.0
- cross-border-transfers-skillsv1.0.0
- data-breach-response-skillsv1.0.0
- data-classification-skillsv1.0.0
- data-retention-skillsv1.0.0
- data-subject-rights-skillsv1.0.0
- employee-privacy-skillsv1.0.0
- gdpr-compliance-skillsv1.0.0
- global-privacy-regulations-skillsv1.0.0
- healthcare-privacy-skillsv1.0.0
- privacy-audit-skillsv1.0.0
- privacy-by-design-skillsv1.0.0