mukul975 avatar

privacy-engineering-skills

14 privacy engineering skills: differential privacy, PII detection, NIST Privacy Framework, privacy APIs, data sharing, metrics

version
1.0.0
GitHub stars
340

Install

/plugin install privacy-engineering-skills@privacy-data-protection-skills

Run it inside Claude Code after adding the marketplace with /plugin marketplace add mukul975/Privacy-Data-Protection-Skills.

Contents

Discovered from source

Skills 14

Kantara Initiative Consent Receipt Specification

Overview

A consent receipt is a record of a consent transaction provided to the individual (data subject) as evidence that consent was given. The Kantara Initiative Consent Receipt Specification v1.1 defines a standard, machine-readable format that enables individuals to track and manage their consent across multiple organizations. This skill covers the implementation of consent receipts aligned with the Kantara specification and ISO/IEC 27560:2023.

Consent Receipt Structure

Core Fields (Kantara v1.1)

FieldTypeRequiredDescription
versionStringYesSpecification version (e.g., "KI-CR-v1.1.0")
jurisdictionStringYesLegal jurisdiction (ISO 3166-1 alpha-2)
consentTimestampDateTimeYesUTC timestamp of consent grant
collectionMethodStringYesHow consent was collected (web form, verbal, paper)
consentReceiptIDUUIDYesUnique identifier for this receipt
publicKeyStringNoPublic key for receipt verification
languageStringYesLanguage of the consent interaction (BCP 47)
piiPrincipalIdStringYesIdentifier for the data subject
piiControllersArrayYesList of data controllers
policyUrlURLYesLink to the privacy policy
servicesArrayYesServices for which consent is given
sensitiveBooleanYesWhether special category data is involved
spiCatArrayNoSpecial categories of data processed

PII Controller Object

FieldTypeRequiredDescription
piiControllerStringYesName of the controller organization
onBehalfBooleanNoWhether acting on behalf of another controller
contactStringYesContact information for the controller
addressObjectYesPhysical address of the controller
emailStringYesContact email address
phoneStringNoContact phone number
piiControllerUrlURLNoURL of the controller's website

Service Object

FieldTypeRequiredDescription
serviceStringYesName of the service
purposesArrayYesList of processing purposes

Purpose Object

FieldTypeRequiredDescription
purposeStringYesDescription of the processing purpose
purposeCategoryArrayYesCategory codes for the purpose
consentTypeStringYes"explicit" or "implicit"
piiCategoryArrayYesCategories of PII processed
primaryPurposeBooleanYesWhether this is the primary purpose
terminationStringYesHow consent can be withdrawn
thirdPartyDisclosureBooleanYesWhether data is shared with third parties
thirdPartyNameStringNoName of third party (if applicable)

Machine-Readable Receipt Format

JSON Consent Receipt

{
  "version": "KI-CR-v1.1.0",
  "jurisdiction": "EU",
  "consentTimestamp": "2026-03-14T10:30:00.000Z",
  "collectionMethod": "web_form",
  "consentReceiptID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "language": "en",
  "piiPrincipalId": "user-98765",
  "piiControllers": [
    {
      "piiController": "Cipher Engineering Labs",
      "onBehalf": false,
      "contact": "Data Protection Officer",
      "address": {
        "streetAddress": "100 Technology Drive",
        "locality": "London",
        "region": "Greater London",
        "postalCode": "EC2A 1NT",
        "country": "GB"
      },
      "email": "dpo@cipherengineeringlabs.com",
      "phone": "+44-20-7946-0958",
      "piiControllerUrl": "https://www.cipherengineeringlabs.com"
    }
  ],
  "policyUrl": "https://www.cipherengineeringlabs.com/privacy-policy",
  "services": [
    {
      "service": "Privacy Analytics Platform",
      "purposes": [
        {
          "purpose": "Provide personalized privacy compliance recommendations",
          "purposeCategory": ["core_service"],
          "consentType": "explicit",
          "piiCategory": ["contact_information", "professional_information"],
          "primaryPurpose": true,
          "termination": "Account settings > Privacy > Withdraw consent",
          "thirdPartyDisclosure": false
        },
        {
          "purpose": "Send product updates and feature announcements",
          "purposeCategory": ["marketing"],
          "consentType": "explicit",
          "piiCategory": ["contact_information"],
          "primaryPurpose": false,
          "termination": "Unsubscribe link in email or Account settings",
          "thirdPartyDisclosure": false
        },
        {
          "purpose": "Aggregate usage analytics to improve platform features",
          "purposeCategory": ["analytics"],
          "consentType": "explicit",
          "piiCategory": ["usage_data", "device_information"],
          "primaryPurpose": false,
          "termination": "Account settings > Privacy > Analytics opt-out",
          "thirdPartyDisclosure": true,
          "thirdPartyName": "Cipher Analytics Processing Ltd"
        }
      ]
    }
  ],
  "sensitive": false,
  "spiCat": []
}

JWT-Based Verification Mechanism

Signed Consent Receipt Implementation

"""
Consent receipt generation and verification using JWT (JSON Web Tokens).
Implements Kantara Initiative Consent Receipt Specification v1.1
with cryptographic verification.
"""

import json
import uuid
from datetime import datetime, timezone
from typing import Optional
import jwt  # PyJWT
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.backends import default_backend


class ConsentReceiptIssuer:
    """
    Issue and sign consent receipts as JWTs.
    Uses RS256 (RSA with SHA-256) for signing.
    """

    def __init__(self, private_key_pem: bytes, issuer_name: str):
        """
        Args:
            private_key_pem: PEM-encoded RSA private key
            issuer_name: Name of the issuing organization
        """
        self.private_key = serialization.load_pem_private_key(
            private_key_pem, password=None, backend=default_backend()
        )
        self.public_key = self.private_key.public_key()
        self.issuer_name = issuer_name

    def issue_receipt(
        self,
        principal_id: str,
        services: list[dict],
        jurisdiction: str,
        collection_method: str,
        policy_url: str,
        sensitive: bool = False,
        language: str = "en",
        controller_info: dict = None
    ) -> tuple[str, str]:
        """
        Generate a signed consent receipt.

        Args:
            principal_id: Data subject identifier
            services: List of service/purpose objects
            jurisdiction: Legal jurisdiction code
            collection_method: How consent was collected
            policy_url: URL of the privacy policy
            sensitive: Whether special categories are involved
            language: Language of consent interaction
            controller_info: PII controller details

        Returns:
            Tuple of (receipt_id, signed_jwt_string)
        """
        receipt_id = str(uuid.uuid4())

        if controller_info is None:
            controller_info = {
                "piiController": self.issuer_name,
                "onBehalf": False,
                "contact": "Data Protection Officer",
                "email": f"dpo@{self.issuer_name.lower().replace(' ', '')}.com"
            }

        receipt_payload = {
            "version": "KI-CR-v1.1.0",
            "jurisdiction": jurisdiction,
            "consentTimestamp": datetime.now(timezone.utc).isoformat(),
            "collectionMethod": collection_method,
            "consentReceiptID": receipt_id,
            "language": language,
            "piiPrincipalId": principal_id,
            "piiControllers": [controller_info],
            "policyUrl": policy_url,
            "services": services,
            "sensitive": sensitive,
            "spiCat": [],
            # JWT standard claims
            "iss": self.issuer_name,
            "sub": principal_id,
            "iat": int(datetime.now(timezone.utc).timestamp()),
            "jti": receipt_id,
        }

        # Sign the receipt as a JWT
        signed_token = jwt.encode(
            receipt_payload,
            self.private_key,
            algorithm="RS256",
            headers={"kid": f"{self.issuer_name}-consent-key-1"}
        )

        return receipt_id, signed_token

    def get_public_key_pem(self) -> str:
        """Export the public key for verification by data subjects."""
        return self.public_key.public_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PublicFormat.SubjectPublicKeyInfo
        ).decode("utf-8")


class ConsentReceiptVerifier:
    """
    Verify signed consent receipts.
    Used by data subjects or auditors to confirm receipt authenticity.
    """

    def __init__(self):
        self.trusted_keys: dict[str, bytes] = {}

    def register_issuer_key(self, issuer_name: str, public_key_pem: str):
        """Register a trusted issuer's public key."""
        self.trusted_keys[issuer_name] = public_key_pem.encode("utf-8")

    def verify_receipt(self, token: str) -> tuple[bool, Optional[dict], Optional[str]]:
        """
        Verify a signed consent receipt JWT.

        Returns:
            Tuple of (is_valid, decoded_payload_or_None, error_message_or_None)
        """
        try:
            # Decode without verification first to get the issuer
            unverified = jwt.decode(token, options={"verify_signature": False})
            issuer = unverified.get("iss")

            if issuer not in self.trusted_keys:
                return (False, None, f"Unknown issuer: {issuer}")

            public_key = serialization.load_pem_public_key(
                self.trusted_keys[issuer],
                backend=default_backend()
            )

            # Verify the signature
            decoded = jwt.decode(
                token,
                public_key,
                algorithms=["RS256"],
                issuer=issuer
            )

            # Validate required Kantara fields
            required_fields = [
                "version", "jurisdiction", "consentTimestamp",
                "collectionMethod", "consentReceiptID", "piiPrincipalId",
                "piiControllers", "policyUrl", "services", "sensitive"
            ]

            missing = [f for f in required_fields if f not in decoded]
            if missing:
                return (False, decoded, f"Missing required fields: {missing}")

            return (True, decoded, None)

        except jwt.ExpiredSignatureError:
            return (False, None, "Receipt JWT has expired")
        except jwt.InvalidSignatureError:
            return (False, None, "Invalid signature — receipt may have been tampered with")
        except jwt.DecodeError as e:
            return (False, None, f"Failed to decode JWT: {str(e)}")

Receipt Lifecycle Management

Lifecycle States

ISSUED --> ACTIVE --> WITHDRAWN
  |          |           |
  |          v           v
  |       UPDATED    ARCHIVED
  |          |
  v          v
EXPIRED   ACTIVE (new version)

Lifecycle Manager

"""
Manage the lifecycle of consent receipts including
updates, withdrawals, and expiration tracking.
"""

from dataclasses import dataclass
from datetime import datetime, timezone
from enum import Enum


class ReceiptStatus(Enum):
    ISSUED = "issued"
    ACTIVE = "active"
    UPDATED = "updated"
    WITHDRAWN = "withdrawn"
    EXPIRED = "expired"
    ARCHIVED = "archived"


@dataclass
class ReceiptRecord:
    receipt_id: str
    principal_id: str
    status: ReceiptStatus
    issued_at: datetime
    last_updated: datetime
    withdrawn_at: datetime | None
    jwt_token: str
    version: int
    superseded_by: str | None
    purposes: list[str]


class ConsentReceiptLifecycle:
    """
    Manage consent receipt state transitions and history.
    """

    def __init__(self, receipt_store, issuer: ConsentReceiptIssuer):
        self.store = receipt_store
        self.issuer = issuer

    def activate_receipt(self, receipt_id: str) -> bool:
        """Transition a receipt from ISSUED to ACTIVE."""
        record = self.store.get(receipt_id)
        if record and record.status == ReceiptStatus.ISSUED:
            record.status = ReceiptStatus.ACTIVE
            record.last_updated = datetime.now(timezone.utc)
            self.store.update(record)
            return True
        return False

    def update_receipt(
        self,
        receipt_id: str,
        updated_services: list[dict],
        jurisdiction: str,
        policy_url: str
    ) -> str | None:
        """
        Update an existing receipt by issuing a new version.
        The old receipt is marked as UPDATED and linked to the new one.

        Returns the new receipt ID or None if update failed.
        """
        old_record = self.store.get(receipt_id)
        if not old_record or old_record.status not in [
            ReceiptStatus.ACTIVE, ReceiptStatus.ISSUED
        ]:
            return None

        # Issue new receipt
        new_receipt_id, new_jwt = self.issuer.issue_receipt(
            principal_id=old_record.principal_id,
            services=updated_services,
            jurisdiction=jurisdiction,
            collection_method="consent_update",
            policy_url=policy_url
        )

        # Create new record
        new_record = ReceiptRecord(
            receipt_id=new_receipt_id,
            principal_id=old_record.principal_id,
            status=ReceiptStatus.ACTIVE,
            issued_at=datetime.now(timezone.utc),
            last_updated=datetime.now(timezone.utc),
            withdrawn_at=None,
            jwt_token=new_jwt,
            version=old_record.version + 1,
            superseded_by=None,
            purposes=[p["purpose"] for s in updated_services for p in s.get("purposes", [])]
        )
        self.store.save(new_record)

        # Mark old receipt as updated
        old_record.status = ReceiptStatus.UPDATED
        old_record.superseded_by = new_receipt_id
        old_record.last_updated = datetime.now(timezone.utc)
        self.store.update(old_record)

        return new_receipt_id

    def withdraw_consent(self, receipt_id: str, reason: str = "") -> bool:
        """
        Withdraw consent and mark the receipt accordingly.

        Returns True if withdrawal was successful.
        """
        record = self.store.get(receipt_id)
        if not record or record.status not in [
            ReceiptStatus.ACTIVE, ReceiptStatus.ISSUED
        ]:
            return False

        record.status = ReceiptStatus.WITHDRAWN
        record.withdrawn_at = datetime.now(timezone.utc)
        record.last_updated = datetime.now(timezone.utc)
        self.store.update(record)

        return True

    def get_active_receipts(self, principal_id: str) -> list[ReceiptRecord]:
        """Get all active consent receipts for a data subject."""
        return self.store.find_by_principal(
            principal_id, status=ReceiptStatus.ACTIVE
        )

    def get_receipt_history(self, principal_id: str) -> list[ReceiptRecord]:
        """Get the complete consent receipt history for a data subject."""
        return self.store.find_by_principal(principal_id)

ISO/IEC 27560:2023 Alignment

Kantara CR FieldISO 27560 EquivalentNotes
consentReceiptIDConsent Record IdentifierUnique identifier for the record
consentTimestampDate and time of consentWhen consent was collected
piiPrincipalIdPII Principal IdentifierData subject identifier
piiControllersPII ControllerOrganization processing data
services.purposesPurpose(s)Processing purposes
services.purposes.piiCategoryPII CategoriesTypes of personal data
sensitiveSensitive PII indicatorSpecial category flag
jurisdictionJurisdictionApplicable legal framework
collectionMethodMechanism of consentHow consent was obtained
policyUrlPrivacy notice referenceLink to privacy notice

References

  • Kantara Initiative Consent Receipt Specification v1.1.0 (2018)
  • ISO/IEC 27560:2023 — Privacy Technologies — Consent Record Information Structure
  • ISO/IEC 29184:2020 — Guidelines for Online Privacy Notices and Consent
  • RFC 7519 — JSON Web Token (JWT)
  • W3C Data Privacy Vocabularies and Controls Community Group
  • IEEE P7012 — Standard for Machine Readable Personal Privacy Terms

Differential Privacy in Production

Overview

Differential privacy is a mathematical framework for quantifying and bounding the privacy loss incurred when publishing statistical information about a dataset. It provides a provable guarantee that the output of a computation does not significantly depend on whether any single individual's data is included. This skill covers the practical engineering of differential privacy systems for production deployment.

Core Definitions

(epsilon, delta)-Differential Privacy

A randomized mechanism M satisfies (epsilon, delta)-differential privacy if for all neighboring datasets D and D' (differing in at most one record) and for all possible outputs S:

P[M(D) in S] <= e^epsilon * P[M(D') in S] + delta

Where:

  • epsilon (privacy loss budget): Quantifies the maximum information leakage. Lower epsilon = stronger privacy.
  • delta: Probability of an additional privacy breach beyond the epsilon guarantee. Should be cryptographically small (< 1/n^2 where n is the dataset size).

Sensitivity

The sensitivity of a function f measures how much one individual can affect the output:

  • Global sensitivity: max over all neighboring D,D' of |f(D) - f(D')|
  • Local sensitivity: for a specific D, max over all neighboring D' of |f(D) - f(D')|

Epsilon Selection Strategy

Epsilon Guidelines by Use Case

Use CaseEpsilon RangeRationale
Census/government statistics0.1 - 1.0Maximum protection for mandatory participation
Healthcare analytics0.5 - 2.0High sensitivity, regulatory requirements
Location analytics1.0 - 3.0Moderate sensitivity, aggregate insights
Product analytics1.0 - 5.0Lower sensitivity, business utility needs
A/B testing2.0 - 8.0Statistical significance requirements
Aggregate reporting0.5 - 3.0Public-facing outputs need stronger guarantees

Factors Influencing Epsilon Choice

  1. Data sensitivity: More sensitive data requires lower epsilon
  2. Dataset size: Larger datasets tolerate lower epsilon with acceptable utility
  3. Query frequency: More queries consume more budget (see composition)
  4. Regulatory requirements: Some regulations imply specific privacy levels
  5. Utility requirements: Business needs may set a floor on acceptable accuracy
  6. Audience: Public release needs stronger guarantees than internal analytics

Noise Mechanisms

Laplace Mechanism

For numeric queries with bounded global sensitivity. Provides pure epsilon-differential privacy.

import numpy as np

def laplace_mechanism(true_value: float, sensitivity: float, epsilon: float) -> float:
    """
    Apply Laplace noise for epsilon-differential privacy.

    Args:
        true_value: The true query result
        sensitivity: Global sensitivity of the query (L1)
        epsilon: Privacy parameter

    Returns:
        Noisy result satisfying epsilon-differential privacy
    """
    scale = sensitivity / epsilon
    noise = np.random.laplace(loc=0, scale=scale)
    return true_value + noise


def laplace_mechanism_vector(true_values: np.ndarray, sensitivity: float, epsilon: float) -> np.ndarray:
    """Apply Laplace noise to a vector of values."""
    scale = sensitivity / epsilon
    noise = np.random.laplace(loc=0, scale=scale, size=true_values.shape)
    return true_values + noise

Gaussian Mechanism

For numeric queries. Provides (epsilon, delta)-differential privacy with tighter noise for high-dimensional outputs.

import numpy as np
import math

def gaussian_mechanism(true_value: float, sensitivity: float, epsilon: float, delta: float) -> float:
    """
    Apply Gaussian noise for (epsilon, delta)-differential privacy.

    Uses the analytic Gaussian mechanism calibration.

    Args:
        true_value: The true query result
        sensitivity: Global sensitivity of the query (L2)
        epsilon: Privacy parameter
        delta: Failure probability parameter

    Returns:
        Noisy result satisfying (epsilon, delta)-differential privacy
    """
    sigma = sensitivity * math.sqrt(2 * math.log(1.25 / delta)) / epsilon
    noise = np.random.normal(loc=0, scale=sigma)
    return true_value + noise

Exponential Mechanism

For non-numeric outputs (categorical selection) where adding noise directly is not meaningful.

import numpy as np

def exponential_mechanism(
    candidates: list,
    utility_scores: np.ndarray,
    sensitivity: float,
    epsilon: float
) -> object:
    """
    Select an output using the exponential mechanism.

    Args:
        candidates: List of possible outputs
        utility_scores: Utility score for each candidate
        sensitivity: Global sensitivity of the utility function
        epsilon: Privacy parameter

    Returns:
        Selected candidate satisfying epsilon-differential privacy
    """
    # Calculate selection probabilities
    probabilities = np.exp(epsilon * utility_scores / (2 * sensitivity))
    probabilities = probabilities / probabilities.sum()

    # Sample according to probabilities
    index = np.random.choice(len(candidates), p=probabilities)
    return candidates[index]

Randomized Response (Local DP)

For collecting individual data points with local differential privacy.

import random
import math

def randomized_response(true_bit: bool, epsilon: float) -> bool:
    """
    Apply randomized response for local differential privacy.

    Args:
        true_bit: The individual's true binary response
        epsilon: Privacy parameter

    Returns:
        Randomized response satisfying epsilon-local-DP
    """
    p = math.exp(epsilon) / (math.exp(epsilon) + 1)

    if random.random() < p:
        return true_bit  # Report truthfully
    else:
        return not true_bit  # Flip the answer


def estimate_from_randomized_responses(
    responses: list,
    epsilon: float
) -> float:
    """
    Estimate true proportion from randomized responses.

    Args:
        responses: List of randomized boolean responses
        epsilon: The epsilon used during collection

    Returns:
        Estimated true proportion
    """
    n = len(responses)
    observed_proportion = sum(responses) / n
    p = math.exp(epsilon) / (math.exp(epsilon) + 1)

    # Correct for randomization bias
    estimated_proportion = (observed_proportion - (1 - p)) / (2 * p - 1)
    return max(0.0, min(1.0, estimated_proportion))

Privacy Budget Tracking

Budget Manager Implementation

import threading
from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional
import math


@dataclass
class BudgetAllocation:
    query_id: str
    epsilon_spent: float
    delta_spent: float
    timestamp: datetime
    query_description: str
    analyst_id: str


class PrivacyBudgetManager:
    """
    Track and enforce differential privacy budget across queries.

    Supports both basic and advanced composition theorems.
    """

    def __init__(
        self,
        total_epsilon: float,
        total_delta: float,
        composition_method: str = "advanced"
    ):
        self.total_epsilon = total_epsilon
        self.total_delta = total_delta
        self.composition_method = composition_method
        self.allocations: list[BudgetAllocation] = []
        self._lock = threading.Lock()

    def remaining_budget(self) -> tuple[float, float]:
        """Calculate remaining (epsilon, delta) budget."""
        if self.composition_method == "basic":
            return self._basic_composition_remaining()
        elif self.composition_method == "advanced":
            return self._advanced_composition_remaining()
        elif self.composition_method == "rdp":
            return self._rdp_composition_remaining()
        else:
            raise ValueError(f"Unknown composition method: {self.composition_method}")

    def _basic_composition_remaining(self) -> tuple[float, float]:
        """Basic sequential composition: epsilons and deltas sum."""
        spent_epsilon = sum(a.epsilon_spent for a in self.allocations)
        spent_delta = sum(a.delta_spent for a in self.allocations)
        return (self.total_epsilon - spent_epsilon, self.total_delta - spent_delta)

    def _advanced_composition_remaining(self) -> tuple[float, float]:
        """
        Advanced composition theorem:
        k queries each with epsilon_i satisfy
        (sqrt(2k * ln(1/delta')) * max(epsilon_i) + k * epsilon_i * (e^epsilon_i - 1),
         k * delta_i + delta')-DP
        """
        k = len(self.allocations)
        if k == 0:
            return (self.total_epsilon, self.total_delta)

        epsilons = [a.epsilon_spent for a in self.allocations]
        deltas = [a.delta_spent for a in self.allocations]

        max_eps = max(epsilons)
        sum_delta = sum(deltas)

        # Reserve delta' for composition overhead
        delta_prime = (self.total_delta - sum_delta) / 2
        if delta_prime <= 0:
            return (0.0, 0.0)

        composed_epsilon = (
            math.sqrt(2 * k * math.log(1 / delta_prime)) * max_eps
            + k * max_eps * (math.exp(max_eps) - 1)
        )
        composed_delta = sum_delta + delta_prime

        return (
            max(0.0, self.total_epsilon - composed_epsilon),
            max(0.0, self.total_delta - composed_delta)
        )

    def _rdp_composition_remaining(self) -> tuple[float, float]:
        """Renyi DP composition (simplified)."""
        # RDP provides tighter bounds through Renyi divergence
        spent_epsilon = sum(a.epsilon_spent for a in self.allocations)
        spent_delta = sum(a.delta_spent for a in self.allocations)
        return (self.total_epsilon - spent_epsilon, self.total_delta - spent_delta)

    def request_budget(
        self,
        query_id: str,
        epsilon_requested: float,
        delta_requested: float,
        query_description: str,
        analyst_id: str
    ) -> bool:
        """
        Request budget allocation for a query.

        Returns True if budget is available and allocated, False otherwise.
        """
        with self._lock:
            remaining_eps, remaining_delta = self.remaining_budget()

            if epsilon_requested > remaining_eps or delta_requested > remaining_delta:
                return False

            allocation = BudgetAllocation(
                query_id=query_id,
                epsilon_spent=epsilon_requested,
                delta_spent=delta_requested,
                timestamp=datetime.utcnow(),
                query_description=query_description,
                analyst_id=analyst_id
            )
            self.allocations.append(allocation)
            return True

    def get_usage_report(self) -> dict:
        """Generate budget usage report."""
        remaining_eps, remaining_delta = self.remaining_budget()
        return {
            "total_epsilon": self.total_epsilon,
            "total_delta": self.total_delta,
            "remaining_epsilon": remaining_eps,
            "remaining_delta": remaining_delta,
            "utilization_pct": (1 - remaining_eps / self.total_epsilon) * 100,
            "num_queries": len(self.allocations),
            "composition_method": self.composition_method,
            "allocations": [
                {
                    "query_id": a.query_id,
                    "epsilon": a.epsilon_spent,
                    "delta": a.delta_spent,
                    "timestamp": a.timestamp.isoformat(),
                    "analyst": a.analyst_id,
                }
                for a in self.allocations
            ],
        }

Composition Theorems

Basic Composition

If M1 satisfies (e1, d1)-DP and M2 satisfies (e2, d2)-DP, then releasing both M1(D) and M2(D) satisfies (e1 + e2, d1 + d2)-DP.

Advanced Composition

For k mechanisms each satisfying (epsilon, delta)-DP, the composed mechanism satisfies (epsilon', k*delta + delta')-DP where:

epsilon' = sqrt(2k * ln(1/delta')) * epsilon + k * epsilon * (e^epsilon - 1)

Renyi Differential Privacy (RDP) Composition

RDP provides tighter composition bounds by tracking privacy loss through Renyi divergence of order alpha. Convert to (epsilon, delta)-DP at the end:

For alpha > 1: epsilon(delta) = RDP_alpha - ln(delta) / (alpha - 1)

Moments Accountant

Used by TensorFlow Privacy and Opacus. Tracks the log of the moment-generating function of the privacy loss variable. Provides the tightest known bounds for iterative mechanisms (e.g., DP-SGD).

Production Deployment Patterns

Pattern 1: Central DP Analytics Pipeline

Raw Data Store --> Sensitivity Calibration --> DP Mechanism --> Result Cache
                         |                         |
                         v                         v
                  Budget Manager <------------- Audit Log

Pattern 2: Local DP Collection

Client Device --> Local Randomizer --> Aggregation Server --> Estimator
                  (epsilon-LDP)              |                    |
                                             v                    v
                                       Budget Tracker      Utility Monitor

Pattern 3: DP Model Training (DP-SGD)

Training Data --> Mini-batch Sampling --> Per-example Gradient
                  (Poisson sampling)            |
                                                v
                                        Gradient Clipping (norm bound C)
                                                |
                                                v
                                        Gaussian Noise Addition (sigma * C)
                                                |
                                                v
                                        Model Update --> Privacy Accountant

Utility Measurement

MetricDescriptionFormula
Mean Absolute ErrorAverage absolute difference from true valueMAE = (1/n) * sum(
Relative ErrorError as fraction of true valueRE =
CoverageFraction of true values within confidence intervalCount(true in CI) / n
Utility RatioRatio of noisy to true signal-to-noise ratioSNR_noisy / SNR_true

References

  • Dwork, C. and Roth, A. "The Algorithmic Foundations of Differential Privacy." Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407, 2014.
  • Mironov, I. "Renyi Differential Privacy." IEEE CSF, 2017.
  • Abadi, M. et al. "Deep Learning with Differential Privacy." ACM CCS, 2016.
  • Google Differential Privacy Library: github.com/google/differential-privacy
  • OpenDP Project: opendp.org
  • Apple Differential Privacy Technical Overview (2017)
  • U.S. Census Bureau Disclosure Avoidance System (2020 Census)

LINDDUN Privacy Threat Modeling

Overview

LINDDUN is a systematic privacy threat modeling methodology developed by the DistriNet research group at KU Leuven. It provides a structured approach to identify and mitigate privacy threats in software systems. The acronym represents seven privacy threat categories that map to violations of privacy properties defined in ISO/IEC 29100.

The Seven LINDDUN Threat Categories

L — Linking

Definition: The ability to associate two or more data items or actions with an individual or group, beyond what is intended by the data subject.

Privacy Property Violated: Unlinkability

Threat Scenarios:

  • Cross-referencing anonymized datasets to re-identify individuals
  • Correlating browsing behavior across websites using fingerprinting
  • Linking social media profiles to real identities through metadata
  • Combining location data points to infer home/work addresses

Mitigation Strategies:

StrategyTechniqueImplementation
Data minimizationCollect only necessary attributesReview each data field against stated purpose
PseudonymizationReplace identifiers with tokensUse cryptographic pseudonymization with key separation
Mix networksObscure communication patternsRoute messages through anonymity networks
AggregationPresent only group-level dataEnforce minimum group size (k>=5) for any query result
Session unlinkabilityPrevent cross-session trackingRotate session tokens, avoid persistent identifiers

I — Identifying

Definition: The ability to identify a data subject from a set of data items, connecting them to a known individual.

Privacy Property Violated: Anonymity

Threat Scenarios:

  • Direct identification from unmasked PII in logs
  • Quasi-identifier attacks (combining age, ZIP code, gender)
  • Facial recognition from images in datasets
  • Voice identification from audio recordings
  • Writing style analysis (stylometry) in anonymous forums

Mitigation Strategies:

StrategyTechniqueImplementation
AnonymizationRemove direct identifiersStrip names, emails, SSNs before processing
k-AnonymityGeneralize quasi-identifiersEnsure each record shares attributes with k-1 others
Differential privacyAdd calibrated noiseApply epsilon-differential privacy to query responses
Data maskingObscure identifying fieldsReplace with realistic synthetic values
Access controlRestrict who can see raw dataImplement need-to-know access with purpose verification

N — Non-repudiation

Definition: The inability of a data subject to deny having performed an action, even when such denial would be desirable for privacy.

Privacy Property Violated: Plausible deniability

Threat Scenarios:

  • Immutable audit logs linking users to actions
  • Digital signatures proving authorship of documents
  • Non-repudiable transaction records
  • Email delivery confirmations and read receipts
  • Blockchain-based records that cannot be denied or deleted

Mitigation Strategies:

StrategyTechniqueImplementation
Deniable encryptionEnable plausible deniabilityUse deniable encryption schemes for sensitive data
Group signaturesHide individual identity in groupImplement group signature schemes for authenticated actions
Minimal loggingLog only what is legally requiredReview and minimize audit trail scope
Aggregate reportingReport actions at group levelAggregate activity reports rather than individual-level
Configurable receiptsLet users control acknowledgmentsAllow opt-out of read receipts and delivery confirmations

D — Detecting

Definition: The ability to determine whether a data subject has been involved in an action or is present in a dataset, even without identifying them specifically.

Privacy Property Violated: Undetectability

Threat Scenarios:

  • Traffic analysis revealing communication patterns
  • Database membership inference attacks
  • Timing attacks revealing user activity
  • Side-channel attacks revealing data access patterns
  • Presence detection through network metadata

Mitigation Strategies:

StrategyTechniqueImplementation
Traffic paddingMask communication patternsGenerate dummy traffic to obscure real patterns
SteganographyHide data within other dataEmbed sensitive communications in innocuous content
Constant-time operationsPrevent timing analysisImplement constant-time algorithms for sensitive operations
Oblivious RAMHide access patternsUse ORAM protocols for privacy-critical data access
Differential privacyProvide membership privacyApply differential privacy to prevent membership inference

D — Data Disclosure

Definition: Unauthorized exposure of personal data to parties who should not have access.

Privacy Property Violated: Confidentiality

Threat Scenarios:

  • SQL injection exposing database contents
  • Misconfigured cloud storage buckets
  • Insider threats with excessive access privileges
  • Man-in-the-middle attacks on unencrypted channels
  • API responses returning excessive data fields
  • Backup media loss or theft

Mitigation Strategies:

StrategyTechniqueImplementation
EncryptionProtect data at rest and in transitAES-256 at rest, TLS 1.3 in transit
Access controlEnforce least privilegeRBAC with regular access reviews
Input validationPrevent injection attacksParameterized queries, input sanitization
API field filteringReturn only requested fieldsImplement field-level access control in APIs
DLPDetect and prevent data exfiltrationDeploy DLP at network egress and endpoints

U — Unawareness

Definition: Data subjects being insufficiently aware of data processing activities, their rights, or the consequences of providing or withholding data.

Privacy Property Violated: Transparency, Intervenability

Threat Scenarios:

  • Hidden data collection through tracking pixels
  • Opaque algorithmic decision-making
  • Buried privacy policies with complex legal language
  • Undisclosed data sharing with third parties
  • Lack of notification when processing purposes change
  • No mechanism for data subjects to exercise rights

Mitigation Strategies:

StrategyTechniqueImplementation
Layered noticesProgressive disclosureShort notice + full policy + just-in-time
Privacy dashboardsCentralized visibilityUser-facing dashboard showing all data held
Consent managementGranular, informed consentPurpose-specific consent with clear descriptions
Explainable AIAlgorithmic transparencyProvide meaningful explanations of automated decisions
Right facilitationEasy rights exerciseSelf-service portal for access, correction, deletion

N — Non-compliance

Definition: Processing personal data in ways that violate applicable laws, regulations, standards, or organizational policies.

Privacy Property Violated: Policy and consent compliance

Threat Scenarios:

  • Processing without valid legal basis
  • Failing to honor data subject rights within deadlines
  • Cross-border transfers without adequate safeguards
  • Retaining data beyond the stated retention period
  • Processing children's data without parental consent
  • Failing to conduct required impact assessments

Mitigation Strategies:

StrategyTechniqueImplementation
Compliance mappingMap processing to legal basesDocument legal basis per processing activity per jurisdiction
Automated enforcementTechnical compliance controlsAutomated retention enforcement, consent verification
DPIA processImpact assessment for high-risk processingMandatory DPIA before deploying new high-risk processing
Regulatory monitoringTrack legal developmentsSubscribe to regulatory updates, conduct periodic gap analysis
Audit programVerify ongoing complianceAnnual compliance audits with corrective action tracking

DFD-Based Threat Analysis Process

Step 1: Create Privacy-Annotated Data Flow Diagram

[Data Subject] ---(personal data)--> [Web Application]
     ^                                      |
     |                                      v
  [Notice]                          [Application Server]
                                          |
                            +-------------+-------------+
                            |             |             |
                            v             v             v
                    [User Database] [Analytics DB] [Third-Party API]
                    (encrypted)     (pseudonymized) (data sharing)

Annotate each element with:

  • Data types flowing through (PII categories)
  • Trust boundaries crossed
  • Storage locations and durations
  • Processing purposes
  • Access controls in place

Step 2: Map Threats to DFD Elements

DFD ElementLIND(etect)D(isclose)UN(on-comply)
Data flowsXXXX
Data storesXXXX
ProcessesXXXXXXX
External entitiesXXXXX

Step 3: Build Threat Trees

For each applicable threat category per DFD element, construct a threat tree:

Identifying Threat to User Database
├── Direct identifier exposure
│   ├── SQL injection reveals raw PII
│   ├── Backup media contains unencrypted PII
│   └── Admin access to production database
├── Quasi-identifier attack
│   ├── Combination of age + ZIP + gender
│   └── Temporal correlation of records
└── Inference attack
    ├── Aggregate query with small group size
    └── Differential attack across query results

Step 4: Prioritize and Mitigate

Use a risk matrix to prioritize identified threats:

Likelihood / ImpactNegligibleLimitedSignificantMaximum
Very LikelyMediumHighCriticalCritical
LikelyLowMediumHighCritical
PossibleLowMediumMediumHigh
UnlikelyLowLowMediumMedium
RareLowLowLowMedium

LINDDUN and STRIDE Integration

LINDDUN CategoryRelated STRIDE CategoryOverlap Area
Data DisclosureInformation DisclosureBoth address unauthorized data exposure
Non-complianceTamperingIntegrity of consent records
DetectingInformation DisclosureMetadata leakage
IdentifyingInformation DisclosurePII exposure
Non-repudiationRepudiationOpposing perspectives on the same property

Practical Application at Cipher Engineering Labs

Threat Modeling Workshop Agenda (4 hours)

  1. System Overview (30 min): Present the system architecture and data flows
  2. DFD Construction (45 min): Collaboratively build privacy-annotated DFD
  3. Threat Identification (90 min): Walk through each LINDDUN category per DFD element
  4. Prioritization (30 min): Score threats on risk matrix
  5. Mitigation Planning (45 min): Identify controls for high and critical risks

Deliverables

  • Privacy-annotated Data Flow Diagram
  • LINDDUN Threat Register (all identified threats with risk scores)
  • Mitigation Plan (prioritized controls for high/critical threats)
  • Residual Risk Statement

References

  • Deng, M., Wuyts, K., Scandariato, R., Preneel, B., and Joosen, W. "A Privacy Threat Analysis Framework: Supporting the Elicitation and Fulfillment of Privacy Requirements." Requirements Engineering, 16(1):3-32, 2011.
  • Wuyts, K. and Joosen, W. "LINDDUN Privacy Threat Modeling: A Tutorial." CW Reports, KU Leuven, 2015.
  • LINDDUN GO — Lightweight approach: linddun.org
  • ISO/IEC 29100:2011 — Information Technology — Security Techniques — Privacy Framework
  • OWASP Privacy Risks Project

NIST Privacy Framework — COMMUNICATE Function

Overview

The COMMUNICATE function develops and implements appropriate activities to enable organizations and individuals to have a reliable understanding and engage in a dialogue about how data are processed and associated privacy risks. This function is essential for building trust and ensuring transparency with data subjects, regulators, and business partners.

COMMUNICATE Function Subcategories

CM.AW — Awareness

Raising awareness of data processing practices and privacy risks among stakeholders.

SubcategoryDescriptionImplementation Guidance
CM.AW-P1Mechanisms (e.g., notices, internal policies) for communicating data processing purposes, practices, and associated privacy risks are established and in servicePublish layered privacy notices. Maintain internal data processing documentation. Establish communication channels for privacy inquiries.
CM.AW-P2Mechanisms for obtaining feedback from individuals about data processing and associated privacy risks are established and in serviceDeploy feedback forms on privacy pages. Monitor privacy-related inquiries through support channels. Conduct periodic privacy perception surveys.
CM.AW-P3System/product/service design enables data processing visibilityImplement privacy dashboards for data subjects. Provide real-time data processing status. Enable granular consent visibility.
CM.AW-P4Records of data disclosures and sharing are maintained and can be accessed for review or transmissionMaintain disclosure registers. Enable data subjects to view sharing history. Automate disclosure logging.
CM.AW-P5Data corrections or deletions can be communicated to individuals or organizations that have received the dataImplement downstream notification workflows. Track data recipients for correction propagation. Automate deletion verification across partners.
CM.AW-P6Data provenance and lineage are maintained and can be accessed for review or transmissionDocument data origins and transformation history. Provide lineage visualization. Enable audit trail access for authorized parties.
CM.AW-P7Impacted individuals and organizations are notified about a privacy breach or eventDevelop breach notification templates. Establish notification timelines per jurisdiction. Maintain contact information for notification delivery.
CM.AW-P8Individuals are provided with mitigation mechanisms for observed privacy risksOffer identity protection services post-breach. Provide self-service privacy controls. Communicate available remediation options.

CM.PO — Communication Policies

Policies governing how privacy-related information is communicated.

SubcategoryDescriptionImplementation Guidance
CM.PO-P1Transparency policies, processes, and procedures for communicating data processing purposes, practices, associated privacy risks, and options for enabling data subjects' data processing preferences are established and in serviceDevelop layered transparency framework. Define communication standards for each audience.
CM.PO-P2Roles and responsibilities for communicating privacy-related information are establishedAssign communications ownership. Define approval workflows for external privacy communications.

Transparency Framework

Layered Privacy Notice Architecture

Layer 1: Short Notice (Banner/Summary)
├── Who we are
├── What data we collect
├── Key purposes
├── Core rights
└── Link to full notice

Layer 2: Full Privacy Notice
├── Detailed purposes and legal bases
├── Data categories and sources
├── Recipients and transfers
├── Retention periods
├── Complete rights information
├── Contact information
└── Links to supplementary notices

Layer 3: Supplementary Notices
├── Cookie Policy
├── Children's Privacy Notice
├── Employee Privacy Notice
├── Specific Service Notices
└── Jurisdiction-Specific Addenda

Layer 4: Just-in-Time Notices
├── Collection-point notifications
├── Consent requests with context
├── Purpose-specific explanations
└── Processing change notifications

Communication Channel Matrix

AudienceChannelFrequencyContent TypeOwner
Data subjects (general)Website privacy centerContinuousNotices, policies, FAQsPrivacy Team
Data subjects (customers)Email, in-app notificationsEvent-drivenConsent changes, updatesMarketing + Privacy
Data subjects (employees)Intranet, HR portalSemi-annual + eventEmployee privacy infoHR + Privacy
RegulatorsFormal correspondenceAs requiredCompliance reports, notificationsLegal + DPO
Business partnersContract addenda, portalAnnual + eventProcessing terms, audit rightsLegal + Procurement
Board/executivesDashboards, presentationsQuarterlyRisk reports, KPIsCPO
Internal teamsConfluence, trainingMonthlyGuidelines, best practicesPrivacy Champions

Privacy Notice Content Requirements

Mandatory Elements by Regulation

ElementGDPR Art. 13/14CCPA/CPRALGPDPIPEDA
Controller identityRequiredRequiredRequiredRequired
DPO contactRequiredN/ARequired (DPO equivalent)N/A
Processing purposesRequiredRequiredRequiredRequired
Legal basisRequiredN/ARequiredN/A
Data categoriesRequiredRequiredRequiredRequired
RecipientsRequiredRequiredRequiredRequired
International transfersRequiredN/ARequiredRequired
Retention periodsRequiredDisclosed on requestRequiredRequired
Data subject rightsRequiredRequiredRequiredRequired
Right to complainRequiredN/ARequiredRequired
Automated decision-makingRequiredCPRA: RequiredRequiredN/A
Data source (indirect collection)RequiredRequiredRequiredRequired
Sale/sharing opt-outN/ARequiredN/AN/A
Financial incentive termsN/ARequiredN/AN/A

Stakeholder Engagement Workflows

Data Subject Inquiry Handling

Inquiry Received
    |
    v
Classify Inquiry Type
├── General Privacy Question → FAQ/Knowledge Base → Respond within 5 business days
├── Data Subject Request → Route to DSR Workflow → Respond within regulatory deadline
├── Complaint → Escalate to DPO → Investigate → Respond within 15 business days
├── Breach Inquiry → Route to Incident Team → Coordinate response
└── Consent Change → Process immediately → Confirm within 24 hours

Privacy Communication Approval Workflow

Communication TypeDraftingReviewApprovalDistribution
Privacy policy updatePrivacy TeamLegalCPO + General CounselWeb team
Breach notificationIncident LeadLegal + DPOCEO + General CounselCommunications
Regulatory responseDPOLegalGeneral CounselDPO
Marketing consent requestMarketingPrivacy TeamCPOMarketing
Internal policy updatePrivacy TeamStakeholdersCPOInternal comms

Feedback Mechanisms

Privacy Perception Survey Template (Annual)

  1. How aware are you of our privacy practices? (1-5 scale)
  2. How easy is it to find information about how we use your data? (1-5 scale)
  3. How confident are you that we protect your personal data? (1-5 scale)
  4. Have you exercised any privacy rights in the past 12 months? (Yes/No, which ones)
  5. How satisfied were you with the response? (1-5 scale, if applicable)
  6. What privacy improvements would you like to see? (Open text)
  7. How likely are you to recommend us based on our privacy practices? (0-10 NPS)

Control Mapping

NIST PF COMMUNICATEISO 27701GDPR ArticleAPEC CBPR
CM.AW-P1A.7.3.2Art. 12, 13, 14Principle 5
CM.AW-P2A.7.3.8Art. 77Principle 9
CM.AW-P3A.7.2.5Art. 12(1)Principle 5
CM.AW-P4A.7.5.3Art. 19Principle 7
CM.AW-P7A.7.3.9Art. 34Principle 8
CM.PO-P1A.7.3.2Art. 12Principle 5
CM.PO-P25.3Art. 37-39Principle 9

References

  • NIST Privacy Framework Version 1.0 (January 16, 2020)
  • Article 29 Working Party WP260 — Guidelines on Transparency
  • EDPB Guidelines 04/2022 on Calculation of Administrative Fines
  • ISO/IEC 29100:2011 — Privacy Framework (Privacy Principles)

NIST Privacy Framework — CONTROL Function

Overview

The CONTROL function develops and implements appropriate activities to enable organizations and individuals to manage data with sufficient granularity to manage privacy risks. It addresses data management, data processing governance, and disassociated processing techniques.

CONTROL Function Subcategories

CT.DM — Data Management

Policies and procedures for managing data throughout its lifecycle.

SubcategoryDescriptionImplementation Guidance
CT.DM-P1Data elements can be accessed for reviewImplement data subject access portals. Provide export in machine-readable formats (JSON, CSV). Response within regulatory timelines.
CT.DM-P2Data elements can be accessed for transmission or disclosureSupport data portability requests. Implement secure transfer mechanisms. Maintain disclosure logs.
CT.DM-P3Data elements can be accessed for alterationEnable data correction workflows. Propagate corrections across all systems holding the data.
CT.DM-P4Data elements can be accessed for deletionImplement cascading deletion across primary and backup systems. Maintain deletion verification logs. Address data in analytics pipelines.
CT.DM-P5Data processing activities are configured to meet the organization's data processing principlesDefault to most privacy-protective settings. Implement purpose limitation at the system level.
CT.DM-P6Retention schedules are established and in placeDefine retention periods per data category. Implement automated purging. Document legal hold exceptions.
CT.DM-P7Data are processed in accordance with established retention schedulesAutomated enforcement of retention policies. Regular audits of data stores. Reporting on retention compliance.
CT.DM-P8Integrity of data elements is maintainedImplement checksums and validation rules. Detect and resolve data quality issues. Track data lineage.
CT.DM-P9Audit/log records are determined, documented, implemented, and reviewedPrivacy-specific audit logging for all data access and modifications. Log retention per regulatory requirements.
CT.DM-P10Mechanisms for data subjects to authorize processing existConsent collection and management infrastructure. Granular consent options. Consent withdrawal capability.

CT.DP — Data Processing Policies and Procedures

Governance controls for data processing activities.

SubcategoryDescriptionImplementation Guidance
CT.DP-P1Policies, processes, and procedures for authorizing data processing are maintained and enforcedDocument lawful basis for each processing activity. Implement access controls aligned with authorized purposes.
CT.DP-P2Data processing environment is identified, managed, and monitoredMaintain environment inventory (production, staging, development). Monitor for unauthorized processing. Separate test data from production data.
CT.DP-P3Data processing procedures are maintained and monitored to meet established termsStandard operating procedures for each processing type. Automated compliance checking. Exception management workflow.
CT.DP-P4System or device configurations are set to authorized specificationsSecure configuration baselines for systems processing personal data. Configuration drift detection. Change management for privacy-relevant settings.
CT.DP-P5Data processing activities are monitored and loggedReal-time monitoring of data processing operations. Anomaly detection for unusual processing patterns. Alerting for policy violations.

CT.PO — Disassociated Processing

Techniques to process data while minimizing identifiability.

SubcategoryDescriptionImplementation Guidance
CT.PO-P1De-identification is applied to data elements to limit observability or linkabilityApply pseudonymization, anonymization, or generalization based on use case. Document de-identification methodology.
CT.PO-P2Technical measures for disassociated processing are implementedDeploy differential privacy, secure multi-party computation, or homomorphic encryption as appropriate. Select technique based on data utility requirements.

Data Management Architecture

Lifecycle Control Points

Collection --> Storage --> Processing --> Sharing --> Retention --> Deletion
    |            |           |             |            |            |
    v            v           v             v            v            v
 Consent     Encryption   Purpose      Access       Schedule    Cascading
 Capture     at Rest      Limitation   Controls     Enforcement  Purge
    |            |           |             |            |            |
    v            v           v             v            v            v
 Audit Log   Access Log   Process Log  Transfer Log Retention Log Deletion Log

Data Element Access Control Matrix

OperationData SubjectData StewardProcessorAnalystAdmin
View own dataYesYesNoNoYes
Export own dataYesYesNoNoYes
Correct own dataRequestYesNoNoYes
Delete own dataRequestYesNoNoYes
View aggregateNoYesYesYesYes
Process dataNoNoYesNoYes
Analyze dataNoNoNoYesYes
Configure systemNoNoNoNoYes

Disassociated Processing Techniques

Technique Selection Guide

TechniqueIdentifiability ReductionData UtilityComputational CostReversible
PseudonymizationMediumHighLowYes (with key)
k-AnonymityMedium-HighMediumMediumNo
l-DiversityHighMediumMediumNo
t-ClosenessHighMedium-LowHighNo
Differential PrivacyVery HighMediumMediumNo
Data MaskingLow-MediumMediumLowDepends
GeneralizationMediumMediumLowNo
SuppressionHighLowLowNo
Synthetic DataVery HighVariableHighNo

De-identification Decision Framework

Is the data being shared externally?
├── Yes → Is re-identification risk acceptable?
│   ├── No → Apply formal anonymization (differential privacy, k-anonymity with k>=5)
│   └── Yes → Apply pseudonymization with contractual controls
└── No → Is the data used for analytics?
    ├── Yes → Apply aggregation or differential privacy
    └── No → Apply pseudonymization with access controls

Retention Schedule Template

Data CategoryRetention PeriodLegal BasisDeletion MethodVerification
Customer transaction records7 years from transactionTax/accounting regulationsSecure overwriteAutomated audit
Marketing consent recordsDuration of consent + 3 yearsLegitimate interest (proof)Logical deletionAnnual review
Employee personnel filesEmployment + 7 yearsEmployment lawSecure destructionHR verification
Website analytics26 monthsConsentAutomated purgeMonthly check
Support tickets3 years from resolutionContract performanceSecure overwriteQuarterly audit
CCTV footage30 daysLegitimate interestAutomated overwriteWeekly check
Backup tapes90 days rollingBusiness continuityTape degaussingQuarterly audit

Audit Logging Requirements

Privacy-Specific Events to Log

Event CategorySpecific EventsRequired Fields
Data AccessView, export, downloadUser ID, timestamp, data subject ID, data elements accessed, purpose
Data ModificationCreate, update, correctUser ID, timestamp, data subject ID, before/after values, purpose
Data DeletionDelete, purge, anonymizeUser ID, timestamp, data subject ID, deletion scope, verification
Consent ChangesGrant, withdraw, modifyData subject ID, timestamp, consent scope, collection channel
Data TransferInternal transfer, external disclosureUser ID, timestamp, recipient, data elements, legal basis
ConfigurationAccess control changes, retention policy changesAdmin ID, timestamp, before/after settings, approval reference

Control Mapping

NIST PF CONTROLISO 27701GDPR ArticleNIST 800-53
CT.DM-P1A.7.3.3Art. 15PM-25
CT.DM-P2A.7.3.7Art. 20PM-25
CT.DM-P3A.7.3.4Art. 16PM-25
CT.DM-P4A.7.3.5Art. 17PM-25
CT.DM-P6A.7.4.7Art. 5(1)(e)SI-12
CT.DM-P9A.7.2.8Art. 30AU-2
CT.DM-P10A.7.2.3Art. 6, 7IP-1
CT.DP-P1A.7.2.2Art. 6(1)PT-3
CT.DP-P5A.7.2.8Art. 30AU-6
CT.PO-P1A.7.4.5Art. 25(1)PM-24
CT.PO-P2A.7.4.5Art. 25(1)PM-24

References

  • NIST Privacy Framework Version 1.0 (January 16, 2020)
  • NIST SP 800-188 — De-Identifying Government Datasets
  • NIST IR 8053 — De-Identification of Personal Information
  • ISO/IEC 20889:2018 — Privacy Enhancing Data De-identification Techniques

NIST Privacy Framework — GOVERN Function

Overview

The GOVERN function establishes and monitors the organization's privacy governance structure, including policies, processes, and procedures for managing privacy risk. It is the foundational function that enables all other NIST Privacy Framework functions.

GOVERN Function Subcategories

GV.AT — Awareness and Training

Building organizational awareness and capability for privacy risk management.

SubcategoryDescriptionImplementation Guidance
GV.AT-P1The workforce is informed and trained on its roles and responsibilitiesDevelop role-based privacy training curricula. New hire training within 30 days, annual refresher for all staff.
GV.AT-P2Senior executives understand their roles and responsibilitiesExecutive briefings quarterly. Board-level privacy risk reporting semi-annually.
GV.AT-P3Privacy personnel understand their roles and responsibilitiesSpecialized training for DPOs, privacy engineers, and legal counsel. Professional certification support (CIPP, CIPM, CIPT).
GV.AT-P4Third parties (service providers, customers, partners) understand their roles and responsibilitiesInclude privacy requirements in contracts. Provide vendor privacy guidelines. Verify third-party training compliance.

GV.MT — Monitoring and Review

Ongoing assessment of privacy governance effectiveness.

SubcategoryDescriptionImplementation Guidance
GV.MT-P1Privacy risk is re-assessed on an ongoing basis and as key factors changeTrigger-based reassessment: new processing activities, regulatory changes, incidents, organizational changes.
GV.MT-P2Privacy risk management activities are periodically reviewedConduct quarterly operational reviews and annual strategic reviews. Benchmark against industry peers.
GV.MT-P3Privacy policies, processes, and procedures are maintained and used to manage privacy riskVersion control for all privacy documentation. Annual policy review cycle with defined approval workflow.
GV.MT-P4Privacy risk assessment results are shared with stakeholdersDistribute risk dashboards to executive leadership monthly. Share relevant findings with operational teams weekly.

GV.PO — Policy

Privacy policies governing organizational approach to privacy risk management.

SubcategoryDescriptionImplementation Guidance
GV.PO-P1Organizational privacy values and policies are established and communicatedPublish a privacy mission statement. Develop comprehensive privacy policy suite aligned with organizational values.
GV.PO-P2Processes to instill organizational privacy values within data processing activities are establishedEmbed privacy requirements into SDLC. Require privacy review at design stage for all new data processing.
GV.PO-P3Roles and responsibilities for privacy workforce are establishedDefine RACI matrices for privacy functions. Document authority levels and escalation paths.
GV.PO-P4Privacy is integrated into organizational risk managementInclude privacy in enterprise risk register. Align privacy risk appetite with organizational risk tolerance.
GV.PO-P5Legal, regulatory, and contractual requirements regarding privacy are understood and managedMaintain regulatory tracking register. Map obligations to controls. Conduct annual compliance assessments.
GV.PO-P6Governance and risk management policies address privacy risksPrivacy risk addressed in board risk committee charter. Executive-level accountability for privacy.

GV.RR — Roles and Responsibilities

Establishing accountability structures for privacy risk management.

SubcategoryDescriptionImplementation Guidance
GV.RR-P1Organizational leadership is responsible and accountable for privacy riskC-suite privacy accountability documented. Chief Privacy Officer reports to CEO or General Counsel.
GV.RR-P2Adequate resources for privacy programs are allocatedAnnual privacy budget planning. Headcount planning for privacy team. Technology investment for privacy tools.
GV.RR-P3Processes are in place to determine, assess, and manage privacy risksFormal privacy risk management methodology adopted. Standard operating procedures for risk assessment.

Governance Structure Template

Organizational Hierarchy

Board of Directors / Audit Committee
    |
Chief Executive Officer
    |
Chief Privacy Officer (CPO) / Data Protection Officer (DPO)
    |
    +-- Privacy Legal Counsel
    +-- Privacy Engineering Lead
    +-- Privacy Operations Manager
    +-- Privacy Compliance Analyst
    |
Business Unit Privacy Champions (distributed model)

RACI Matrix for Key Privacy Activities

ActivityCPO/DPOLegalEngineeringBusiness UnitsExecutive
Privacy strategyACCIR
Policy developmentRRCCA
DPIA executionACRRI
Incident responseRRCCI
Training deliveryACIRI
Regulatory monitoringCRIIA
Vendor assessmentARCRI
Metrics reportingRICCA

R = Responsible, A = Accountable, C = Consulted, I = Informed

Training Program Framework

Tier 1 — General Awareness (All Staff)

  • Frequency: Annual, plus new hire onboarding
  • Duration: 45 minutes
  • Topics: Privacy principles, data handling basics, incident reporting, individual rights
  • Assessment: Quiz with 80% pass threshold

Tier 2 — Role-Specific (Data Handlers)

  • Frequency: Semi-annual
  • Duration: 2 hours
  • Topics: Data classification, access controls, secure processing, retention schedules
  • Assessment: Scenario-based exercises

Tier 3 — Specialist (Privacy Team)

  • Frequency: Quarterly
  • Duration: 4 hours
  • Topics: Regulatory updates, advanced risk assessment, PET implementation, incident management
  • Assessment: Practical exercises and case studies

Tier 4 — Executive (Leadership)

  • Frequency: Semi-annual
  • Duration: 1 hour
  • Topics: Privacy risk landscape, regulatory exposure, strategic alignment, board reporting
  • Assessment: N/A (discussion-based)

Policy Suite Inventory

PolicyOwnerReview CycleApproval Authority
Enterprise Privacy PolicyCPOAnnualCEO
Data Classification PolicyCISO/CPOAnnualCTO
Data Retention PolicyCPOAnnualGeneral Counsel
Data Subject Rights PolicyCPOAnnualGeneral Counsel
Privacy Incident Response PolicyCPOSemi-AnnualCISO
Third-Party Data Processing PolicyCPOAnnualProcurement Lead
Cross-Border Transfer PolicyLegalAnnualGeneral Counsel
Cookie and Tracking PolicyCPOSemi-AnnualCMO
Employee Privacy PolicyCPO/HRAnnualCHRO
Privacy-by-Design StandardsPrivacy EngineeringAnnualCTO

Control Mapping

NIST PF GOVERNISO 27701GDPR ArticleSOC 2 TSC
GV.AT-P17.2.2Art. 39(1)(b)CC1.4
GV.AT-P25.1Art. 38(3)CC1.2
GV.MT-P19.1Art. 24(1)CC4.1
GV.MT-P35.2Art. 24(2)CC1.3
GV.PO-P15.2Art. 5(1)CC1.1
GV.PO-P45.3.2Art. 24(1)CC3.1
GV.PO-P55.2.1Art. 5(1)(a)CC2.2
GV.RR-P15.3Art. 37-39CC1.2
GV.RR-P27.1Art. 38(2)CC1.1

References

  • NIST Privacy Framework Version 1.0 (January 16, 2020)
  • NIST SP 800-53 Rev. 5 — Security and Privacy Controls (PM family)
  • ISO/IEC 27701:2019 — Clause 5 (PIMS-specific requirements)
  • IAPP CIPM Body of Knowledge — Privacy Program Management

NIST Privacy Framework — IDENTIFY Function

Overview

The IDENTIFY function in the NIST Privacy Framework (Version 1.0, January 2020) enables organizations to develop organizational understanding of privacy risk arising from data processing. This skill covers all four subcategories: Business Environment (ID.BE), Data Actions (ID.DA), Improvement (ID.IM), and Risk Assessment (ID.RA).

IDENTIFY Function Subcategories

ID.BE — Business Environment

Understanding the organization's mission, objectives, stakeholders, and activities to prioritize privacy risk management decisions.

SubcategoryDescriptionImplementation Guidance
ID.BE-P1The organization's role(s) in the data processing ecosystem are identified and communicatedDocument whether the organization acts as data controller, processor, or both. Map all data flows identifying organizational role at each stage.
ID.BE-P2Priorities for organizational mission, objectives, and activities are established and communicatedAlign privacy objectives with business strategy. Ensure executive leadership endorses privacy as a business priority.
ID.BE-P3Systems/products/services that process data are identified and prioritizedMaintain an inventory of all systems processing personal data. Classify by risk tier based on data sensitivity and volume.

ID.DA — Data Actions

Understanding the data actions the organization performs and associated privacy risks.

SubcategoryDescriptionImplementation Guidance
ID.DA-P1A data processing ecosystem inventory is created and maintainedCatalog all data processing activities including collection points, storage locations, sharing partners, and retention periods.
ID.DA-P2Owners of data actions are identifiedAssign clear ownership for each data processing activity. Document accountability chains from operational to executive level.
ID.DA-P3Problematic data actions are identified and prioritized for managementUse the NIST problematic data actions catalog to assess risk. Score based on likelihood and impact to individuals.

ID.IM — Improvement

Continuous improvement of privacy risk management.

SubcategoryDescriptionImplementation Guidance
ID.IM-P1A process for continuous improvement of the privacy risk assessment approach is establishedSchedule quarterly reviews of risk methodology. Incorporate lessons from incidents and regulatory developments.
ID.IM-P2Privacy risk assessment findings are incorporated into improvement plansTrack remediation actions in a centralized register. Assign deadlines and owners for each improvement item.

ID.RA — Risk Assessment

Understanding and evaluating privacy risks.

SubcategoryDescriptionImplementation Guidance
ID.RA-P1Data actions and their expected problematic data actions are identifiedMap each data action to potential problematic outcomes using NIST's catalog of problematic data actions.
ID.RA-P2Organizational systems, products, and services are monitored for problematic data actionsDeploy continuous monitoring for unauthorized access, unexpected data flows, and policy violations.
ID.RA-P3Risk responses are identified and prioritizedDefine response strategies: mitigate, transfer, avoid, or accept. Document rationale for each risk response decision.

Implementation Workflow

Phase 1: Inventory and Discovery (Weeks 1-4)

  1. Stakeholder Identification: Identify all internal and external stakeholders involved in data processing
  2. System Inventory: Catalog all systems, applications, and services that process personal data
  3. Data Flow Mapping: Document how data moves through the organization from collection to deletion
  4. Role Classification: Determine the organization's role (controller/processor) for each data processing activity

Phase 2: Data Action Analysis (Weeks 5-8)

  1. Data Action Catalog: Document all data actions (collection, retention, logging, generation, disclosure, transfer)
  2. Ownership Assignment: Assign data stewards for each data action category
  3. Problematic Data Action Screening: Evaluate each data action against NIST's problematic data actions list
  4. Impact Assessment: Score each problematic data action on likelihood and severity scales (1-5)

Phase 3: Risk Assessment (Weeks 9-12)

  1. Risk Scoring: Calculate risk scores as Likelihood x Impact for each identified problematic data action
  2. Risk Prioritization: Rank risks and identify those exceeding organizational risk tolerance
  3. Response Planning: Select and document risk response strategies for each high-priority risk
  4. Control Mapping: Map existing and planned controls to identified risks

Phase 4: Continuous Improvement (Ongoing)

  1. Quarterly Reviews: Reassess risk landscape and update risk register
  2. Incident Integration: Incorporate findings from privacy incidents into risk assessments
  3. Regulatory Monitoring: Track changes in applicable privacy regulations
  4. Metrics Tracking: Monitor key risk indicators and report to governance bodies

Control Mapping to Other Frameworks

NIST PF IDENTIFYISO 27701GDPR ArticleCCPA Section
ID.BE-P15.2.1Art. 24, 26, 281798.140(d),(v)
ID.BE-P25.2.2Art. 5(2)1798.100
ID.BE-P3A.7.2.1Art. 301798.110
ID.DA-P1A.7.2.8Art. 30(1)1798.110(c)
ID.DA-P25.3Art. 37-391798.130
ID.DA-P3A.7.2.5Art. 351798.185
ID.RA-P16.1.2Art. 35(7)1798.185(a)(15)
ID.RA-P29.1Art. 32(1)(d)1798.150
ID.RA-P36.1.3Art. 35(7)(d)1798.185(a)(15)

Maturity Assessment

Level 1 — Initial

  • Ad hoc privacy risk identification
  • No formal inventory of data processing activities
  • Reactive approach to privacy issues

Level 2 — Managed

  • Basic data inventory exists
  • Some data actions documented
  • Risk assessments conducted periodically

Level 3 — Defined

  • Comprehensive data processing inventory maintained
  • All data actions cataloged with owners
  • Structured risk assessment process in place

Level 4 — Quantitatively Managed

  • Metrics-driven risk assessment
  • Automated monitoring of data actions
  • Quantitative risk scoring methodology

Level 5 — Optimizing

  • Continuous improvement process embedded
  • Predictive risk analytics
  • Industry-leading privacy risk management practices

References

  • NIST Privacy Framework Version 1.0 (January 16, 2020)
  • NIST SP 800-37 Rev. 2 — Risk Management Framework
  • NIST IR 8062 — An Introduction to Privacy Engineering and Risk Management in Federal Systems
  • ISO/IEC 27701:2019 — Privacy Information Management

NIST Privacy Framework — PROTECT Function

Overview

The PROTECT function develops and implements appropriate data processing safeguards to prevent cybersecurity-related privacy events. This function directly bridges the NIST Cybersecurity Framework and Privacy Framework, addressing the overlap where security controls serve privacy objectives. It covers access control, data security, and protective policies.

PROTECT Function Subcategories

PR.AC — Access Control

Managing access to data and systems to protect privacy.

SubcategoryDescriptionImplementation Guidance
PR.AC-P1Identities and credentials are issued, managed, verified, revoked, and audited for authorized individuals, processes, and devicesImplement identity lifecycle management. Enforce MFA for systems processing personal data. Conduct quarterly access reviews.
PR.AC-P2Physical access to data and devices is managedRestrict physical access to data centers and server rooms. Implement visitor logging. Secure endpoint devices with encryption and remote wipe capability.
PR.AC-P3Remote access is managedEnforce VPN for remote access to personal data systems. Implement zero-trust network architecture. Monitor and log all remote sessions.
PR.AC-P4Access permissions and authorizations are managed with least privilege and separation of dutiesImplement role-based access control (RBAC). Enforce separation of duties for sensitive operations. Regular privileged access reviews.
PR.AC-P5Network integrity is protectedNetwork segmentation for systems processing personal data. Intrusion detection/prevention systems. Encrypted network communications.
PR.AC-P6Individuals and devices are proofed and bound to credentialsImplement strong identity verification for data subject account creation. Device certificate management. Biometric authentication where appropriate.

PR.DS — Data Security

Protecting data at rest, in transit, and during processing.

SubcategoryDescriptionImplementation Guidance
PR.DS-P1Data-at-rest are protectedAES-256 encryption for stored personal data. Key management with HSM. Database-level encryption (TDE).
PR.DS-P2Data-in-transit are protectedTLS 1.3 for all data transmissions. Certificate pinning for mobile applications. End-to-end encryption for sensitive communications.
PR.DS-P3Systems/products/services and associated data are formally managed throughout removal, transfers, and dispositionSecure media sanitization (NIST SP 800-88). Certificate of destruction for physical media. Verified data deletion from cloud services.
PR.DS-P4Adequate capacity to ensure availability is maintainedCapacity planning for privacy-critical systems. DDoS protection. Redundancy for data subject rights portals.
PR.DS-P5Protections against data leaks are implementedDLP solutions monitoring egress points. Email DLP for personal data. Cloud access security broker (CASB) deployment.
PR.DS-P6Integrity checking mechanisms are used to verify software, firmware, and information integrityFile integrity monitoring for privacy-critical systems. Code signing for privacy-related applications. Checksum verification for data transfers.
PR.DS-P7The development and testing environment(s) are separate from the production environmentSeparate environments with no production personal data in dev/test. Synthetic data generation for testing. Access controls between environments.
PR.DS-P8Data disclosures are consistent with the purposes for which the data were collectedTechnical enforcement of purpose limitation. API-level access controls tied to documented purposes. Automated purpose verification at query time.

PR.PO — Protective Policies

Policies governing protective measures for privacy.

SubcategoryDescriptionImplementation Guidance
PR.PO-P1A baseline configuration of information technology/industrial control systems is created and maintainedEstablish secure baselines for all systems processing personal data. Configuration management database. Automated compliance scanning.
PR.PO-P2Configuration change control processes are in placeChange advisory board review for privacy-impacting changes. Impact assessment for configuration changes. Rollback procedures documented.
PR.PO-P3Backups of information are conducted, maintained, and testedEncrypted backups of personal data. Regular restoration testing. Backup retention aligned with data retention policies.
PR.PO-P4Policy and regulations regarding the physical operating environment are metEnvironmental controls for data processing facilities. Compliance with data residency requirements. Physical security audits.
PR.PO-P5Data destruction processes are defined and managedDocumented destruction procedures per data classification. Verification and certification of destruction. Compliant disposal of hardware containing personal data.
PR.PO-P6Maintenance and repair of organizational assets are performed and loggedVendor supervision during maintenance. Data sanitization before equipment service. Maintenance activity logging.
PR.PO-P7Response plans (Incident Response and Business Continuity) and recovery plans (Incident Recovery and Disaster Recovery) are in place and managedPrivacy-integrated incident response plan. Business continuity for privacy-critical services. Regular tabletop exercises.
PR.PO-P8Response and recovery plans are testedAnnual incident response exercises including privacy scenarios. DR testing with privacy recovery objectives. Post-exercise improvement plans.
PR.PO-P9Privacy procedures are included in human resources practicesBackground checks for privacy-sensitive roles. Confidentiality agreements. Exit procedures including access revocation.
PR.PO-P10A vulnerability management plan is developed and implementedVulnerability scanning for privacy-critical systems. Patch management prioritizing privacy-impacting vulnerabilities. Penetration testing including privacy controls.

Technical Control Architecture

Access Control Architecture

Identity Provider (IdP)
    |
    v
Multi-Factor Authentication
    |
    v
Authorization Engine (RBAC + ABAC)
    |
    +-- Role-Based Access Control
    |   ├── Privacy Administrator
    |   ├── Data Steward
    |   ├── Data Processor
    |   ├── Data Analyst (aggregated only)
    |   └── Data Subject (own data only)
    |
    +-- Attribute-Based Access Control
    |   ├── Purpose attribute (why accessing)
    |   ├── Data classification (sensitivity level)
    |   ├── Time-based restrictions
    |   └── Location-based restrictions
    |
    v
Audit Logging Engine

Encryption Standards

Data StateMinimum StandardRecommended StandardKey Management
At RestAES-128AES-256HSM-managed keys, annual rotation
In TransitTLS 1.2TLS 1.3Certificate management with auto-renewal
In ProcessingApplication-level encryptionEnvelope encryptionPer-session keys
BackupAES-256AES-256 + separate keyOffline key escrow
ArchiveAES-256AES-256 + integrity checkLong-term key management

Data Loss Prevention Configuration

ChannelDLP ControlDetection MethodAction
EmailEmail gateway DLPPattern matching (SSN, CC, IBAN)Block + notify
Web uploadWeb proxy DLPContent inspectionBlock + notify
Cloud storageCASBClassification labelsEncrypt + restrict sharing
EndpointEndpoint DLPFile content scanningPrevent copy to removable media
DatabaseDatabase activity monitoringQuery analysisAlert on bulk extraction
APIAPI gatewayPayload inspectionRate limit + log

Network Security for Privacy

Segmentation Architecture

Internet
    |
    v
[DMZ / WAF]
    |
    v
[Application Tier] ── TLS 1.3 ── [API Gateway]
    |                                    |
    v                                    v
[Business Logic Tier]            [Privacy Services Tier]
    |                            ├── Consent Manager
    v                            ├── DSR Processor
[Data Tier]                      ├── Anonymization Engine
├── PII Database (encrypted)     └── Audit Logger
├── Analytics Database (aggregated)
└── Backup Storage (encrypted, isolated)

Control Mapping

NIST PF PROTECTNIST CSFISO 27701GDPR Article
PR.AC-P1PR.AC-1A.7.2.2Art. 32(1)(b)
PR.AC-P4PR.AC-4A.7.2.2Art. 25(2)
PR.DS-P1PR.DS-1A.7.4.9Art. 32(1)(a)
PR.DS-P2PR.DS-2A.7.4.9Art. 32(1)(a)
PR.DS-P5PR.DS-5A.7.4.9Art. 32(1)(b)
PR.DS-P7PR.DS-7A.7.4.9Art. 32(1)(a)
PR.DS-P8N/AA.7.2.2Art. 5(1)(b)
PR.PO-P3PR.IP-4A.7.4.9Art. 32(1)(c)
PR.PO-P5PR.IP-6A.7.4.7Art. 17(1)
PR.PO-P7PR.IP-9A.7.2.8Art. 33, 34
PR.PO-P10PR.IP-12A.7.4.9Art. 32(1)(d)

References

  • NIST Privacy Framework Version 1.0 (January 16, 2020)
  • NIST Cybersecurity Framework Version 1.1 (April 16, 2018)
  • NIST SP 800-53 Rev. 5 — Security and Privacy Controls for Information Systems
  • NIST SP 800-88 Rev. 1 — Guidelines for Media Sanitization
  • NIST SP 800-175B — Guideline for Using Cryptographic Standards

Automated PII Detection and Redaction Pipeline

Overview

Automated PII detection is a foundational capability for privacy engineering, enabling organizations to discover, classify, and protect personal data at scale. This skill covers building production-grade PII detection pipelines that combine rule-based pattern matching, machine learning-based Named Entity Recognition (NER), and cloud-native discovery services.

PII Entity Types Catalog

Direct Identifiers

Entity TypeExamplesDetection MethodRisk Level
PERSON_NAME"John Smith", "Maria Garcia"NER modelHigh
EMAIL_ADDRESS"j.smith@cipherengineeringlabs.com"Regex patternHigh
PHONE_NUMBER"+1-555-0123", "(555) 012-3456"Regex + validationHigh
SSN"123-45-6789"Regex + checksumCritical
PASSPORT_NUMBER"AB1234567"Regex per country formatCritical
DRIVER_LICENSE"D123-4567-8901"Regex per state/countryCritical
CREDIT_CARD"4111-1111-1111-1111"Regex + Luhn checksumCritical
IBAN"GB82 WEST 1234 5698 7654 32"Regex + modulo-97 checkHigh
IP_ADDRESS"192.168.1.1", "2001:db8::1"Regex (IPv4/IPv6)Medium
MAC_ADDRESS"00:1A:2B:3C:4D:5E"Regex patternMedium

Quasi-Identifiers

Entity TypeExamplesDetection MethodRisk Level
DATE_OF_BIRTH"1990-01-15", "January 15, 1990"NER + date parsingMedium
POSTAL_CODE"10001", "SW1A 1AA"Regex per countryMedium
AGE"35 years old", "age: 42"NER + contextLow-Medium
GENDER"male", "female", "non-binary"Dictionary + contextLow
NATIONALITY"British", "Japanese"Dictionary + NERLow-Medium
LOCATION"123 Main St", "New York"NER modelMedium

Sensitive Categories

Entity TypeExamplesDetection MethodRisk Level
MEDICAL_RECORD"MRN: 12345678"Regex + contextCritical
HEALTH_CONDITION"diabetes", "HIV positive"Medical NER + dictionaryCritical
RELIGIOUS_BELIEF"Muslim", "Catholic"Dictionary + contextHigh
POLITICAL_OPINION"Democratic Party member"Dictionary + contextHigh
SEXUAL_ORIENTATION"gay", "bisexual"Dictionary + contextHigh
BIOMETRIC_DATA"fingerprint hash: ..."Context + patternCritical
GENETIC_DATA"BRCA1 positive"Medical dictionaryCritical

Microsoft Presidio Implementation

Pipeline Architecture

Input Data --> Presidio Analyzer --> Detected Entities --> Presidio Anonymizer --> Redacted Output
                    |                      |                      |
                    v                      v                      v
             +------------+        +-------------+        +---------------+
             | Recognizers|        | Score Filter |        | Operators     |
             | - Pattern  |        | (threshold)  |        | - Replace     |
             | - NER      |        |              |        | - Redact      |
             | - Custom   |        |              |        | - Hash        |
             +------------+        +-------------+        | - Mask        |
                                                          | - Encrypt     |
                                                          +---------------+

Core Implementation

"""
PII detection and redaction pipeline using Microsoft Presidio.
Supports structured and unstructured text with configurable
entity types, confidence thresholds, and redaction strategies.
"""

from presidio_analyzer import AnalyzerEngine, PatternRecognizer, Pattern
from presidio_analyzer.nlp_engine import NlpEngineProvider
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig
import json
from dataclasses import dataclass, field


@dataclass
class DetectionResult:
    entity_type: str
    text: str
    start: int
    end: int
    score: float
    source: str


@dataclass
class PipelineConfig:
    language: str = "en"
    score_threshold: float = 0.5
    entities_to_detect: list[str] = field(default_factory=lambda: [
        "PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD",
        "US_SSN", "US_DRIVER_LICENSE", "IBAN_CODE", "IP_ADDRESS",
        "LOCATION", "DATE_TIME", "NRP", "MEDICAL_LICENSE",
        "US_PASSPORT", "US_BANK_NUMBER", "UK_NHS"
    ])
    redaction_strategy: str = "replace"  # replace, redact, hash, mask, encrypt


class PIIDetectionPipeline:
    """
    Production PII detection pipeline built on Microsoft Presidio.
    Supports custom entity types and configurable redaction strategies.
    """

    def __init__(self, config: PipelineConfig):
        self.config = config

        # Initialize NLP engine with spaCy
        nlp_config = {
            "nlp_engine_name": "spacy",
            "models": [{"lang_code": config.language, "model_name": "en_core_web_lg"}]
        }
        nlp_engine = NlpEngineProvider(nlp_configuration=nlp_config).create_engine()

        # Initialize analyzer
        self.analyzer = AnalyzerEngine(nlp_engine=nlp_engine)

        # Register custom recognizers
        self._register_custom_recognizers()

        # Initialize anonymizer
        self.anonymizer = AnonymizerEngine()

    def _register_custom_recognizers(self):
        """Register custom PII recognizers beyond built-in types."""

        # UK National Insurance Number
        nino_pattern = Pattern(
            name="uk_nino",
            regex=r"\b[A-CEGHJ-PR-TW-Z]{2}\s?\d{2}\s?\d{2}\s?\d{2}\s?[A-D]\b",
            score=0.85
        )
        nino_recognizer = PatternRecognizer(
            supported_entity="UK_NINO",
            patterns=[nino_pattern],
            supported_language="en"
        )
        self.analyzer.registry.add_recognizer(nino_recognizer)

        # Vehicle Registration Number (UK format)
        vrn_pattern = Pattern(
            name="uk_vrn",
            regex=r"\b[A-Z]{2}\d{2}\s?[A-Z]{3}\b",
            score=0.6
        )
        vrn_recognizer = PatternRecognizer(
            supported_entity="UK_VRN",
            patterns=[vrn_pattern],
            supported_language="en"
        )
        self.analyzer.registry.add_recognizer(vrn_recognizer)

        # Employee ID (Cipher Engineering Labs format: CEL-XXXXX)
        emp_id_pattern = Pattern(
            name="employee_id",
            regex=r"\bCEL-\d{5}\b",
            score=0.95
        )
        emp_id_recognizer = PatternRecognizer(
            supported_entity="EMPLOYEE_ID",
            patterns=[emp_id_pattern],
            supported_language="en"
        )
        self.analyzer.registry.add_recognizer(emp_id_recognizer)

    def detect(self, text: str) -> list[DetectionResult]:
        """
        Detect PII entities in text.

        Args:
            text: Input text to scan

        Returns:
            List of detected PII entities with confidence scores
        """
        results = self.analyzer.analyze(
            text=text,
            entities=self.config.entities_to_detect,
            language=self.config.language,
            score_threshold=self.config.score_threshold
        )

        return [
            DetectionResult(
                entity_type=r.entity_type,
                text=text[r.start:r.end],
                start=r.start,
                end=r.end,
                score=r.score,
                source=r.analysis_explanation.recognizer if r.analysis_explanation else "unknown"
            )
            for r in results
        ]

    def redact(self, text: str) -> tuple[str, list[DetectionResult]]:
        """
        Detect and redact PII from text.

        Returns:
            Tuple of (redacted_text, list_of_detections)
        """
        # Detect entities
        analyzer_results = self.analyzer.analyze(
            text=text,
            entities=self.config.entities_to_detect,
            language=self.config.language,
            score_threshold=self.config.score_threshold
        )

        # Configure redaction operator
        operators = self._get_operators()

        # Apply redaction
        anonymized = self.anonymizer.anonymize(
            text=text,
            analyzer_results=analyzer_results,
            operators=operators
        )

        detections = [
            DetectionResult(
                entity_type=r.entity_type,
                text=text[r.start:r.end],
                start=r.start,
                end=r.end,
                score=r.score,
                source="presidio"
            )
            for r in analyzer_results
        ]

        return anonymized.text, detections

    def _get_operators(self) -> dict:
        """Configure anonymization operators based on strategy."""
        if self.config.redaction_strategy == "replace":
            return {"DEFAULT": OperatorConfig("replace", {"new_value": "<REDACTED>"})}
        elif self.config.redaction_strategy == "hash":
            return {"DEFAULT": OperatorConfig("hash", {"hash_type": "sha256"})}
        elif self.config.redaction_strategy == "mask":
            return {"DEFAULT": OperatorConfig("mask", {
                "type": "mask",
                "masking_char": "*",
                "chars_to_mask": 100,
                "from_end": False
            })}
        elif self.config.redaction_strategy == "redact":
            return {"DEFAULT": OperatorConfig("redact", {})}
        else:
            return {"DEFAULT": OperatorConfig("replace", {"new_value": "<REDACTED>"})}


class BatchPIIProcessor:
    """
    Process large volumes of documents for PII detection.
    Supports structured (CSV/JSON) and unstructured (text) formats.
    """

    def __init__(self, pipeline: PIIDetectionPipeline):
        self.pipeline = pipeline

    def process_csv(
        self,
        input_path: str,
        output_path: str,
        columns_to_scan: list[str] = None
    ) -> dict:
        """
        Scan and redact PII in a CSV file.

        Args:
            input_path: Path to input CSV
            output_path: Path to write redacted CSV
            columns_to_scan: Columns to scan (all if None)

        Returns:
            Summary statistics of detections
        """
        import pandas as pd

        df = pd.read_csv(input_path)
        stats = {"total_rows": len(df), "detections": {}, "columns_scanned": []}

        scan_columns = columns_to_scan or df.columns.tolist()

        for col in scan_columns:
            if col not in df.columns:
                continue

            stats["columns_scanned"].append(col)
            col_detections = []

            for idx, value in df[col].items():
                if pd.isna(value):
                    continue

                text = str(value)
                redacted, detections = self.pipeline.redact(text)
                df.at[idx, col] = redacted

                for d in detections:
                    col_detections.append(d.entity_type)

            # Count detection types for this column
            for entity_type in set(col_detections):
                count = col_detections.count(entity_type)
                key = f"{col}:{entity_type}"
                stats["detections"][key] = count

        df.to_csv(output_path, index=False)
        stats["total_detections"] = sum(stats["detections"].values())

        return stats

    def process_text_files(
        self,
        file_paths: list[str],
        output_dir: str
    ) -> dict:
        """
        Batch process text files for PII detection and redaction.

        Returns summary statistics.
        """
        import os

        stats = {
            "files_processed": 0,
            "total_detections": 0,
            "entity_counts": {},
            "high_risk_files": []
        }

        for file_path in file_paths:
            with open(file_path, "r", encoding="utf-8") as f:
                text = f.read()

            redacted, detections = self.pipeline.redact(text)

            # Write redacted output
            output_path = os.path.join(output_dir, os.path.basename(file_path))
            with open(output_path, "w", encoding="utf-8") as f:
                f.write(redacted)

            # Update statistics
            stats["files_processed"] += 1
            stats["total_detections"] += len(detections)

            for d in detections:
                stats["entity_counts"][d.entity_type] = (
                    stats["entity_counts"].get(d.entity_type, 0) + 1
                )

            # Flag high-risk files (contain critical PII)
            critical_types = {"US_SSN", "CREDIT_CARD", "MEDICAL_LICENSE", "UK_NHS"}
            if any(d.entity_type in critical_types for d in detections):
                stats["high_risk_files"].append(file_path)

        return stats

spaCy NER Custom Training

Training Custom Entity Types

"""
Train custom spaCy NER model for domain-specific PII entities.
"""

import spacy
from spacy.training import Example
import random


def create_training_data() -> list[tuple[str, dict]]:
    """
    Create training examples for custom PII entity types.

    Returns list of (text, annotations) tuples.
    """
    training_data = [
        (
            "Patient MRN 12345678 was admitted on 2024-01-15",
            {"entities": [(12, 20, "MEDICAL_RECORD_NUMBER")]}
        ),
        (
            "Employee CEL-00142 reported the incident",
            {"entities": [(9, 18, "EMPLOYEE_ID")]}
        ),
        (
            "Policy holder number PLH-2024-99887 filed a claim",
            {"entities": [(22, 35, "POLICY_NUMBER")]}
        ),
        (
            "The customer with loyalty ID LYL-A1B2C3 requested data export",
            {"entities": [(29, 39, "LOYALTY_ID")]}
        ),
    ]
    return training_data


def train_custom_ner(
    base_model: str = "en_core_web_lg",
    training_data: list = None,
    n_iter: int = 30,
    output_dir: str = "./custom_ner_model"
):
    """
    Fine-tune spaCy NER model with custom PII entity types.

    Args:
        base_model: Base spaCy model to fine-tune
        training_data: List of (text, annotations) tuples
        n_iter: Number of training iterations
        output_dir: Directory to save trained model
    """
    if training_data is None:
        training_data = create_training_data()

    nlp = spacy.load(base_model)

    # Get or create NER pipe
    if "ner" not in nlp.pipe_names:
        ner = nlp.add_pipe("ner", last=True)
    else:
        ner = nlp.get_pipe("ner")

    # Add custom entity labels
    custom_labels = set()
    for _, annotations in training_data:
        for ent in annotations.get("entities", []):
            custom_labels.add(ent[2])

    for label in custom_labels:
        ner.add_label(label)

    # Train
    optimizer = nlp.resume_training()
    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]

    with nlp.disable_pipes(*other_pipes):
        for iteration in range(n_iter):
            random.shuffle(training_data)
            losses = {}

            for text, annotations in training_data:
                doc = nlp.make_doc(text)
                example = Example.from_dict(doc, annotations)
                nlp.update([example], drop=0.35, sgd=optimizer, losses=losses)

            if iteration % 10 == 0:
                print(f"Iteration {iteration}, Losses: {losses}")

    nlp.to_disk(output_dir)
    return nlp

AWS Macie Integration

Architecture

S3 Buckets --> Macie Classification Jobs --> Findings --> EventBridge --> Lambda
                                                |                          |
                                                v                          v
                                        Security Hub              Remediation
                                        (centralized)             - Tag sensitive
                                                                  - Encrypt
                                                                  - Notify owner
                                                                  - Quarantine

Macie Job Configuration

"""
AWS Macie integration for cloud-native PII detection in S3 buckets.
"""

import boto3
from datetime import datetime


class MacieIntegration:
    """
    Configure and manage AWS Macie classification jobs for
    automated PII detection across S3 data stores.
    """

    def __init__(self, region: str = "us-east-1"):
        self.macie_client = boto3.client("macie2", region_name=region)

    def create_classification_job(
        self,
        bucket_name: str,
        job_name: str,
        custom_data_identifiers: list[str] = None,
        schedule: str = "ONE_TIME"
    ) -> str:
        """
        Create a Macie classification job for an S3 bucket.

        Args:
            bucket_name: Target S3 bucket
            job_name: Descriptive job name
            custom_data_identifiers: IDs of custom data identifier resources
            schedule: ONE_TIME or SCHEDULED

        Returns:
            Job ID
        """
        job_config = {
            "name": job_name,
            "description": f"PII detection scan for {bucket_name}",
            "jobType": schedule,
            "s3JobDefinition": {
                "bucketDefinitions": [
                    {
                        "accountId": self._get_account_id(),
                        "buckets": [bucket_name]
                    }
                ],
                "scoping": {
                    "includes": {
                        "and": [
                            {
                                "simpleScopeTerm": {
                                    "comparator": "STARTS_WITH",
                                    "key": "OBJECT_EXTENSION",
                                    "values": ["csv", "json", "txt", "pdf", "docx", "xlsx", "parquet"]
                                }
                            }
                        ]
                    }
                }
            },
            "managedDataIdentifierSelector": "ALL",
            "tags": {
                "Team": "Privacy Engineering",
                "Purpose": "PII Detection"
            }
        }

        if custom_data_identifiers:
            job_config["customDataIdentifierIds"] = custom_data_identifiers

        response = self.macie_client.create_classification_job(**job_config)
        return response["jobId"]

    def create_custom_data_identifier(
        self,
        name: str,
        description: str,
        regex: str,
        keywords: list[str] = None,
        maximum_match_distance: int = 50
    ) -> str:
        """
        Create a custom data identifier for organization-specific PII patterns.

        Returns:
            Custom data identifier ID
        """
        params = {
            "name": name,
            "description": description,
            "regex": regex,
            "maximumMatchDistance": maximum_match_distance
        }

        if keywords:
            params["keywords"] = keywords

        response = self.macie_client.create_custom_data_identifier(**params)
        return response["customDataIdentifierId"]

    def _get_account_id(self) -> str:
        """Get the current AWS account ID."""
        sts = boto3.client("sts")
        return sts.get_caller_identity()["Account"]

    def get_findings_summary(self, job_id: str) -> dict:
        """Get summary of findings from a classification job."""
        response = self.macie_client.list_findings(
            findingCriteria={
                "criterion": {
                    "classificationDetails.jobId": {
                        "eq": [job_id]
                    }
                }
            }
        )

        findings = []
        if response["findingIds"]:
            details = self.macie_client.get_findings(findingIds=response["findingIds"])
            findings = details["findings"]

        summary = {
            "total_findings": len(findings),
            "severity_counts": {},
            "entity_type_counts": {},
            "affected_objects": []
        }

        for finding in findings:
            severity = finding.get("severity", {}).get("description", "unknown")
            summary["severity_counts"][severity] = (
                summary["severity_counts"].get(severity, 0) + 1
            )

            sensitive_data = finding.get("classificationDetails", {}).get(
                "result", {}
            ).get("sensitiveData", [])

            for sd in sensitive_data:
                category = sd.get("category", "unknown")
                summary["entity_type_counts"][category] = (
                    summary["entity_type_counts"].get(category, 0)
                    + sd.get("totalCount", 0)
                )

            resource = finding.get("resourcesAffected", {}).get("s3Object", {})
            if resource:
                summary["affected_objects"].append(resource.get("key", "unknown"))

        return summary

Confidence Scoring Framework

Score RangeConfidence LevelRecommended Action
0.95 - 1.00Very HighAuto-redact
0.80 - 0.94HighAuto-redact with logging
0.60 - 0.79MediumFlag for human review
0.40 - 0.59LowLog only, no action
0.00 - 0.39Very LowIgnore

References

  • Microsoft Presidio Documentation: microsoft.github.io/presidio
  • spaCy NER Documentation: spacy.io/usage/linguistic-features#named-entities
  • AWS Macie Documentation: docs.aws.amazon.com/macie
  • Google Cloud DLP API Documentation
  • NIST SP 800-188 — De-Identifying Government Datasets
  • Article 29 WP Opinion 05/2014 on Anonymisation Techniques

Privacy API Design Patterns

Overview

Privacy APIs provide programmatic interfaces for data subject rights exercise, consent management, data deletion, and audit logging. Well-designed privacy APIs enable organizations to automate compliance workflows, integrate with consent management platforms, and provide data subjects with self-service privacy controls. This skill covers API design patterns, OpenAPI specifications, authentication, and error handling for privacy-critical endpoints.

API Architecture Overview

External Consumers                    Privacy API Gateway                    Backend Services
+------------------+                 +--------------------+                 +------------------+
| Data Subject App |----HTTPS/TLS--->| Authentication     |                 | DSAR Service     |
+------------------+                 | Rate Limiting      |----internal---->| Consent Service  |
                                     | Request Validation |                 | Deletion Service |
+------------------+                 | Audit Logging      |                 | Audit Service    |
| Partner Portal   |----HTTPS/TLS--->| Versioning         |                 | Identity Service |
+------------------+                 +--------------------+                 +------------------+
                                            |
+------------------+                        v
| Internal Systems |----mTLS-------->+--------------------+
+------------------+                 | Privacy Event Bus  |
                                     | (async processing) |
                                     +--------------------+

1. Data Subject API (DSAR Endpoints)

OpenAPI Specification

openapi: 3.1.0
info:
  title: Cipher Engineering Labs - Data Subject Rights API
  version: 1.0.0
  description: >
    API for data subjects to exercise their privacy rights under GDPR,
    CCPA/CPRA, and other applicable privacy regulations.
  contact:
    name: Privacy Engineering Team
    email: privacy-engineering@cipherengineeringlabs.com

servers:
  - url: https://api.cipherengineeringlabs.com/privacy/v1
    description: Production

security:
  - BearerAuth: []
  - OAuth2: [dsar:read, dsar:write]

paths:
  /dsar/requests:
    post:
      operationId: createDSARRequest
      summary: Submit a new data subject access request
      tags: [DSAR]
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/DSARCreateRequest'
      responses:
        '202':
          description: Request accepted for processing
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DSARResponse'
          headers:
            X-Request-ID:
              schema:
                type: string
                format: uuid
            Location:
              schema:
                type: string
                format: uri
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
        '429':
          $ref: '#/components/responses/RateLimited'

    get:
      operationId: listDSARRequests
      summary: List data subject's own requests
      tags: [DSAR]
      parameters:
        - name: status
          in: query
          schema:
            type: string
            enum: [pending, in_progress, completed, rejected]
        - name: type
          in: query
          schema:
            type: string
            enum: [access, deletion, portability, rectification, restriction, objection]
        - name: page
          in: query
          schema:
            type: integer
            minimum: 1
            default: 1
        - name: per_page
          in: query
          schema:
            type: integer
            minimum: 1
            maximum: 100
            default: 20
      responses:
        '200':
          description: List of DSAR requests
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DSARListResponse'

  /dsar/requests/{requestId}:
    get:
      operationId: getDSARRequest
      summary: Get status and details of a specific DSAR
      tags: [DSAR]
      parameters:
        - name: requestId
          in: path
          required: true
          schema:
            type: string
            format: uuid
      responses:
        '200':
          description: DSAR details
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/DSARDetailResponse'
        '404':
          $ref: '#/components/responses/NotFound'

  /dsar/requests/{requestId}/download:
    get:
      operationId: downloadDSARData
      summary: Download the data package for a completed access request
      tags: [DSAR]
      parameters:
        - name: requestId
          in: path
          required: true
          schema:
            type: string
            format: uuid
      responses:
        '200':
          description: Data package download
          content:
            application/zip:
              schema:
                type: string
                format: binary
          headers:
            Content-Disposition:
              schema:
                type: string
        '404':
          $ref: '#/components/responses/NotFound'
        '410':
          description: Download link has expired

components:
  schemas:
    DSARCreateRequest:
      type: object
      required:
        - requestType
        - email
        - identityVerification
      properties:
        requestType:
          type: string
          enum: [access, deletion, portability, rectification, restriction, objection]
          description: Type of data subject right being exercised
        email:
          type: string
          format: email
          description: Email address associated with the account
        identityVerification:
          type: object
          properties:
            method:
              type: string
              enum: [account_login, email_verification, document_upload]
            verificationToken:
              type: string
          required: [method]
        details:
          type: string
          maxLength: 2000
          description: Additional details about the request
        dataCategories:
          type: array
          items:
            type: string
            enum: [profile, transactions, communications, analytics, marketing, all]
          description: Specific data categories (for access/portability)
        rectificationData:
          type: object
          additionalProperties: true
          description: Corrected data values (for rectification requests)
        preferredFormat:
          type: string
          enum: [json, csv, pdf]
          default: json
          description: Preferred data export format (for access/portability)

    DSARResponse:
      type: object
      properties:
        requestId:
          type: string
          format: uuid
        status:
          type: string
          enum: [pending, identity_verification, in_progress, completed, rejected]
        requestType:
          type: string
        submittedAt:
          type: string
          format: date-time
        estimatedCompletionDate:
          type: string
          format: date-time
        regulatoryDeadline:
          type: string
          format: date-time
          description: Maximum date by which response must be provided

    DSARDetailResponse:
      allOf:
        - $ref: '#/components/schemas/DSARResponse'
        - type: object
          properties:
            statusHistory:
              type: array
              items:
                type: object
                properties:
                  status:
                    type: string
                  timestamp:
                    type: string
                    format: date-time
                  note:
                    type: string
            downloadAvailable:
              type: boolean
            downloadExpiresAt:
              type: string
              format: date-time

    DSARListResponse:
      type: object
      properties:
        data:
          type: array
          items:
            $ref: '#/components/schemas/DSARResponse'
        pagination:
          $ref: '#/components/schemas/Pagination'

    Pagination:
      type: object
      properties:
        page:
          type: integer
        perPage:
          type: integer
        totalPages:
          type: integer
        totalItems:
          type: integer

  responses:
    BadRequest:
      description: Invalid request parameters
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
    Unauthorized:
      description: Authentication required
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
    NotFound:
      description: Resource not found
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
    RateLimited:
      description: Rate limit exceeded
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
      headers:
        Retry-After:
          schema:
            type: integer
        X-RateLimit-Limit:
          schema:
            type: integer
        X-RateLimit-Remaining:
          schema:
            type: integer

  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT
    OAuth2:
      type: oauth2
      flows:
        authorizationCode:
          authorizationUrl: https://auth.cipherengineeringlabs.com/oauth2/authorize
          tokenUrl: https://auth.cipherengineeringlabs.com/oauth2/token
          scopes:
            dsar:read: Read DSAR request status
            dsar:write: Submit DSAR requests
            consent:read: Read consent preferences
            consent:write: Update consent preferences
            audit:read: Read audit logs

2. Consent API

Consent Preference Management

paths:
  /consent/preferences:
    get:
      operationId: getConsentPreferences
      summary: Get current consent preferences for the authenticated user
      tags: [Consent]
      responses:
        '200':
          description: Current consent preferences
          content:
            application/json:
              schema:
                type: object
                properties:
                  subjectId:
                    type: string
                  preferences:
                    type: array
                    items:
                      $ref: '#/components/schemas/ConsentPreference'
                  lastUpdated:
                    type: string
                    format: date-time

    put:
      operationId: updateConsentPreferences
      summary: Update consent preferences (bulk update)
      tags: [Consent]
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [preferences]
              properties:
                preferences:
                  type: array
                  items:
                    type: object
                    required: [purposeId, granted]
                    properties:
                      purposeId:
                        type: string
                      granted:
                        type: boolean
                      channels:
                        type: array
                        items:
                          type: string
                          enum: [email, sms, push, phone, postal]
      responses:
        '200':
          description: Preferences updated
          content:
            application/json:
              schema:
                type: object
                properties:
                  updated:
                    type: array
                    items:
                      $ref: '#/components/schemas/ConsentPreference'
                  consentReceiptId:
                    type: string
                    format: uuid
                    description: ID of the issued consent receipt

  /consent/purposes:
    get:
      operationId: listConsentPurposes
      summary: List all available consent purposes
      tags: [Consent]
      responses:
        '200':
          description: List of consent purposes
          content:
            application/json:
              schema:
                type: object
                properties:
                  purposes:
                    type: array
                    items:
                      type: object
                      properties:
                        purposeId:
                          type: string
                        name:
                          type: string
                        description:
                          type: string
                        category:
                          type: string
                          enum: [essential, functional, analytics, marketing]
                        mandatory:
                          type: boolean
                        defaultState:
                          type: boolean
                        dataCategories:
                          type: array
                          items:
                            type: string

  /consent/receipts/{receiptId}:
    get:
      operationId: getConsentReceipt
      summary: Retrieve a specific consent receipt
      tags: [Consent]
      parameters:
        - name: receiptId
          in: path
          required: true
          schema:
            type: string
            format: uuid
      responses:
        '200':
          description: Consent receipt
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ConsentReceipt'

3. Deletion API

Cascading Delete Orchestration

"""
Privacy deletion API with cascading delete orchestration.
Handles deletion requests across multiple backend services
with verification and audit logging.
"""

from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Optional
import uuid


class DeletionStatus(Enum):
    PENDING = "pending"
    IN_PROGRESS = "in_progress"
    PARTIALLY_COMPLETED = "partially_completed"
    COMPLETED = "completed"
    FAILED = "failed"
    VERIFIED = "verified"


class ServiceDeletionStatus(Enum):
    PENDING = "pending"
    IN_PROGRESS = "in_progress"
    COMPLETED = "completed"
    FAILED = "failed"
    SKIPPED = "skipped"  # For legally exempt data


@dataclass
class DeletionTarget:
    service_name: str
    data_categories: list[str]
    priority: int  # Lower number = delete first
    legal_hold_check: bool = True
    retention_exempt: bool = False  # True if data must be retained (e.g., tax records)


@dataclass
class ServiceDeletionResult:
    service_name: str
    status: ServiceDeletionStatus
    records_deleted: int
    records_retained: int
    retention_reason: Optional[str]
    completed_at: Optional[datetime]
    error_message: Optional[str]


@dataclass
class DeletionRequest:
    request_id: str
    subject_id: str
    status: DeletionStatus
    created_at: datetime
    scope: list[str]  # Data categories to delete
    exclude: list[str]  # Data categories exempt from deletion
    service_results: list[ServiceDeletionResult] = field(default_factory=list)
    verified_at: Optional[datetime] = None
    verification_method: Optional[str] = None


class CascadingDeletionOrchestrator:
    """
    Orchestrate data deletion across multiple backend services
    in the correct order, handling dependencies, legal holds,
    and retention requirements.
    """

    def __init__(self, service_registry: list[DeletionTarget], audit_logger):
        self.targets = sorted(service_registry, key=lambda t: t.priority)
        self.audit = audit_logger

    def execute_deletion(
        self,
        subject_id: str,
        scope: list[str],
        exclude: list[str] = None
    ) -> DeletionRequest:
        """
        Execute cascading deletion across all registered services.

        Args:
            subject_id: The data subject whose data should be deleted
            scope: Data categories to delete (or ["all"])
            exclude: Data categories to exclude from deletion

        Returns:
            DeletionRequest with results from each service
        """
        exclude = exclude or []

        request = DeletionRequest(
            request_id=str(uuid.uuid4()),
            subject_id=subject_id,
            status=DeletionStatus.IN_PROGRESS,
            created_at=datetime.now(timezone.utc),
            scope=scope,
            exclude=exclude
        )

        self.audit.log_deletion_start(request)

        for target in self.targets:
            # Skip if data category not in scope
            if "all" not in scope:
                relevant_categories = set(target.data_categories) & set(scope)
                if not relevant_categories:
                    continue

            # Skip if explicitly excluded
            if set(target.data_categories) & set(exclude):
                result = ServiceDeletionResult(
                    service_name=target.service_name,
                    status=ServiceDeletionStatus.SKIPPED,
                    records_deleted=0,
                    records_retained=0,
                    retention_reason="Excluded by request",
                    completed_at=datetime.now(timezone.utc),
                    error_message=None
                )
                request.service_results.append(result)
                continue

            # Check retention exemptions
            if target.retention_exempt:
                result = ServiceDeletionResult(
                    service_name=target.service_name,
                    status=ServiceDeletionStatus.SKIPPED,
                    records_deleted=0,
                    records_retained=0,
                    retention_reason="Legal retention requirement",
                    completed_at=datetime.now(timezone.utc),
                    error_message=None
                )
                request.service_results.append(result)
                continue

            # Check legal holds
            if target.legal_hold_check:
                hold = self._check_legal_hold(subject_id, target.service_name)
                if hold:
                    result = ServiceDeletionResult(
                        service_name=target.service_name,
                        status=ServiceDeletionStatus.SKIPPED,
                        records_deleted=0,
                        records_retained=0,
                        retention_reason=f"Legal hold: {hold}",
                        completed_at=datetime.now(timezone.utc),
                        error_message=None
                    )
                    request.service_results.append(result)
                    continue

            # Execute deletion for this service
            result = self._delete_from_service(subject_id, target)
            request.service_results.append(result)

        # Determine overall status
        statuses = [r.status for r in request.service_results]
        if all(s in (ServiceDeletionStatus.COMPLETED, ServiceDeletionStatus.SKIPPED) for s in statuses):
            request.status = DeletionStatus.COMPLETED
        elif any(s == ServiceDeletionStatus.FAILED for s in statuses):
            request.status = DeletionStatus.PARTIALLY_COMPLETED
        else:
            request.status = DeletionStatus.COMPLETED

        self.audit.log_deletion_complete(request)
        return request

    def verify_deletion(self, request: DeletionRequest) -> DeletionRequest:
        """
        Verify that data was actually deleted from all services.
        Run after deletion to confirm no data remnants exist.
        """
        for result in request.service_results:
            if result.status == ServiceDeletionStatus.COMPLETED:
                # Attempt to query for the subject's data
                remaining = self._check_data_exists(
                    request.subject_id, result.service_name
                )
                if remaining > 0:
                    result.status = ServiceDeletionStatus.FAILED
                    result.error_message = f"Verification failed: {remaining} records still exist"

        request.verified_at = datetime.now(timezone.utc)
        request.verification_method = "post_deletion_query"

        all_verified = all(
            r.status in (ServiceDeletionStatus.COMPLETED, ServiceDeletionStatus.SKIPPED)
            for r in request.service_results
        )
        request.status = DeletionStatus.VERIFIED if all_verified else DeletionStatus.PARTIALLY_COMPLETED

        self.audit.log_deletion_verification(request)
        return request

    def _check_legal_hold(self, subject_id: str, service: str) -> Optional[str]:
        """Check if any legal hold applies to this subject's data."""
        # Integration point with legal hold management system
        return None  # Override in production with actual legal hold check

    def _delete_from_service(
        self, subject_id: str, target: DeletionTarget
    ) -> ServiceDeletionResult:
        """Execute deletion against a specific backend service."""
        # Integration point with actual backend service deletion API
        return ServiceDeletionResult(
            service_name=target.service_name,
            status=ServiceDeletionStatus.COMPLETED,
            records_deleted=0,
            records_retained=0,
            retention_reason=None,
            completed_at=datetime.now(timezone.utc),
            error_message=None
        )

    def _check_data_exists(self, subject_id: str, service: str) -> int:
        """Check if data still exists for a subject in a service."""
        # Integration point with service data existence check
        return 0

4. Audit API

Audit Log Query Endpoints

paths:
  /audit/events:
    get:
      operationId: queryAuditEvents
      summary: Query privacy audit events
      tags: [Audit]
      security:
        - OAuth2: [audit:read]
      parameters:
        - name: subjectId
          in: query
          schema:
            type: string
        - name: eventType
          in: query
          schema:
            type: string
            enum: [data_access, consent_change, deletion, dsar_submitted, dsar_completed, breach_detected]
        - name: startDate
          in: query
          required: true
          schema:
            type: string
            format: date-time
        - name: endDate
          in: query
          required: true
          schema:
            type: string
            format: date-time
        - name: actorId
          in: query
          schema:
            type: string
        - name: page
          in: query
          schema:
            type: integer
            default: 1
        - name: perPage
          in: query
          schema:
            type: integer
            default: 50
            maximum: 200
      responses:
        '200':
          description: Audit events
          content:
            application/json:
              schema:
                type: object
                properties:
                  events:
                    type: array
                    items:
                      type: object
                      properties:
                        eventId:
                          type: string
                          format: uuid
                        eventType:
                          type: string
                        timestamp:
                          type: string
                          format: date-time
                        actorId:
                          type: string
                        actorType:
                          type: string
                          enum: [user, system, admin, api_client]
                        subjectId:
                          type: string
                        resource:
                          type: string
                        action:
                          type: string
                        purpose:
                          type: string
                        outcome:
                          type: string
                          enum: [success, failure, denied]
                        details:
                          type: object
                  pagination:
                    $ref: '#/components/schemas/Pagination'

  /audit/reports/compliance:
    get:
      operationId: getComplianceReport
      summary: Generate compliance audit report for a date range
      tags: [Audit]
      parameters:
        - name: startDate
          in: query
          required: true
          schema:
            type: string
            format: date
        - name: endDate
          in: query
          required: true
          schema:
            type: string
            format: date
        - name: format
          in: query
          schema:
            type: string
            enum: [json, pdf, csv]
            default: json
      responses:
        '200':
          description: Compliance report
          content:
            application/json:
              schema:
                type: object
                properties:
                  reportId:
                    type: string
                  generatedAt:
                    type: string
                    format: date-time
                  period:
                    type: object
                    properties:
                      start:
                        type: string
                      end:
                        type: string
                  summary:
                    type: object
                    properties:
                      totalDataAccesses:
                        type: integer
                      totalConsentChanges:
                        type: integer
                      totalDeletions:
                        type: integer
                      totalDSARs:
                        type: integer
                      accessDenials:
                        type: integer
                      policyViolations:
                        type: integer

API Security Patterns

Authentication and Authorization

Endpoint CategoryAuth MethodScopes RequiredRate Limit
DSAR (data subject)OAuth2 PKCE / JWTdsar:read, dsar:write10 req/min
DSAR (admin)OAuth2 Client Credentialsdsar:admin100 req/min
Consent (data subject)OAuth2 PKCE / JWTconsent:read, consent:write30 req/min
Deletion (internal)mTLS + API keydeletion:execute5 req/min
Audit (compliance)OAuth2 Client Credentialsaudit:read50 req/min

Error Response Format

{
  "error": {
    "code": "PRIVACY_001",
    "type": "identity_verification_required",
    "message": "Identity verification is required before processing this request",
    "details": {
      "verificationMethods": ["email_verification", "document_upload"],
      "supportUrl": "https://support.cipherengineeringlabs.com/privacy"
    },
    "requestId": "req_a1b2c3d4",
    "timestamp": "2026-03-14T10:30:00.000Z"
  }
}

References

  • OpenAPI Specification 3.1.0
  • GDPR Articles 12-22 (Data Subject Rights)
  • CCPA/CPRA Sections 1798.100-1798.135
  • RFC 6749 — OAuth 2.0 Authorization Framework
  • RFC 7519 — JSON Web Token (JWT)
  • OWASP API Security Top 10 (2023)
  • Google Privacy Sandbox API Design Guidelines
  • Apple App Tracking Transparency API Design Patterns

Privacy-Preserving Data Sharing Platform

Overview

Privacy-preserving data sharing enables organizations to derive analytical value from combined datasets without exposing raw personal data. This skill covers four primary approaches: synthetic data generation, data clean rooms, secure enclaves, and federated analytics, along with utility measurement frameworks to ensure shared data remains useful.

Approach Selection Framework

ApproachPrivacy GuaranteeData UtilityComputational CostTrust Model
Synthetic DataStatistical (configurable)High for distributions, lower for edge casesMedium (training)No trust required
Data Clean RoomsContractual + technicalHigh (real data, restricted queries)Low-MediumTrusted operator
Secure Enclaves (TEE)Hardware-backed isolationVery high (real data)MediumTrust hardware vendor
Federated AnalyticsCryptographic/DPMedium-HighHigh (communication)Minimal trust
Homomorphic EncryptionCryptographicHighVery HighNo trust required
Secure Multi-Party ComputationCryptographicHighHighHonest majority

Synthetic Data Generation with SDV

Architecture

Source Data --> Statistical Profiling --> Model Training --> Synthetic Generation
                    |                        |                      |
                    v                        v                      v
            Metadata Analysis        Model Selection         Quality Assessment
            - Column types           - GaussianCopula        - Statistical tests
            - Distributions          - CTGAN                 - Privacy metrics
            - Correlations           - CopulaGAN             - Utility metrics
            - Constraints            - TVAE                  - Visual comparison

SDV Implementation

"""
Synthetic data generation using the Synthetic Data Vault (SDV) library.
Generates privacy-preserving synthetic datasets that maintain statistical
properties of the original data.
"""

import pandas as pd
import numpy as np
from sdv.metadata import SingleTableMetadata
from sdv.single_table import GaussianCopulaSynthesizer, CTGANSynthesizer, TVAESynthesizer
from sdv.evaluation.single_table import run_diagnostic, evaluate_quality
from sdmetrics.reports.single_table import QualityReport


def create_metadata(df: pd.DataFrame) -> SingleTableMetadata:
    """Auto-detect and create metadata for a DataFrame."""
    metadata = SingleTableMetadata()
    metadata.detect_from_dataframe(df)
    return metadata


def train_gaussian_copula(
    df: pd.DataFrame,
    metadata: SingleTableMetadata,
    enforce_min_max: bool = True
) -> GaussianCopulaSynthesizer:
    """
    Train a Gaussian Copula model for synthetic data generation.
    Best for: Datasets with mostly numerical data and linear correlations.
    """
    synthesizer = GaussianCopulaSynthesizer(
        metadata,
        enforce_min_max_values=enforce_min_max,
        numerical_distributions={
            "norm": "beta",  # Fit beta distributions for bounded numerical data
        }
    )
    synthesizer.fit(df)
    return synthesizer


def train_ctgan(
    df: pd.DataFrame,
    metadata: SingleTableMetadata,
    epochs: int = 300,
    batch_size: int = 500
) -> CTGANSynthesizer:
    """
    Train a CTGAN model for synthetic data generation.
    Best for: Complex distributions, mixed data types, mode-specific patterns.
    """
    synthesizer = CTGANSynthesizer(
        metadata,
        epochs=epochs,
        batch_size=batch_size,
        verbose=True
    )
    synthesizer.fit(df)
    return synthesizer


def train_tvae(
    df: pd.DataFrame,
    metadata: SingleTableMetadata,
    epochs: int = 300
) -> TVAESynthesizer:
    """
    Train a TVAE model for synthetic data generation.
    Best for: Datasets where CTGAN struggles, faster training than CTGAN.
    """
    synthesizer = TVAESynthesizer(
        metadata,
        epochs=epochs
    )
    synthesizer.fit(df)
    return synthesizer


def generate_synthetic_data(
    synthesizer,
    num_rows: int
) -> pd.DataFrame:
    """Generate synthetic data from a trained synthesizer."""
    return synthesizer.sample(num_rows=num_rows)


def evaluate_synthetic_quality(
    real_data: pd.DataFrame,
    synthetic_data: pd.DataFrame,
    metadata: SingleTableMetadata
) -> dict:
    """
    Evaluate the quality of synthetic data against real data.

    Returns diagnostic and quality scores.
    """
    # Run diagnostic checks
    diagnostic = run_diagnostic(
        real_data=real_data,
        synthetic_data=synthetic_data,
        metadata=metadata
    )

    # Run quality evaluation
    quality = evaluate_quality(
        real_data=real_data,
        synthetic_data=synthetic_data,
        metadata=metadata
    )

    return {
        "diagnostic_score": diagnostic.get_score(),
        "quality_score": quality.get_score(),
    }


def measure_privacy_risk(
    real_data: pd.DataFrame,
    synthetic_data: pd.DataFrame,
    metadata: SingleTableMetadata,
    key_fields: list[str]
) -> dict:
    """
    Measure re-identification risk in synthetic data.

    Checks for exact matches and nearest-neighbor distances
    between real and synthetic records.
    """
    # Check for exact record matches
    merged = real_data.merge(synthetic_data, how="inner")
    exact_match_rate = len(merged) / len(real_data)

    # Check key field matches
    if key_fields:
        key_merged = real_data[key_fields].merge(
            synthetic_data[key_fields], how="inner"
        )
        key_match_rate = len(key_merged) / len(real_data)
    else:
        key_match_rate = 0.0

    return {
        "exact_match_rate": exact_match_rate,
        "key_match_rate": key_match_rate,
        "privacy_safe": exact_match_rate < 0.01 and key_match_rate < 0.05,
    }

Model Selection Guide

FactorGaussianCopulaCTGANTVAE
Training speedFast (minutes)Slow (hours)Medium (30-60 min)
Small datasets (<1K rows)GoodPoorFair
Large datasets (>100K rows)GoodGoodGood
Numerical dataExcellentGoodGood
Categorical data (high cardinality)FairGoodGood
Complex correlationsFairGoodGood
Constraint handlingGoodFairFair
ReproducibilityExcellentFair (seed-dependent)Fair

Data Clean Room Architecture

Components

Organization A                    Clean Room                    Organization B
+-------------+    encrypted    +------------------+    encrypted    +-------------+
| Source Data  |  ----------->  | Ingestion Layer  |  <-----------  | Source Data  |
+-------------+                 +------------------+                 +-------------+
                                        |
                                        v
                                +------------------+
                                | Data Preparation |
                                | - Schema mapping |
                                | - Normalization  |
                                | - Deduplication  |
                                +------------------+
                                        |
                                        v
                                +------------------+
                                | Approved Queries |
                                | - Pre-approved   |
                                |   query templates|
                                | - Aggregate only |
                                | - Min group size |
                                +------------------+
                                        |
                                        v
                                +------------------+
                                | Output Validation|
                                | - k-anonymity    |
                                | - DP noise       |
                                | - Disclosure risk|
                                +------------------+
                                        |
                            +-----------+-----------+
                            |                       |
                            v                       v
                    Results for Org A        Results for Org B

Clean Room Policy Engine

"""
Policy engine for data clean room query validation.
Enforces privacy rules on all queries before execution.
"""

from dataclasses import dataclass


@dataclass
class CleanRoomPolicy:
    min_group_size: int = 50
    allowed_operations: list[str] = None
    blocked_columns: list[str] = None
    max_output_rows: int = 1000
    require_aggregation: bool = True
    dp_epsilon: float = 1.0

    def __post_init__(self):
        if self.allowed_operations is None:
            self.allowed_operations = ["COUNT", "SUM", "AVG", "MEDIAN", "PERCENTILE"]
        if self.blocked_columns is None:
            self.blocked_columns = ["ssn", "email", "phone", "full_name", "address"]


class QueryValidator:
    """Validate clean room queries against privacy policies."""

    def __init__(self, policy: CleanRoomPolicy):
        self.policy = policy

    def validate(self, query_ast: dict) -> tuple[bool, list[str]]:
        """
        Validate a parsed query against the policy.

        Returns (is_valid, list_of_violations).
        """
        violations = []

        # Check for blocked columns
        referenced_columns = query_ast.get("columns", [])
        for col in referenced_columns:
            if col.lower() in self.policy.blocked_columns:
                violations.append(f"Column '{col}' is blocked by policy")

        # Check aggregation requirement
        if self.policy.require_aggregation:
            if not query_ast.get("has_aggregation", False):
                violations.append("Query must include aggregation (no raw record output)")

        # Check operations
        operations = query_ast.get("operations", [])
        for op in operations:
            if op.upper() not in self.policy.allowed_operations:
                violations.append(f"Operation '{op}' is not in allowed operations list")

        # Check output size
        if query_ast.get("limit", float("inf")) > self.policy.max_output_rows:
            violations.append(
                f"Output exceeds max rows ({self.policy.max_output_rows})"
            )

        return (len(violations) == 0, violations)

Secure Enclave Integration

Intel SGX / Azure Confidential Computing

Data Owner A           Confidential Computing          Data Owner B
                       +----------------------+
Data (encrypted) ----> | Enclave Environment  | <---- Data (encrypted)
                       | - Decryption in TEE  |
                       | - Join/Analysis      |
                       | - Re-encrypt results |
                       +----------------------+
                               |
                       Encrypted Results
                       (only to authorized parties)

Key Properties

  • Confidentiality: Data is encrypted outside the enclave; only decrypted within the TEE
  • Integrity: Enclave code is measured and attested; tampering is detectable
  • Attestation: Remote parties can verify the enclave is running approved code
  • Isolation: Even the cloud provider cannot access data inside the enclave

Utility Measurement Framework

Statistical Utility Metrics

MetricDescriptionTarget
Column ShapesDistribution similarity per column (KS test)> 0.85
Column Pair TrendsCorrelation preservation between column pairs> 0.80
Boundary AdherenceValues within real data min/max ranges> 0.95
Category CoverageAll categories in real data appear in synthetic> 0.90
Range CoverageNumeric ranges adequately represented> 0.85

Privacy Metrics

MetricDescriptionTarget
Exact Match Rate% of synthetic records identical to real records< 1%
Nearest Neighbor DistanceMinimum distance from synthetic to nearest real record> threshold
Membership Inference AUCAbility of attack model to determine membership< 0.55
Attribute Inference AccuracyAbility to infer sensitive attributes< random + 5%
k-Anonymity of outputMinimum equivalence class sizek >= 5

References

  • Patki, N., Wedge, R., and Veeramachaneni, K. "The Synthetic Data Vault." IEEE DSAA, 2016.
  • SDV Documentation: docs.sdv.dev
  • Xu, L. et al. "Modeling Tabular Data Using Conditional GAN." NeurIPS, 2019.
  • Google BigQuery Clean Rooms Documentation
  • AWS Clean Rooms Service Documentation
  • Microsoft Azure Confidential Computing Documentation
  • Stadler, T. et al. "Synthetic Data — Anonymisation Groundhog Day." USENIX Security, 2022.

Privacy Metrics Dashboard

Overview

A privacy metrics dashboard provides quantitative visibility into the effectiveness and maturity of an organization's privacy program. This skill defines the key privacy performance indicators (KPIs), data collection methods, visualization patterns, and reporting cadences needed to measure, monitor, and continuously improve privacy operations.

Core Privacy KPI Categories

1. Data Subject Access Requests (DSARs)

MetricDefinitionTargetFrequency
DSAR VolumeTotal DSARs received per periodTrend monitoringMonthly
DSAR by TypeBreakdown by right exercised (access, deletion, portability, rectification)N/A (distribution)Monthly
Average Response TimeMean calendar days from receipt to completion< 25 days (GDPR: 30 day limit)Monthly
Median Response TimeMedian calendar days from receipt to completion< 20 daysMonthly
P95 Response Time95th percentile response time< 28 daysMonthly
On-Time Completion Rate% completed within regulatory deadline> 98%Monthly
DSAR BacklogOpen DSARs at period end< 10% of monthly volumeWeekly
Automated Fulfillment Rate% of DSARs fulfilled without manual intervention> 60%Quarterly
Identity Verification Failure Rate% of DSARs where identity could not be verified< 5%Monthly
Cost per DSARAverage cost to process one DSARTrend reductionQuarterly

2. Data Breaches

MetricDefinitionTargetFrequency
Breach CountTotal confirmed breaches per period0 (minimize)Monthly
Breach Severity DistributionCount by severity level (low/medium/high/critical)No critical breachesMonthly
Mean Time to Detect (MTTD)Average hours from breach occurrence to detection< 24 hoursPer incident
Mean Time to Notify (MTTN)Average hours from detection to regulatory notification< 48 hours (GDPR: 72 hours)Per incident
Mean Time to Contain (MTTC)Average hours from detection to containment< 12 hoursPer incident
Records AffectedTotal data subject records affectedMinimizePer incident
Repeat Breach Rate% of breaches with same root cause as prior breach0%Quarterly
Regulatory FinesTotal fines imposed due to breaches$0Annual
Breach Simulation ScoreScore from most recent tabletop exercise> 85%Semi-annual

3. Privacy Impact Assessments (DPIAs)

MetricDefinitionTargetFrequency
DPIA Completion Rate% of high-risk processing with completed DPIA100%Quarterly
DPIA BacklogDPIAs required but not yet started0Monthly
Average DPIA DurationMean days from initiation to sign-off< 30 daysQuarterly
DPIA Findings Remediated% of identified risks with implemented mitigations> 90%Quarterly
DPIA Review Rate% of existing DPIAs reviewed within their review cycle> 95%Quarterly
New Processing Without DPIACount of new high-risk processing launched without DPIA0Monthly

4. Training and Awareness

MetricDefinitionTargetFrequency
Training Completion Rate% of staff who completed required privacy training> 95%Quarterly
Training Pass Rate% passing the assessment on first attempt> 85%Quarterly
New Hire Training TimeAverage days from start date to training completion< 14 daysMonthly
Specialist Certification Rate% of privacy team with professional certification> 80%Annual
Privacy Champion Coverage% of business units with designated privacy champion100%Quarterly
Phishing Simulation Pass Rate% of staff correctly identifying privacy-related phishing> 90%Quarterly

5. Consent Management

MetricDefinitionTargetFrequency
Consent Rate% of users providing consent (by purpose)Trend monitoringMonthly
Opt-in Rate% choosing to opt in to optional processingTrend monitoringMonthly
Opt-out/Withdrawal Rate% withdrawing previously given consent< 5% monthlyMonthly
Consent Freshness% of active consents collected within last 12 months> 70%Quarterly
Consent Collection Coverage% of data processing with documented consent/legal basis100%Quarterly
Cookie Consent Rate% of website visitors accepting cookies (by category)Trend monitoringMonthly
Consent Record Accuracy% of consent records matching actual processing> 99%Quarterly

6. Operational Metrics

MetricDefinitionTargetFrequency
Privacy Incidents ReportedInternal privacy incidents reported (not just breaches)Trend monitoringMonthly
Vendor Privacy Assessments Completed% of high-risk vendors assessed100%Quarterly
Data Inventory Completeness% of processing activities documented in ROPA> 95%Quarterly
Policy Acknowledgment Rate% of staff acknowledging updated privacy policies> 98%Per update
Privacy-by-Design Gate Pass Rate% of projects passing privacy review at design gate> 85%Quarterly
Regulatory Inquiry Response TimeAverage days to respond to regulatory inquiries< 5 daysPer event

Data Collection Architecture

Data Sources                    Collection Layer              Dashboard Layer
+------------------+
| DSAR Platform    |---+
+------------------+   |
                       |    +-------------------+    +-----------------------+
+------------------+   +--->|                   |    |                       |
| Breach Tracker   |------->| Data Integration  |--->| Visualization Engine  |
+------------------+   +--->| (ETL Pipeline)    |    | (Grafana / Tableau /  |
                       |    |                   |    |  Power BI / Metabase) |
+------------------+   |    +-------------------+    +-----------------------+
| DPIA Register    |---+           |                          |
+------------------+   |           v                          v
                       |    +-------------------+    +-----------------------+
+------------------+   |    | Metrics Database  |    | Alerting System       |
| LMS (Training)   |---+    | (Time-series /    |    | - Threshold breaches  |
+------------------+   |    |  PostgreSQL)      |    | - Trend deviations    |
                       |    +-------------------+    | - SLA warnings        |
+------------------+   |                              +-----------------------+
| Consent Platform |---+
+------------------+

Metric Collection Implementation

"""
Privacy metrics collection and aggregation engine.
Collects KPI data from multiple privacy operational systems
and computes dashboard-ready metrics.
"""

from dataclasses import dataclass
from datetime import datetime, timedelta
from typing import Optional
import statistics


@dataclass
class DSARMetrics:
    period_start: datetime
    period_end: datetime
    total_received: int
    total_completed: int
    total_overdue: int
    avg_response_days: float
    median_response_days: float
    p95_response_days: float
    on_time_rate: float
    by_type: dict[str, int]
    backlog: int
    automated_rate: float


@dataclass
class BreachMetrics:
    period_start: datetime
    period_end: datetime
    total_breaches: int
    by_severity: dict[str, int]
    avg_detection_hours: float
    avg_notification_hours: float
    avg_containment_hours: float
    total_records_affected: int
    repeat_breach_count: int


@dataclass
class ConsentMetrics:
    period_start: datetime
    period_end: datetime
    total_consent_events: int
    opt_in_rate: float
    withdrawal_rate: float
    consent_rate_by_purpose: dict[str, float]
    cookie_acceptance_rate: float
    consent_freshness_pct: float


class PrivacyMetricsCollector:
    """
    Collect and compute privacy KPIs from operational data sources.
    """

    def __init__(self, dsar_repo, breach_repo, dpia_repo, training_repo, consent_repo):
        self.dsar_repo = dsar_repo
        self.breach_repo = breach_repo
        self.dpia_repo = dpia_repo
        self.training_repo = training_repo
        self.consent_repo = consent_repo

    def compute_dsar_metrics(
        self,
        start_date: datetime,
        end_date: datetime,
        regulatory_deadline_days: int = 30
    ) -> DSARMetrics:
        """Compute DSAR KPIs for a given period."""
        dsars = self.dsar_repo.get_by_period(start_date, end_date)

        completed = [d for d in dsars if d.status == "completed"]
        response_times = [
            (d.completed_at - d.received_at).days
            for d in completed
            if d.completed_at and d.received_at
        ]

        on_time = [
            d for d in completed
            if (d.completed_at - d.received_at).days <= regulatory_deadline_days
        ]

        automated = [d for d in completed if d.automated]
        open_dsars = [d for d in dsars if d.status in ("open", "in_progress")]

        by_type = {}
        for d in dsars:
            by_type[d.request_type] = by_type.get(d.request_type, 0) + 1

        return DSARMetrics(
            period_start=start_date,
            period_end=end_date,
            total_received=len(dsars),
            total_completed=len(completed),
            total_overdue=len(dsars) - len(on_time) - len(open_dsars),
            avg_response_days=statistics.mean(response_times) if response_times else 0,
            median_response_days=statistics.median(response_times) if response_times else 0,
            p95_response_days=(
                sorted(response_times)[int(len(response_times) * 0.95)]
                if len(response_times) >= 20 else
                max(response_times) if response_times else 0
            ),
            on_time_rate=len(on_time) / len(completed) if completed else 1.0,
            by_type=by_type,
            backlog=len(open_dsars),
            automated_rate=len(automated) / len(completed) if completed else 0,
        )

    def compute_breach_metrics(
        self,
        start_date: datetime,
        end_date: datetime
    ) -> BreachMetrics:
        """Compute breach KPIs for a given period."""
        breaches = self.breach_repo.get_by_period(start_date, end_date)

        by_severity = {}
        detection_hours = []
        notification_hours = []
        containment_hours = []
        total_records = 0

        for b in breaches:
            by_severity[b.severity] = by_severity.get(b.severity, 0) + 1
            total_records += b.records_affected or 0

            if b.detected_at and b.occurred_at:
                detection_hours.append(
                    (b.detected_at - b.occurred_at).total_seconds() / 3600
                )
            if b.notified_at and b.detected_at:
                notification_hours.append(
                    (b.notified_at - b.detected_at).total_seconds() / 3600
                )
            if b.contained_at and b.detected_at:
                containment_hours.append(
                    (b.contained_at - b.detected_at).total_seconds() / 3600
                )

        # Count repeat breaches (same root cause as previous)
        root_causes = [b.root_cause for b in breaches if b.root_cause]
        repeat_count = len(root_causes) - len(set(root_causes))

        return BreachMetrics(
            period_start=start_date,
            period_end=end_date,
            total_breaches=len(breaches),
            by_severity=by_severity,
            avg_detection_hours=statistics.mean(detection_hours) if detection_hours else 0,
            avg_notification_hours=statistics.mean(notification_hours) if notification_hours else 0,
            avg_containment_hours=statistics.mean(containment_hours) if containment_hours else 0,
            total_records_affected=total_records,
            repeat_breach_count=repeat_count,
        )

    def generate_executive_report(
        self,
        period_start: datetime,
        period_end: datetime
    ) -> dict:
        """
        Generate a comprehensive executive privacy report.
        Combines all KPI categories into a single report structure.
        """
        dsar = self.compute_dsar_metrics(period_start, period_end)
        breach = self.compute_breach_metrics(period_start, period_end)

        # Calculate overall privacy health score (0-100)
        health_components = []

        # DSAR component (25% weight)
        dsar_score = min(100, dsar.on_time_rate * 100)
        health_components.append(("DSAR Compliance", dsar_score, 0.25))

        # Breach component (30% weight)
        breach_score = 100 if breach.total_breaches == 0 else max(0, 100 - breach.total_breaches * 20)
        health_components.append(("Breach Prevention", breach_score, 0.30))

        # Response time component (20% weight)
        response_score = min(100, max(0, (30 - dsar.avg_response_days) / 30 * 100))
        health_components.append(("Response Timeliness", response_score, 0.20))

        # Detection speed component (25% weight)
        detection_score = min(100, max(0, (72 - breach.avg_detection_hours) / 72 * 100)) if breach.total_breaches > 0 else 100
        health_components.append(("Detection Speed", detection_score, 0.25))

        overall_score = sum(score * weight for _, score, weight in health_components)

        return {
            "period": {
                "start": period_start.isoformat(),
                "end": period_end.isoformat()
            },
            "overall_privacy_health_score": round(overall_score, 1),
            "health_rating": (
                "Excellent" if overall_score >= 90 else
                "Good" if overall_score >= 75 else
                "Needs Improvement" if overall_score >= 60 else
                "Critical"
            ),
            "components": {
                name: {"score": round(score, 1), "weight": f"{weight*100:.0f}%"}
                for name, score, weight in health_components
            },
            "dsar_summary": {
                "volume": dsar.total_received,
                "on_time_rate": f"{dsar.on_time_rate*100:.1f}%",
                "avg_response_days": round(dsar.avg_response_days, 1),
                "backlog": dsar.backlog
            },
            "breach_summary": {
                "count": breach.total_breaches,
                "severity_distribution": breach.by_severity,
                "avg_detection_hours": round(breach.avg_detection_hours, 1),
                "records_affected": breach.total_records_affected
            },
        }

Dashboard Visualization Patterns

Executive Summary View

PanelChart TypeMetrics DisplayedRefresh Rate
Privacy Health ScoreGauge (0-100)Overall weighted scoreDaily
DSAR Volume TrendLine chartMonthly DSAR count (12-month rolling)Daily
Breach TimelineEvent timelineBreaches by severity over timeReal-time
On-Time RateSingle stat with spark lineDSAR on-time completion %Daily
Training CoverageHorizontal barCompletion rate by departmentWeekly
Consent Rate TrendMulti-line chartOpt-in rate by purpose (6-month)Daily

Operational View

PanelChart TypeMetrics DisplayedRefresh Rate
Active DSARsTable with statusOpen requests with deadlinesHourly
DSAR AgingHistogramDistribution of open DSAR agesDaily
Response Time Box PlotBox plotResponse time distribution by monthDaily
Breach Detection FunnelFunnel chartDetection > Assessment > Notification > ResolutionPer event
DPIA PipelineKanbanDPIAs by status (draft, review, approved)Daily
Vendor Risk Heat MapHeat mapVendors by risk score and last assessment dateWeekly

Alerting Rules

AlertConditionSeverityRecipient
DSAR SLA WarningAny DSAR open > 21 daysWarningDSAR team
DSAR SLA BreachAny DSAR open > 28 daysCriticalCPO + Legal
Breach DetectedNew breach record createdCriticalIncident team + CPO
Training Compliance DropDepartment completion < 90%WarningHR + Privacy
Consent Rate AnomalyOpt-in rate drops > 20% in a weekWarningPrivacy + Marketing
DPIA OverdueDPIA review past scheduled dateWarningDPIA owner + CPO
Vendor Assessment DueHigh-risk vendor assessment > 11 months oldWarningProcurement + Privacy

Executive Reporting Template

Monthly Privacy Board Report Structure

  1. Privacy Health Score: Overall score with trend arrow
  2. Key Highlights: 3-5 bullet points on notable events
  3. DSAR Performance: Volume, on-time rate, top request types
  4. Breach Status: Count, severity, response times, lessons learned
  5. Compliance Posture: DPIA coverage, training rates, policy updates
  6. Risk Outlook: Emerging risks, regulatory developments, planned changes
  7. Resource Needs: Budget requests, headcount, tooling

References

  • IAPP Privacy Program Management Guide (CIPM Body of Knowledge)
  • NIST Privacy Framework — GV.MT Monitoring and Review
  • ISO/IEC 27701:2019 — Clause 9 (Performance Evaluation)
  • EDPB Guidelines on Data Breach Notification (WP250 rev.01)
  • Gartner Privacy Program Metrics Framework
  • ISACA Privacy Audit and Assurance Program

Privacy-Preserving Record Linkage

Overview

Privacy-Preserving Record Linkage (PPRL) enables two or more organizations to identify matching records across their datasets without revealing the underlying personal data to each other. This is critical for healthcare research, fraud detection, national statistics, and cross-organizational analytics where direct data sharing is prohibited by privacy regulations.

Approach Comparison

ApproachPrivacy LevelAccuracyScalabilityCommunication Cost
Bloom Filter EncodingHighGood (>95% F1)Very HighLow
Secure Hash MatchingVery HighHigh (exact match only)Very HighVery Low
Secure Multi-Party ComputationCryptographicVery HighMediumHigh
Trusted Third PartyDepends on TTPVery HighHighMedium
Differential Privacy LinkageFormally privateModerateHighLow

Bloom Filter-Based PPRL

How It Works

  1. Each organization encodes their quasi-identifiers (name, date of birth, address) into Bloom filters
  2. The Bloom filter encoding uses cryptographic hash functions with a shared secret key
  3. Encoded Bloom filters are compared using similarity metrics (Dice coefficient, Jaccard)
  4. Matching pairs above a threshold are identified as linked records
  5. Raw data is never exchanged — only Bloom filter bit arrays

Bloom Filter Encoding Implementation

"""
Privacy-preserving record linkage using Bloom filter encoding.
Implements the approach described by Schnell, Bachteler, and Reiher (2009).
"""

import hashlib
import hmac
import math
from typing import Optional
import numpy as np


class BloomFilterEncoder:
    """
    Encode string attributes into Bloom filters for privacy-preserving
    record linkage using cryptographic keyed hashing.
    """

    def __init__(
        self,
        filter_size: int = 1024,
        num_hash_functions: int = 30,
        ngram_size: int = 2,
        secret_key: bytes = b""
    ):
        """
        Args:
            filter_size: Number of bits in the Bloom filter
            num_hash_functions: Number of hash functions (k)
            ngram_size: Size of character n-grams (bigrams = 2)
            secret_key: Shared secret key for HMAC hashing
        """
        self.filter_size = filter_size
        self.num_hash_functions = num_hash_functions
        self.ngram_size = ngram_size
        self.secret_key = secret_key

    def _generate_ngrams(self, value: str) -> list[str]:
        """Generate character n-grams from a string value."""
        # Pad the string to handle edge characters
        padded = f"_{value}_"
        return [
            padded[i:i + self.ngram_size]
            for i in range(len(padded) - self.ngram_size + 1)
        ]

    def _hash_ngram(self, ngram: str, hash_index: int) -> int:
        """
        Hash an n-gram using HMAC with the shared key and hash index.
        Returns a bit position in the Bloom filter.
        """
        message = f"{hash_index}:{ngram}".encode("utf-8")
        digest = hmac.new(self.secret_key, message, hashlib.sha256).digest()
        # Convert first 4 bytes to integer and modulo by filter size
        position = int.from_bytes(digest[:4], byteorder="big") % self.filter_size
        return position

    def encode_value(self, value: str) -> np.ndarray:
        """
        Encode a single attribute value into a Bloom filter.

        Args:
            value: The string value to encode (e.g., a name)

        Returns:
            Numpy array of bits (0/1) representing the Bloom filter
        """
        bloom_filter = np.zeros(self.filter_size, dtype=np.uint8)

        # Normalize the input
        normalized = value.strip().lower()

        # Generate n-grams
        ngrams = self._generate_ngrams(normalized)

        # Hash each n-gram with each hash function
        for ngram in ngrams:
            for h in range(self.num_hash_functions):
                position = self._hash_ngram(ngram, h)
                bloom_filter[position] = 1

        return bloom_filter

    def encode_record(self, attributes: dict[str, str]) -> np.ndarray:
        """
        Encode multiple attributes into a single composite Bloom filter
        using Cryptographic Longterm Key (CLK) approach.

        Args:
            attributes: Dict mapping attribute names to values
                       e.g., {"first_name": "John", "last_name": "Smith", "dob": "1990-01-15"}

        Returns:
            Composite Bloom filter as numpy array
        """
        composite = np.zeros(self.filter_size, dtype=np.uint8)

        for attr_name, attr_value in attributes.items():
            if attr_value:
                # Use attribute name as additional salt
                salted_key = self.secret_key + attr_name.encode("utf-8")
                encoder = BloomFilterEncoder(
                    filter_size=self.filter_size,
                    num_hash_functions=self.num_hash_functions,
                    ngram_size=self.ngram_size,
                    secret_key=salted_key
                )
                attr_bf = encoder.encode_value(attr_value)
                composite = np.bitwise_or(composite, attr_bf)

        return composite


class BloomFilterMatcher:
    """
    Compare Bloom filter-encoded records to find matching pairs.
    """

    @staticmethod
    def dice_coefficient(bf1: np.ndarray, bf2: np.ndarray) -> float:
        """
        Calculate the Dice coefficient between two Bloom filters.

        Dice = 2 * |bf1 AND bf2| / (|bf1| + |bf2|)

        Returns a similarity score between 0 and 1.
        """
        intersection = np.sum(np.bitwise_and(bf1, bf2))
        cardinality_sum = np.sum(bf1) + np.sum(bf2)

        if cardinality_sum == 0:
            return 0.0

        return 2.0 * intersection / cardinality_sum

    @staticmethod
    def jaccard_similarity(bf1: np.ndarray, bf2: np.ndarray) -> float:
        """
        Calculate the Jaccard similarity between two Bloom filters.

        Jaccard = |bf1 AND bf2| / |bf1 OR bf2|
        """
        intersection = np.sum(np.bitwise_and(bf1, bf2))
        union = np.sum(np.bitwise_or(bf1, bf2))

        if union == 0:
            return 0.0

        return intersection / union

    def find_matches(
        self,
        encodings_a: list[tuple[str, np.ndarray]],
        encodings_b: list[tuple[str, np.ndarray]],
        threshold: float = 0.8,
        similarity_metric: str = "dice"
    ) -> list[tuple[str, str, float]]:
        """
        Find matching record pairs between two encoded datasets.

        Args:
            encodings_a: List of (record_id, bloom_filter) from organization A
            encodings_b: List of (record_id, bloom_filter) from organization B
            threshold: Minimum similarity score for a match
            similarity_metric: "dice" or "jaccard"

        Returns:
            List of (id_a, id_b, similarity_score) for matching pairs
        """
        metric_fn = (
            self.dice_coefficient if similarity_metric == "dice"
            else self.jaccard_similarity
        )

        matches = []

        for id_a, bf_a in encodings_a:
            best_score = 0.0
            best_id_b = None

            for id_b, bf_b in encodings_b:
                score = metric_fn(bf_a, bf_b)
                if score > best_score:
                    best_score = score
                    best_id_b = id_b

            if best_score >= threshold and best_id_b is not None:
                matches.append((id_a, best_id_b, best_score))

        return matches

Secure Hash Matching

For exact matching scenarios where approximate matching is not needed.

"""
Secure hash-based record linkage for exact matching.
Uses keyed HMAC to prevent rainbow table attacks.
"""

import hashlib
import hmac


class SecureHashLinker:
    """
    Link records across organizations using keyed hash matching.
    Suitable for exact match on standardized identifiers.
    """

    def __init__(self, shared_key: bytes):
        self.shared_key = shared_key

    def hash_identifier(self, *fields: str) -> str:
        """
        Create a keyed hash of concatenated identifier fields.

        Args:
            fields: Identifier fields in standardized order
                    e.g., ("john", "smith", "19900115")

        Returns:
            Hex-encoded HMAC-SHA256 hash
        """
        # Normalize and concatenate fields
        normalized = "|".join(f.strip().lower() for f in fields)

        # Generate keyed hash
        digest = hmac.new(
            self.shared_key,
            normalized.encode("utf-8"),
            hashlib.sha256
        ).hexdigest()

        return digest

    def hash_dataset(
        self,
        records: list[dict],
        id_field: str,
        linkage_fields: list[str]
    ) -> dict[str, str]:
        """
        Hash all records in a dataset for linkage.

        Returns mapping of hash -> record_id.
        """
        hash_map = {}

        for record in records:
            fields = [str(record.get(f, "")) for f in linkage_fields]
            record_hash = self.hash_identifier(*fields)
            hash_map[record_hash] = record[id_field]

        return hash_map

    @staticmethod
    def find_exact_matches(
        hashes_a: dict[str, str],
        hashes_b: dict[str, str]
    ) -> list[tuple[str, str]]:
        """
        Find exact matches between two hash maps.

        Returns list of (id_a, id_b) pairs.
        """
        common_hashes = set(hashes_a.keys()) & set(hashes_b.keys())
        return [(hashes_a[h], hashes_b[h]) for h in common_hashes]

Threshold Tuning

Methodology

Threshold RangePrecisionRecallUse Case
0.90 - 1.00Very HighLowHigh-stakes decisions (medical records)
0.80 - 0.90HighMediumStandard record linkage
0.70 - 0.80MediumHighExploratory analysis, broad matching
0.60 - 0.70LowVery HighCandidate generation (with manual review)

Optimal Threshold Selection Process

  1. Generate labeled pairs: Create a sample of known matches and non-matches
  2. Compute similarity scores: Calculate Dice/Jaccard for all pairs in the sample
  3. Plot precision-recall curve: Sweep threshold from 0 to 1
  4. Select threshold: Choose based on acceptable false positive rate for the use case
  5. Validate: Test on held-out labeled data

False Positive Management

"""
Post-linkage false positive reduction through multi-stage verification.
"""


class FalsePositiveManager:
    """
    Reduce false positive matches through additional verification stages
    without revealing raw data between parties.
    """

    def __init__(self, primary_threshold: float = 0.8, verification_threshold: float = 0.9):
        self.primary_threshold = primary_threshold
        self.verification_threshold = verification_threshold

    def multi_field_verification(
        self,
        candidate_pairs: list[tuple[str, str, float]],
        secondary_encodings_a: dict[str, dict[str, np.ndarray]],
        secondary_encodings_b: dict[str, dict[str, np.ndarray]],
        matcher: BloomFilterMatcher
    ) -> list[tuple[str, str, float, bool]]:
        """
        Verify candidate matches using additional encoded fields.

        Args:
            candidate_pairs: (id_a, id_b, primary_score) from initial matching
            secondary_encodings_a: {record_id: {field: bloom_filter}} from org A
            secondary_encodings_b: {record_id: {field: bloom_filter}} from org B

        Returns:
            (id_a, id_b, composite_score, verified) for each candidate
        """
        verified_pairs = []

        for id_a, id_b, primary_score in candidate_pairs:
            secondary_scores = []

            fields_a = secondary_encodings_a.get(id_a, {})
            fields_b = secondary_encodings_b.get(id_b, {})

            common_fields = set(fields_a.keys()) & set(fields_b.keys())

            for field_name in common_fields:
                score = matcher.dice_coefficient(
                    fields_a[field_name],
                    fields_b[field_name]
                )
                secondary_scores.append(score)

            if secondary_scores:
                avg_secondary = sum(secondary_scores) / len(secondary_scores)
                composite = 0.6 * primary_score + 0.4 * avg_secondary
                verified = composite >= self.verification_threshold
            else:
                composite = primary_score
                verified = primary_score >= self.verification_threshold

            verified_pairs.append((id_a, id_b, composite, verified))

        return verified_pairs

Security Considerations

AttackDescriptionMitigation
Frequency analysisAnalyzing bit patterns to infer common valuesUse composite Bloom filters (CLK), add noise bits
Dictionary attackPre-computing Bloom filters for known valuesUse strong shared secret keys, rotate keys periodically
Bit pattern cryptanalysisExploiting structure in Bloom filter bit patternsSufficient filter size (>= 1024), adequate hash functions (>= 20)
Collision exploitationDeliberately crafting records to match target hashesHMAC-based hashing, input validation

References

  • Schnell, R., Bachteler, T., and Reiher, J. "Privacy-Preserving Record Linkage Using Bloom Filters." BMC Medical Informatics and Decision Making, 9(1):41, 2009.
  • Vatsalan, D., Christen, P., and Verykios, V.S. "A Taxonomy of Privacy-Preserving Record Linkage Techniques." Information Systems, 38(6):946-969, 2013.
  • Randall, S.M. et al. "Privacy-Preserving Record Linkage on Large Real World Datasets." Journal of Biomedical Informatics, 50:205-212, 2014.
  • AIHW (Australian Institute of Health and Welfare) PPRL Implementation Guide
  • Christen, P. "Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection." Springer, 2012.

Purpose-Based Access Control (PBAC) Architecture

Overview

Purpose-Based Access Control (PBAC) extends traditional access control models (RBAC, ABAC) by adding purpose as a mandatory dimension in every access decision. Under PBAC, data access is granted only when the requester can demonstrate a valid, pre-authorized purpose that aligns with the basis under which the data was collected. This directly implements GDPR Article 5(1)(b) purpose limitation, CCPA purpose restrictions, and similar requirements across global privacy regulations.

PBAC vs Traditional Access Control

DimensionRBACABACPBAC
Access decision based onRole membershipAttributes (user, resource, environment)Purpose + attributes
Answers the question"Who can access?""Under what conditions?""Why is this access needed?"
Purpose enforcementNone (implicit)Possible as attributeCore requirement
Audit trailWho accessed whatWho, what, when, whereWho, what, when, where, WHY
Privacy alignmentLowMediumHigh
Consent integrationNonePossibleNative

Purpose Ontology

Purpose Hierarchy

Root Purpose
├── Service Delivery
│   ├── Order Fulfillment
│   │   ├── Payment Processing
│   │   ├── Shipping Logistics
│   │   └── Order Confirmation
│   ├── Account Management
│   │   ├── Account Creation
│   │   ├── Account Maintenance
│   │   └── Account Recovery
│   └── Customer Support
│       ├── Ticket Resolution
│       └── Escalation Handling
├── Legal Compliance
│   ├── Regulatory Reporting
│   │   ├── Tax Reporting
│   │   ├── AML Compliance
│   │   └── Regulatory Audit
│   ├── Data Subject Rights
│   │   ├── Access Request
│   │   ├── Deletion Request
│   │   ├── Portability Request
│   │   └── Rectification Request
│   └── Litigation Hold
├── Analytics
│   ├── Product Analytics
│   │   ├── Feature Usage Analysis
│   │   └── UX Research
│   ├── Business Intelligence
│   │   ├── Revenue Reporting
│   │   └── Forecasting
│   └── Aggregate Reporting
│       ├── Board Reporting
│       └── Investor Reporting
├── Marketing
│   ├── Direct Marketing
│   │   ├── Email Campaigns
│   │   └── Personalized Offers
│   ├── Market Research
│   │   └── Survey Analysis
│   └── Advertising
│       ├── Audience Segmentation
│       └── Campaign Measurement
└── Security
    ├── Fraud Detection
    ├── Incident Investigation
    └── Access Auditing

Purpose Definition Schema

from dataclasses import dataclass, field
from enum import Enum
from datetime import datetime


class LegalBasis(Enum):
    CONSENT = "consent"
    CONTRACT = "contract"
    LEGAL_OBLIGATION = "legal_obligation"
    VITAL_INTEREST = "vital_interest"
    PUBLIC_INTEREST = "public_interest"
    LEGITIMATE_INTEREST = "legitimate_interest"


class PurposeStatus(Enum):
    ACTIVE = "active"
    DEPRECATED = "deprecated"
    PENDING_APPROVAL = "pending_approval"


@dataclass
class Purpose:
    purpose_id: str
    name: str
    description: str
    parent_purpose_id: str | None
    legal_bases: list[LegalBasis]
    data_categories_allowed: list[str]
    retention_period_days: int
    requires_consent: bool
    status: PurposeStatus
    owner: str
    created_date: datetime
    review_date: datetime
    jurisdictions: list[str] = field(default_factory=lambda: ["global"])
    compatible_purposes: list[str] = field(default_factory=list)
    incompatible_purposes: list[str] = field(default_factory=list)

Policy Engine Architecture

System Design

Access Request
    |
    v
+-------------------+
| Request Parser    |
| - Identity (who)  |
| - Resource (what) |
| - Action (how)    |
| - Purpose (why)   |
| - Context (when/  |
|   where)          |
+-------------------+
    |
    v
+-------------------+     +--------------------+
| Purpose Validator |---->| Purpose Registry   |
| - Purpose exists  |     | - Ontology tree    |
| - Purpose active  |     | - Legal bases      |
| - Purpose applies |     | - Data categories  |
|   to data category|     +--------------------+
+-------------------+
    |
    v
+-------------------+     +--------------------+
| Consent Checker   |---->| Consent Store      |
| - Consent exists  |     | - Data subject     |
| - Consent active  |     |   preferences      |
| - Scope matches   |     | - Consent records  |
+-------------------+     +--------------------+
    |
    v
+-------------------+     +--------------------+
| ABAC Policy       |---->| Policy Repository  |
| Evaluator         |     | - XACML policies   |
| - Role check      |     | - Purpose-data     |
| - Attribute check  |     |   mappings        |
| - Environment check|     +--------------------+
+-------------------+
    |
    v
+-------------------+
| Decision Engine   |
| - PERMIT / DENY   |
| - Obligations     |
| - Advice          |
+-------------------+
    |
    +------+-------+
    |              |
    v              v
PERMIT          DENY
+ Audit Log     + Audit Log
+ Obligations   + Denial Reason

Policy Engine Implementation

"""
Purpose-Based Access Control policy engine.
Evaluates access requests against purpose definitions,
consent records, and attribute-based policies.
"""

from dataclasses import dataclass
from datetime import datetime
from enum import Enum
from typing import Optional


class Decision(Enum):
    PERMIT = "PERMIT"
    DENY = "DENY"


@dataclass
class AccessRequest:
    requester_id: str
    requester_roles: list[str]
    resource_id: str
    data_category: str
    action: str  # read, write, delete, export
    purpose_id: str
    data_subject_id: Optional[str]
    timestamp: datetime
    source_ip: str
    justification: str


@dataclass
class AccessDecision:
    decision: Decision
    reason: str
    obligations: list[str]
    request: AccessRequest
    evaluated_at: datetime
    policy_version: str


class PurposeBasedAccessController:
    """
    Core PBAC engine that evaluates access requests against
    purpose definitions, consent records, and organizational policies.
    """

    def __init__(
        self,
        purpose_registry,
        consent_store,
        policy_repository,
        audit_logger
    ):
        self.purpose_registry = purpose_registry
        self.consent_store = consent_store
        self.policy_repository = policy_repository
        self.audit_logger = audit_logger

    def evaluate(self, request: AccessRequest) -> AccessDecision:
        """
        Evaluate an access request through the PBAC pipeline.

        Pipeline stages:
        1. Purpose validation
        2. Consent verification
        3. Attribute-based policy evaluation
        4. Decision with obligations
        """
        now = datetime.utcnow()

        # Stage 1: Validate purpose
        purpose = self.purpose_registry.get_purpose(request.purpose_id)
        if purpose is None:
            return self._deny(request, "Purpose not found in registry", now)

        if purpose.status != PurposeStatus.ACTIVE:
            return self._deny(request, f"Purpose is {purpose.status.value}", now)

        if request.data_category not in purpose.data_categories_allowed:
            return self._deny(
                request,
                f"Data category '{request.data_category}' not allowed for purpose '{purpose.name}'",
                now
            )

        # Stage 2: Verify consent (if required)
        if purpose.requires_consent and request.data_subject_id:
            consent = self.consent_store.get_active_consent(
                data_subject_id=request.data_subject_id,
                purpose_id=request.purpose_id
            )
            if consent is None:
                return self._deny(
                    request,
                    f"No active consent for purpose '{purpose.name}' from data subject",
                    now
                )

        # Stage 3: Evaluate attribute-based policies
        policy_result = self.policy_repository.evaluate(
            roles=request.requester_roles,
            resource=request.resource_id,
            action=request.action,
            purpose=request.purpose_id,
            data_category=request.data_category,
            context={"source_ip": request.source_ip, "time": now}
        )

        if not policy_result.permitted:
            return self._deny(request, policy_result.denial_reason, now)

        # Stage 4: Permit with obligations
        obligations = self._determine_obligations(purpose, request)
        decision = AccessDecision(
            decision=Decision.PERMIT,
            reason=f"Access permitted for purpose '{purpose.name}'",
            obligations=obligations,
            request=request,
            evaluated_at=now,
            policy_version=self.policy_repository.version
        )

        self.audit_logger.log_access_decision(decision)
        return decision

    def _deny(
        self, request: AccessRequest, reason: str, timestamp: datetime
    ) -> AccessDecision:
        """Create a DENY decision and log it."""
        decision = AccessDecision(
            decision=Decision.DENY,
            reason=reason,
            obligations=[],
            request=request,
            evaluated_at=timestamp,
            policy_version=self.policy_repository.version
        )
        self.audit_logger.log_access_decision(decision)
        return decision

    def _determine_obligations(self, purpose, request: AccessRequest) -> list[str]:
        """Determine post-access obligations based on purpose and context."""
        obligations = []

        if request.action == "export":
            obligations.append("LOG_DATA_EXPORT")
            obligations.append("APPLY_ENCRYPTION_TO_EXPORT")

        if request.action == "read" and request.data_category in [
            "health_data", "financial_data", "biometric_data"
        ]:
            obligations.append("MASK_SENSITIVE_FIELDS")

        obligations.append(f"ENFORCE_RETENTION_{purpose.retention_period_days}_DAYS")
        obligations.append("LOG_PURPOSE_USAGE")

        return obligations

Purpose Verification at Query Time

SQL Proxy with Purpose Enforcement

"""
SQL proxy that intercepts database queries and enforces
purpose-based access control at the query level.
"""

import re
from datetime import datetime


class PurposeAwareSQLProxy:
    """
    Intercepts SQL queries, validates purpose, and rewrites
    queries to enforce data category restrictions.
    """

    def __init__(self, pbac_controller, column_category_map: dict):
        self.pbac = pbac_controller
        self.column_category_map = column_category_map

    def execute_query(
        self,
        sql: str,
        requester_id: str,
        requester_roles: list[str],
        purpose_id: str,
        justification: str
    ) -> tuple[bool, object]:
        """
        Execute a SQL query with purpose verification.

        Returns (success, result_or_error).
        """
        # Parse referenced columns/tables from SQL
        referenced_columns = self._extract_columns(sql)
        referenced_categories = set()

        for col in referenced_columns:
            category = self.column_category_map.get(col, "general")
            referenced_categories.add(category)

        # Validate access for each data category
        denied_categories = []
        for category in referenced_categories:
            request = AccessRequest(
                requester_id=requester_id,
                requester_roles=requester_roles,
                resource_id="database",
                data_category=category,
                action="read",
                purpose_id=purpose_id,
                data_subject_id=None,
                timestamp=datetime.utcnow(),
                source_ip="internal",
                justification=justification
            )

            decision = self.pbac.evaluate(request)
            if decision.decision == Decision.DENY:
                denied_categories.append((category, decision.reason))

        if denied_categories:
            reasons = "; ".join(
                f"{cat}: {reason}" for cat, reason in denied_categories
            )
            return (False, f"Access denied for categories: {reasons}")

        # Rewrite query to mask unauthorized columns if needed
        rewritten_sql = self._apply_column_masking(sql, purpose_id)

        # Execute the rewritten query (actual DB execution would go here)
        return (True, rewritten_sql)

    def _extract_columns(self, sql: str) -> list[str]:
        """Extract column references from SQL (simplified parser)."""
        select_pattern = re.compile(
            r"SELECT\s+(.+?)\s+FROM", re.IGNORECASE | re.DOTALL
        )
        match = select_pattern.search(sql)
        if not match:
            return []

        columns_str = match.group(1)
        if columns_str.strip() == "*":
            return list(self.column_category_map.keys())

        columns = [
            col.strip().split(".")[-1].split(" ")[0]
            for col in columns_str.split(",")
        ]
        return columns

    def _apply_column_masking(self, sql: str, purpose_id: str) -> str:
        """Rewrite query to mask columns not needed for the given purpose."""
        return sql  # Production implementation would rewrite SELECT columns

Audit Logging for PBAC

Audit Record Schema

FieldTypeDescription
audit_idUUIDUnique audit record identifier
timestampDateTimeWhen the access decision was made
requester_idStringIdentity of the requester
requester_rolesArray[String]Roles held by requester
resource_idStringResource being accessed
data_categoryStringCategory of data being accessed
actionStringType of access (read/write/delete/export)
purpose_idStringStated purpose for access
purpose_nameStringHuman-readable purpose name
decisionEnumPERMIT or DENY
denial_reasonStringReason for denial (if denied)
obligationsArray[String]Post-access obligations applied
justificationStringFree-text justification from requester
policy_versionStringVersion of policy used for decision

Integration with Existing IAM

PBAC Overlay Architecture

Existing IAM (RBAC/ABAC)
    |
    v
+---------------------------+
| PBAC Middleware Layer      |
| (intercepts access calls) |
+---------------------------+
    |
    +-- Purpose verification
    +-- Consent validation
    +-- Data category mapping
    +-- Obligation enforcement
    +-- Enhanced audit logging
    |
    v
Resource (Database / API / File System)

References

  • Byun, J. and Li, N. "Purpose Based Access Control of Complex Data for Privacy Protection." ACM SACMAT, 2008.
  • GDPR Article 5(1)(b) — Purpose Limitation Principle
  • ISO/IEC 29101:2018 — Privacy Architecture Framework
  • NIST SP 800-162 — Guide to Attribute Based Access Control Definition and Considerations
  • Ni, Q. et al. "Privacy-Aware Role-Based Access Control." ACM TOPS, 2010.

Sponsors

Details

Marketplace
mukul975/Privacy-Data-Protection-Skills
Source
plugins/privacy-engineering-skills
Author
Mahipal
Category
Engineering
License
Apache-2.0
Keywords
privacy-engineering, differential-privacy, pii-detection, nist-privacy-framework

More from mukul975/Privacy-Data-Protection-Skills

All 21