+150 XP

SaaS · Consent, purpose limitation, and the AI feature trap

In August 2023, Zoom updated its terms of service in a way that suggested customer content could train AI models, then reversed course within days after a visible user backlash. The episode became a textbook case. A SaaS company had a mountain of customer data, a new AI feature to build, and no clean legal path to use one for the other. That gap between "we have the data" and "we may use the data" is where many AI copilot projects quietly break the law.

The trap, defined

Support tickets, chat logs, and usage data are usually collected for one purpose: running the product and helping the customer. That is the purpose you told users about when they signed up.

Training an AI copilot is a different purpose. Under the GDPR (General Data Protection Regulation, the EU's core privacy law), this triggers purpose limitation. The principle says personal data collected for one specified purpose cannot be reused for an incompatible new purpose without a fresh legal basis (see GDPR Article 5(1)(b)).

Support tickets are especially risky because they often contain:

  • Names, emails, account details (personal data by default)
  • Pasted screenshots or logs with customer secrets, API keys, or end-user PII (personally identifiable information)
  • Sensitive complaints (health, financial, HR issues) if your SaaS serves those verticals

Feeding this corpus into a large language model (LLM) fine-tuning job is a new processing purpose. So is using it for retrieval-augmented generation (RAG, where a model pulls live snippets from a database to answer questions) in a way that surfaces one customer's data to another. If your original privacy notice said "to provide customer support," you likely do not have a basis to also say "to train our AI features."

Why this is not only an EU problem

US companies often assume this is a European headache. It is not.

  • The FTC (Federal Trade Commission) warned in February 2024 that companies quietly changing their terms to allow AI training on previously collected data may be engaging in unfair or deceptive practices under Section 5 of the FTC Act. Read the post: AI (and other) Companies: Quietly Changing Your Terms of Service Could Be Unfair or Deceptive.
  • California's CCPA/CPRA (California Consumer Privacy Act, amended by the California Privacy Rights Act) gives users a right to limit use of sensitive personal information and requires disclosure of new processing purposes. Its rules on automated decision-making technology, finalized in 2025, add further obligations for some AI uses.
  • Sector rules stack on top: HIPAA (health data, US) and GLBA (financial data, US) restrict secondary use regardless of AI involvement.

The common thread across the EU, the US, and most other frameworks: the problem is silent repurposing of data collected under a narrower promise. AI is simply the most common reason companies now try it.

What "purpose limitation" actually requires

Two lawful paths exist once you want to reuse data for a new purpose.

  1. Compatibility test: GDPR Article 6(4) allows reuse if the new purpose is "compatible" with the original one, weighing context, reasonable user expectations, and safeguards. Training a model on aggregated, anonymized ticket categories to improve routing might pass. Training a generative copilot that could regurgitate a user's exact complaint text to another customer almost certainly does not.
  2. Fresh consent or another lawful basis: If the use is not compatible, you need a new basis. In practice that means specific, informed consent, or a documented legitimate interest assessment (LIA).

The EU regulators' main reference here is EDPB Opinion 28/2024 (December 2024). It confirms legitimate interest can work for AI development, but only after a real three-step test (legitimate purpose, necessity, balancing) that weighs what users would reasonably expect. Meta relied on this route when it resumed training on public EU posts in May 2025, paired with an opt-out. Note that the European Commission's Digital Omnibus proposal (November 2025) would spell out legitimate interest for AI training in the GDPR text; check its status before relying on it, since a proposal is not law.

Anonymization matters but is often overstated. True anonymization (irreversible, no re-identification possible) takes data outside the GDPR. Pseudonymization (replacing identifiers with tokens) does not, because pseudonymized data is still personal data if re-identification is feasible. The same EDPB opinion adds a twist: a trained model is only "anonymous" if personal data cannot be extracted from it with reasonable effort, so memorization risk is a legal question too.

Building the re-permissioning workflow

When product wants to ship an AI feature on top of existing data, run this sequence before a single training job starts.

Step 1: Data mapping. Identify every dataset the feature will touch (tickets, chat transcripts, usage logs) and its original stated purpose. A data inventory or a Records of Processing Activities (RoPA, a GDPR-mandated log of what data you process and why) makes this traceable.

Step 2: Compatibility assessment. Document, in writing, whether the AI use case is compatible with the original purpose. The UK Information Commissioner's Office (ICO) publishes practical guidance worth adapting: ICO guidance on purpose limitation.

Step 3: Choose the basis.

  • Compatible and low-risk (aggregated analytics, for example): proceed, with a Data Protection Impact Assessment (DPIA) if risk is non-trivial.
  • Incompatible or high-risk (raw ticket text into an LLM): trigger re-permissioning, or document an LIA strong enough to survive the EDPB test.

Step 4: Re-permissioning campaign. This is the operational core:

  • Update the privacy notice with a plain-language description of the AI use case.
  • Present an explicit, granular opt-in, not a bundled "I agree to updated terms" checkbox. Regulators increasingly reject bundled consent as not "freely given."
  • Offer a real opt-out that does not degrade core product function. GDPR requires consent to be as easy to withdraw as to give.
  • Log consent with timestamp, notice version shown, and mechanism, and keep the log for audits.

Step 5: Technical enforcement. Consent decisions must reach the data pipeline. A common failure: legal approves an opt-in flow, but engineering trains on the whole ticket table because no flag propagates downstream.

A minimal schema fix looks like this:

sql
-- ticket table gains a consent flag that ETL/training jobs must filter on
ALTER TABLE support_tickets
ADD COLUMN ai_training_consent BOOLEAN DEFAULT FALSE;

-- training extraction query respects the flag
SELECT ticket_id, body_text
FROM support_tickets
WHERE ai_training_consent = TRUE
  AND anonymization_status = 'completed';

Without an enforced filter like this, consent exists in a policy document and nowhere else.

Audits to run before and after launch

  • Pre-launch DPIA: mandatory under GDPR when processing is "likely to result in high risk," which training generative models on user content typically is.
  • Data minimization check: does the model need raw ticket text, or would summaries and structured fields do?
  • Leakage test: prompt the trained model to see if it reproduces verbatim customer text (a known LLM memorization risk, and now part of the EU anonymity analysis).
  • Consent coverage audit: a quarterly check that the share of training data with valid, current consent matches what compliance believes it to be.
  • Vendor flow-down: if a third-party AI vendor (OpenAI, Anthropic, Google) processes the data, verify your Data Processing Agreement (DPA) bars them from training their own models on your customers' data.

Knowledge check

1. A SaaS company collected support tickets under a privacy notice that says data is used 'to provide customer support.' It now wants to use that same data to fine-tune an AI copilot. What is the core legal problem under GDPR's purpose limitation principle?

2. Why are support tickets and chat logs described as especially risky source material for AI training, compared to other operational data?

3. A product team argues: 'We're not fine-tuning a model, we're just using RAG to let the copilot pull live snippets from our support ticket database to answer questions.' From a purpose limitation standpoint, why doesn't this framing avoid the legal issue?

MULTIPLE CHOICE

4. Select ALL correct answers about why the 'we have the data' vs. 'we may use the data' gap matters for AI copilot projects.

Select all the correct answers.

MULTIPLE CHOICE

5. Select ALL correct answers about why US-based SaaS companies should not assume purpose limitation concerns are 'Europe-only.'

Select all the correct answers.

The business tension, honestly stated

Re-permissioning is friction, and friction shrinks the training data pool. Product teams push back: "If we ask again, adoption drops and the copilot gets worse." That tradeoff is real, but so is the enforcement record. Italy's Garante fined OpenAI 15 million euros in December 2024, partly for training ChatGPT without an adequate legal basis. The Dutch regulator fined Clearview AI 30.5 million euros in 2024 for building its database from scraped photos. Meta paused its EU AI training plans in 2024 after pressure from the Irish regulator and only resumed once it had a documented basis and an opt-out.

A better framing for product teams: build re-permissioning into the feature announcement. "We're launching an AI assistant trained on support patterns. Here's exactly what it uses, and here's your control." That turns a compliance step into a trust signal, which matters in enterprise SaaS deals where procurement teams now routinely ask about AI training data provenance.

🎬 [VIDEO: "GDPR and AI: Purpose Limitation Explained" - youtube.com - search for IAPP (International Association of Privacy Professionals) or ICO channel explainers on purpose limitation and AI training data, useful for a visual walkthrough of the compatibility test]

Key Takeaways

  • Purpose limitation (GDPR Article 5(1)(b)) means data collected for support cannot automatically be reused to train AI features. This is a live enforcement area in the EU and a stated FTC concern in the US.
  • Legitimate interest can justify AI training in the EU, but only with a documented three-step test along the lines of EDPB Opinion 28/2024. Treat the Digital Omnibus changes as pending until adopted.
  • Anonymization removes the legal problem only if it is irreversible, and a model that can leak personal data is not anonymous. Pseudonymized data is still personal data.
  • The workflow has five steps: data mapping, compatibility assessment, choosing a lawful basis, an explicit opt-in campaign, and consent flags enforced in the actual pipeline.
  • Run a DPIA before training, test for verbatim memorization after, and audit consent coverage every quarter.
  • Present re-permissioning as a trust signal in the launch, since enterprise buyers now check AI training data provenance during procurement.