Data privacy when everything goes to a model: the blind spots your legal team isn't catching

Every large organization is now feeding data to models. Customer records, internal memos, financial forecasts, HR files, legal drafts. The volume going into LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition →-powered tools has grown faster than the governance structures designed to control it. Procurement teams sign enterprise agreements with OpenAI, Anthropic, or Microsoft, legal reviews the data processing addendum, and everyone assumes the problem is handled. It mostly isn't.

The consensus view: treat LLMs like any other SaaS vendor

The standard position in corporate AI governance circles runs roughly as follows. LLMs are software services. You apply the same data classification rules you'd apply to Salesforce or Workday. You check whether the vendor trains on your inputs (most enterprise contracts now say they don't). You make sure the contract covers GDPR Article 28 processor requirements, or CCPA obligations if you're operating in California. You run a data protection impact assessment. You get sign-off from the DPO. Done.

This view isn't wrong. Those steps matter. Enterprise agreements with OpenAI (via Azure OpenAI Service) and Anthropic do contain substantive commitments around data retention and training exclusions. Microsoft's published documentation on its enterprise data boundary, updated through 2025, gives IT teams a reasonable basis for scoping what stays where. The DPIA process, when done properly, forces business units to articulate what data they're actually sending and why.

The compliance-first framing has real value. Without it, organizations end up with shadow AI usage, employees pasting sensitive data into consumer-grade ChatGPT accounts because no approved tool exists. That's a documented, concrete risk. The consensus addresses it.

Where the consensus breaks down

The problem is that the compliance checklist treats privacy as a static property of data, when the real exposure comes from what models do with data in combination and over time.

Three blind spots stand out.

First,inference and reconstruction. When a model receives a prompt containing partial information, it can reconstruct sensitive attributes that were never explicitly shared. A 2023 paper from researchers at Google DeepMind and Carnegie Mellon (published in a peer-reviewed NLP venue, not vendor marketing) demonstrated that LLMs can infer demographic attributes, health conditions, and financial status from conversational text with accuracy that would surprise most privacy officers. The implication: stripping direct identifiers from a dataset before feeding it to a model is not the same as anonymizing it. Many DPIAs are still written as if it were.

Second, the retrieval-augmented generation problem. RAG architectures, now standard in enterprise deployments, connect models to internal knowledge bases at query time. The access control logic that governs which documents a user can retrieve through the RAG system is often less rigorous than the access controls on the underlying document repository. A junior analyst asking a natural language question may surface content from HR policy documents, M&A planning files, or legal opinions that they would never have found through the document management system itself. The model doesn't enforce need-to-know; it retrieves what's relevant.

Third, prompt logs. Most enterprise LLM deployments log prompts for quality, safety, or debugging purposes. Those logs accumulate a detailed record of what employees are thinking about, what customers are asking, and what problems the business is actually trying to solve. The data governancedata governanceData governance is the set of policies, roles, and processes that ensure data is accurate, secure, well-defined, and used responsibly across an organization.View full definition → applied to those logs is typically far weaker than the governance applied to the primary data sources. A breach of the prompt log database is potentially more revealing than a breach of the CRMCRMCustomer Relationship Management: software and strategy to manage and analyse customer interactions throughout their lifecycle.View full definition →.

The legal frameworks aren't keeping pace. The EU AI Act, fully applicable from August 2026, creates obligations around high-risk AI systems and transparency, but its provisions on privacy interact awkwardly with GDPR rather than replacing or clarifying it. Organizations are now managing two overlapping regulatory regimes with different risk concepts, different timelines, and different enforcement bodies. The compliance-as-checkbox approach doesn't handle that ambiguity well.

There's also a subtler issue that rarely gets discussed in governance meetings: organizational incentives. The teams building and deploying LLM tools are rewarded for speed and capability. The teams responsible for privacy are rewarded for avoiding incidents. Those incentives diverge, and in most organizations the builders have more resources, more executive attention, and faster cycle times than the governance function. The DPIA process becomes a formality reviewed after the product is already in use.

What a sharp operator should actually do

Governance frameworks built for SaaS need structural modifications to work for LLMs, not just additional checklist items.

On architecture: treat the prompt log as regulated data from day one. Apply the same retention limits, access controls, and breach notification triggers you'd apply to a database containing personal health information. Most organizations currently treat logs as operational data with weak controls. That needs to change before an incident makes it change.

On RAG deployments: mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → the access control gap explicitly. Before deploying any RAG system, run a structured test where users at different permission levels query the system and document what they can retrieve. Compare that against what they're authorized to see through conventional access controls. The gap is usually larger than anyone expected.

On inference risk: stop treating data minimization as a binary. The question isn't whether you've removed names and email addresses. The question is whether the residual data, in combination with what the model already knows from pretraining, can reconstruct protected attributes. That requires input from someone who understands model behavior, not just from a legal team applying a data classification rubric.

On vendor relationships: when you're working with a vendor like Microsoft (whose Azure OpenAI Service is now the most common enterprise deployment path), read the data boundary documentation yourself rather than relying on the vendor's compliance summary. The details on what telemetry flows where, and under what conditions Microsoft employees can access prompt content for safety review, are in the technical documentation. They're worth understanding. Treat vendor-provided compliance summaries as a starting point, not a conclusion.

The organizations that will manage this well are the ones that close the loop between their model deployment teams and their privacy function before deployment, not after the first user complaint or regulatory inquiry.