Piloting AI without risking client trust or confidentiality

The phased pilot structure

Phase 0: Anonymized past matters only

Before any live client data is touched, build a test set from closed, anonymized matters, engagements the firm already completed, where client names, counterparties, deal values, and identifying details are stripped or replaced.

Anonymization here means more than deleting a company name. It includes:

• Removing dates that could identify the deal via public records.
• Replacing dollar figures and percentages with placeholder ranges.
• Scrubbing metadata (file authors, timestamps, tracked changes).

This phase answers one question only: does the tool perform the task at all, before asking whether it's safe with real data.

Phase 1: Guardrails defined before testing begins

Before Phase 0 data even loads, the pilot team should lock in:

1. Data residency and retention terms: where does the vendor process and store data, and for how long? Under the EU's GDPR (General Data Protection Regulation), and for US firms under state privacy laws and client confidentiality rules (e.g., ABA Model Rule 1.6 for lawyers), this must be answered in writing, not assumed.

2. No-training clause: contractual confirmation the vendor will not use firm or client data to train models used by other customers. This is standard in enterprise contracts from Harvey, OpenAI's enterprise tier, and Microsoft Azure OpenAI, but must be verified, not assumed from marketing pages.

3. Human-in-the-loop checkpoints: every AI output (a flagged clause, a risk summary) is reviewed by a qualified professional before it reaches a workpaper or deliverable.

4. Access control list: name exactly who on the pilot team can use the tool and with which document set. No open access.

5. Kill switch: a documented process to immediately disable the tool if a data leak, hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → pattern, or vendor breach is detected.

None of this is exotic. It mirrors how firms already govern outsourcing to third-party vendors, applied here to software.

Phase 2: Synthetic and redacted live-adjacent data

Once anonymized historical data passes Phase 0, the next step is testing against synthetic documents: contracts generated to resemble real deal structures but containing no real party. This checks the tool's behavior on edge cases (unusual clause language, foreign-language contracts, scanned PDFs) without any confidentiality exposure at all.

Phase 3: Limited live pilot, with client consent

Only after Phases 0 to 2 succeed does a live client matter enter the picture, and only:

• With the client's informed consent (many engagement letters now include an AI-use clause).
• On a single, low-stakes matter first, not a live billion-dollar merger.
• With outputs treated as a draft input to human review, never as the final deliverable.

Some firms, including several Big Four accounting networks, now disclose AI tool use directly in engagement letters, following guidance trends from bodies like the AICPA (American Institute of CPAs) on technology in audit quality control.

Evaluating the pilot: what "success" actually means

Resist judging a pilot purely on speed. Measure across four dimensions:

| Dimension | Sample metric |

|---|---|

| Accuracy | % of clauses correctly flagged vs. human-reviewed baseline |

| Time saved | Hours per 100 contracts reviewed, AI-assisted vs. manual |

| Error type | False negatives (missed risk) weighted more heavily than false positives |

| Trust impact | Client feedback, willingness to consent to future AI use |

A simple worked example: if a senior associate reviews 100 contracts manually in an estimated 40 hours (a commonly cited due-diligence benchmark, not a fixed rule), and an AI-assisted first pass cuts initial review to 15 hours with a human verifying flagged clauses in another 10 hours, that's 25 hours instead of 40, a 37.5% time reduction. That number only matters if accuracy on high-risk clauses (change-of-control, liability caps) holds at or above the manual baseline. A faster review that misses one indemnity clause is not a win, it's a liability.

What tends to go wrong

• Skipping Phase 0: teams under deal pressure jump straight to live data because "we're behind schedule." This is the single most common governance failure cited in professional services AI adoption surveys.
• Vendor claims taken at face value: "SOC 2 compliant" (a common security audit standard) does not automatically mean no-training or full data deletion. Read the actual data processing addendum.
• No sunset clause on pilot data: anonymized test sets linger on shared drives indefinitely. Set a deletion date at pilot kickoff.
• Treating human review as a formality: if reviewers rubber-stamp AI output to hit deadlines, the human-in-the-loop control is theatre, not a safeguard.

🎬 [VIDEO: "How Law Firms Are Using AI for Due Diligence" - youtube.com - search for recent legal-tech conference panels (e.g., Legalweek or ILTA sessions) discussing real due-diligence AI deployments and vendor selection criteria]

A minimal technical guardrail: redaction check

Even a basic automated scan before data upload catches obvious leaks. A simple pattern-matching check (illustrative, not production-grade):

import re

PII_PATTERNS = {
    "email": r"[\w\.-]+@[\w\.-]+\.\w+",
    "dollar_amount": r"\$\s?\d[\d,]*(\.\d{2})?",
    "phone": r"\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}",
}

def flag_sensitive(text):
    findings = {}
    for label, pattern in PII_PATTERNS.items():
        matches = re.findall(pattern, text)
        if matches:
            findings[label] = len(matches)
    return findings

This is not a substitute for legal review or proper anonymization tooling, it's a first-pass tripwire before any document reaches Phase 0 testing.

Key Takeaways

• Never test a due-diligence AI tool on live client data first. Start with anonymized historical matters, then synthetic data, then a limited live pilot with client consent.
• Define guardrails (data residency, no-training clauses, human review, access control, kill switch) in writing before Phase 0 begins, not after a problem occurs.
• Judge pilots on accuracy and error type, not just speed. A missed high-risk clause outweighs hours saved.
• Vendor security certifications (like SOC 2) are necessary but not sufficient. Verify data training and retention terms explicitly.
• Treat human-in-the-loop review as a real control, not a formality, or the entire guardrail structure collapses under deadline pressure.