when the model is wrong and a guest gets hurt

# When the model is wrong and a guest gets hurt

A family of five lands at 11 p.m. after a nine-hour flight. Their hotel room does not exist. Not "not ready", gone: the property sold it twice, and the overbooking model decided their reservation was the one to bump. No human made that call in real time. An algorithm did, hours earlier, based on a no-show forecast that turned out to be wrong.

This is what model risk looks like in travel and hospitality: not a rogue AI headline, but a quiet miscalculation that turns into a guest standing at a front desk at midnight with nowhere to sleep.

This lesson traces three real failure patterns so you can classify AI risk by severity (how bad is the harm) and blast radius (how many bookings, guests, or routes it touches at once).

Pattern 1: the overbooking model that miscounts no-shows

Airlines and hotels intentionally overbook. It is a legitimate, decades-old practice: some percentage of guests will not show, so selling slightly over capacity maximizes revenue. The math is a forecasting problem, and forecasting models can be wrong.

How it breaks: a no-show prediction model is trained on historical patterns (season, day of week, booking channel, lead time). If the pattern shifts, say, a new travel advisory changes cancellation behavior, or a corporate client group behaves differently than leisure travelers, the model keeps forecasting yesterday's world. Overbooking is calibrated too aggressively, and paying, checked-in-eligible guests get denied boarding or denied rooms.

Severity: medium to high. Involuntary denied boarding (getting bumped from a flight you have a valid ticket for) and walked hotel guests (relocated to another property, often at the hotel's cost) create real financial and reputational damage. The US Department of Transportation (DOT) requires airlines to compensate involuntarily bumped passengers, currently up to 4x the ticket price capped by federal rule, as of 2024 (source: transportation.gov). In the EU, Regulation EC 261/2004 mandates fixed compensation (up to €600 per passenger depending on flight distance) for denied boarding.

Blast radius: one flight or one property, but repeated daily across a network. A systematically miscalibrated model does not cause one bad night, it causes hundreds, silently, until someone audits the pattern.

The guardrail: back-test the no-show model against realized outcomes weekly, not quarterly. Set a hard ceiling on overbooking percentage that a model cannot exceed regardless of its confidence score. Require human sign-off when the model recommends overbooking above a threshold (e.g., more than 8 percent over capacity, illustrative, not a universal industry number).

Pattern 2: the chatbot that invents refund policies

Air Canada's chatbot told a customer he could apply for a bereavement fare discount after his flight, contradicting the airline's actual policy. The customer sued. In 2024, a Canadian tribunal ruled Air Canada liable for its chatbot's false statement (source: widely reported, including CBC News). This is now a textbook case in AI governance training.

Why it happens: large language models (LLMs, AI systems trained to generate human-like text) generate plausible-sounding answers even when they lack the actual policy in context. This is called hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition →: the model produces a confident, fabricated fact. A chatbot without retrieval grounding (pulling the actual policy document at answer time, rather than relying on the model's trained memory) is a fabricationfabricationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → machine wearing a customer-service uniform.

Severity: low per incident, but legally binding. Courts and regulators have signaled companies are responsible for what their AI agentsAI agentsAgentic AI refers to AI systems that pursue goals autonomously by planning, taking actions through tools, and adapting based on results, with minimal step-by-step human direction.View full definition → say, the same way they are responsible for what a human employee says. The chatbot's words are treated as the company's words.

Blast radius: potentially enormous. One flawed prompt template or one stale knowledge base can generate the same wrong answer to thousands of customers simultaneously, all before anyone notices.

The guardrail:

• Ground the chatbot in a live, versioned policy database (retrieval-augmented generation, RAG), not the model's general training.
• Log every customer-facing answer for audit.
• Add a confidence gate: if the model is answering a refund, compensation, or safety question, route to human review or cite the source document verbatim rather than paraphrasing.
• Disclose to users they are speaking with an AI, required under the EU AI Act for many customer-interaction systems as of 2026 rollout phases (source: EU AI Act).

Pattern 3: the route-optimization model that fails during a hurricane

Cruise lines, airlines, and logistics providers use route and capacity optimization models to reposition aircraft, reroute cruise itineraries, and rebalance hotel demand during disruptions. These models are trained mostly on normal operating conditions.

How it breaks: a hurricane, a volcanic ash cloud, or a geopolitical closure of airspace is an out-of-distribution event, a situation unlike anything in the training data. A model optimized for "typical disruption" (a storm delaying one hub for a day) can make dangerously wrong recommendations in a systemic event (a storm closing three hubs for a week), because it has never seen that scale of correlated failure.

Severity: high. This is the pattern with the most direct line to physical guest safety, stranded passengers, crews exceeding duty-time limits, cruise ships rerouted into worse weather because the model underweighted a rapidly intensifying storm.

Blast radius: network-wide and time-compressed. Unlike the chatbot case, there is no time to quietly patch it. Decisions must be made in hours.

The guardrail: maintain a manual override protocol staffed by operations experts, tested through regular tabletop exercises (simulated crisis drills), not just written in a manual. Require the model to output a confidence interval and flag when live conditions fall outside its training distribution. The NIST AI Risk Management Framework (a US voluntary framework for identifying and mitigating AI risks) explicitly recommends this kind of out-of-distribution monitoring.

Building a simple severity × blast radius grid

You do not need a complex model to triage AI risk. A 2x2 grid works for a board presentation or an internal audit:

| | Low blast radius | High blast radius |

|---|---|---|

| Low severity | Monitor, log | Monitor closely, set alert thresholds |

| High severity | Human-in-the-loop required | Kill switch + mandatory human override + regulator disclosure |

Applying it:

• Overbooking miscalibration: medium severity, high blast radius (repeats daily) → sits in the top-right to bottom-right, monitor closely with hard caps.
• Chatbot hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition →: low severity per instance, potentially high blast radius → alert thresholds plus mandatory grounding.
• Route optimization during a hurricane: high severity, high blast radius → kill switch and human override, non-negotiable.

A "kill switch" here means an operational ability to instantly revert to manual routing or a prior stable model version, not a metaphor. If your team cannot describe how they would disable a model within minutes, the guardrail does not actually exist.