Reading vendor claims: what AI travel tools really do under the hood

# Reading vendor claims: what AI travel tools really do under the hood

A hotel group's CIO gets a pitch deck. Slide four says "our AI concierge understands guest intent with 94% accuracy." Slide seven says "powered by deep learning and natural language processing." Slide nine has a client logo wall with three recognizable hotel brands. Nobody in the room asks what model the tool actually runs, what data trained it, or what happens when a guest asks something the vendor didn't anticipate. Six months later, the "AI concierge" breaks the first time someone phrases a request slightly differently than the demo. This lesson is about catching that gap before the contract is signed.

Why this matters more in travel than most sectors

Travel and hospitality buy an unusual amount of "AI-washed" software: rules-based systems, decision trees, or simple keyword matching rebranded with AI language because the label sells. This happens because:

• The sector has thousands of small and mid-size buyers (independent hotels, regional airlines, tour operators) with limited in-house technical teams to interrogate vendor claims.
• Guest-facing chat and voice tools look impressive in a scripted demo, which is exactly the environment rules engines perform best in.
• "AI" has become a checkbox on RFPs (request for proposal, the formal document buyers issue to solicit vendor bids), so vendors have a direct incentive to use the word regardless of what's inside.

Knowing the right questions is a core fluency skill, not a technical one.

Rules engine vs. machine learning: the real distinction

Rules engine (a.k.a. decision tree or expert system): a human writes explicit "if this, then that" logic. If guest says "cancel," show cancellation policy. If guest says "late checkout," show late checkout options. It cannot handle inputs its authors didn't anticipate, and it does not improve with use unless someone manually adds more rules.

Machine learning (ML): the system is trained on data and infers patterns, so it can generalize to inputs it has never seen. A well-trained hotel chatbot can handle "can I stay past 11" and "is there any way I could leave a bit later" as the same request, without a human coding both phrasings.

Large language models (LLMs): the current generation of ML used in most "generative AI" concierge and support tools (examples: OpenAI's GPT models, Anthropic's Claude, Google's Gemini). These are trained on massive text datasets and can handle open-ended conversation, but they can also "hallucinate" (generate fluent, confident, but factually wrong answers), which matters a lot when the answer involves a cancellation deadline or a visa requirement.

Both approaches can look identical in a five-minute demo. The difference shows up in edge cases, which is exactly what vendors don't showcase.

The five questions that separate substance from label

1. "What happens when a guest asks something outside the expected script?"

Ask the vendor to type an odd, real request live: "my flight got cancelled and I have a dog, what are my options." A rules engine stalls, loops, or hands off to a human immediately. An ML-based system attempts a reasonable, if imperfect, response.

2. "What data was this trained on, and is any of it ours?"

Generic LLMs trained on public internet text know nothing about your specific loyalty tiers, room categories, or local partner network unless the vendor has done retrieval-augmented generation (RAG: a method where the model looks up your proprietary documents, like your policy PDFs, before answering) or fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → (further training the model on your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.Voir la définition complète →). If the answer is vague, the tool is likely a thin wrapper around a generic model with no real customization.

3. "How does performance improve over time, and what does that improvement cost?"

True ML systems get better as more interaction data accumulates, but that requires a feedback loop (someone reviewing flagged conversations, retraining periodically). Ask who does that work and how often. If nobody can describe a retraining cadence, the "learning" claim is marketing.

4. "What's the fallback when the model is uncertain?"

Reputable systems have confidence thresholds and escalate to a human agent when uncertain. If a vendor claims their bot "always has an answer," that's a red flag: it usually means the system will guess rather than admit not knowing, a serious problem for anything touching refunds, medical accommodation, or safety.

5. "Can you show us the accuracy metric's definition, not just the number?"

"94% accuracy" is meaningless without knowing the test set (how many samples, in what language, on what task) and what counts as "correct." Ask for the benchmark methodology. If it's proprietary and undisclosed, treat the number as unverified marketing.

A worked mini-example: comparing two chatbot claims

Say a vendor tells you their intent-classification model, the layer that decides what a guest is asking about, hits 90% accuracy on a 500-message test set of real guest queries. A second vendor claims 96% but tested on 50 messages they wrote themselves.

Simple comparison of reliability:

Vendor A: 90% of 500 real guest messages correctly classified
  -> 450 correct, 50 misclassified, tested on realistic variety

Vendor B: 96% of 50 self-written test messages correctly classified
  -> 48 correct, 2 misclassified, tested on narrow, curated inputs

Vendor A's lower headline number is the more trustworthy signal because the sample size and source are larger and closer to real conditions. This is the kind of arithmetic a non-technical buyer can and should do before comparing vendor scorecards at face value.

Where genuine AI does add value in travel, right now

To calibrate expectations, it helps to know where the technology is proven, not hypothetical, as of 2026 (estimates, vendor-reported figures vary by source):

• Dynamic pricing and revenue management: airlines and hotels have used ML-driven demand forecasting for years (this predates the "generative AI" wave). Companies like Duetto and IDeaS are established players here.
• Personalization in marketing and upsell: recommending room upgrades or excursions based on booking history and browsing behavior.
• Multilingual guest messaging: translation and intent handling across languages, useful for international travel brands.
• Operational forecasting: predicting occupancy, staffing needs, and no-show rates.

Where claims deserve the most scrutiny: fully autonomous "AI concierges" replacing front desk or call center staff end to end. These often still rely on heavy human-authored scripting behind an AI-sounding interface.

For a grounded, non-vendor explainer on how modern conversational AI actually works, see this resource from Google's Machine Learning Crash Course on NLP fundamentals, useful for building enough vocabulary to hold vendors accountable.

Reading the pitch deck itself

A few deck-level tells worth flagging in any vendor meeting:

• Logo walls without usage detail. A hotel brand's logo on a slide might mean a company-wide deployment, or it might mean one property ran a three-month pilot. Ask which.
• "Powered by GPT/Gemini/Claude" as the entire technical explanation. This tells you which foundation model they call, not how (or whether) they've adapted it to your business, checked it for hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.Voir la définition complète → on your policies, or built any guardrails.
• Case studies with no baseline. "Increased guest satisfaction by 20%" is only meaningful if you know the measurement period, sample size, and what satisfaction was before.
• No mention of failure modes. Every deployed AI system fails sometimes. A vendor who cannot describe a real failure and how it was handled either lacks deployment experience or is not being candid.

Key Takeaways

• The core diagnostic question is behavior under unscripted input: a rules engine breaks on unexpected phrasing, genuine ML degrades more gracefully.