Reading vendor claims: what AI travel tools really do under the hood

Machine learning (ML): the system is trained on data and infers patterns, so it can generalize to inputs it has never seen. A well-trained hotel chatbot can handle "can I stay past 11" and "is there any way I could leave a bit later" as the same request, without a human coding both phrasings.

Large language models (LLMs): the current generation of ML used in most "generative AI" concierge and support tools (examples: OpenAI's GPT models, Anthropic's Claude, Google's Gemini). These are trained on massive text datasets and can handle open-ended conversation, but they can also "hallucinate" (generate fluent, confident, but factually wrong answers), which matters a lot when the answer involves a cancellation deadline or a visa requirement.

Both approaches can look identical in a five-minute demo. The difference shows up in edge cases, which is exactly what vendors don't showcase.

The five questions that separate substance from label

1. "What happens when a guest asks something outside the expected script?"

Ask the vendor to type an odd, real request live: "my flight got cancelled and I have a dog, what are my options." A rules engine stalls, loops, or hands off to a human immediately. An ML-based system attempts a reasonable, if imperfect, response.

2. "What data was this trained on, and is any of it ours?"

Generic LLMs trained on public internet text know nothing about your specific loyalty tiers, room categories, or local partner network unless the vendor has done retrieval-augmented generation (RAG: a method where the model looks up your proprietary documents, like your policy PDFs, before answering) or fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.View full definition → (further training the model on your own dataown dataData collected directly from your own customers and prospects through your own channels: your most reliable and privacy-compliant source.View full definition →). If the answer is vague, the tool is likely a thin wrapper around a generic model with no real customization.

3. "How does performance improve over time, and what does that improvement cost?"

True ML systems get better as more interaction data accumulates, but that requires a feedback loop (someone reviewing flagged conversations, retraining periodically). Ask who does that work and how often. If nobody can describe a retraining cadence, the "learning" claim is marketing.

4. "What's the fallback when the model is uncertain?"

Reputable systems have confidence thresholds and escalate to a human agent when uncertain. If a vendor claims their bot "always has an answer," that's a red flag: it usually means the system will guess rather than admit not knowing, a serious problem for anything touching refunds, medical accommodation, or safety.

5. "Can you show us the accuracy metric's definition, not just the number?"

"94% accuracy" is meaningless without knowing the test set (how many samples, in what language, on what task) and what counts as "correct." Ask for the benchmark methodology. If it's proprietary and undisclosed, treat the number as unverified marketing.

A worked mini-example: comparing two chatbot claims

Say a vendor tells you their intent-classification model, the layer that decides what a guest is asking about, hits 90% accuracy on a 500-message test set of real guest queries. A second vendor claims 96% but tested on 50 messages they wrote themselves.

Simple comparison of reliability:

Vendor A's lower headline number is the more trustworthy signal because the sample size and source are larger and closer to real conditions. This is the kind of arithmetic a non-technical buyer can and should do before comparing vendor scorecards at face value.

Where genuine AI does add value in travel, right now

To calibrate expectations, it helps to know where the technology is proven, not hypothetical, as of 2026 (estimates, vendor-reported figures vary by source):

• Dynamic pricing and revenue management: airlines and hotels have used ML-driven demand forecasting for years (this predates the "generative AI" wave). Companies like Duetto and IDeaS are established players here.
• Personalization in marketing and upsell: recommending room upgrades or excursions based on booking history and browsing behavior.
• Multilingual guest messaging: translation and intent handling across languages, useful for international travel brands.
• Operational forecasting: predicting occupancy, staffing needs, and no-show rates.

Where claims deserve the most scrutiny: fully autonomous "AI concierges" replacing front desk or call center staff end to end. These often still rely on heavy human-authored scripting behind an AI-sounding interface.

For a grounded, non-vendor explainer on how modern conversational AI actually works, see this resource from Google's Machine Learning Crash Course on NLP fundamentals, useful for building enough vocabulary to hold vendors accountable.

Reading the pitch deck itself

A few deck-level tells worth flagging in any vendor meeting:

• Logo walls without usage detail. A hotel brand's logo on a slide might mean a company-wide deployment, or it might mean one property ran a three-month pilot. Ask which.
• "Powered by GPT/Gemini/Claude" as the entire technical explanation. This tells you which foundation model they call, not how (or whether) they've adapted it to your business, checked it for hallucinationhallucinationA hallucination is when an AI model generates output that is fluent and confident but factually wrong, fabricated, or unsupported by its source data.View full definition → on your policies, or built any guardrails.
• Case studies with no baseline. "Increased guest satisfaction by 20%" is only meaningful if you know the measurement period, sample size, and what satisfaction was before.
• No mention of failure modes. Every deployed AI system fails sometimes. A vendor who cannot describe a real failure and how it was handled either lacks deployment experience or is not being candid.

Key Takeaways

• The core diagnostic question is behavior under unscripted input: a rules engine breaks on unexpected phrasing, genuine ML degrades more gracefully.
• Always ask for the training data source, the retraining cadence, and the fallback behavior when the system is uncertain. Vague answers on any of these are a warning sign.
• Demand the methodology behind any accuracy number (sample size, test set source, definition of "correct"), not just the headline percentage. A large, realistic test set beating a small, curated one is often the more trustworthy claim even at a lower percentage.
• Proven, low-hype AI use cases in travel and hospitality include dynamic pricingdynamic pricingAutomatically adjusting prices in real time based on demand, competition or user behaviour to optimise revenue, margin or conversion.View full definition →, demand forecasting, and personalization. Fully autonomous guest-facing "AI concierges" deserve the highest level of scrutiny.
• Vendor decks reveal as much through omission (no failure modes, no baseline metrics, vague logo walls) as through what they claim outright.