Substantiating technical performance claims before publication

In October 2017 Kobe Steel admitted that inspection data for aluminium, copper and steel products had been falsified, in some cases for decades. The certificates on file said one thing; the metal shipped to several hundred customers, including carmakers, aircraft builders and rail operators, was another. The cost was never mainly the criminal fine. It was the re-certification work forced on customers who had designed parts around numbers that had no test behind them, and a chief executive resignation.

That is the failure this lesson is built to prevent, in a smaller and more common form: a performance number published with nothing behind it. What follows is the evidence file standard, meaning what test, on how many units, under which stated conditions, verified by whom, has to exist before a number goes on a datasheet, a booth banner or a product page.

Where puffery ends and a substantiated claim begins

The rule that matters is prior substantiation: an objective performance claim needs a reasonable basis on file at the moment it is published, not assembled later when someone complains. Which authority would police it in each market is the subject of the cross-border lesson; the evidence standard itself barely moves between jurisdictions.

The line to watch is not tone, it is checkability. "Built to last" is unverifiable opinion. "Tested for the equivalent of 20 years of use" is a factual assertion, and Miele publishes exactly that kind of durability claim for its washing machines on the basis of testing equivalent to 20 years at five loads a week. The duty cycle assumption is part of the claim. Strip it out and the same sentence becomes indefensible, because a laundrette running 30 loads a week is not making an unreasonable reading.

The moment you attach a number, a unit of measure, or a named competitor, you have left puffery.

What counts as substantiation

1. Test methodology, documented in advance. Was the protocol (load, duration, ambient conditions, failure definition) written before results came in, or does the claim describe what a test happened to show? The second pattern is what enforcement files look like from the inside.

2. Sample size that supports the precision of the number. This is where most claims quietly fail, and the arithmetic is unforgiving. To claim an MTBF of 50,000 hours you need to accumulate at least 50,000 unit-hours of testing to expect a single failure: 100 units for 500 hours, or 50 units for 1,000. Even then, zero failures across 50,000 unit-hours only supports an MTBF of roughly 22,000 hours at 90 percent confidence. Five pilot installations do not produce a 99.9 percent uptime figure; they produce a hypothesis.

3. Third-party verification where feasible. Accredited labs (UL, TÜV, Intertek) carry weight that in-house benches do not, and for safety, emissions or energy claims the testing is often mandatory rather than a credibility upgrade.

4. Reproducibility. If a second lab cannot repeat the protocol and land in the same range, the data is weak in a dispute regardless of how carefully it was collected.

5. Currency. A 2019 report does not substantiate a claim about a 2025 revision of the same product. Component substitutions on the supply side are the usual culprit: the marketing number survives a bill-of-materials change that the test file never saw.

The FTC's guidance on advertising and marketing basics sets out the same "reasonable basis" test that gets applied to industrial claims, even though the drafting is consumer-facing.

The disclaimer is risk transfer, not decoration

Qualifiers do real work when they are specific and sit next to the claim. Compare:

• Weak: "*Results may vary.*"
• Stronger: "*Based on internal testing of 12 units under continuous load at 25°C ambient over 90 days. Field performance varies by installation and duty cycle.*"

The second tells the buyer which conditions produced the number, which is what an investigator checks first. A footnote in 6-point type at the bottom of a banner does not rescue a headline number, and it will not satisfy the duties toward business buyers that the fair-treatment lesson sets out.

Test-cycle claims are the standard trap. Tesla's advertised range figures come from a defined regulatory drive cycle, and in January 2023 Korea's Fair Trade Commission fined the company roughly 2.85 billion won for not disclosing how far real driving range falls in cold weather. The lab number was not fabricated. The conditions attached to it were not carried through to the buyer.

Where this bites in manufacturing

Datasheets. These are durable, quotable documents with a long tail. A distributor citing your 2024 datasheet in a 2027 tender is still quoting you. Version and date every sheet, and actively withdraw superseded ones.

There is a second-order consequence that outweighs the advertising exposure. Published performance figures get pulled into supply contracts as specifications, sometimes verbatim from the PDF. At that point an unsupported number is not an advertising problem, it is a warranty and breach question with the customer's line-down costs attached. Kobe Steel's fine was small next to what its customers spent verifying parts they had already built.

Trade show graphics. Booth panels are frequently produced by an agency working from an old press release rather than the current test file, and printed graphics are the hardest asset to correct once a claim moves.

Comparative claims. "50 percent more efficient than Model X" invites the fastest challenge, because the named competitor has standing and a motive. Matched conditions on both products are the minimum, ideally run by an outside lab, and the competitor unit tested has to be the one currently on sale.

An evidence adequacy check

Before a performance number is cleared for external use:

1. Is there a written, dated test report that predates the drafting of the claim?
2. Does the sample size support the precision implied by the number? "99.9 percent" asserts far more than "around 99 percent."
3. Are the qualifying conditions stated adjacent to the claim rather than three pages away?
4. Does the file cover the product configuration actually shipping, and has it gone through the pre-ship review the operational lesson lays out?

If any answer is no, the claim is softened, qualified, or held until the evidence exists.

The evidence file as a standing process

Most manufacturers treat substantiation as a one-off legal review before a launch. The stronger model is a living file per product line:

• One folder per active performance claim: report, protocol, sample size, date, testing body.
• A retirement rule triggered by engineering change orders, not by the calendar. When a spec changes, the linked claim is flagged for re-test or withdrawal.
• A named owner, usually product marketing plus a quality or compliance contact, who approves any new external use of a number.

The failure mode this catches is internal drift. An engineer's hedged note ("we're seeing about 99.9 percent across the early pilots") becomes a hard datasheet claim four handoffs later, with the caveat and the sample size stripped off somewhere in between. Nobody lied. The claim still has no basis.

The related edge case is accelerated life testing. Compressing 20 years into a few thousand chamber hours requires an acceleration model, and the model is an assumption, not a measurement. If the field stress profile differs from the one assumed, the claim collapses even though the test was run correctly. Record the model and its assumed duty cycle in the file, because that is the first thing a challenger will attack.

🎬 [VIDEO: "How the FTC Evaluates Advertising Claims" - youtube.com - search for FTC or American Bar Association explainer videos on advertising substantiation standards, useful for seeing how regulators actually apply the "reasonable basis" test to specific claim types]

Key Takeaways

• Objective performance claims (uptime, efficiency, MTBF, comparisons) need a reasonable basis on file before publication; "we'll defend it if challenged" is not a position.
• Sample size has to match the precision claimed: a 50,000 hour MTBF figure requires tens of thousands of unit-hours, and zero failures across 50,000 unit-hours only supports about 22,000 hours at 90 percent confidence.
• Test conditions travel with the claim. Miele's 20-year durability claim rests on a stated duty cycle; Tesla's range figures cost it a Korean fine when cold-weather conditions were not disclosed.
• The larger exposure is contractual, not regulatory: published numbers get written into supply agreements, where a bad claim becomes a warranty liability.
• Keep a living evidence file per claim with a named owner and a retirement rule tied to engineering changes, and record acceleration assumptions behind any life-test claim.