AIThis week in AI

Fireworks AI breach reaches Notion chats through a vendor's vendor

Fireworks AI first said it had found no evidence of access to models, datasets, training data, inference traffic, or personal information. A month later it confirmed attackers had reached customer datasets and partial model weights, and Notion disclosed that the breach exposed Notion AI chat transcripts in a small number of workspaces. Anyone who turns on AI features at work is trusting companies they can't see, and the promises in their contracts may cover less than they assume.

Server rack inside a rented hangar with a pried-open wall, violet chat bubbles escaping as an office worker watches.

Listen to the podcast

9 min

Chapters

Key takeaways

  • Ask AI vendors to name the providers they rely on, including who hosts their machines, since each layer has its own exposure.
  • Check whether a zero data retention promise covers only query answering or also stored data kept for features like analytics and categorization.
  • Never paste passwords or API keys into an AI chat, and rotate any you already have if a vendor is breached.
  • Treat an early "no evidence of access" breach bulletin as provisional before telling leadership you are safe.
  • Act on a vendor incident before full confirmation, as Notion did by cutting traffic and deleting its data from Fireworks the next day.

Episode page and full transcript →

Fireworks AI is an inference provider: it runs AI models on its own hardware for other companies. On Monday it said some of its customers' datasets and models were accessed by hackers over the course of the past month, according to Forbes. Forbes says Fireworks counts roughly 10,000 enterprise customers, including Uber, Shopify, Upwork and DoorDash, as well as marquee startups like Cursor and Cognition.

The trouble is that the story changed. According to The Register, Fireworks first disclosed an incident on October 9 after detecting unauthorized use of internal credentials on October 6. The new update describes a second intrusion: a third-party cloud provider hosting its research and development machines was compromised on October 23. Attackers used that access the following day to reach Fireworks development systems and run an unauthorized workload.

The damage also reached people who never signed up with Fireworks. BleepingComputer reports that Notion got an alert from Fireworks on November 2 about an exposed dataset said to hold only synthetic test data. Notion responded by cutting off traffic to Fireworks, deleting all data from the vendor's systems, and halting training runs by November 3. However, Fireworks confirmed on November 8 that customer data had been accessed, including that of Notion users. SecurityWeek reports that fewer than 1% of Notion workspaces were affected, and most of the transcripts were unattributable to a customer or the workspace they came from.

The detail drawing the most argument is retention. Fireworks had contractually committed to zero data retention for Notion data, per BleepingComputer. VentureBeat puts the tension plainly: Fireworks says its zero data retention default held. Attackers reached stored customer data instead, and Notion found AI chat transcripts in it.

People also disagree about how serious it is. Fireworks suspects links to financially motivated criminals involved in LLMJacking, selling unauthorized access to AI models through illicit gateways. CEO Lin Qiao told The Register "From what we see attackers are not after weights". Customers are split. Forbes found two customers who had already substantially moved their business off Fireworks' systems, with one describing a series of 'very bad hacks' and poor communication, while two other customers said they were not aware of any Fireworks hacks.

Next, watch whether other downstream customers report exposure, and whether "limited set of accounts" becomes an actual count. The company is working with Unit 42, the cybersecurity division of Palo Alto Networks to investigate.

Finished reading?

Validate your read to earn XP and feed your radar.