Fireworks AI breach: how Notion AI chats leaked via a vendor
November 12, 2026 · 9 min · Free, with full transcript
How did Notion AI chat transcripts end up exposed through a company most users never heard of? The episode traces the chain from user to Notion to the inference provider Fireworks AI to a third-party cloud host, and argues that turning on AI features at work means trusting vendors you cannot see. It covers Fireworks' first "no evidence" bulletin, the later confirmation of exposed datasets and partial model weights, and why a zero data retention contract did not protect stored data used for Notion's Insights feature. You will be able to question AI vendors about their own subprocessors, read zero data retention clauses for what they exclude, and handle secrets pasted into AI chats, with context on LLMjacking and Unit 42.
Listen to the podcast
9 min
Chapters
Key takeaways
- Ask AI vendors to name the providers they rely on, including who hosts their machines, since each layer has its own exposure.
- Check whether a zero data retention promise covers only query answering or also stored data kept for features like analytics and categorization.
- Never paste passwords or API keys into an AI chat, and rotate any you already have if a vendor is breached.
- Treat an early "no evidence of access" breach bulletin as provisional before telling leadership you are safe.
- Act on a vendor incident before full confirmation, as Notion did by cutting traffic and deleting its data from Fireworks the next day.
Read the full transcript
Host:You're listening to MBA Training. Today's subject: Fireworks AI breach reaches Notion chats through a vendor's vendor. So the feature Notion built to show managers how their teams use AI is the part that leaked.
Expert:Yes, the Insights feature. The dashboard for watching people's AI use got watched.
Host:Oh, come on. Nobody would pitch that as satire.
Expert:Nobody had to. Notion published it this week. Chat transcripts it used for Insights were sitting with a vendor called Fireworks AI, and attackers got to them.
Host:Okay, back up. Who is Fireworks, and why has nobody at my lunch table heard of them until now?
Expert:Because they're plumbing. Fireworks is an inference provider, meaning it runs AI models on its own hardware for companies that don't want to babysit the chips. Forbes says it has roughly ten thousand enterprise customers: Uber, Shopify, DoorDash, and startups like Cursor and Cognition.
Host:So when someone types into Notion AI, they might be typing into Fireworks.
Expert:For some features. Notion used it to serve open-weight models (models whose internals are published so anyone can run them) and to sort chats into categories for Insights.
Host:Fine. Companies get hacked every week. Why is this the one people are fighting about?
Expert:Because of the order of the statements. In early October, Fireworks caught someone misusing internal credentials. Its first bulletin, as The Register reports it, found no evidence of access to models, datasets, training data or personal information.
Host:Ah. The famous "no evidence."
Expert:The famous "no evidence." Then this Monday, about a month later, an update: billing records, plus stored secrets, datasets and incomplete model weights for what it calls a limited set of accounts.
Host:Weights being...
Expert:...the learned numbers inside a model. If a company fine-tuned a model, meaning trained it further on its own material, the weights are where that material ends up.
Host:So "no evidence" aged like milk on a radiator.
Expert:To be fair, this was a second intrusion. A third-party cloud provider hosting Fireworks' research machines got compromised, and the next day attackers used that foothold to reach Fireworks' development machines.
Host:Wait, what? Hang on, say that again slowly.
Expert:The user, then Notion, then Fireworks, then whoever hosts Fireworks' research machines.
Host:Russian dolls. Each with its own privacy policy.
Expert:And none of the dolls has read the others'.
Host:Okay, let me be the annoying one. Notion says fewer than one percent of workspaces were touched, and most of the transcripts weren't tied to a user or a company. Small incident, big headline.
Expert:That's the strongest counter-argument, and Notion makes it well. SecurityWeek quotes them saying nothing outside the transcript itself links a chat to a workspace or a person.
Host:"Outside the transcript itself." That clause is doing a lot of work.
Expert:That's the whole problem. The transcript is what people typed. Paste in a client contract or a salary table and the chat identifies itself, label or no label.
Host:And Notion is telling affected customers to rotate any passwords or API keys they pasted in.
Expert:API keys are the secret strings that let software log into other services on someone's behalf.
Host:Who pastes a password into a chatbot?
Expert:Anyone with a deadline and a login that won't work.
Host:Wonderful. So the real attack surface is Friday afternoon.
Expert:Partly. But I won't let this turn into a story about careless users. Notion did something bold. Fireworks first told it the exposed dataset held only synthetic test data. According to BleepingComputer, Notion cut off traffic anyway, deleted its data from Fireworks and halted training runs by the next day.
Host:Before they knew real chats were in there.
Expert:Before. Fireworks only confirmed real customer data later.
Host:Credit where it's due. That's a company that didn't wait for its lawyers.
Expert:Now the part that makes me grind my teeth. BleepingComputer says Fireworks had contractually committed to zero data retention for Notion data, which means a promise not to keep anything once the answer is served.
Host:Hang on. Zero retention, and the chats were in storage?
Expert:That's exactly how VentureBeat frames it. Fireworks says its zero-retention default held for the products that answer queries. The attackers reached stored data instead, the data kept for jobs like the Insights sorting.
Host:So the promise held and the data leaked.
Expert:Apparently both are true.
Host:That's the most lawyerly sentence I've heard all year.
Expert:And it's probably accurate, which is worse. "Zero data retention" in a contract covers one pipe. Nobody was asking about the bucket next to it.
Host:Okay, motive. Who does this?
Expert:Fireworks blames LLMjacking, which is stealing access to AI models and reselling it through illicit gateways.
Host:So it's a stolen gym membership, except the gym is a server farm.
Expert:Close enough. And Forbes reports the CEO, Lin Qiao, said on X that the attackers' behaviour suggested AI agents were involved.
Host:Criminals using AI to break into an AI company so they can resell AI. Somebody check whether the ransom note was autocompleted.
Expert:Then Cantina Security's CEO, Hari Mulackal, raised the stolen-weights worry, and The Register quotes Qiao's answer: "From what we see attackers are not after weights."
Host:I find that weirdly unreassuring.
Expert:Same. Why they broke in and what they could see are separate questions. A burglar who came for the telly still had a look at your diary.
Host:Here's where I land. If I ran a team on Fireworks with private data, I'd already be halfway out the door. Notion walked. That's the signal.
Expert:I don't buy it. Forbes found two customers who said they'd substantially moved their business off, one complaining of "very bad hacks" and poor communication. It found the same number who hadn't heard of any hack at all.
Host:Which is its own scandal.
Expert:Maybe. But that's anecdote, not a stampede. Fireworks caught the second intrusion the same day, brought in Unit 42, Palo Alto Networks' incident team, and told customers when it found the bad news.
Host:After telling them the opposite.
Expert:And if the correction gets punished harder than the original mistake, every vendor learns to say less.
Host:Or learns to look properly the first time. That first bulletin was what customers made decisions on.
Expert:Leaving just moves the risk somewhere else, though. Every inference provider rents machines from someone. The next vendor has its own fourth doll.
Host:Still. I'd rather bet on the doll that hasn't cracked.
Expert:You don't know which one hasn't cracked. You only know which one told you.
Host:We're not settling this.
Expert:No, we're not.
Host:Honestly, the part that gets me is whoever read the October bulletin, told their boss "we're fine," and now has to walk back into that room.
Expert:Yeah. They did the job right with bad information. That one stings.
Host:Okay. What are you watching?
Expert:Whether more downstream names come out. If any of the startups Forbes lists find their own users' data in the pile, this stops being a Notion story. I'm also watching whether the Unit 42 work turns "limited set of accounts" into an actual count.
Host:And contracts. I want to see who starts asking for the vendor's vendors by name.
Expert:That's the quiet consequence. Buyers trusted the phrase "zero data retention", and now it has a footnote.
Host:Which brings us back to Insights.
Expert:The feature that tells you how your team uses AI.
Host:Turns out somebody else got the better insights.
Expert:Better than the customers did, anyway. And that leaves the question nobody has answered yet: when someone switches on an AI feature inside a tool they trust, how many companies they're actually trusting, and whose job it is to count them.
Host:Drawn from The Register, BleepingComputer and Forbes, links in the show notes. That's the episode. If you want the structured version, the AI track is at mba-training.com.
Read and go deeper
- The article behind this episode: Fireworks AI breach reaches Notion chats through a vendor's vendor
- The free AI track
- Watch on YouTube · the full playlist