Open data portals: publish fewer datasets as products

September 9, 2026 · 5 min · Free, with full transcript

Government portals hold thousands of datasets that get downloaded once and never opened again. The position here is that dataset count is a vanity metric and repeat use is the real measure of a portal's health, citing Data.gov's 300,000 plus datasets and their dead long tail. The answer is to treat a dataset like a product with an owner, a release schedule and documentation, and to retire the rest. You will learn how to separate the needs of journalists, advocates and oversight bodies, why Transport for London's live feed produced CityMapper and tens of millions of pounds in value, how to read public records requests as a demand signal, and why a plain language data dictionary decides whether a dataset gets cited.

Listen to the podcast

5 min

Chapters

Key takeaways

  • Measure repeat downloads and living audiences instead of the number of datasets published.
  • Pick one audience per dataset, since journalists, advocates and auditors need opposite things.
  • Retire 90 percent of datasets and fund the ten people fight over, such as spending, permits, inspections and police stops.
  • Use recurring public records requests as a free backlog of what to publish well.
  • Open your download logs, find the five datasets people return to, and give each a named owner and publishing schedule.
Read the full transcript

Host:Welcome to the Leaders Insights Podcast. Today's episode, publishing open data that citizens, journalists, and oversight bodies actually use. There are government open data portals with thousands of datasets, and most of them get downloaded once, probably by the person who uploaded them. Why do we keep building these graveyards?

Expert:Because publishing looks like progress, a politician cuts a ribbon on a portal with 4,000 datasets and calls it transparency. Nobody measures whether a single journalist ever opened file number 3,000. It's theater with a CSV. That's a comma-separated file, plain text spreadsheet, attached.

Host:So the number of datasets is a vanity metric? What should they be counting instead?

Expert:Repeat use. Does anyone come back to a dataset a second time? Data.gov, comma, the US federal portal, has hosted well over 300,000 datasets, and internal reviews have shown the long tail is basically dead weight, thousands with essentially zero downloads. The health of a portal isn't how much you dumped in, it's how many datasets have a living audience.

Host:Fine, but living audience sounds like a slogan. Who actually uses this stuff, and what do they need?

Expert:Three groups, and they want opposite things. Advocates want a story-shaped file today, clean enough to open and excel without crying. Advocates want a stable series they can track over years. Oversight bodies, auditors, inspectors general, want completeness and provenance, meaning where every number came from. If you design for all three at once, you serve none. You pick.

Host:That's uncomfortable. You're telling public agencies to prioritize instead of publishing everything?

Expert:Yes, because publish everything is how you get the graveyard. The durable lesson the good operators internalized years ago, treat a dataset like a product, not a file. A product has an owner, a release schedule, documentation, and someone whose job is ruined if it breaks. A file just sits there rotting.

Host:Give me a concrete case where someone did this right.

Expert:Transport for London, the transit agency. They didn't just dump timetables. They published a live feed with a real contract behind it, stable formats, and they let developers build on it. CityMapper, the journey planning app, was built on that data. TFL estimated the economic value to travelers and the city in the tens of millions of pounds a year. That's what happens when you maintain data like it matters instead of archiving it.

Host:The counter argument, agencies are broke and understaffed. Maintaining 100 datasets as products is a fantasy.

Expert:Which is exactly why you kill 90 of them. Take your budget and pour it into the 10 datasets people actually fight over. Spending, permits, inspections, police stops. DBBT Labs, the data tooling vendor, argues that maintenance eats something like half of a data team's time, though they sell the software that promises to cut that, so cross-check it against independent work. MIT Sloan Management Review has been blunt for years that data quality neglect is where value quietly dies. The point stands either way. Fewer things kept alive.

Host:How do you know which 10? Every department will claim theirs is essential.

Expert:Don't ask departments. Ask the users. Look at what journalists file public records requests for over and over. That's your backlog, handed to you free. If reporters keep suing you for the same inspection data, publish it well, and you've turned a legal headache into a product. The demand signal is already screaming at you. Agencies just refuse to listen.

Host:And documentation. Everyone says it matters. Nobody does it. Why does it actually change usage?

Expert:Because an undocumented column is a landmine. If a field is called status underscore two and nobody knows what value four means, a journalist either guesses wrong and prints something false or walks away. A data dictionary, a plain language description of every field, is the difference between a data set that gets cited in an investigation and one that gets abandoned in an afternoon. It's boring. It's also the whole game.

Host:So if a CDO listening today does one thing this week, what is it?

Expert:Open your portal's download logs, find the five data sets people actually return to, and assign each one a named human owner with a publishing schedule. Then let the dead one stay dead. Serve the five you have brilliantly before you publish a sixth nobody wanted.

Host:Sources for today's episode. The new stack towards data science, MIT technology review, MIT Sloan Management Review, DBT Labs, vendor data tooling. That's your five. The full CDO track, structured no fluff, is at MBA-training.com.

Read and go deeper