# Building open data that citizens and journalists actually use
In 2013, a reporter filed a records request with a mid-sized US city asking for every restaurant inspection score from the past five years. The city responded eight months later with a locked PDF: 4,000 pages, scanned sideways, no way to search or sort. Technically, they complied. Practically, they buried the story.
That gap between "we released it" and "someone can use it" is the entire subject of this lesson.
FOIA (the Freedom of Information Act, the US federal law giving the public a right to government records; most states have their own equivalent, often called sunshine or public records laws) is reactive. Someone has to ask. Open data flips the model: publish proactively so nobody has to ask.
Smart agencies watch their FOIA logs as a demand signal. If the health department gets the same request for inspection data twelve times a year, that dataset should live on a portal, refreshed automatically, so the twelfth requester (and the thousandth citizen) just downloads it.
Here is the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → a good agency builds:
1. Source system: the inspection database the sanitarians already use in the field.
2. Extract: a scheduled pull (nightly or weekly) into a staging area.
3. Clean and standardize: fix addresses, normalize categories, strip anything legally exempt (personal phone numbers, informant identities).
4. Publish: push to a public portal as a machine-readable file plus an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → (Application Programming InterfaceApplication Programming InterfaceApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →, a way for software to fetch the data directly).
5. Document: attach metadata so a stranger understands what each field means.
The scanned PDF failed at steps 3, 4, and 5. Let's fix each.
A compliance dump is human-readable at best. Open data is machine-readable: software can parse it without a person retyping anything.
The practical hierarchy, roughly from worst to best:
Tim Berners-Lee's 5-star open data model is the classic reference: one star for any open license, three stars for a non-proprietary format like CSV, five stars for linked data connected to other datasets. Aim for at least three stars. Most civic value gets unlocked there.
One field that punches above its weight: a stable unique identifier for every record. If each restaurant has a permanent ID, a journalist can track one location across years even when it changes owners or names. Without it, they are guessing.
Metadata is data about the data. It is boring to write and it is the difference between a dataset people trust and one they abandon.
Every dataset needs a data dictionary: a plain-language definition of each column. Consider this snippet from a well-documented inspection file:
field_name description example
------------------------------------------------------------------------
facility_id permanent unique ID for the location REST-004821
inspection_date date of inspection (YYYY-MM-DD) 2026-03-14
score 0-100, higher is better 92
result PASS, CONDITIONAL, or CLOSED CONDITIONAL
violation_codes semicolon-separated codes; see codebook 3-501; 4-602Notice what this prevents. Without the note that "higher is better," a reporter might publish that a beloved diner scored 92 as if it failed. Without the codebook reference, violation codes are meaningless strings.
Also document the provenance and cadence: where the data comes from, how often it updates, and when it was last refreshed. A portal showing "last updated: 2021" tells users the data is dead. A visible "updated weekly, last run Monday 6am" signals the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → is alive.
The federal standard here is the DCAT-US metadata schema, which structures fields like publisher, contact email, update frequency, and license. Follow an established schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.View full definition → instead of inventing your own so that aggregators and search engines can index your catalog automatically.
Publishing once is a press release. Publishing on a reliable schedule is infrastructure.
Journalists and app developers will only build on data they can count on. A transit agency that publishes vehicle locations every 30 seconds enables the arrival-prediction apps millions of riders use. If that feed goes dark intermittently, developers abandon it.
Set a cadence you can actually hold: real-time, daily, weekly, monthly, or quarterly. Then automate it so it survives staff turnover. The most common failure mode is a heroic analyst who updates a dataset by hand until they take a new job, and then it silently rots.
Proactive publishing does not mean publishing everything. The obligation to redact is real, and getting it wrong causes lasting damage.
The danger is re-identification: even without names, a combination of fields (ZIP code, birth date, gender) can pinpoint an individual. A famous demonstration showed that these three fields alone can uniquely identify a large share of the US population (a widely cited estimate is around 87 percent from 1990 census work, and the exact figure is debated).
Practical guardrails before you publish:
The goal is a defensible balance: maximum accountability data (how government performs), minimum personal exposure.
Knowledge check
1. The lesson describes a city that fulfilled a records request with a locked, sideways-scanned PDF. What core distinction does this example illustrate?
2. How does the open data model fundamentally differ from the FOIA model?
3. Why does the lesson suggest agencies should watch their FOIA logs as a 'demand signal'?
4. Select ALL correct answers. In the agency pipeline described, what should happen during the 'clean and standardize' step?
Select all the correct answers.
5. Select ALL correct answers. The scanned PDF failed the pipeline at the clean, publish, and document steps. What consequences follow from these specific failures?
Select all the correct answers.
Citizens and journalists want different things from the same portal. Serve both.
Journalists and analysts want bulk download. Give them a single CSV of the full dataset, a clear license (Creative Commons Zero or a public domain dedication removes all doubt about reuse), and a changelog so they can see what shifted between versions. Do not force them to click through 400 pages of a web viewer.
Civic developers want an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → with documentation and stable field names. If you rename score to inspection_score without warning, every app built on your data breaks overnight. Version your APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → and announce changes.
General citizens want a simple search and a readable page: type in a restaurant name, see its recent scores. Most will never touch the raw file, but the raw file is what powers the tools they eventually use.
A useful test: can a competent outsider, with no phone call to your office, download your data, understand every field, and reproduce a basic finding? If yes, you built open data. If they need to email you to decode a column, you built a slightly nicer compliance dump.
You will know the portal works when the outputs show up in the world:
Track reuse the way a product team tracks adoption: download counts, APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → call volume, and a public gallery of projects built on your data. Cities like New York with its Open Data portal publish usage and showcase civic projects, which builds a virtuous cycle: visible reuse justifies the budget to keep the pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → running.