Analyze Boston, the City of Boston open data portal (232 datasets)

Provider
City of Boston (Department of Innovation and Technology)
License
Open Data Commons Public Domain Dedication and License (PDDL) 1.0 for 216 of 232 datasets; 14 unspecified, 1 ODbL
Size
403 files
Records
232 datasets, 403 SQL-queryable resources
Stored at
data.boston.gov
Access
Free and public. No account with the data provider is needed.
Category
Urban & Civic
Source
https://data.boston.gov/

About Analyze Boston, the City of Boston open data portal (232 datasets)

Analyze Boston is the City of Boston's official open data portal, a CKAN 2.x instance at data.boston.gov publishing 232 datasets from 32 city departments across 1,261 resources. 403 of those resources are loaded into the CKAN DataStore, which means they are queryable over HTTP with SQL rather than only downloadable as files. 216 of the 232 datasets are released under the Open Data Commons Public Domain Dedication and License (PDDL) 1.0; 14 carry no stated license and one is ODbL. This archive is the portal itself rather than a single dataset: launch it to explore the whole catalog, discover which resource holds the series you need, and query it in place. The largest publisher by dataset count is Boston Maps (123 datasets, mostly ArcGIS-backed geospatial layers), followed by the Planning Department (14), Public Works (9), the Boston Police Department (9), and the Department of Innovation and Technology (8). The catalog spans 311 service requests, crime and fire incidents, property assessment rolls, building permits, food inspections, payroll, municipal expenditures, building energy and emissions disclosure, road safety, housing conditions, parcels, and administrative and climate-risk boundaries. Access is entirely anonymous and unauthenticated over three endpoints: package_search and package_show for the catalog, datastore_search for filtered and full-text row reads, and datastore_search_sql for server-side SQL including joins and aggregation. The schema card documents the two behaviours that trip up every first attempt: both datastore endpoints cap a single response at 32,000 rows (the SQL endpoint sets records_truncated: true when it does), and every DataStore column is typed text no matter what it holds, so numeric and date predicates need an explicit cast. Scigantic mounts nothing for this archive; the notebook talks to the portal directly over HTTPS. The same portal is what the OpenContext project (github.com/thealphacubicle/OpenContext) wraps when it exposes municipal CKAN, ArcGIS Hub and Socrata catalogs to AI assistants over MCP.

How to load Analyze Boston, the City of Boston open data portal (232 datasets) in Python

import io, json, urllib.error, urllib.parse, urllib.request
import pandas as pd

CKAN = "https://data.boston.gov/api/3/action"

def ckan(action, **params):
    """Any CKAN action endpoint. Plain urllib works; no auth, no headers."""
    url = f"{CKAN}/{action}?" + urllib.parse.urlencode(params)
    with urllib.request.urlopen(url, timeout=300) as r:
        return json.load(r)["result"]

def sql(query):
    """Server-side SQL over the DataStore. Warns when the 32,000-row cap clips."""
    res = ckan("datastore_search_sql", sql=query)
    if res.get("records_truncated"):
        print("WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET")
    return pd.DataFrame(res["records"])

# What is on the portal, and what of it can be queried with SQL.
catalog = ckan("package_search", rows=1000)
rows = []
for p in catalog["results"]:
    live = [r for r in p["resources"] if r.get("datastore_active")]
    rows.append({
        "package": p["name"],
        "title": p["title"],
        "org": (p.get("organization") or {}).get("title"),
        "license": p.get("license_title"),
        "resources": len(p["resources"]),
        "queryable": len(live),
    })
df = pd.DataFrame(rows)
print(f'{len(df)} datasets, {df["resources"].sum()} resources, {df["queryable"].sum()} queryable')
print(df.sort_values("queryable", ascending=False).head(15).to_string(index=False))

Data structure

Files

The Analyze Boston portal itself, not one dataset. CKAN 2.x at https://data.boston.gov, 232 datasets from 32 city departments across 1261 resources, of which 403 are loaded into the CKAN DataStore and therefore queryable with SQL rather than only downloadable.
FOUR ENDPOINTS DO EVERYTHING. All are anonymous GET, all return JSON, all work with bare urllib.
  package_search?q=&rows=1000    the whole catalog in one call (pass rows=0 for just the count)
  package_show?id=<name>         one dataset: notes, license, and every resource with its id,
                                 format, url and datastore_active flag
  datastore_search?resource_id=  filtered row reads: filters={"neighborhood":"Roxbury"},
                                 q=<full text>, fields=, sort=, limit=, offset=
  datastore_search_sql?sql=      server-side SQL, including JOINs across resources and
                                 GROUP BY aggregation. This is the one that scales.
FORMATS ACROSS THE CATALOG: 384 CSV, 188 HTML, 125 GeoJSON, 120 ArcGIS GeoServices REST API, 118 KML, 105 Shapefile, 73 ZIP, 68 PDF, 47 XLSX. The GeoJSON, KML, Shapefile and ArcGIS entries are overwhelmingly Boston Maps layers, which are mirrors of one ArcGIS Hub item and are best read from the FeatureServer rather than the Hub download API (see the Climate Ready Boston archive for why).
BIGGEST PUBLISHERS BY DATASET COUNT: Boston Maps 123, Planning Department 14, Public Works 9, Boston Police Department 9, Department of Innovation and Technology 8, Inspectional Services 6, Environment Department 6, DoIT Data & Analytics 5, Archives and Record Management 5, Boston Transportation Department 5.
LICENSING: 216 of 232 datasets are PDDL 1.0 (public domain dedication), 14 state no license, and 1 is ODbL. Check package_show(...)["license_title"] before redistributing anything from the unspecified 14; absence of a license is not a grant.
Fourteen of the highest-value datasets on this portal are onboarded to Scigantic as their own archives with their own deep cards (311, crime, property assessment, building permits, food inspections, employee earnings, checkbook, BERDO, Vision Zero, fire, BPD FIO, RentSmart, Climate Ready Boston, parcels and boundaries). Use this archive to find the rest.
SEVEN OF THOSE FOURTEEN ARE NO LONGER READ FROM THIS API. Property assessment, BPD FIO, fire incidents, employee earnings, parcels and boundaries, BERDO and Climate Ready Boston are published annually or have stopped updating, so Scigantic serves them from a mirror at s3://scigantic-boston-open-data/ (raw/ byte-verbatim plus parquet/) that FUSE-mounts at ~/archive with no row cap and no presigned-URL 403. The seven that change daily (311, crime, building permits, food inspections, checkbook, Vision Zero, RentSmart) are deliberately NOT mirrored and still come from this API, because a stale mirror is worse than a slow query. If you are working with one of the mirrored seven, open that archive rather than querying the DataStore through this one.

Example file: Find every dataset mentioning "housing", then list what is actually queryable in one of them: ckan("package_search", q="housing", rows=50)["results"] [r for r in ckan("package_show", id="rentsmart")["resources"] if r["datastore_active"]]

Sample notebook

Sample notebook for Analyze Boston, the City of Boston open data portal (232 datasets): worked code with the output it printed.

Analyze this dataset without downloading it

On Scigantic, Analyze Boston, the City of Boston open data portal (232 datasets) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.

This page is also available as Markdown and as JSON.

Related Urban & Civic datasets