Analyze Boston, the City of Boston open data portal (232 datasets)
- Provider
- City of Boston (Department of Innovation and Technology)
- License
- Open Data Commons Public Domain Dedication and License (PDDL) 1.0 for 216 of 232 datasets; 14 unspecified, 1 ODbL
- Size
- 403 files
- Records
- 232 datasets, 403 SQL-queryable resources
- Stored at
- data.boston.gov
- Access
- Free and public. No account with the data provider is needed.
- Category
- Urban & Civic
- Source
- https://data.boston.gov/
About Analyze Boston, the City of Boston open data portal (232 datasets)
Analyze Boston is the City of Boston's official open data portal, a CKAN 2.x instance at data.boston.gov publishing 232 datasets from 32 city departments across 1,261 resources. 403 of those resources are loaded into the CKAN DataStore, which means they are queryable over HTTP with SQL rather than only downloadable as files. 216 of the 232 datasets are released under the Open Data Commons Public Domain Dedication and License (PDDL) 1.0; 14 carry no stated license and one is ODbL. This archive is the portal itself rather than a single dataset: launch it to explore the whole catalog, discover which resource holds the series you need, and query it in place. The largest publisher by dataset count is Boston Maps (123 datasets, mostly ArcGIS-backed geospatial layers), followed by the Planning Department (14), Public Works (9), the Boston Police Department (9), and the Department of Innovation and Technology (8). The catalog spans 311 service requests, crime and fire incidents, property assessment rolls, building permits, food inspections, payroll, municipal expenditures, building energy and emissions disclosure, road safety, housing conditions, parcels, and administrative and climate-risk boundaries. Access is entirely anonymous and unauthenticated over three endpoints: package_search and package_show for the catalog, datastore_search for filtered and full-text row reads, and datastore_search_sql for server-side SQL including joins and aggregation. The schema card documents the two behaviours that trip up every first attempt: both datastore endpoints cap a single response at 32,000 rows (the SQL endpoint sets records_truncated: true when it does), and every DataStore column is typed text no matter what it holds, so numeric and date predicates need an explicit cast. Scigantic mounts nothing for this archive; the notebook talks to the portal directly over HTTPS. The same portal is what the OpenContext project (github.com/thealphacubicle/OpenContext) wraps when it exposes municipal CKAN, ArcGIS Hub and Socrata catalogs to AI assistants over MCP.
How to load Analyze Boston, the City of Boston open data portal (232 datasets) in Python
import io, json, urllib.error, urllib.parse, urllib.request
import pandas as pd
CKAN = "https://data.boston.gov/api/3/action"
def ckan(action, **params):
"""Any CKAN action endpoint. Plain urllib works; no auth, no headers."""
url = f"{CKAN}/{action}?" + urllib.parse.urlencode(params)
with urllib.request.urlopen(url, timeout=300) as r:
return json.load(r)["result"]
def sql(query):
"""Server-side SQL over the DataStore. Warns when the 32,000-row cap clips."""
res = ckan("datastore_search_sql", sql=query)
if res.get("records_truncated"):
print("WARNING: clipped at the 32,000-row cap; aggregate or page with OFFSET")
return pd.DataFrame(res["records"])
# What is on the portal, and what of it can be queried with SQL.
catalog = ckan("package_search", rows=1000)
rows = []
for p in catalog["results"]:
live = [r for r in p["resources"] if r.get("datastore_active")]
rows.append({
"package": p["name"],
"title": p["title"],
"org": (p.get("organization") or {}).get("title"),
"license": p.get("license_title"),
"resources": len(p["resources"]),
"queryable": len(live),
})
df = pd.DataFrame(rows)
print(f'{len(df)} datasets, {df["resources"].sum()} resources, {df["queryable"].sum()} queryable')
print(df.sort_values("queryable", ascending=False).head(15).to_string(index=False))Data structure
Files
The Analyze Boston portal itself, not one dataset. CKAN 2.x at https://data.boston.gov, 232 datasets from 32 city departments across 1261 resources, of which 403 are loaded into the CKAN DataStore and therefore queryable with SQL rather than only downloadable.
FOUR ENDPOINTS DO EVERYTHING. All are anonymous GET, all return JSON, all work with bare urllib.
package_search?q=&rows=1000 the whole catalog in one call (pass rows=0 for just the count)
package_show?id=<name> one dataset: notes, license, and every resource with its id,
format, url and datastore_active flag
datastore_search?resource_id= filtered row reads: filters={"neighborhood":"Roxbury"},
q=<full text>, fields=, sort=, limit=, offset=
datastore_search_sql?sql= server-side SQL, including JOINs across resources and
GROUP BY aggregation. This is the one that scales.
FORMATS ACROSS THE CATALOG: 384 CSV, 188 HTML, 125 GeoJSON, 120 ArcGIS GeoServices REST API, 118 KML, 105 Shapefile, 73 ZIP, 68 PDF, 47 XLSX. The GeoJSON, KML, Shapefile and ArcGIS entries are overwhelmingly Boston Maps layers, which are mirrors of one ArcGIS Hub item and are best read from the FeatureServer rather than the Hub download API (see the Climate Ready Boston archive for why).
BIGGEST PUBLISHERS BY DATASET COUNT: Boston Maps 123, Planning Department 14, Public Works 9, Boston Police Department 9, Department of Innovation and Technology 8, Inspectional Services 6, Environment Department 6, DoIT Data & Analytics 5, Archives and Record Management 5, Boston Transportation Department 5.
LICENSING: 216 of 232 datasets are PDDL 1.0 (public domain dedication), 14 state no license, and 1 is ODbL. Check package_show(...)["license_title"] before redistributing anything from the unspecified 14; absence of a license is not a grant.
Fourteen of the highest-value datasets on this portal are onboarded to Scigantic as their own archives with their own deep cards (311, crime, property assessment, building permits, food inspections, employee earnings, checkbook, BERDO, Vision Zero, fire, BPD FIO, RentSmart, Climate Ready Boston, parcels and boundaries). Use this archive to find the rest.
SEVEN OF THOSE FOURTEEN ARE NO LONGER READ FROM THIS API. Property assessment, BPD FIO, fire incidents, employee earnings, parcels and boundaries, BERDO and Climate Ready Boston are published annually or have stopped updating, so Scigantic serves them from a mirror at s3://scigantic-boston-open-data/ (raw/ byte-verbatim plus parquet/) that FUSE-mounts at ~/archive with no row cap and no presigned-URL 403. The seven that change daily (311, crime, building permits, food inspections, checkbook, Vision Zero, RentSmart) are deliberately NOT mirrored and still come from this API, because a stale mirror is worse than a slow query. If you are working with one of the mirrored seven, open that archive rather than querying the DataStore through this one.Example file: Find every dataset mentioning "housing", then list what is actually queryable in one of them:
ckan("package_search", q="housing", rows=50)["results"]
[r for r in ckan("package_show", id="rentsmart")["resources"] if r["datastore_active"]]
Sample notebook
Sample notebook for Analyze Boston, the City of Boston open data portal (232 datasets): worked code with the output it printed.
Analyze this dataset without downloading it
On Scigantic, Analyze Boston, the City of Boston open data portal (232 datasets) opens in a hosted JupyterLab notebook with the data already mounted, so you can read files where they are stored instead of copying them first. Each notebook includes an AI assistant that can see the mounted files and write and run analysis code. Open the dataset page to launch a notebook, or browse the full catalog.
This page is also available as Markdown and as JSON.
Related Urban & Civic datasets
- Boston 311 Service Requests, 2011 to present
- Philadelphia Zoning Districts
- Boston Fire Incident Reporting, 2012 to 2025
- Flateau V2: PLATEAU Building Footprints and Attributes for 303 Japanese Cities (GeoParquet)
- National Archives NSDAP membership records (microfilm publication A3340, MFKL and MFOK)
- Boston Vision Zero crash and fatality records, 2015 to present
- Boston Approved Building Permits, 2006 to present
- City Safety Data: 311 and 911 Fire Calls for Five US Cities