Where do LLMs think neighborhoods are?
A dashboard comparing where different LLMs place San Francisco neighborhoods, and where they don't, against reference boundaries.
How precise a place needs to be depends on the context. In law, property and real estate transactions, zoning and administration, places need exact, defensible boundaries.
In everyday use, geographic research and vernacular geography, a neighborhood is often a vague region: a recognizable core with a fuzzy edge, and boundaries that differ by source.
So this tool doesn't assume one ground truth. It scores each model's answer against SF Planning's official boundaries, and compares the reference sources themselves (SF Planning, OpenStreetMap, Zillow, Click That Hood) to show how much they disagree.
March 2026 vs. November 2026
Medians across 24 neighborhoods Γ 3 runs, bounding-box answers scored against sf_planning. Higher IoU is better; lower centroid error is better.
| Model | Median IoU | Median centroid error | ||||
|---|---|---|---|---|---|---|
| March 2026 | November 2026 | Ξ | March 2026 | November 2026 | Ξ | |
| GPT-4o | 0.427 | 0.398 | β0.029 | 271 m | 265 m | β6 m |
| Claude Sonnet 4.6 | 0.421 | 0.439 | +0.018 | 304 m | 284 m | β20 m |
| Gemini 2.5 Flash | 0.359 | 0.398 | +0.039 | 384 m | 416 m | +32 m |
| GPT-4o Mini | 0.266 | 0.354 | +0.088 | 627 m | 593 m | β34 m |
| Claude Haiku 4.5 | 0.225 | 0.215 | β0.010 | 476 m | 523 m | +47 m |
By place tier
Tier 1 Β· famous, well defined
| Model | Median IoU | Median centroid error | ||||
|---|---|---|---|---|---|---|
| March 2026 | November 2026 | Ξ | March 2026 | November 2026 | Ξ | |
| GPT-4o | 0.595 | 0.603 | +0.008 | 212 m | 215 m | +3 m |
| Claude Sonnet 4.6 | 0.619 | 0.613 | β0.006 | 203 m | 203 m | Β±0 m |
| Gemini 2.5 Flash | 0.576 | 0.504 | β0.072 | 258 m | 258 m | Β±0 m |
| GPT-4o Mini | 0.421 | 0.412 | β0.009 | 203 m | 163 m | β40 m |
| Claude Haiku 4.5 | 0.321 | 0.321 | Β±0.000 | 212 m | 185 m | β27 m |
Tier 2 Β· common, moderately vague
| Model | Median IoU | Median centroid error | ||||
|---|---|---|---|---|---|---|
| March 2026 | November 2026 | Ξ | March 2026 | November 2026 | Ξ | |
| GPT-4o | 0.445 | 0.407 | β0.038 | 246 m | 259 m | +13 m |
| Claude Sonnet 4.6 | 0.437 | 0.432 | β0.005 | 199 m | 199 m | Β±0 m |
| Gemini 2.5 Flash | 0.449 | 0.415 | β0.034 | 333 m | 350 m | +17 m |
| GPT-4o Mini | 0.342 | 0.399 | +0.057 | 389 m | 259 m | β130 m |
| Claude Haiku 4.5 | 0.179 | 0.167 | β0.012 | 817 m | 608 m | β209 m |
Tier 3 Β· local knowledge
| Model | Median IoU | Median centroid error | ||||
|---|---|---|---|---|---|---|
| March 2026 | November 2026 | Ξ | March 2026 | November 2026 | Ξ | |
| GPT-4o | 0.445 | 0.429 | β0.016 | 378 m | 380 m | +2 m |
| Claude Sonnet 4.6 | 0.394 | 0.394 | Β±0.000 | 403 m | 403 m | Β±0 m |
| Gemini 2.5 Flash | 0.419 | 0.423 | +0.004 | 432 m | 485 m | +53 m |
| GPT-4o Mini | 0.227 | 0.251 | +0.024 | 882 m | 868 m | β14 m |
| Claude Haiku 4.5 | 0.191 | 0.245 | +0.054 | 465 m | 465 m | Β±0 m |
Tier 4 Β· hyper-local vernacular
| Model | Median IoU | Median centroid error | ||||
|---|---|---|---|---|---|---|
| March 2026 | November 2026 | Ξ | March 2026 | November 2026 | Ξ | |
| GPT-4o | 0.245 | 0.212 | β0.033 | 334 m | 395 m | +61 m |
| Claude Sonnet 4.6 | 0.405 | 0.439 | +0.034 | 482 m | 487 m | +5 m |
| Gemini 2.5 Flash | 0.255 | 0.238 | β0.017 | 673 m | 503 m | β170 m |
| GPT-4o Mini | 0.220 | 0.369 | +0.149 | 566 m | 511 m | β55 m |
| Claude Haiku 4.5 | 0.293 | 0.259 | β0.034 | 632 m | 776 m | +144 m |