Thomas' Learning Hub
πŸ—ΊοΈ

Where do LLMs think neighborhoods are?

A dashboard comparing where different LLMs place San Francisco neighborhoods, and where they don't, against reference boundaries.

How precise a place needs to be depends on the context. In law, property and real estate transactions, zoning and administration, places need exact, defensible boundaries.

In everyday use, geographic research and vernacular geography, a neighborhood is often a vague region: a recognizable core with a fuzzy edge, and boundaries that differ by source.

So this tool doesn't assume one ground truth. It scores each model's answer against SF Planning's official boundaries, and compares the reference sources themselves (SF Planning, OpenStreetMap, Zillow, Click That Hood) to show how much they disagree.

Launch Streamlit Dashboard β†—
Models
5 LLMs scored
Places
24 SF neighborhoods
Tiers
4 tiers, iconic to vernacular

March 2026 vs. November 2026

Medians across 24 neighborhoods Γ— 3 runs, bounding-box answers scored against sf_planning. Higher IoU is better; lower centroid error is better.

ModelMedian IoUMedian centroid error
March 2026November 2026Ξ”March 2026November 2026Ξ”
GPT-4o0.4270.398βˆ’0.029271 m265 mβˆ’6 m
Claude Sonnet 4.60.4210.439+0.018304 m284 mβˆ’20 m
Gemini 2.5 Flash0.3590.398+0.039384 m416 m+32 m
GPT-4o Mini0.2660.354+0.088627 m593 mβˆ’34 m
Claude Haiku 4.50.2250.215βˆ’0.010476 m523 m+47 m
By place tier

Tier 1 Β· famous, well defined

ModelMedian IoUMedian centroid error
March 2026November 2026Ξ”March 2026November 2026Ξ”
GPT-4o0.5950.603+0.008212 m215 m+3 m
Claude Sonnet 4.60.6190.613βˆ’0.006203 m203 mΒ±0 m
Gemini 2.5 Flash0.5760.504βˆ’0.072258 m258 mΒ±0 m
GPT-4o Mini0.4210.412βˆ’0.009203 m163 mβˆ’40 m
Claude Haiku 4.50.3210.321Β±0.000212 m185 mβˆ’27 m

Tier 2 Β· common, moderately vague

ModelMedian IoUMedian centroid error
March 2026November 2026Ξ”March 2026November 2026Ξ”
GPT-4o0.4450.407βˆ’0.038246 m259 m+13 m
Claude Sonnet 4.60.4370.432βˆ’0.005199 m199 mΒ±0 m
Gemini 2.5 Flash0.4490.415βˆ’0.034333 m350 m+17 m
GPT-4o Mini0.3420.399+0.057389 m259 mβˆ’130 m
Claude Haiku 4.50.1790.167βˆ’0.012817 m608 mβˆ’209 m

Tier 3 Β· local knowledge

ModelMedian IoUMedian centroid error
March 2026November 2026Ξ”March 2026November 2026Ξ”
GPT-4o0.4450.429βˆ’0.016378 m380 m+2 m
Claude Sonnet 4.60.3940.394Β±0.000403 m403 mΒ±0 m
Gemini 2.5 Flash0.4190.423+0.004432 m485 m+53 m
GPT-4o Mini0.2270.251+0.024882 m868 mβˆ’14 m
Claude Haiku 4.50.1910.245+0.054465 m465 mΒ±0 m

Tier 4 Β· hyper-local vernacular

ModelMedian IoUMedian centroid error
March 2026November 2026Ξ”March 2026November 2026Ξ”
GPT-4o0.2450.212βˆ’0.033334 m395 m+61 m
Claude Sonnet 4.60.4050.439+0.034482 m487 m+5 m
Gemini 2.5 Flash0.2550.238βˆ’0.017673 m503 mβˆ’170 m
GPT-4o Mini0.2200.369+0.149566 m511 mβˆ’55 m
Claude Haiku 4.50.2930.259βˆ’0.034632 m776 m+144 m
Where do LLMs think neighborhoods are? | Spatial Eval