The data
A public dataset of 693,071 ride price quotes from Uber and Lyft in Boston, collected between November 26 and December 18, 2018, alongside hourly weather readings for the same neighborhoods.
| Dimension | What's in it |
|---|---|
| Providers | Uber (385,663 quotes) and Lyft (307,408) |
| Ride types | 13, from shared rides (UberPool, Lyft Shared) to premium (Black SUV, Lux Black XL) |
| Locations | 12 Boston neighborhoods as pickup and drop-off points, including Back Bay, Fenway, the Financial District and Northeastern University |
| Trip features | Distance (median 2.2 miles), surge multiplier, timestamp, source and destination |
| Weather | 6,276 hourly readings: temperature, rain, clouds, pressure, humidity and wind |
Two quirks shape any model built on it. Every Uber Taxi quote (55,095 rows) comes without a price, and 97% of rides have no surge, so the surge pricing that riders notice most is rare in the data.
The pipeline
The data lived in Google BigQuery. At 690K+ records, the initial queries took more than four minutes.
Moving the heavy processing to PySpark brought query latency down from 4+ minutes to 12 seconds, which made fast iteration on features and models possible.
The models
I built and validated two models with feature engineering and cross-validation:
| Model | Mean absolute error |
|---|---|
| XGBoost (selected) | $6.69 |
| Random Forest | $7.11 |
Serving it
Predictions were served through an interactive Flask dashboard, so fares and model outputs could be explored interactively.
This is my strongest data engineering work: cloud warehouse SQL, ETL, and performance tuning at scale.