Citi Bike Lakehouse
In progress2026 — · Python · Redpanda · SeaweedFS · Apache Iceberg · DuckDB · dbt · Dagster · Streamlit / Evidence
The problem
When a Citi Bike station runs out of bikes, the next rider doesn't file a report — they walk away. The system records a station sitting at zero, and then... nothing. No rentals, because there's nothing to rent. The failure is invisible in the obvious data: observed demand goes to zero at exactly the moment real demand peaks. Statisticians call this censored demand, and it's why you can't answer "how many bikes did this station actually need?" by counting rides.
The same thing happens in reverse when a station is full and there's nowhere to dock.
These failures cost real money — rebalancing trucks dispatched too late, lost rides, riders who churn to other transport. So the question this project exists to answer: when will a station run out of bikes or docks, before it happens? The approach is to combine live station status with historical demand, and to look hard at what the data is missing rather than only what it says.
Why build it this way
This is a learning-and-showcase project, and the goal is to learn the plumbing, not rent it. Every layer is something I run myself:
| Layer | Tool | Why |
|---|---|---|
| Ingestion & glue | Python | The lingua franca of data engineering |
| Streaming | Redpanda | Kafka-compatible event streaming for the live station feeds |
| Object storage | SeaweedFS | S3-compatible storage I operate myself instead of renting |
| Table format | Apache Iceberg | Real lakehouse semantics: schema evolution, time travel, atomic commits |
| Query & transform | DuckDB + dbt | Fast local analytics with tested, versioned models (dbt-duckdb) |
| Orchestration | Dagster | Asset-based scheduling and lineage |
| Serving | Streamlit or Evidence | A dashboard for the predictions |
How I'm building it
Tracer-bullet style: get the thinnest possible end-to-end slice working first, then widen each segment. Every phase produces something that runs, and every phase gets written up as a post here as I finish it.
Phase 0 — one bullet, end to end (current): fetch the station feed once, write the raw response bytes to a local file named by the feed's own update timestamp, convert that file to Parquet with DuckDB, query the Parquet, print the answer. No streaming, no cluster, no orchestrator — just proof that fetch, store, transform, and query connect.
After that (rough shape, firming up as I go): continuous ingestion through Redpanda; a raw zone on SeaweedFS with Iceberg tables on top; dbt models for cleaning and aggregation; joining live availability against historical trip data to model the censored demand; runout predictions; and a dashboard to watch them land or miss.
Follow along
Each completed phase becomes a post on the blog, linked here as the project grows.