NASDAQ TotalView-ITCH 5.0 → Polars, pandas, Arrow or Parquet. At native speed, in one line.
A real NASDAQ day (Dec 30, 2019): 268.7M messages counted in 4 s, AAPL's orders pulled in 5 s, its full-day order book rebuilt in 4 s.
import fastitch
frames = fastitch.read("12302019.NASDAQ_ITCH50.gz", types="orders", symbols=["AAPL", "NVDA"])
frames["add_order"] # a Polars DataFrame, one row per Add Order messageNASDAQ publishes full-depth, message-by-message order book data as TotalView-ITCH 5.0 files: a few gigabytes per day, hundreds of millions of binary messages. Existing Python parsers hand you one Python object per message, which is fine for a demo and painful for a trading day.
fastitch decodes straight into columnar Arrow buffers in C++, so a whole day becomes ordinary DataFrames you can filter, group and join, and it never builds a Python object per message.
| Task (first 10M messages of a real NASDAQ day, 340 MB) | fastitch | itchcpp (C++) | itchfeed (Python) | meatpy (Python) |
|---|---|---|---|---|
| Add orders → pandas DataFrame | 0.85 s | 43.6 s (51x slower) | 57.6 s (67x slower) | n/a |
| Decode every message | 1.11 s | 5.1 s (4.6x slower) | 17.8 s (16x slower) | 24.9 s (22x slower) |
Full trading day (Dec 30, 2019 · 7.7 GB · 268,744,780 messages):
Count every message by type (fastitch info) |
4.1 s (65M msgs/s) |
| All AAPL add orders → DataFrame (695,972 rows) | 5.0 s |
| AAPL 5-level order book, 1-second grid, full session | 4.4 s |
2-vCPU Linux VM, Python 3.13, file in the OS page cache, timings include imports. Comparison table is best of 2, each tool in its own process;
fastitch's "decode" also builds Arrow tables for all 23 types, while the others only create message objects.
On the same real data, fastitch's output matches itchcpp's decoding exactly, and the rebuilt AAPL book is never crossed.
Reproduce with python benchmarks/bench.py <ITCH file>; real NASDAQ days are free at emi.nasdaq.com.
pip install fastitch # prebuilt wheels for Linux, macOS, Windows · Python 3.10–3.13
pip install "fastitch[all]" # + polars, pandas, and ISA-L for faster .gz reading- All 23 ITCH 5.0 message types, every field, named and typed (
add_order,order_executed,noii, …) - DataFrames out: Polars (default), pandas or a zero-copy Arrow table
- Filters pushed into C++: by symbol, by time of day (
start="09:30", end="16:00"), by message type or group ("orders","trades","auction","admin") - Order-book reconstruction: top-N levels, emitted on every change or sampled on a time grid
- Constant-memory streaming:
iter_batches()andto_parquet()handle full days on a laptop - Real timestamps: the trading date is read from NASDAQ file names, so you get
datetime[ns], not "nanoseconds since midnight" - Reads
.gzdirectly, plus raw bytes and file objects - A CLI for when you don't want to open Python at all
- Synthetic sessions built in, so you can try everything without a 5 GB download
import fastitch
# Everything (one DataFrame per message type)
frames = fastitch.read("12302019.NASDAQ_ITCH50.gz")
# Only what you need: filtering happens in C++ before anything reaches Python
fills = fastitch.read(
"12302019.NASDAQ_ITCH50.gz",
types=["order_executed", "order_executed_with_price", "trade"],
symbols=["AAPL", "MSFT"],
start="09:30", end="16:00",
)
# A single type, as pandas
adds = fastitch.read_frame("12302019.NASDAQ_ITCH50.gz", "add_order", backend="pandas")types accepts codes ("A"), names ("add_order") or groups: orders, trades, auction, admin, all.
Prices are floats in dollars (pass raw_prices=True for the exact integer ticks).
book = fastitch.book("12302019.NASDAQ_ITCH50.gz", "AAPL", depth=5, every="100ms")every=None (the default) emits a row each time the top depth levels change; a duration such as
"1s" or "100ms" samples on a regular grid. The book is always built from the open, so a
start/end window never desyncs it. Correctness is tested row-for-row against an independent
pure-Python reference implementation.
# Stream to one Parquet file per message type
fastitch.to_parquet("12302019.NASDAQ_ITCH50.gz", "parquet/", types="orders")
# Or process batch by batch
for batch in fastitch.iter_batches("12302019.NASDAQ_ITCH50.gz", "trades", batch_rows=5_000_000):
handle(batch["order_executed"])fastitch info 12302019.NASDAQ_ITCH50.gz # message counts by type
fastitch convert 12302019.NASDAQ_ITCH50.gz -o parquet/ -t orders -s AAPL MSFT
fastitch book 12302019.NASDAQ_ITCH50.gz -s AAPL -d 10 -e 1s -o aapl_book.parquet
fastitch sample demo.itch -n 1000000 # synthetic data to play withfrom fastitch.synthetic import generate
data = generate(1_000_000, symbols=["AAPL", "MSFT"]) # a self-consistent ITCH session, as bytes
fastitch.read(data)Real sample days are published by NASDAQ at https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/.
See examples/quickstart.py for spreads, imbalance and volume bars in a dozen lines.
_spec.pyholds the byte layout of every ITCH 5.0 message: the single source of truth, checked against the spec's message lengths at import.- The C++ core is layout-driven: it frames the length-prefixed stream (messages may straddle read chunks), applies symbol/time filters, and appends each field to a contiguous native-endian column. The GIL is released while parsing.
- Python wraps those columns as Arrow arrays without copying them again, then hands them to Polars or pandas.
- The order book keeps live orders in a hash map and price levels in sorted maps per symbol, applying adds, executions, cancels, deletes and replaces exactly as the spec defines.
Run them yourself: python benchmarks/bench.py (synthetic) or python benchmarks/bench.py <ITCH file>.
Each tool runs in its own process; best of three.
Issues and PRs welcome. Good first contributions: MoldUDP64 packet captures as input, more book analytics (queue position, order flow imbalance), and a Rust or DuckDB front end.
pip install -e ".[test]" && pytestMIT. ITCH is a trademark of Nasdaq, Inc.; this project is not affiliated with or endorsed by Nasdaq.