Vision Pipeline

Historical market data replay

Binance Vision
event server

On-demand fetch from data.binance.vision, parse ZIP/CSV archives, and stream Binance-shaped events over WebSocket, NDJSON, or SSE — so any app that already speaks Binance WS can replay history.

How it works

  1. Request — client or agent specifies market, symbol, data type, and date range.
  2. Resolve — server probes Vision; missing days (delisted coins, gaps) are reported, not fatal.
  3. Fetch — downloads only available ZIPs into a local cache (checksum verified).
  4. Stream — parses CSV → Binance WS JSON and paces events by timestamp × speed.

WebSocket streaming

Connect, send one JSON config, receive events. Control frames mark missing days.

const ws = new WebSocket(`ws://${location.host}/ws/replay`);
ws.onopen = () => ws.send(JSON.stringify({
  market: "um",
  data_type: "aggTrades",
  symbol: "BTCUSDT",
  start_date: "2024-01-01",
  end_date: "2024-01-03",
  speed: 100,
  limit: 500
}));
ws.onmessage = (m) => {
  const ev = JSON.parse(m.data);
  if (ev.e === "datasetMissing") console.warn("gap", ev.date, ev.key);
  else if (ev.e === "aggTrade") console.log(ev.s, ev.p, ev.q, ev.T);
  else if (ev.e === "replayEnd") console.log("done");
};
EndpointShape
WS /ws/replayRaw Binance events
WS /market/wsOptional SUBSCRIBE, then replay config / REPLAY
WS /market/stream{"stream","data"} envelope

HTTP API

MethodPathPurpose
GET/healthLiveness
GET/api/catalogKnown dataset families + schemas
GET/api/symbols?market=um&data_type=aggTradesSymbols that have that dataset
GET/api/resolve?...Which days exist in a range (sparse-safe)
GET/api/browse?prefix=data/Live S3 listing
GET/stream/ndjson?...Paced NDJSON body
GET/stream/sse?...Server-Sent Events

Shared query fields: market, data_type, symbol, start_date, end_date, frequency, interval (klines), speed, limit.

curl -N "http://${location.host}/stream/ndjson?\
market=um&data_type=metrics&symbol=BTCUSDT&\
start_date=2024-01-01&end_date=2024-01-02&speed=1000&limit=5"

Missing datasets

Not every symbol has every data type for every day. When an archive is absent, the stream emits {"e":"datasetMissing","date":"...","key":"..."} and continues. Use /api/resolve first to preview coverage.

Event shapes

// aggTrade
{"e":"aggTrade","E":1704067200038,"a":1965151407,"s":"BTCUSDT",
 "p":"42313.9","q":"0.046","f":4426785111,"l":4426785111,"T":1704067200038,"m":true}

// kline (historical bars are closed: k.x=true)
{"e":"kline","s":"BTCUSDT","k":{"t":...,"T":...,"i":"1m","o":"...","c":"...","x":true}}