/projects / market-data-connectors

Market Data Connectors

A standalone Rust workspace for consuming public cryptocurrency and FX feeds through one domain model, validating order-book integrity, and delivering events to application-owned consumers.

What it is

Market Data Connectors sits between venue protocols and an application. Provider adapters parse each venue's wire format, apply its integrity rules, and emit a common MarketEvent<MarketDataKind> envelope. Consumers can then use the same event model for snapshots, deltas, book tickers, trades, reference rates, and integrity alerts.

The workspace is deliberately a connector and normalization layer. It leaves storage, pricing, freshness, and business decisions to the application that consumes the events.

Core design choices

Integration choices

MarketDataSink and FnSink deliver owned events when a complete value is convenient. StreamingSink receives book levels incrementally and borrows payloads for the lower-allocation path. Its callback runs synchronously on the connector task, so blocking work belongs in an asynchronous consumer behind the bounded channel.

mdc-sinks also provides L2Book and TopOfBookProjector. The projector stages recovery snapshots, applies deltas, rejects crossed or incomplete books, and returns either Available or Unavailable. It does not impose a maximum quote age: the application compares effective_ts with its own clock and decides when a quote expires.

Workspace crates

mdc-core

Domain models, event envelopes, sink traits, integrity primitives, resynchronization, catalog, and REST-quote contracts. It has no I/O runtime dependency.

mdc-providers

Feature-gated WebSocket and REST connectors, parsers, recovery, polling, catalog discovery, and quote sources.

mdc-sinks

Book and top-of-book projections, bounded asynchronous delivery, and application-facing sink adapters.

mdc-testkit

Deterministic mock WebSocket and REST servers, fixtures, fault injection, property strategies, and conformance helpers.

mdc-inspect

A terminal application that exercises provider sessions through the same consumer path as an integrating application.

Supported providers

Provider support is feature-gated. Each module owns its endpoint, configuration, stream names, protocol rules, and recovery behavior.

Provider Feature Data Distinct handling
Binance Spot binance Depth, book ticker, trades REST snapshot reconciliation and silence watchdog
Kraken v2 kraken Book, trades Book checksum verification, resubscription, quarantine
Coinbase Exchange coinbase Level 2, matches Heartbeat and sequence monitoring with resubscription
Coinbase Advanced Trade coinbase-advanced Level 2, market trades Connection-wide sequence tracking and resubscription
Bitfinex v2 bitfinex Book, trades Signed book checksum verification and channel recovery
Gate.io Spot v4 gateio Order-book updates, trades Sequence continuity and in-place resubscription
Bitstamp v2 bitstamp Order-book updates, trades Timestamp-gated snapshot reconciliation
Huobi / HTX Spot huobi MBP book, trades In-band snapshots and bounded decompression
LMAX Digital lmax Full book updates, trades Validated book replacement and silence monitoring
VWD / Infront vwd Polled indicative FX quotes Validation with optional inverse and scaling transforms
Frankfurter frankfurter Polled reference FX rates Complete response validation and UTC date normalization

Terminal demos

These recordings show the interactive inspector and the states it exposes while market data is loading, synchronized, interrupted, or historical. Each demo starts paused. Use the native controls to play, pause, seek, or open it fullscreen; starting one pauses the others.

1. Guided setup

Select a provider, enter an invalid symbol, then a valid one. The inspector shows validation feedback before opening the symbol list.

cargo run -p mdc-inspect --features all

2. Live multi-symbol order book

Binance provides BTCUSDT and ETHUSDT order books. The view moves through loading, awaiting snapshot, and in sync; then it demonstrates switching symbols, focusing panes, and scrolling.

cargo run -p mdc-inspect -- --provider binance \
  --symbol BTCUSDT --symbol ETHUSDT

3. Integrity failure and historical data

A controlled disconnect creates a connection loss and sequence gap. The affected view becomes unavailable; historical data is labeled [HISTORICAL] instead of being treated as live.

A local mock or controlled disconnect is used for this scenario.

4. Help and responsive layout

Press ? to open the keyboard-help overlay. When the terminal is narrower than 70 columns, the side-by-side panes stack vertically to fit the available space.

cargo run -p mdc-inspect

5. Reference-rate mode

The Frankfurter provider displays EUR, USD, and GBP reference rates through the same inspector workflow.

cargo run -p mdc-inspect --features frankfurter -- \
  --provider frankfurter --symbol EUR --symbol USD --symbol GBP

Integrity model and boundaries

A provider can emit only the alerts relevant to its protocol. The public alert model includes GapDetected, ChecksumMismatch, SilenceTimeout, HeartbeatTimeout, ResyncStarted, ResyncCompleted, Quarantined, SourceError, ConnectionLost, and SinkFailure. Consumers should handle these alerts before using an L2-derived quote.

The integrity path is provider-owned and cannot be disabled through one universal switch while keeping the same L2 feed. If an application does not need depth, it can select a provider's ticker or trade stream where available. If it does need depth, it keeps the venue-specific sequence and checksum rules and owns only the downstream book, freshness, and business policies.

Engineering and verification

The workspace uses Rust edition 2024, declares Rust 1.85 as its minimum supported version, and keeps the lockfile committed. Its verification gate covers formatting, Clippy with warnings denied, all-feature tests, documentation, per-feature checks, benchmark compilation, dependency and license checks, and secret scanning.

Tests terminate networking at local mock servers rather than live venues. Fixtures, property-based tests, deterministic virtual time, fault injection, conformance runners, and allocation benchmarks cover malformed data, gaps, bad checksums, stale snapshots, timeouts, reconnects, cancellation, and bounded delivery.

Read the repository's provider usage guide , engineering standards , and contribution guide for the full integration and verification contract.

Explore the code

The repository contains the connectors, shared domain model, downstream sinks, test kit, provider examples, and engineering standards.

open on GitHub →