Real-Time Processing on Fast-Moving Trading Data: Ingesting MT5 and Acting on It

When I joined the Islero Capital project, the core challenge was obvious: MetaTrader 5 (MT5) generates a continuous stream of tick data, position updates, and account events, but the existing system was polling it every 60 seconds. By the time an alert fired, the market had already moved.

We needed real-time ingestion—with the ability to act on data within milliseconds of it arriving. Here's how we built it.

Why MT5 Data Is Hard to Work With

MT5 is a mature desktop trading platform built in C++. It wasn't designed to feed data to web services. Getting data out reliably requires one of three approaches:

1. MQL5 Expert Advisors (EAs) — Scripts that run inside MT5 and can push data via sockets or HTTP.
2. Python MetaTrader5 package — The official Python binding from MetaQuotes, Windows-only.
3. Third-party bridges — Commercial products that wrap MT5's DLL.

We went with option 1 (MQL5 EA) pushing to a Python relay, which then forwarded over WebSocket to our Node.js backend. It's a few extra hops, but it's the most reliable and doesn't require a Windows VM running 24/7 on our infra—just on the broker-side Windows machine where MT5 already runs.

The Data Bridge Architecture

MT5 (Windows) ──── MQL5 EA ──── TCP Socket
                                    │
                           Python relay (Windows)
                                    │
                         WebSocket (wss://) over TLS
                                    │
                          Node.js ingestion server
                                    │
                         Event bus (EventEmitter / Redis Pub/Sub)
                                    │
                    ┌───────────────┼───────────────┐
                  Alerts        Storage          AI Agent

The MQL5 EA fires on every OnTick() event—that's every price update. It serializes the current tick, open positions, and account equity into a JSON payload and sends it over a raw TCP socket to the Python relay on localhost.

// Simplified EA tick handler
void OnTick() {
    MqlTick tick;
    SymbolInfoTick(_Symbol, tick);
 
    string payload = StringFormat(
        "{\"type\":\"tick\",\"symbol\":\"%s\",\"bid\":%.5f,\"ask\":%.5f,"
        "\"time\":%lld,\"spread\":%d}",
        _Symbol, tick.bid, tick.ask,
        (long)tick.time, (int)((tick.ask - tick.bid) / _Point)
    );
 
    sendToRelay(payload);
}

The Python relay is minimal—it just receives from the TCP socket and re-emits over an authenticated WebSocket connection:

import asyncio
import websockets
import socket
 
async def relay_to_ws(ws_uri: str, mt5_host: str, mt5_port: int):
    tcp_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    tcp_sock.connect((mt5_host, mt5_port))
 
    async with websockets.connect(ws_uri) as ws:
        while True:
            data = tcp_sock.recv(4096).decode("utf-8")
            if data:
                await ws.send(data)

Normalization Layer

The raw EA output is per-symbol, per-event. Before anything downstream consumes it, we normalize it into a consistent schema and attach metadata:

type TradeEvent =
  | { type: "tick"; symbol: string; bid: number; ask: number; spread: number; ts: number }
  | { type: "position_opened"; ticket: number; symbol: string; lots: number; price: number; sl: number; tp: number; ts: number }
  | { type: "position_closed"; ticket: number; profit: number; pips: number; ts: number }
  | { type: "account_update"; equity: number; margin: number; free_margin: number; ts: number };
 
function normalize(raw: unknown): TradeEvent {
  const obj = raw as Record<string, unknown>;
  switch (obj.type) {
    case "tick":
      return {
        type: "tick",
        symbol: String(obj.symbol),
        bid: Number(obj.bid),
        ask: Number(obj.ask),
        spread: Number(obj.spread),
        ts: Number(obj.time) * 1000, // convert MT5 seconds to ms
      };
    case "TRADE":
      return parseTradeEvent(obj);
    default:
      throw new Error(`Unknown event type: ${obj.type}`);
  }
}

The normalization step is also where we drop duplicate ticks. MT5 can fire multiple OnTick() events for the same price if multiple symbols update simultaneously. We deduplicate by (symbol, ts) pair with a small in-memory window.

Event-Driven Processing

Downstream consumers subscribe to specific event types using Node's EventEmitter. This keeps the processing clean—each handler deals with exactly the events it cares about:

import { EventEmitter } from "events";
 
const bus = new EventEmitter();
bus.setMaxListeners(50);
 
// Alert engine subscribes to ticks
bus.on("tick", async (event: Extract<TradeEvent, { type: "tick" }>) => {
  await alertEngine.evaluate(event);
});
 
// Persistence subscribes to everything
bus.on("tick", persist);
bus.on("position_opened", persist);
bus.on("position_closed", async (event) => {
  await persist(event);
  await analyticsPipeline.recordClose(event);
});

For anything requiring fan-out to multiple services (analytics, AI agent, alerts), we push to Redis Pub/Sub instead, which decouples the Node process from downstream consumers that might be slow or fail:

await redis.publish(
  `trading:events:${event.type}`,
  JSON.stringify(event)
);

Handling Reconnects and Gaps

WebSocket connections drop. MT5 restarts on broker server maintenance. We built automatic reconnect logic with gap detection:

class MT5WebSocketClient {
  private lastTs = 0;
  private reconnectDelay = 1000;
 
  async connect(uri: string) {
    while (true) {
      try {
        const ws = new WebSocket(uri);
        await this.handleConnection(ws);
        this.reconnectDelay = 1000; // reset on clean connect
      } catch {
        await sleep(this.reconnectDelay);
        this.reconnectDelay = Math.min(this.reconnectDelay * 2, 30_000);
      }
    }
  }
 
  private onMessage(event: TradeEvent) {
    const gap = event.ts - this.lastTs;
    if (gap > 60_000 && this.lastTs > 0) {
      // Gap over 1 minute—fetch missing history from MT5 REST endpoint
      this.backfillHistory(this.lastTs, event.ts);
    }
    this.lastTs = event.ts;
    bus.emit(event.type, event);
  }
}

Gap detection was crucial. Without it, the alert engine would silently miss position closes that happened during disconnects.

Storage Strategy

We store two types of data differently:

• Tick data: Time-series in TimescaleDB. We partition by symbol and day. Tick volume means a plain PostgreSQL table would be unmanageable within weeks.
• Positions and account history: Standard PostgreSQL tables with full JSONB events for audit trail.

The AI agent (covered in a separate post) reads from both, joining position outcomes with the tick history around the open/close timestamps.

Key Takeaways

• Pull-based polling is wrong for trading data—a 60-second lag in a forex market is an eternity.
• The MQL5 EA + Python relay pattern is battle-tested and avoids needing Windows infra beyond the broker machine that already runs MT5.
• Deduplication matters at the source—MT5 fires multiple ticks per wall-clock second during volatile periods.
• Build gap detection from day one. Connection drops are guaranteed; knowing you missed data is better than silently missing it.
• Separate hot path (EventEmitter / Redis Pub/Sub) from cold path (TimescaleDB)—the alert engine and the historical store have completely different latency requirements.