Part IV — Inside the Event Stream Pipeline

In the previous article we examined how the order book structures and organises incoming orders, enabling the matching engine to deterministically produce trades. However, producing a trade is only one part of the system. Once a trade is generated by the matching engine, it becomes an externalised event that must be propagated beyond the in-memory state of the engine.

Recall the main system diagram from part I. In this article we will cover what happens after a trade has been emitted from the engine.

The following diagram shows the full path of a trade event, from generation inside the matching engine to consumption by external systems:

flowchart TD

OB["Insert order in order book (not of interest here)"]
E{Engine}
EC["External Consumers"]
AL[Analytics Layer]
PL[Persistence Layer]
RS[Redis Stream]

E -->|No| OB 

E -->|Yes| J[Emit Trade Event]

J --> BCT[Background Consumer Thread: Receive Trade Event]

BCT --> EP[Event Publisher: publish event to Redis Stream]

EP --> RS

RS --> EC

EC --> AL

EC --> PL

The key boundary in this system is the point where the matching engine emits a Trade event. From this moment onward, the engine is no longer responsible for what happens to the data. Its only role is to guarantee that the event is produced deterministically.

This boundary is implemented using an asynchronous multi-producer, single-consumer (MPSC) channel. In this context, the engine will be the producer and another component will be the single consumer. That component will be a long-running background worker.

This background worker receives these events and acts as a bridge between the Rust runtime and external systems. Its responsibility is to forward each Trade event to an external event stream (Redis Streams in this prototype).

Event Consumer Implementation

#

The event consumer worker runs as a long-lived asynchronous task. It continuously receives Trade events from the matching engine and forwards them to Redis.

The worker definition is:

use deadpool_redis::{Pool as RedisPool};
use tokio::sync::mpsc::{Receiver as TokioReceiver};

use crate::domain::trade::Trade;

pub async fn evt_consumer_worker(mut receiver: TokioReceiver<Trade>, redis_pool: &RedisPool) {
    let mut connection = create_redis_connection(redis_pool).await
        .expect("Failed to create Redis connection");

    while let Some(event) = receiver.recv().await {
        write_to_redis(&mut connection, event).await
            .expect("Failed to stream event to Redis");
    };
}

The worker continuously listens on the MPSC receiver. Each incoming Trade event is processed sequentially and forwarded to Redis.

The function responsible for publishing events to Redis is:

use deadpool_redis::{Connection};
use redis::RedisResult;

use crate::domain::trade::Trade;

pub async fn write_to_redis(connection: &mut Connection, event: Trade) -> RedisResult<()> {
    let evt =  serde_json::to_string(&event).unwrap();

    let result = redis::cmd("XADD")
        .arg("events_stream")
        .arg("*")
        .arg("event")
        .arg(evt)
        .query_async(connection)
        .await?;

    println!("[redis] pushed event");

    result
}

Each event is serialised and appended using the XADD command.

Supporting code for connection management and pool creation has been omitted for brevity. The important architectural detail is that the worker maintains a Redis connection and continuously forwards Trade events from the MPSC channel to a Redis Stream.

This design introduces a clear separation between deterministic computation and external side effects.

Design decisions

#

Engine does not perform I/O

#

The matching engine is deliberately isolated from any external side effects. Its only responsibility is to generate deterministic Trade events.

Asynchronous boundary via MPSC channel

#

Trade events are passed out of the engine using a \( Tokio \) MPSC channel. The engine acts as the producer, while a single background worker consumes events asynchronously.

Non-blocking event emission (try_send)

#

Event emission uses try_send rather than a blocking .send(). This ensures the matching engine remains responsive and is not stalled by downstream latency or backpressure from the consumer pipeline.

Single consumer design

#

The system uses a single consumer to maintain ordering guarantees for Trade events. This simplifies consistency at the cost of horizontal scalability.

Redis Streams as event log

#

Redis Streams is treated as an external event sink rather than part of the matching system. It provides an append-only log of trade activity for downstream services.

This completes the journey from order submission to external event propagation. We have now seen how a deterministic in-memory matching engine produces trades and how those trades are streamed out of the system for downstream processing.

In the next article we will move beyond the matching engine and explore how downstream services consume trade events from Redis Streams.