Skip to content
dgbit GitHub

← back to writing

Why dgbit uses an NNG service bus, not HTTP between workers

dgbit Team · ·
architectureinfrastructure

The architecture diagram in the dgbit README has a detail that is easy to miss on a first read. The FastAPI backend talks to its workers — the data service, the backtest worker, and the strategy service — over NNG IPC, not HTTP. The configuration file makes the choice explicit:

NNG_COMMAND_ADDRESS=ipc:///tmp/dgbit_cmd.ipc
NNG_EVENT_ADDRESS=ipc:///tmp/dgbit_evt.ipc

Two unix-domain sockets, one for commands and one for events. No http://localhost:8001 between services. That is a deliberate choice and it is worth unpacking, because it tells you something about the kind of system dgbit is trying to be.

What NNG is

NNG (nanomsg-next-gen) is a low-latency messaging library that implements the classic Scalability Protocols: REQ/REP, PUB/SUB, PUSH/PULL, SURVEY, BUS, PAIR. You bind a socket to an address and the library handles framing, retry, and the connection state machine. It can run over TCP, but it can also run over a unix domain socket (ipc://), in-process (inproc://), or over websockets. dgbit’s defaults are unix sockets, which is the right choice when all the workers run on the same machine.

For our purposes the key properties of an NNG socket are: messages are framed, the pattern is enforced by the socket type (you cannot accidentally send a REQ without a REP coming back), and the per-message overhead is dominated by the kernel’s IPC cost rather than by header parsing.

What an HTTP service bus would look like instead

The default for a Python microservice stack is HTTP between every service. The data service would expose a REST endpoint at :8001/klines, the backtest worker would hit :8001/klines?symbol=BTCUSDT&interval=15, and the response would come back as JSON. This works and it is well-understood. It has three costs:

  1. Per-call latency floor. Every HTTP round-trip pays for header parsing, content-length negotiation, TLS (if enabled), and the socket overhead. On localhost the round-trip cost is small but not zero; under a tight backtest loop it adds up.
  2. JSON marshalling. A kline tick is naturally a small struct: timestamp, open, high, low, close, volume. JSON encodes it as a verbose, parsed-by-string object. The encoding overhead is real.
  3. State machine for failure. HTTP gives you “the request failed somewhere between client and server” as one signal. You have to layer retry logic, idempotency keys, and timeout policy on top.

NNG removes all three. The framing is binary, the transport is a kernel pipe, and the socket pattern (REQ/REP for commands, PUB/SUB for events) makes the failure semantics explicit at the protocol level.

Why two addresses

The README configures two separate NNG addresses:

  • NNG_COMMAND_ADDRESS carries requests. The API server asks the data service for klines, asks the backtest worker to run a backtest, asks the strategy service to generate a signal. The pattern is request/response.
  • NNG_EVENT_ADDRESS carries broadcasts. When a backtest finishes, when a trade enters or exits, when a strategy generates a signal, the responsible worker publishes an event. Whoever is subscribed picks it up.

Splitting commands and events on separate sockets matters because they have different semantics. A command is one-to-one and back. An event is one-to-many and fire-and-forget. Mixing them on one socket would force either pattern to compromise: either commands get publish-style “maybe nobody is listening” semantics, or events get blocked behind in-flight commands. The split keeps both flows clean.

The event socket is also what feeds the FastAPI WebSocket endpoint at /api/ws/events. The README lists the event vocabulary: job.created, job.completed, job.failed, trade.entered, trade.exited, signal.generated. Each one is published on the NNG event bus by whichever worker produced it, the API server consumes from that bus, and the WebSocket clients see the same event almost immediately. There is no separate fan-out queue; NNG’s PUB/SUB does that work natively.

What this means for the strategy author

If you are writing a strategy and never touching the worker layer, none of this affects you directly. The framework’s surface to a strategy author is still: subclass BaseStrategy, implement generate_signal, register with strategy_registry. The NNG bus runs underneath you.

But the bus does affect three things that matter to anyone running the stack:

  1. Latency under load. Because the bus is binary and the transport is a kernel pipe, signal generation under high message rates does not stall on header parsing. If you are running a strategy that fires many signal evaluations per second across many symbols, the worker layer keeps up.
  2. Backpressure model. PUB/SUB on NNG drops messages for slow subscribers rather than blocking publishers. This is the right default for a trading event stream — you want the strategy service to keep producing signals even if a dashboard tab is slow. If you need delivery guarantees, you layer them on top.
  3. Deployment topology. All defaults are ipc://, which means everything runs on one host. If you want to spread workers across hosts, you change the address to tcp://0.0.0.0:5555 and the protocol semantics survive intact. Nothing in the strategy code or the API contracts changes.

What this means for the operator

Three operational implications are worth naming up front.

You need a writable /tmp. The default addresses are unix sockets under /tmp. The Docker Compose deployment handles this by mounting a shared volume. If you run the services on bare metal, every service needs read/write access to the same /tmp directory. The README is explicit about the path: ipc:///tmp/dgbit_cmd.ipc.

Restart order matters less than it would with HTTP. NNG’s REQ/REP sockets handle the case where one side comes up before the other. You can bring up the API server before the data service, and the first request will block briefly until the data service binds its end of the socket. There is no “service unreachable” failure cascade.

Observability looks different. You cannot tail /var/log/nginx/access.log to see who called what. The framework’s observability layer for the bus is the WebSocket event stream itself: subscribe to /api/ws/events and you see every meaningful event the workers produced. For lower-level debugging, NNG’s per-socket stats are available, but most of the time you want the event stream view, not the wire view.

A note on the choice

It is fair to ask why the project picked NNG specifically. The honest answer is that the choice is one of a small set of reasonable answers — ZeroMQ, raw sockets with a framing protocol, or a managed broker like NATS would all be defensible. NNG has three properties that matter for an open-source framework that wants to be easy to run: it is a small library (no broker to deploy), it has good Python bindings, and its Scalability Protocols cover the patterns this codebase actually needs. The decision is pragmatic, not religious.

If you wanted to swap NNG out, the contract surface is the two addresses in .env and the message shapes on each socket. The strategy interface and the REST API would not need to change. That separation — the message bus is an implementation detail of the worker layer, not part of the strategy contract — is the property that makes the choice safe to revisit.

Closing

The reason the NNG choice is worth a blog post is that it is the kind of decision that shows up everywhere downstream and never explicitly. Latency under tight backtest loops, throughput on the event stream, the operational story for deployment, and the upgrade path if the codebase ever needs to scale across hosts all flow from the same socket configuration in .env. The README does not call attention to it, but it is doing a lot of load-bearing work. Now you know it is there.