Rust Modbus Production Engineering: Reconnect, Retryable Errors, Backpressure, Observability

A Modbus client that works in a test is easy. A Modbus client that works for a year on a plant floor is a different problem — the serial cable gets unplugged, the gateway reboots, a sensor answers slowly, and your process has to survive all of it without a human watching. This article covers the four disciplines that close that gap, and shows them in one runnable Rust program built on oms-modbus.

Discipline 1: auto-reconnect — and why the interval is fixed

Reconnection is not a nice-to-have; on a USB serial link, unplugging the dongle and plugging it back in is a routine event. oms-modbus gives you a reconnect state machine through with_reconnect, configured with a maximum retry count and an interval: ClientOptions::default().with_reconnect(3, Duration::from_millis(100)). Two design choices here are deliberate, and worth understanding before you reach for them:

• max_retries = 0 means infinite. For a diagnostic tool you often want bounded retries (fail loudly so the operator sees it). For a long-running poller, 0 keeps reconnecting until the device comes back.
• The interval is fixed, not exponential. Exponential backoff is the right answer for congested HTTP servers where retries add load to a shared service. An industrial bus is different: you want predictable timing so a polling loop stays aligned to a schedule, and a device coming back online is answered the moment the next fixed-interval attempt lands. No jitter, no surprise 64-second gaps.

Reconnect is not a retry-everything knob. It only triggers on hard transport failures — see the next discipline for the exact boundary.

Discipline 2: know which errors are retryable

The single biggest production mistake with Modbus is retrying the wrong errors. A CRC mismatch will not be fixed by sending the same bytes again, and re-reading a register that returned "Illegal Data Address" is guaranteed to fail identically. oms-modbus ships is_retryable to encode this boundary as a pure function:

The Timeout row is the one that surprises people. A timeout looks like a transport failure, but it is genuinely ambiguous: it might mean the device is slow, or you addressed a slave that doesn't exist, or the device deliberately ignores a malformed request. Blindly retrying a timeout can turn a 100 ms stall into a 30-second hang while a scanner-like poll loops. So is_retryable returns false for it, and the reconnect state machine honors that — it rebuilds the connection on Connection/Serial, and leaves everything else to you to classify.

Discipline 3: backpressure with a bounded semaphore

One connection handles one request at a time. When you poll a fleet of devices concurrently, the natural temptation is to spawn a task per device and let them all run free — until a slow device means 200 in-flight requests hammering one gateway. A tokio::sync::Semaphore puts a ceiling on in-flight work — Arc::new(Semaphore::new(2)) means at most two polls in flight.

This is the difference between "bounded concurrency" and "unbounded concurrency". The semaphore bounds your load; it doesn't make a single connection faster, but it stops a fleet of pollers from collectively overwhelming a gateway or a half-duplex RS-485 bus.

Discipline 4: passive observability — see the wire, not just your code

When a poller reports a mysterious error at 3 a.m., the error type tells you what, but not what actually went over the wire. oms-modbus's WireTap attaches a read-only observer at the I/O boundary and records every frame — direction, raw bytes, microsecond timestamps — without the client or server participating. Attach a BusCapture through ClientOptions and you get live counters plus a drainable record log.

Everything together

One self-contained program that exercises all four. It stands up an in-memory device, configures reconnect + capture through a single ClientOptions, polls three devices under a two-permit semaphore, prints the error taxonomy, and reports what the passive monitor recorded:

use oms_modbus::*;
use oms_modbus::reconnect::is_retryable;
use std::net::{IpAddr, Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::Semaphore;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // 1. In-memory device (stand-in for a field sensor or gateway).
    let store = Arc::new(SlaveStore::with_holding_registers(&[(0, 100), (1, 200)]));
    let server = tcp::TcpServer::bind(SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0)).await?;
    let addr = server.local_addr()?;
    tokio::spawn(async move { server.serve_forever(store).await.ok(); });

    // 2. Observability + timeout + reconnect, all through ClientOptions.
    let tap = Arc::new(BusCapture::unbounded());
    let opts = ClientOptions::default()
        .with_timeout(Duration::from_secs(1))
        .with_tap(tap.clone())
        .with_reconnect(3, Duration::from_millis(100));

    // 3. Backpressure: at most 2 polls in flight across 3 poller tasks.
    let sem = Arc::new(Semaphore::new(2));
    let mut pollers = Vec::new();
    for device in 0..3u8 {
        let client = tcp::with_options(addr, opts.clone()).await?;
        let sem = sem.clone();
        pollers.push(tokio::spawn(async move {
            for i in 0..3u32 {
                let _permit = sem.acquire().await.unwrap();
                match client.read_holding_registers(1, 0, 1).await {
                    Ok(regs) => println!("device {} poll {i}: {regs:?}", device + 1),
                    Err(e) => println!(
                        "device {} poll {i}: {} (retryable={})",
                        device + 1,
                        e.label(),
                        is_retryable(&e)
                    ),
                }
            }
        }));
    }
    for p in pollers {
        p.await.unwrap();
    }

    // 4. Error taxonomy — label, retryability, and short Display for each kind.
    for e in [
        ModbusError::timeout("recv timed out"),
        ModbusError::connection("connection reset"),
        ModbusError::protocol("CRC mismatch"),
        ModbusError::exception(3, 2),
        ModbusError::serial("device disconnected"),
    ] {
        println!("{:14} retryable={}  {}", e.label(), is_retryable(&e), e);
    }

    // 5. What the passive monitor recorded on the wire.
    println!(
        "capture: {} requests, {} responses, {} errors",
        tap.count_requests(),
        tap.count_responses(),
        tap.count_errors()
    );
    Ok(())
}

The interesting part of the output is the taxonomy block, which is a compact reference for the whole error model:

TIMEOUT          retryable=false  TIMEOUT
CONNECTION ERROR retryable=true   CONNECTION ERROR
PROTOCOL ERROR   retryable=false  PROTOCOL ERROR
MODBUS EXCEPTION retryable=false  Illegal Data Address (code=2)
PORT ERROR       retryable=true   PORT ERROR
capture: 9 requests, 9 responses, 0 errors

Notice the three distinct representations of an error in play: label() gives a short machine-readable tag for a status bar or a metrics key, Display gives a compact human sentence ("Illegal Data Address (code=2)"), and detail() (used in the scanner article) gives the full prefixed log line. They serve different consumers, and a production service wants all three — the short label for dashboards, the Display for operator screens, the detail() for logs.

The three representations, spelled out

ModbusError is a single enum, but it renders three ways on purpose:

Keeping them separate means your log line can carry the function code and exception code while your status bar stays a single short token — you never have to parse a string back apart, which is the failure mode label() exists to prevent.

What's missing — honestly

These four disciplines are the foundation, not the whole building. A genuinely production-grade service also needs, in rough order of importance:

• A poll schedule, not a free-running loop — align reads to a fixed cadence so the bus load is predictable, and reconnect's fixed interval composes cleanly with it.
• A connection pool if a single gateway sees sustained load from many pollers; a semaphore bounds in-flight work but a single TcpClient still serializes on one socket.
• Graceful shutdown — draining in-flight polls before the process exits, rather than dropping connections mid-request.
• Metrics export — the BusCapture counters (count_requests / count_responses / count_errors) are already there; wiring them into your metrics system is the next obvious step.

The reconnect machine, the retryable/not-retryable boundary, bounded concurrency, and passive capture are the four things that make the difference between "works in the lab" and "runs unattended in the plant" — and they are all available through a single ClientOptions builder plus one free function.

This article is part of the oms-modbus tutorial series. oms-modbus is a transport-generic Modbus library in Rust — TCP, RTU, and ASCII through one API — used at OrangeHorse to build diagnostic tools for our Modbus-enabled sensors. MIT/Apache-2.0, on crates.io.

Read next: Building a Modbus Device Scanner — the discovery step that often precedes a long-running service. WireTap Bus Monitor Deep-Dive — the observability discipline in detail.