Observability

Rhei provides structured logging via the tracing crate, optional Prometheus metrics via the metrics-exporter feature, and a live terminal dashboard via rh dashboard.

Logging

Rhei's log output is built on tracing. The format and verbosity are controlled at runtime.

Log format

Set log_format in the [engine] section of your TOML config (see Configuration):

[engine]
log_format = "json"   # structured JSON — default is "text"

Value	Output style	Use with
"text" (default)	Human-readable, coloured terminal output	Development, rh serve interactive use
"json"	One JSON object per line, newline-delimited	Loki, Datadog, ELK / OpenSearch, Grafana Agent

JSON log lines include fields like timestamp, level, target, message, and any span/event fields attached by the emitting code (e.g. sql, table, events_processed).

Log level

Control verbosity with the RUST_LOG environment variable, following the standard tracing filter syntax:

# Info level for everything (default)
RUST_LOG=info rh serve --config htap.toml

# Debug for rhei-sync, info for everything else
RUST_LOG=info,rhei_sync=debug rh serve --config htap.toml

# Trace for flight SQL service
RUST_LOG=info,rhei_flight=trace rh serve --config htap.toml

Production recommendation

Run with log_format = "json" and RUST_LOG=info in production. Pipe stdout to your log aggregator. JSON format lets you filter and alert on fields like level, table, or error without regex.

Prometheus metrics

The metrics-exporter feature (on rhei-tui) starts an HTTP server that exposes counters and gauges in the Prometheus text format.

Enabling

Build with the feature:

cargo build -p rhei-tui --features metrics-exporter --release

Add metrics_port to your config:

[engine]
oltp_path    = "app.db"
olap_backend = "datafusion"
metrics_port = 9100   # exposes http://0.0.0.0:9100/metrics

The endpoint is available at http://0.0.0.0:{metrics_port}/metrics once rh serve starts.

Available metrics

The following metrics are emitted under the metrics feature flag by rhei-sync and rhei:

Counters (monotonically increasing):

Metric	Description
rhei.sync.events_processed	Total CDC events consumed across all sync cycles
rhei.sync.rows_inserted	Rows INSERTed into the OLAP backend
rhei.sync.rows_updated	Rows UPDATEd in the OLAP backend
rhei.sync.rows_deleted	Rows DELETEd from the OLAP backend
rhei.sync.rows_pruned	CDC log entries pruned after successful sync
rhei.query.routed_oltp	Queries routed to the OLTP (Rusqlite) engine
rhei.query.routed_olap	Queries routed to the OLAP (DataFusion / DuckDB) engine

Gauges (current value):

Metric	Description
rhei.sync.cdc_lag	Number of CDC events currently pending sync (backlog depth)

Source

Metric names are defined in crates/rhei-sync/src/sync_engine.rs and crates/rhei/src/lib.rs. They are only emitted when built with --features metrics (implied by metrics-exporter).

Prometheus scrape config

# prometheus.yml
scrape_configs:
  - job_name: rhei
    static_configs:
      - targets: ["localhost:9100"]

TUI dashboard

The rh dashboard command opens a live terminal UI that shows engine status without requiring Prometheus:

rh dashboard

The rh dashboard subcommand takes no flags — it runs an in-memory demo workload so you can preview the live TUI layout without touching real data. To run the dashboard against an actual deployment, use rh serve with a TOML config (it renders the same dashboard when stdout is a TTY).

The dashboard displays sync lag, CDC backlog depth, recent OLTP/OLAP query rates, and sync cycle timing. It refreshes in real time using ratatui + crossterm.

See the CLI reference for all dashboard flags.

Full observability config example

[engine]
oltp_path        = "app.db"
olap_backend     = "datafusion"
sync_interval_ms = 500
log_format       = "json"       # structured output for log aggregators
metrics_port     = 9100         # Prometheus endpoint (requires metrics-exporter feature)

# Optional Flight SQL server
# flight_port = 50051

Start the service:

RUST_LOG=info rh serve --config htap.toml

Logs flow to stdout as JSON. Prometheus scrapes http://host:9100/metrics.

Grafana dashboard sketch

If you ship metrics to Grafana via the Prometheus datasource, useful panels include:

• Sync throughput — rate of rhei.sync.events_processed over time (events/sec)
• CDC backlog — current value of rhei.sync.cdc_lag (alert when sustained > threshold)
• Write breakdown — stacked rate chart of rows_inserted, rows_updated, rows_deleted
• Query routing split — pie or time-series of routed_oltp vs routed_olap to validate the router is behaving as expected
• Prune rate — rate of rhei.sync.rows_pruned to confirm the CDC log is not growing unboundedly

These panels give a complete picture of sync health, CDC backpressure, and query routing correctness.