OLAP Backends

Rhei supports two OLAP engines: Apache DataFusion (default) and DuckDB. Both implement the same OlapEngine trait, so switching backends requires only a config change.

---

Quick Comparison

Feature	DataFusion	DuckDB
Feature flag	datafusion-backend (default)	duckdb-backend
C++ dependency	None — pure Rust	Yes — bundled via duckdb-sys
Async model	Natively async	spawn_blocking wrappers
Transactions	No (no-ops)	Yes — full ACID
Bulk-load path	Arrow batch → load_arrow	Arrow Appender API (zero-copy)
Storage modes	InMemory / Vortex { url } (local + S3-compatible)	InMemory or single .duckdb file
Streaming results	Native (RecordBatchBoxStream)	Collect-then-stream
Primary key enforcement	No	Yes (in destructive mode)
Compile time	Fast	Slower (C++ bundled build)

---

DataFusion

DataFusion is the default OLAP engine. It is a pure-Rust, natively async columnar query engine built on the Apache Arrow in-memory format.

Enable it (or keep the default):

rhei = "2"
# or explicitly:
rhei = { version = "2", features = ["datafusion-backend"] }

Storage Modes

DataFusion's storage is configurable via HtapConfig::datafusion_storage (StorageMode enum). Two modes are available:

Mode	Description	Feature required
InMemory	Default. Data lost on shutdown.	None
Vortex { url }	Unified Vortex storage. Local or S3-compatible depending on the URL scheme.	None for local; cloud-storage for S3

URL auto-classification rules:

URL prefix	Storage backend
./path or /abs/path or file://...	Local filesystem
s3://bucket/prefix	S3-compatible object store (cloud-storage feature required)

use rhei::{HtapConfig, StorageMode};

// In-memory (default) — no persistence
let config = HtapConfig {
    datafusion_storage: StorageMode::InMemory,
    ..HtapConfig::default()
};

// Local Vortex — durable, local filesystem
let config = HtapConfig {
    olap_in_memory: false,
    datafusion_storage: StorageMode::Vortex {
        url: "./rhei-data".to_string(),
    },
    ..HtapConfig::default()
};

// S3-compatible Vortex (requires cloud-storage feature + AWS_* env vars)
let config = HtapConfig {
    datafusion_storage: StorageMode::Vortex {
        url: "s3://my-bucket/analytics/".to_string(),
    },
    ..HtapConfig::default()
};

See Vortex storage for the full reference including S3-compatible service configuration, required environment variables, and the migration story from v1.5.x ArrowIpc/Parquet modes.

Bulk Loading

CDC INSERT batches are loaded via OlapEngine::load_arrow(), which accepts an Arrow RecordBatch directly. DataFusion appends the batch without any SQL string construction. The fallback to SQL only occurs for column types not yet handled by the Arrow builder (Date, Timestamp, Decimal, List, Struct).

Streaming Queries

DataFusion implements OlapEngine::query_stream() natively via DataFrame::execute_stream(), returning a RecordBatchBoxStream that emits batches without buffering the entire result set. The Arrow Flight SQL server uses this for all data transfer.

No Transactions

DataFusion's transaction support is a no-op. In SyncMode::Temporal, the UPDATE (close previous version) + INSERT (new version) pair is not atomic — a crash between the two leaves an open-ended previous version and a missing new version until the engine resumes and reprocesses from the last committed sequence number.

---

DuckDB

DuckDB embeds the full DuckDB columnar engine via duckdb-sys (C++ bundled build). It provides excellent SQL compatibility, MVCC-based concurrent reads, and ACID transactions.

Enable DuckDB:

rhei = { version = "2", features = ["duckdb-backend"] }
# or both backends:
rhei = { version = "2", features = ["full"] }

Configure the engine:

use rhei::{HtapConfig, OlapBackendType};

// In-memory DuckDB
let config = HtapConfig {
    olap_backend: OlapBackendType::DuckDb,
    olap_in_memory: true,
    ..HtapConfig::default()
};

// Persistent DuckDB — single file
let config = HtapConfig {
    olap_backend: OlapBackendType::DuckDb,
    olap_in_memory: false,
    olap_path: Some("/var/lib/myapp/olap.duckdb".to_string()),
    ..HtapConfig::default()
};

Bulk Loading

DuckDB's load_arrow() uses the native Appender API (append_record_batch) for zero-copy Arrow ingestion. All Arrow types — including Date, Timestamp, Binary, and Decimal — are handled natively without SQL literal serialisation. This is the fastest bulk-load path across both backends.

Thread Safety

duckdb::Connection is !Send. Rhei wraps each connection in a Mutex with unsafe impl Send/Sync and dispatches all operations through spawn_blocking. The engine maintains a single write connection and a read pool (default: 4) using DuckDB's MVCC for concurrent read isolation.

Transactions

DuckDB supports full ACID transactions. In SyncMode::Temporal, the close + insert pair for each UPDATE or DELETE is wrapped in a single BEGIN … COMMIT, making it fully atomic. This is the key advantage of DuckDB over DataFusion for temporal workloads.

datafusion_storage is Ignored

When olap_backend = OlapBackendType::DuckDb, the datafusion_storage field has no effect. DuckDB uses either in-memory mode or a single .duckdb file, controlled by olap_in_memory and olap_path.

---

When to Pick Which

Choose DataFusion when:

• You want zero C/C++ dependencies (CI, cross-compilation, WebAssembly targets).
• You need cloud object storage (S3-compatible: AWS S3, MinIO, R2, Wasabi, Ceph) via Vortex.
• Your workload is primarily append-only (INSERTs) — the Arrow batch path is equally fast.
• Temporal mode atomicity is not critical (or you run DuckDB for temporal and DataFusion for destructive).

Choose DuckDB when:

• You need SyncMode::Temporal with atomic UPDATE/DELETE transitions.
• You need primary-key enforcement on the OLAP mirror.
• You want a single durable file for the OLAP store with a familiar SQL dialect.
• You need the DuckDB extension ecosystem (spatial, JSON, HTTPFS, etc.) via raw queries.

---

Next Steps

• Sync Modes — DuckDB atomicity matters most for temporal mode
• Configuration Reference — StorageMode, OlapBackendType, and persistence settings
• Vortex storage — local and S3-compatible storage with URL auto-classification
• Arrow Flight SQL — streaming OLAP queries over gRPC