26 KiB
Architecture Research
Domain: eBPF-based network monitoring CLI (Rust) Researched: 2026-03-21 Confidence: MEDIUM-HIGH
Standard Architecture
System Overview
┌─────────────────────────────────────────────────────────────────────┐
│ User Interface Layer │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Ratatui TUI │ │ CSV Logger │ │ CLI (clap) │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │ │
├─────────┴─────────────────┴──────────────────┴──────────────────────┤
│ Application Core Layer │
│ ┌──────────────┐ ┌───────────────┐ ┌──────────────────┐ │
│ │ Connection │ │ Aggregation │ │ Filter Engine │ │
│ │ State Table │ │ (summaries) │ │ (port/pid/name) │ │
│ └──────┬───────┘ └───────┬───────┘ └──────┬───────────┘ │
│ │ │ │ │
├─────────┴──────────────────┴──────────────────┴─────────────────────┤
│ Platform Abstraction Layer │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ trait DataSource (platform-agnostic) │ │
│ └──────────────────┬───────────────────┬───────────────────┘ │
│ │ │ │
│ ┌────────────┴───┐ ┌─────────┴──────────┐ │
│ │ Linux Backend │ │ macOS Backend │ │
│ │ (eBPF / Aya) │ │ (NetworkStatistics │ │
│ │ │ │ + libproc) │ │
│ └────────┬───────┘ └─────────┬──────────┘ │
├─────────────────┴───────────────────────┴───────────────────────────┤
│ Kernel / OS Layer │
│ ┌───────────────────┐ ┌────────────────────────┐ │
│ │ Linux Kernel │ │ macOS/XNU Kernel │ │
│ │ kprobes on: │ │ com.apple.network │ │
│ │ tcp_sendmsg │ │ .statistics events │ │
│ │ tcp_recvmsg │ │ proc_pidinfo │ │
│ │ tcp_connect │ │ │ │
│ │ tcp_close │ │ │ │
│ │ udp_sendmsg │ │ │ │
│ │ udp_recvmsg │ │ │ │
│ └───────────────────┘ └────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
Component Responsibilities
| Component | Responsibility | Typical Implementation |
|---|---|---|
| CLI (clap) | Parse arguments, select mode (TUI vs CSV), apply filters | clap derive macros, validated at startup |
| Ratatui TUI | Render summary header + sortable connection table, handle keyboard input | ratatui + crossterm, async event loop with tokio |
| CSV Logger | Write connection snapshots to file at configurable interval | Simple std::io::BufWriter, no extra crate needed |
| Connection State Table | In-memory store of all active connections with their stats | HashMap<ConnectionKey, ConnectionStats> behind Arc<RwLock> or channel-based |
| Aggregation | Compute summary stats (total connections, aggregate bandwidth) | Derived from Connection State Table on each render tick |
| Filter Engine | Include/exclude connections by port, PID, or process name | Predicate functions applied during render, not at collection |
Platform Abstraction (trait DataSource) |
Unified interface for receiving connection events | Rust trait with async fn poll_events() or channel-based push |
| Linux Backend (eBPF/Aya) | Kernel-level connection tracing via kprobes + ring buffer | Aya userspace loader + eBPF programs compiled to BPF bytecode |
| macOS Backend | Per-connection stats via NetworkStatistics private framework | libntstat-style approach via com.apple.network.statistics kernel events, plus proc_pidinfo for process resolution |
Recommended Project Structure
tcptop/
├── Cargo.toml # Workspace root
├── tcptop/ # Userspace application (main binary)
│ ├── Cargo.toml
│ └── src/
│ ├── main.rs # Entry point, CLI parsing, privilege check
│ ├── app.rs # Application state, main event loop
│ ├── tui/
│ │ ├── mod.rs # TUI setup/teardown
│ │ ├── widgets.rs # Summary header, connection table widgets
│ │ └── input.rs # Keyboard handler (sort, filter, quit)
│ ├── logger.rs # CSV logging output
│ ├── state/
│ │ ├── mod.rs # Connection state table
│ │ ├── connection.rs # ConnectionKey, ConnectionStats types
│ │ └── aggregation.rs # Summary computation
│ ├── filter.rs # Filter engine (port, pid, name)
│ └── platform/
│ ├── mod.rs # trait DataSource + platform detection
│ ├── linux.rs # eBPF backend (loads + polls ring buffer)
│ └── macos.rs # NetworkStatistics backend
├── tcptop-ebpf/ # eBPF kernel programs (Linux only)
│ ├── Cargo.toml # Targets bpfel-unknown-none
│ └── src/
│ ├── main.rs # eBPF program entry points
│ └── maps.rs # Ring buffer + shared map definitions
├── tcptop-common/ # Shared types between kernel and userspace
│ ├── Cargo.toml # no_std compatible
│ └── src/
│ └── lib.rs # ConnectionEvent, protocol enums
└── xtask/ # Build automation (optional)
├── Cargo.toml
└── src/
└── main.rs # Build eBPF, generate bindings
Structure Rationale
- Three-crate workspace (Aya convention):
tcptop-ebpfcompiles to BPF bytecode (targetbpfel-unknown-none),tcptop-commonisno_stdso it can be used by both kernel and userspace, andtcptopis the standard userspace binary. This is the standard Aya template pattern. platform/module: Contains theDataSourcetrait and platform-specific implementations. Compile-timecfgattributes select the correct backend. This keeps platform specifics out of business logic.tui/vslogger.rs: Output modes are separate -- TUI for interactive use, CSV logger for batch/scripting. They consume the sameConnectionState.state/module: Owns all connection data. Both TUI and logger read from it; only the platform backend writes to it. This is the single source of truth.xtask/: Optional but recommended for Aya projects. Handles cross-compilation of eBPF programs duringcargo build.
Architectural Patterns
Pattern 1: Platform Abstraction via Trait + cfg
What: Define a DataSource trait that abstracts over platform-specific data collection. Use #[cfg(target_os)] to compile only the relevant backend.
When to use: Always -- this is the core architectural decision enabling cross-platform support.
Trade-offs: Adds a layer of indirection, but the alternative (scattered #[cfg] blocks throughout business logic) is far worse. The trait boundary also makes testing possible with a mock backend.
// platform/mod.rs
pub struct ConnectionEvent {
pub key: ConnectionKey,
pub bytes_tx: u64,
pub bytes_rx: u64,
pub packets_tx: u64,
pub packets_rx: u64,
pub pid: Option<u32>,
pub process_name: Option<String>,
pub state: TcpState,
pub rtt_us: Option<u64>,
}
#[async_trait]
pub trait DataSource: Send + 'static {
async fn start(&mut self) -> Result<()>;
async fn next_event(&mut self) -> Result<ConnectionEvent>;
async fn stop(&mut self) -> Result<()>;
}
#[cfg(target_os = "linux")]
mod linux;
#[cfg(target_os = "macos")]
mod macos;
pub fn create_data_source() -> Result<Box<dyn DataSource>> {
#[cfg(target_os = "linux")]
{ Ok(Box::new(linux::EbpfSource::new()?)) }
#[cfg(target_os = "macos")]
{ Ok(Box::new(macos::NetworkStatsSource::new()?)) }
}
Pattern 2: Channel-Based Data Flow (Producer/Consumer)
What: The platform backend runs in its own tokio task and sends ConnectionEvents through an mpsc channel to the application core. The TUI event loop uses tokio::select! to multiplex between data events, keyboard input, and render ticks.
When to use: Always -- this decouples data collection rate from render rate and prevents blocking.
Trade-offs: Slight latency from channel buffering (negligible at human-visible refresh rates). Channel backpressure needs handling if events arrive faster than consumption.
// app.rs - main event loop
loop {
tokio::select! {
Some(event) = data_rx.recv() => {
state.update(event);
}
Some(key) = input_rx.recv() => {
match key {
KeyCode::Char('q') => break,
KeyCode::Char('s') => state.cycle_sort(),
// ...
}
}
_ = render_interval.tick() => {
terminal.draw(|f| ui::render(f, &state, &filters))?;
if let Some(ref mut logger) = csv_logger {
logger.write_snapshot(&state)?;
}
}
}
}
Pattern 3: eBPF Ring Buffer for Kernel-to-Userspace Transfer
What: Use BPF_MAP_TYPE_RINGBUF (not the older perf buffer) for sending events from eBPF kernel programs to userspace. Ring buffer is shared across all CPUs, preserves event ordering, and is more memory-efficient.
When to use: On Linux, for all kernel-to-userspace event delivery.
Trade-offs: Requires Linux kernel 5.8+. Older kernels need perf buffer fallback, though kernel 5.8 is from 2020 so this is rarely a concern in 2026.
// tcptop-ebpf/src/main.rs (kernel side)
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(256 * 1024, 0);
#[kprobe]
pub fn tcp_sendmsg(ctx: ProbeContext) -> u32 {
match try_tcp_sendmsg(ctx) {
Ok(()) => 0,
Err(_) => 1,
}
}
fn try_tcp_sendmsg(ctx: ProbeContext) -> Result<(), i64> {
let sock: *const sock = ctx.arg(0).ok_or(1)?;
// Extract connection info from sock, write to ring buffer
if let Some(mut entry) = EVENTS.reserve::<ConnectionEventRaw>(0) {
// populate entry fields from sock
entry.submit(0);
}
Ok(())
}
Pattern 4: Shared Types in no_std Common Crate
What: Data structures exchanged between eBPF kernel programs and userspace live in a no_std crate that both can depend on. This ensures type-safe serialization without duplicating definitions.
When to use: Always in Aya projects -- this is the standard Aya pattern.
Trade-offs: The common crate must be no_std compatible (no heap allocation, no String, etc.), which limits what types can be shared. Use fixed-size arrays for strings.
// tcptop-common/src/lib.rs
#![no_std]
#[repr(C)]
pub struct ConnectionEventRaw {
pub src_addr: u32, // IPv4 as u32
pub dst_addr: u32,
pub src_port: u16,
pub dst_port: u16,
pub protocol: u8, // IPPROTO_TCP or IPPROTO_UDP
pub event_type: u8, // send, recv, connect, close
pub pid: u32,
pub bytes: u64,
pub comm: [u8; 16], // process name (fixed-size)
}
Data Flow
Primary Data Flow (Linux / eBPF)
Kernel Space User Space
┌──────────────┐
│ tcp_sendmsg │──┐
│ tcp_recvmsg │ │ ┌──────────┐ ┌────────────┐ ┌──────────────┐
│ tcp_connect │──┼──>│ RingBuf │───>│ EbpfSource │───>│ mpsc channel │
│ tcp_close │ │ │ (shared) │ │ (poll loop)│ └──────┬───────┘
│ udp_sendmsg │──┘ └──────────┘ └────────────┘ │
│ udp_recvmsg │ ▼
└──────────────┘ ┌──────────────────┐
│ Connection State │
│ Table │
└────────┬─────────┘
│
┌────────┴─────────┐
│ │
┌─────┴─────┐ ┌───────┴─────┐
│ Ratatui │ │ CSV Logger │
│ TUI │ │ │
└───────────┘ └─────────────┘
Primary Data Flow (macOS)
XNU Kernel User Space
┌──────────────────┐
│ com.apple.network│ ┌─────────────────┐ ┌──────────────┐
│ .statistics │───>│ NetworkStats │───>│ mpsc channel │
│ kernel events │ │ Source │ └──────┬───────┘
└──────────────────┘ │ + proc_pidinfo │ │
│ for PID/name │ ▼
└─────────────────┘ ┌──────────────────┐
│ Connection State │
│ Table │
└────────┬─────────┘
│
(same as Linux from here)
Key Data Flows
-
Connection Event Ingestion: Platform backend detects network activity (eBPF kprobe fires or macOS kernel event arrives) -> raw event is normalized into
ConnectionEvent-> sent viatokio::sync::mpscchannel ->ConnectionStatetable upserts the entry, updating byte/packet counters and timestamps. -
TUI Render Cycle: Every ~250ms, render tick fires -> connection table is read (with current sort + filters applied) -> summary header computed via aggregation -> ratatui draws frame to terminal.
-
CSV Logging: On each render tick (or configurable interval), current snapshot of filtered connections is written as CSV rows to the output file.
-
Connection Lifecycle:
tcp_connectkprobe creates entry ->tcp_sendmsg/tcp_recvmsgupdate byte counters ->tcp_closemarks connection as closed -> after configurable retention period, entry is removed from state table.
Scaling Considerations
| Concern | 100 connections | 10K connections | 100K+ connections |
|---|---|---|---|
| State table memory | Negligible (~10KB) | Moderate (~1MB) | Significant (~10MB), needs eviction policy |
| Ring buffer throughput | No pressure | Monitor for drops | May need larger ring buffer, sampling, or kernel-side filtering |
| TUI rendering | Instant | Needs virtual scrolling | Must limit visible rows, aggressive filtering |
| CPU overhead | Unnoticeable | Low (<1%) | eBPF per-event cost adds up; use tracepoints over kprobes |
Scaling Priorities
- First bottleneck: Connection table grows unboundedly if closed connections are not evicted. Implement a TTL-based eviction (e.g., remove closed connections after 30s).
- Second bottleneck: Ring buffer overflow under extreme event rates. Use kernel-side filtering (port/protocol) in the eBPF program itself to reduce event volume before it reaches userspace.
Anti-Patterns
Anti-Pattern 1: Filtering Only in Userspace
What people do: Collect ALL network events in the eBPF program and filter in userspace.
Why it's wrong: Under high traffic, the ring buffer fills and drops events. Every event has per-event kernel overhead even if userspace ignores it.
Do this instead: When the user specifies --port 443, update an eBPF map with the filter criteria. The eBPF program checks the map and skips non-matching events at the kernel level. This is a well-established pattern -- "push filtering down."
Anti-Pattern 2: Polling Kernel Data from the Render Loop
What people do: Call into the platform backend synchronously on each render tick. Why it's wrong: Blocks the TUI event loop. If the kernel read takes longer than the frame budget, the UI freezes. Also couples data collection rate to render rate. Do this instead: Run data collection in its own tokio task. Feed events through a channel. The render loop only reads from the already-populated state table.
Anti-Pattern 3: Shared Mutable State Between eBPF Loader and TUI
What people do: Use Arc<Mutex<HashMap>> shared directly between the eBPF polling thread and the TUI render thread, locking on every event and every frame.
Why it's wrong: Lock contention under high event rates causes jitter in both data collection and rendering.
Do this instead: Use the channel pattern (Pattern 2). The state table is owned by the main event loop task. Incoming events are applied in batches between render ticks. No shared mutable state, no locks.
Anti-Pattern 4: Putting Platform-Specific Code in Business Logic
What people do: Scatter #[cfg(target_os = "linux")] blocks throughout app.rs, state.rs, etc.
Why it's wrong: Makes cross-platform maintenance painful. Every feature addition requires touching cfg blocks everywhere.
Do this instead: All platform-specific code lives behind the DataSource trait in the platform/ module. The rest of the application is platform-agnostic.
Integration Points
Kernel Integration (Linux)
| Hook Point | What It Captures | Aya Program Type |
|---|---|---|
tcp_sendmsg |
Bytes sent on TCP connection | kprobe |
tcp_recvmsg |
Bytes received on TCP connection | kprobe |
tcp_v4_connect / tcp_v6_connect |
New outbound TCP connection | kprobe |
tcp_close |
Connection teardown | kprobe |
udp_sendmsg |
Bytes sent on UDP | kprobe |
udp_recvmsg |
Bytes received on UDP | kprobe |
inet_csk_accept |
New inbound TCP connection (server-side) | kretprobe |
Alternative: Use tracepoints (sock:inet_sock_set_state, tcp:tcp_sendmsg) where available. Tracepoints are a stable kernel API and survive kernel upgrades better than kprobes. However, tracepoint coverage for byte-level accounting is incomplete, so kprobes on tcp_sendmsg/tcp_recvmsg are still necessary for per-connection byte counters.
Kernel Integration (macOS)
| Mechanism | What It Provides | Notes |
|---|---|---|
com.apple.network.statistics |
Per-connection bytes tx/rx, packets, state | Private framework; used by nettop. Not a public API -- may break across macOS versions |
proc_pidinfo + PROC_PIDFDSOCKETINFO |
Process-to-socket mapping | Public API via libproc. Stable but requires iterating FDs |
getifaddrs |
Per-interface aggregate stats | Public, but no per-connection granularity |
netstat -an (fallback) |
Connection listing | Gross fallback -- spawns process, parses text output |
Recommended macOS approach: Use libntstat pattern (C library wrapping com.apple.network.statistics kernel events) called from Rust via FFI, supplemented by proc_pidinfo for PID-to-process resolution. This gives nettop-equivalent data. The risk is that this is a private API.
Graceful degradation: If the private API is unavailable or breaks, fall back to periodic polling of /usr/sbin/nettop -L 1 -P or netstat output. This is ugly but functional as a last resort.
Internal Boundaries
| Boundary | Communication | Notes |
|---|---|---|
| Platform Backend -> App Core | tokio::sync::mpsc channel of ConnectionEvent |
Async, non-blocking, buffered |
| App Core -> TUI | Direct function call (same task) | State table read on render tick |
| App Core -> CSV Logger | Direct function call (same task) | Snapshot written on tick |
| CLI -> App Core | Configuration struct passed at startup | One-time, not a runtime boundary |
| eBPF Kernel -> eBPF Userspace Loader | Ring buffer (BPF_MAP_TYPE_RINGBUF) |
Shared memory, async poll via Aya |
| Userspace -> eBPF Kernel (filter updates) | eBPF HashMap map | Write filter criteria; eBPF program reads |
Build Order Implications
The architecture has clear dependency ordering that should inform phase structure:
-
Common types first (
tcptop-common): Shared data structures must exist before either kernel or userspace code can use them. This is a small, stable crate. -
eBPF kernel programs second (
tcptop-ebpf): These are the most technically challenging component. Getting kprobes attached and events flowing through the ring buffer is the core proof-of-concept. Build Linux-only first. -
Platform abstraction + state management third (
tcptop/src/platform/,tcptop/src/state/): TheDataSourcetrait and connection state table. Wire up the eBPF backend as the first implementation. -
Basic TUI fourth (
tcptop/src/tui/): Once events flow into the state table, render them. Start with a minimal table view, no sorting or filtering. -
Filtering, sorting, polish fifth: Add
--port,--pidfilters, column sorting, summary header, keyboard controls. -
macOS backend sixth (
tcptop/src/platform/macos.rs): This is the highest-risk component due to private API dependency. Build it after Linux works end-to-end. -
CSV logging, packaging last: These are straightforward and don't block anything.
Critical path: Common types -> eBPF programs -> Ring buffer communication -> Userspace polling -> State table -> TUI rendering. This is the minimum viable pipeline and should be the first milestone.
Sources
- Aya eBPF Framework (GitHub) - PRIMARY, HIGH confidence
- Aya Template - Workspace structure convention, HIGH confidence
- Aya Book - Getting Started - HIGH confidence
- BPF Ring Buffer Design (Nakryiko) - Ring buffer vs perf buffer rationale, HIGH confidence
- Linux Kernel Ring Buffer Docs - HIGH confidence
- TCP Tracepoints (Brendan Gregg) - Hook point selection, HIGH confidence
- RustNet Cross-Platform Monitor - Cross-platform pattern reference, MEDIUM confidence
- libntstat (macOS NetworkStatistics) - macOS backend approach, MEDIUM confidence
- proc_pidinfo Rust crate - macOS process resolution, MEDIUM confidence
- Ratatui Async Event Stream - TUI event loop pattern, HIGH confidence
- eBPF Cerberus Network Monitor - Architecture reference, MEDIUM confidence
- Apple Developer Forums - nettop alternatives - macOS API landscape, LOW confidence (private APIs)
Architecture research for: tcptop - eBPF network monitoring CLI Researched: 2026-03-21