Files
Zachary D. Rowitsch 38e6dcc34a chore: archive v1.0 phase directories to milestones/v1.0-phases/
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 01:33:15 -04:00

595 lines
22 KiB
Markdown

---
phase: 01-data-pipeline
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
- Cargo.toml
- rust-toolchain.toml
- .cargo/config.toml
- tcptop/Cargo.toml
- tcptop/build.rs
- tcptop/src/main.rs
- tcptop/src/privilege.rs
- tcptop/src/collector/mod.rs
- tcptop/src/model.rs
- tcptop-common/Cargo.toml
- tcptop-common/src/lib.rs
- tcptop-ebpf/Cargo.toml
- tcptop-ebpf/src/main.rs
autonomous: true
requirements:
- PLAT-01
- PLAT-03
- OPS-01
- OPS-02
must_haves:
truths:
- "Workspace compiles with cargo build (userspace crate) without errors"
- "eBPF crate compiles to BPF bytecode via aya-build in build.rs"
- "Running without root exits with code 77 and message 'error: tcptop requires root privileges. Run with sudo.'"
- "NetworkCollector trait is defined and importable from collector module"
- "Shared types in tcptop-common are repr(C) and usable from both no_std (eBPF) and std (userspace)"
artifacts:
- path: "Cargo.toml"
provides: "Workspace root with three members"
contains: "members"
- path: "rust-toolchain.toml"
provides: "Pinned nightly toolchain with bpfel-unknown-none target"
contains: "bpfel-unknown-none"
- path: "tcptop-common/src/lib.rs"
provides: "TcptopEvent enum, DataEvent, StateEvent, ConnectionKey structs"
contains: "repr(C)"
- path: "tcptop/src/collector/mod.rs"
provides: "NetworkCollector trait definition"
contains: "trait NetworkCollector"
- path: "tcptop/src/privilege.rs"
provides: "Privilege check with exit code 77"
contains: "exit(77)"
- path: "tcptop/src/model.rs"
provides: "ConnectionRecord, ConnectionKey, Protocol types"
contains: "struct ConnectionRecord"
key_links:
- from: "tcptop/build.rs"
to: "tcptop-ebpf"
via: "aya-build compilation"
pattern: "aya_build"
- from: "tcptop-ebpf/src/main.rs"
to: "tcptop-common/src/lib.rs"
via: "shared event types import"
pattern: "use tcptop_common"
- from: "tcptop/src/main.rs"
to: "tcptop/src/privilege.rs"
via: "privilege check on startup"
pattern: "check_privileges"
---
<objective>
Scaffold the Aya eBPF workspace, define all shared types, establish the platform abstraction trait, and implement the privilege check.
Purpose: Create the foundational project structure that all subsequent plans build on. The workspace must compile, the eBPF build pipeline must work, shared types must be defined for kernel-userspace communication, and the platform trait must be ready for the Linux collector implementation.
Output: Compiling workspace with three crates (userspace, ebpf, common), working eBPF build pipeline, privilege checking, platform trait, and all shared data types.
</objective>
<execution_context>
@/Users/zrowitsch/local_src/tcptop/.claude/get-shit-done/workflows/execute-plan.md
@/Users/zrowitsch/local_src/tcptop/.claude/get-shit-done/templates/summary.md
</execution_context>
<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/01-data-pipeline/01-CONTEXT.md
@.planning/phases/01-data-pipeline/01-RESEARCH.md
</context>
<tasks>
<task type="auto">
<name>Task 1: Create Aya workspace with shared types and eBPF build pipeline</name>
<files>
Cargo.toml,
rust-toolchain.toml,
.cargo/config.toml,
tcptop/Cargo.toml,
tcptop/build.rs,
tcptop/src/main.rs,
tcptop-common/Cargo.toml,
tcptop-common/src/lib.rs,
tcptop-ebpf/Cargo.toml,
tcptop-ebpf/src/main.rs
</files>
<read_first>
.planning/phases/01-data-pipeline/01-RESEARCH.md (architecture patterns, code examples, version numbers),
CLAUDE.md (technology stack, version compatibility)
</read_first>
<action>
Create the full Aya eBPF workspace with three crates. This is a greenfield project -- no existing code.
**rust-toolchain.toml:**
Pin nightly channel (use `nightly-2026-01-15` or a recent stable nightly). Include components: `rust-src`, `rustfmt`, `clippy`. Add `bpfel-unknown-none` as a target.
**.cargo/config.toml:**
Add any needed build flags for the eBPF target (if aya-build handles this automatically, this file may be minimal or empty -- check aya-build behavior).
**Cargo.toml (workspace root):**
```toml
[workspace]
members = ["tcptop", "tcptop-common", "tcptop-ebpf"]
resolver = "2"
[workspace.dependencies]
aya = { version = "0.13.1", features = ["async_tokio"] }
aya-log = "0.2"
tokio = { version = "1", features = ["full"] }
anyhow = "1"
thiserror = "2"
clap = { version = "4.6", features = ["derive"] }
log = "0.4"
env_logger = "0.11"
nix = { version = "0.29", features = ["user"] }
procfs = "0.18"
signal-hook = "0.3"
```
**tcptop-common/Cargo.toml:**
```toml
[package]
name = "tcptop-common"
version = "0.1.0"
edition = "2021"
[features]
default = []
user = ["no-std-compat-off"] # feature flag if needed
[dependencies]
# No dependencies -- this crate must be no_std compatible
```
The common crate must compile under both `no_std` (for eBPF) and `std` (for userspace).
**tcptop-common/src/lib.rs:**
Define ALL shared types used between kernel and userspace. Every struct MUST be `#[repr(C)]` with only primitive fields (no String, Vec, Option, enums with data larger than simple discriminants).
**LOCKED DECISION -- Use `union` for TcptopEventData.** The `#[repr(C)] pub union TcptopEventData` approach is the canonical layout. Plan 02's `parse_event` uses `unsafe { &event.data.data_event }` and `unsafe { &event.data.state_event }` accessors keyed on `event_type`. Do NOT use a flat struct alternative -- the union is the contract between Plans 01 and 02.
Types to define:
```rust
#![no_std]
// IP address family
pub const AF_INET: u16 = 2;
pub const AF_INET6: u16 = 10;
#[repr(C)]
#[derive(Clone, Copy)]
pub struct DataEvent {
pub pid: u32,
pub comm: [u8; 16], // process name from bpf_get_current_comm
pub af_family: u16, // AF_INET or AF_INET6
pub sport: u16,
pub dport: u16,
pub _pad: u16, // alignment padding
pub saddr: [u8; 16], // IPv4 in first 4 bytes, or full IPv6
pub daddr: [u8; 16],
pub bytes: u32, // bytes transferred in this call
pub srtt_us: u32, // smoothed RTT (shifted <<3 from kernel, we store raw)
}
#[repr(C)]
#[derive(Clone, Copy)]
pub struct StateEvent {
pub pid: u32,
pub af_family: u16,
pub sport: u16,
pub dport: u16,
pub _pad: u16,
pub saddr: [u8; 16],
pub daddr: [u8; 16],
pub old_state: u32,
pub new_state: u32,
}
// Event tag for the ring buffer -- simple discriminant
pub const EVENT_TCP_SEND: u32 = 1;
pub const EVENT_TCP_RECV: u32 = 2;
pub const EVENT_UDP_SEND: u32 = 3;
pub const EVENT_UDP_RECV: u32 = 4;
pub const EVENT_TCP_STATE: u32 = 5;
#[repr(C)]
#[derive(Clone, Copy)]
pub struct TcptopEvent {
pub event_type: u32, // one of EVENT_* constants
pub _pad: u32, // alignment to 8 bytes
pub data: TcptopEventData,
}
// Tagged union -- event_type discriminant tells which variant to read.
// Both DataEvent and StateEvent must fit. DataEvent is larger.
// Userspace reads via: unsafe { &event.data.data_event } or unsafe { &event.data.state_event }
#[repr(C)]
#[derive(Clone, Copy)]
pub union TcptopEventData {
pub data_event: DataEvent,
pub state_event: StateEvent,
}
```
If the eBPF crate fails to compile with `union` under `no_std` + `bpfel-unknown-none`, investigate the specific error (likely a missing `Copy` bound or alignment issue) and fix it while preserving the union layout. The union is the locked contract -- do not fall back to a flat struct.
**tcptop-ebpf/Cargo.toml:**
```toml
[package]
name = "tcptop-ebpf"
version = "0.1.0"
edition = "2021"
[dependencies]
aya-ebpf = "0.1"
aya-log-ebpf = "0.1"
tcptop-common = { path = "../tcptop-common" }
[[bin]]
name = "tcptop"
path = "src/main.rs"
```
**tcptop-ebpf/src/main.rs:**
Minimal skeleton that compiles:
```rust
#![no_std]
#![no_main]
use aya_ebpf::{macros::map, maps::RingBuf};
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(256 * 1024, 0);
// Placeholder -- kprobes added in Plan 02
#[panic_handler]
fn panic(_info: &core::panic::PanicInfo) -> ! {
loop {}
}
```
Note: aya-ebpf may provide its own panic handler. If compilation fails due to duplicate panic handlers, remove the manual one.
**tcptop/Cargo.toml:**
```toml
[package]
name = "tcptop"
version = "0.1.0"
edition = "2021"
build = "build.rs"
[dependencies]
aya = { workspace = true }
aya-log = { workspace = true }
tokio = { workspace = true }
anyhow = { workspace = true }
thiserror = { workspace = true }
clap = { workspace = true }
log = { workspace = true }
env_logger = { workspace = true }
nix = { workspace = true }
procfs = { workspace = true }
signal-hook = { workspace = true }
tcptop-common = { path = "../tcptop-common" }
[build-dependencies]
aya-build = "0.1"
```
**tcptop/build.rs:**
Use aya-build to compile the eBPF crate. Based on research:
```rust
fn main() {
aya_build::build_ebpf(&["tcptop-ebpf"])
.expect("Failed to build eBPF programs");
}
```
If the exact API differs, consult aya-build 0.1.3 docs and adapt.
**tcptop/src/main.rs:**
Minimal entry point that calls privilege check and exits:
```rust
mod privilege;
mod collector;
mod model;
fn main() {
privilege::check_privileges();
println!("tcptop: privilege check passed, eBPF loading not yet implemented");
}
```
After creating all files, run `cargo check` (not `cargo build` -- building eBPF requires bpf-linker which may not be installed). If `cargo check` fails on the eBPF crate, that is acceptable -- focus on the userspace crate compiling: `cargo check -p tcptop --lib` or similar.
</action>
<verify>
<automated>cd /Users/zrowitsch/local_src/tcptop && cargo check -p tcptop-common 2>&1 | tail -5 && cargo check -p tcptop 2>&1 | tail -10</automated>
</verify>
<acceptance_criteria>
- Cargo.toml contains `members = ["tcptop", "tcptop-common", "tcptop-ebpf"]`
- rust-toolchain.toml contains `bpfel-unknown-none`
- tcptop-common/src/lib.rs contains `#![no_std]` AND `#[repr(C)]` AND `pub struct DataEvent` AND `pub struct StateEvent` AND `pub struct TcptopEvent`
- tcptop-common/src/lib.rs contains `pub union TcptopEventData` (LOCKED: union layout, not flat struct)
- tcptop-common/src/lib.rs contains `af_family: u16` (IPv6-ready per Pitfall 4)
- tcptop-common/src/lib.rs contains `saddr: [u8; 16]` (16-byte IP fields, not u32)
- tcptop-common/src/lib.rs contains `EVENT_TCP_SEND` AND `EVENT_TCP_RECV` AND `EVENT_UDP_SEND` AND `EVENT_UDP_RECV` AND `EVENT_TCP_STATE`
- tcptop-ebpf/src/main.rs contains `#![no_std]` AND `#![no_main]` AND `RingBuf`
- tcptop/build.rs contains `aya_build`
- tcptop/Cargo.toml contains `aya-build` in build-dependencies
- `cargo check -p tcptop-common` succeeds (exit code 0)
- `cargo check -p tcptop` succeeds (exit code 0) -- verifies workspace resolution, build.rs wiring, and userspace crate compilation
</acceptance_criteria>
<done>Workspace structure exists with three crates; common and userspace crates compile; shared types are defined with repr(C) union layout and IPv6-ready fields; eBPF crate has skeleton with RingBuf map; userspace crate has build.rs with aya-build.</done>
</task>
<task type="auto">
<name>Task 2: Implement privilege check and platform abstraction trait</name>
<files>
tcptop/src/privilege.rs,
tcptop/src/collector/mod.rs,
tcptop/src/model.rs,
tcptop/src/main.rs
</files>
<read_first>
tcptop/src/main.rs (current state from Task 1),
tcptop-common/src/lib.rs (shared types to reference),
.planning/phases/01-data-pipeline/01-RESEARCH.md (privilege check code example, NetworkCollector trait design, ConnectionRecord design),
.planning/phases/01-data-pipeline/01-CONTEXT.md (D-09, D-10, D-11 for privilege; D-05, D-06, D-07, D-08 for model types)
</read_first>
<action>
Implement three modules that Plan 02 and Plan 03 depend on.
**tcptop/src/privilege.rs (per D-09, D-10, D-11):**
```rust
use nix::unistd::geteuid;
use std::process;
pub fn check_privileges() {
if geteuid().is_root() {
return;
}
if has_required_capabilities() {
return;
}
eprintln!("error: tcptop requires root privileges. Run with sudo.");
process::exit(77);
}
fn has_required_capabilities() -> bool {
let status = match std::fs::read_to_string("/proc/self/status") {
Ok(s) => s,
Err(_) => return false,
};
for line in status.lines() {
if let Some(hex) = line.strip_prefix("CapEff:") {
let hex = hex.trim();
if let Ok(caps) = u64::from_str_radix(hex, 16) {
let cap_perfmon = 1u64 << 38;
let cap_bpf = 1u64 << 39;
return (caps & cap_perfmon != 0) && (caps & cap_bpf != 0);
}
}
}
false
}
```
**tcptop/src/model.rs (per D-05, D-06, D-07, D-08, D-12, D-15):**
Define the userspace-side connection model. These are NOT the eBPF shared types (those are in tcptop-common). These are rich Rust types used by the aggregator and output formatter.
```rust
use std::net::IpAddr;
use std::time::Instant;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Protocol {
Tcp,
Udp,
}
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub struct ConnectionKey {
pub protocol: Protocol,
pub local_addr: IpAddr,
pub local_port: u16,
pub remote_addr: IpAddr,
pub remote_port: u16,
}
#[derive(Debug, Clone)]
pub struct ConnectionRecord {
pub key: ConnectionKey,
pub pid: u32,
pub process_name: String,
pub tcp_state: Option<TcpState>, // None for UDP (per D-07)
pub bytes_in: u64,
pub bytes_out: u64,
pub packets_in: u64,
pub packets_out: u64,
pub rate_in: f64, // bytes/sec (per DATA-06)
pub rate_out: f64, // bytes/sec
pub prev_bytes_in: u64, // for rate calculation
pub prev_bytes_out: u64,
pub rtt_us: Option<u32>, // microseconds, None for UDP
pub last_seen: Instant,
pub is_partial: bool, // true for pre-existing connections (per D-15)
pub is_closed: bool, // true when TCP state -> CLOSE (per D-12)
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TcpState {
Established,
SynSent,
SynRecv,
FinWait1,
FinWait2,
TimeWait,
Close,
CloseWait,
LastAck,
Listen,
Closing,
NewSynRecv,
}
impl TcpState {
pub fn from_kernel(state: u32) -> Option<Self> {
// Kernel TCP state values from include/net/tcp_states.h
match state {
1 => Some(TcpState::Established),
2 => Some(TcpState::SynSent),
3 => Some(TcpState::SynRecv),
4 => Some(TcpState::FinWait1),
5 => Some(TcpState::FinWait2),
6 => Some(TcpState::TimeWait),
7 => Some(TcpState::Close),
8 => Some(TcpState::CloseWait),
9 => Some(TcpState::LastAck),
10 => Some(TcpState::Listen),
11 => Some(TcpState::Closing),
12 => Some(TcpState::NewSynRecv),
_ => None,
}
}
pub fn as_str(&self) -> &'static str {
match self {
TcpState::Established => "ESTABLISHED",
TcpState::SynSent => "SYN_SENT",
TcpState::SynRecv => "SYN_RECV",
TcpState::FinWait1 => "FIN_WAIT1",
TcpState::FinWait2 => "FIN_WAIT2",
TcpState::TimeWait => "TIME_WAIT",
TcpState::Close => "CLOSE",
TcpState::CloseWait => "CLOSE_WAIT",
TcpState::LastAck => "LAST_ACK",
TcpState::Listen => "LISTEN",
TcpState::Closing => "CLOSING",
TcpState::NewSynRecv => "NEW_SYN_RECV",
}
}
}
impl std::fmt::Display for TcpState {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.as_str())
}
}
```
**tcptop/src/collector/mod.rs (per PLAT-03):**
Define the platform abstraction trait. Use `tokio::sync::mpsc` channel pattern from research.
```rust
pub mod linux; // Will be implemented in Plan 02
use crate::model::ConnectionRecord;
use anyhow::Result;
use tokio::sync::mpsc;
/// Events emitted by the collector to the aggregator
#[derive(Debug)]
pub enum CollectorEvent {
TcpSend { key: crate::model::ConnectionKey, pid: u32, comm: String, bytes: u32, srtt_us: u32 },
TcpRecv { key: crate::model::ConnectionKey, pid: u32, comm: String, bytes: u32, srtt_us: u32 },
UdpSend { key: crate::model::ConnectionKey, pid: u32, comm: String, bytes: u32 },
UdpRecv { key: crate::model::ConnectionKey, pid: u32, comm: String, bytes: u32 },
TcpStateChange { key: crate::model::ConnectionKey, pid: u32, old_state: u32, new_state: u32 },
}
/// Platform abstraction for network data collection.
/// Linux: eBPF kprobes + tracepoints (Phase 1)
/// macOS: libpcap + PKTAP (Phase 4)
#[async_trait::async_trait]
pub trait NetworkCollector: Send {
/// Start collecting network events. Sends events to the provided channel.
/// Returns when stop() is called or an error occurs.
async fn start(&mut self, tx: mpsc::Sender<CollectorEvent>) -> Result<()>;
/// Stop collecting and clean up kernel resources (detach probes, etc).
async fn stop(&mut self) -> Result<()>;
/// Bootstrap pre-existing connections (e.g., from /proc/net/tcp).
/// Called once before start().
fn bootstrap_existing(&self) -> Result<Vec<ConnectionRecord>>;
}
```
Add `async-trait = "0.1"` to tcptop/Cargo.toml dependencies if not already present.
Create a placeholder `tcptop/src/collector/linux.rs` with a comment: `// Linux eBPF collector -- implemented in Plan 02`
**Update tcptop/src/main.rs:**
Wire up the privilege check and declare all modules:
```rust
mod privilege;
mod collector;
mod model;
fn main() {
privilege::check_privileges();
println!("tcptop: privilege check passed");
// eBPF loading and event loop implemented in Plan 02 and Plan 03
}
```
</action>
<verify>
<automated>cd /Users/zrowitsch/local_src/tcptop && cargo check -p tcptop 2>&1 | tail -10</automated>
</verify>
<acceptance_criteria>
- tcptop/src/privilege.rs contains `eprintln!("error: tcptop requires root privileges. Run with sudo.")`
- tcptop/src/privilege.rs contains `process::exit(77)`
- tcptop/src/privilege.rs contains `geteuid().is_root()`
- tcptop/src/privilege.rs contains `1u64 << 38` AND `1u64 << 39` (CAP_PERFMON and CAP_BPF)
- tcptop/src/privilege.rs contains `CapEff:`
- tcptop/src/collector/mod.rs contains `trait NetworkCollector`
- tcptop/src/collector/mod.rs contains `async fn start`
- tcptop/src/collector/mod.rs contains `fn bootstrap_existing`
- tcptop/src/collector/mod.rs contains `mpsc::Sender<CollectorEvent>`
- tcptop/src/collector/mod.rs contains `enum CollectorEvent`
- tcptop/src/model.rs contains `pub struct ConnectionKey`
- tcptop/src/model.rs contains `pub struct ConnectionRecord`
- tcptop/src/model.rs contains `pub enum Protocol` with `Tcp` and `Udp` variants
- tcptop/src/model.rs contains `pub enum TcpState` with `from_kernel` method
- tcptop/src/model.rs contains `is_partial: bool` (per D-15)
- tcptop/src/model.rs contains `is_closed: bool` (per D-12)
- tcptop/src/model.rs contains `rate_in: f64` AND `rate_out: f64` (per DATA-06)
- tcptop/src/model.rs contains `rtt_us: Option<u32>` (per DATA-05)
- tcptop/src/main.rs contains `privilege::check_privileges()`
- `cargo check -p tcptop` succeeds (exit code 0) -- note: may need to allow dead_code warnings for unused modules
</acceptance_criteria>
<done>Privilege check implemented per D-09/D-10/D-11; NetworkCollector trait defined with mpsc channel pattern per PLAT-03; ConnectionRecord and ConnectionKey model types defined with all required fields for DATA-01 through DATA-07; userspace crate compiles cleanly.</done>
</task>
</tasks>
<verification>
1. `cargo check -p tcptop-common` exits 0 (shared types compile under std)
2. `cargo check -p tcptop` exits 0 (userspace crate compiles with all modules)
3. `grep -r "repr(C)" tcptop-common/src/lib.rs` returns at least 3 matches (DataEvent, StateEvent, TcptopEvent)
4. `grep "union TcptopEventData" tcptop-common/src/lib.rs` returns a match (locked union layout)
5. `grep "trait NetworkCollector" tcptop/src/collector/mod.rs` returns a match
6. `grep "exit(77)" tcptop/src/privilege.rs` returns a match
</verification>
<success_criteria>
- Workspace structure with three crates exists and userspace + common crates compile
- All shared types are #[repr(C)] with IPv6-ready [u8; 16] address fields
- TcptopEventData uses union layout (locked contract for Plan 02 parse_event)
- Privilege check exits 77 with exact error message per D-09
- NetworkCollector trait is defined with start/stop/bootstrap_existing per PLAT-03
- ConnectionRecord has all fields needed for DATA-01 through DATA-07
- eBPF crate has skeleton with RingBuf map declaration
</success_criteria>
<output>
After completion, create `.planning/phases/01-data-pipeline/01-01-SUMMARY.md`
</output>