Software Engineer- GPU Fabric Observability

Details

• Real-time telemetry engine — Build the ingestion, reduction, storage, and query path for high-cardinality fabric, host, GPU, and workload telemetry.
• Service-aware fabric diagnosis — Build collectors, probes, and topology-aware correlation to detect latency, drops, stalls, congestion, bad paths, and degradation.
• Software-aware RDMA diagnosis — Tie network behavior to RDMA operations, GPUDirect paths, KV cache transfers, prefill/decode disaggregation, and request latency.
• Own Baseten’s GPU fabric observability and root-cause analysis architecture.
• Build telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services.
• Model topology, flow paths, service ownership, and failure domains.
• Separate true fabric faults from host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload failures.
• Create clear operator workflows for triage, remediation, and post-incident learning.
• Staff-level or senior staff-level experience building production infrastructure software.
• Strong distributed systems background, especially streaming systems, telemetry pipelines, diagnostics, or control-plane software.
• Experience building systems that process high-volume, high-cardinality, noisy operational data.
• Understanding of networking fundamentals and high-performance networks
• Ability to work with low-level infrastructure signals and build practical correlation, anomaly detection, or root-cause analysis systems.
• Competitive compensation, including meaningful equity.
• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)
• Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.