Software Engineer- Model Products

Details

• Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling and multi-modal serving
• Profile and optimize TensorRT-LLM kernels, analyze CUDA kernel performance, implement custom CUDA operators, tune memory allocation patterns for maximum throughput and optimize communication patterns across multi-GPU setups
• Productionize performance improvements across runtimes with deep understanding of their internals: speculative decoding implementations, guided generation for structured outputs, custom scheduling and routing algorithms for high-performance serving
• Build comprehensive benchmarking frameworks that measure real-world performance across different model architectures, batch sizes, sequence lengths, and hardware configurations
• Productionize performance improvements across runtimes (e.g.TensorRT, TensorRT‑LLM): speculative decoding, quantization, batching, and KV‑cache reuse.
• Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks to measure speed, reliability, and quality.
• Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication.
• Collaborate closely with other teams to deliver robust, developer‑friendly model serving experiences.
• 3+ years experience building and operating distributed systems or large‑scale APIs.
• Proven track record of owning low‑latency, reliable backend services (rate‑limiting, auth, quotas, metering, migrations).
• Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management.
• Comfortable debugging complex systems, from runtime internals to GPU execution traces.
• Strong written communication; able to produce clear design docs and collaborate across functions.
• Experience with LLM runtimes (vLLM, SGLang, TensorRT‑LLM) or contributions to open-source inference engines (vLLM, TensorRT-LLM, SGLang, TGI)
• Knowledge of Kubernetes, service meshes, API gateways, or distributed scheduling.
• Background in developer‑facing infrastructure or open‑source APIs.
• We value infra‑leaning generalists who bring strong engineering fundamentals and curiosity. ML experience is a plus, but not required.
• Competitive compensation, including meaningful equity.
• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)
• Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.