Software Engineer- BIS (Baseten Inference Stack)

Details

• Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference
• Work across the stack, from customer-facing features to low-level infrastructure components
• Build platform capabilities related to routing, autoscaling, scheduling, observability, and runtime management
• Improve the reliability, scalability, and usability of our inference stack
• Collaborate closely with Model Performance engineers to make new inference optimizations broadly available to customers and easy to configure
• Help define best practices around testing, release automation, benchmarking, and operational excellence
• Debug complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads
• Make thoughtful engineering tradeoffs balancing performance, reliability, operational simplicity, and developer experience
• Own projects end-to-end: from architecture and implementation through deployment, monitoring, and iteration based on customer feedback
• Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field
• Strong background in distributed systems, backend infrastructure, or platform engineering
• Experience building and operating production systems where reliability, latency, and scale are first-class concerns
• Strong sense of developer experience: you think about how systems are used, not just how they work
• Motivated and willing to learn new languages, frameworks, and systems as needed
• Ability to debug complex systems across multiple layers of the stack
• Genuine interest in inference engineering. You don’t need to have hands on experience but are willing to learn
• Excellent communication and collaboration skills
• Experience with Kubernetes, including concepts like operators and custom resources
• Prior work on Dynamo, vLLM, SGLang, TensorRT-LLM, or similar inference frameworks
• Experience with distributed scheduling, autoscaling, or service orchestration
• Experience operating GPU workloads in production
• Familiarity with observability tooling, CI/CD systems, or release automation
• Experience contributing to open-source infrastructure or ML systems
• Competitive compensation, including meaningful equity.
• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)
• Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.