Senior Software Engineer- Analytics Data Platform Lakehouse

Details

• Design, build, and operate core components of our lakehouse platform, including Apache Iceberg table management (data compaction, data layout optimization, materialized view scheduling…) and Iceberg catalog
• Drive adoption of open table formats across internal teams, owning the integration of Trino, Spark and other query engines (DuckDB, Puppygraph…) with our Iceberg-based lakehouse at petabyte scale
• Build observability for managed iceberg tables, to identify query performance bottlenecks, cost drivers and contribute fixes back to upstream open-source projects (Iceberg, Trino, Spark, Open Lineage) where relevant
• Build self-serve tooling and abstractions that allow data engineering teams to reliably run thousands of pipelines per day against our lakehouse
• Collaborate with data engineers, analysts, and infrastructure teams to define the roadmap for our lakehouse architecture and shape how Datadog manages analytic data at scale
• You have a BS/MS/PhD in Computer Science, Engineering, or a related field, or equivalent professional experience
• You have deep, production-grade experience with one or more of Apache Iceberg, Trino, or Apache Spark, ideally demonstrated through significant open-source contributions: merged PRs, committer status, or PMC membership on projects
• You have built or operated large-scale distributed data systems
• You have a solid grasp of query planning, columnar file formats (Parquet, ORC), and table format internals (snapshots, manifests, partition evolution)
• You are fluent in Java, Scala or Go and comfortable with Python for pipeline tooling
• You have experience deploying and running data infrastructure on Kubernetes in cloud environments