Lead Data Engineer (Databricks)

Details

• 8+ years of hands-on data engineering experience, designing and delivering production-grade data platforms
• Expert-level in Apache Spark, including runtime internals, performance tuning, and optimisation, you understand what's happening under the hood and use that knowledge to build pipelines that perform at scale.
• You write clean, production-quality code in Python, with Scala experience a strong plus for deeper Spark and performance-critical work.
• You've built and productionalized solutions on the platform, including Delta Lake architectures, Unity Catalog governance, and Databricks Workflows. Databricks certification is a strong plus.
• You have real, working experience across at least two major cloud platforms (AWS, Azure, GCP) with genuine depth in at least one, including cloud-native services such as AWS Redshift/Glue/S3, Azure Synapse/Data Factory/ADLS, or Google BigQuery/Dataflow/GCS.
• You've led data engineering projects end-to-end in a client-facing or consulting context, managing technical scope, navigating stakeholder expectations, and delivering against timelines without cutting corners.
• You bring a DataOps mindset: CI/CD for data pipelines, automated testing, observability, and infrastructure-as-code are standard practice for you, not afterthoughts.
• Your experience spans ETL/ELT design, data warehousing, lakehouse architecture, and data modelling, and you know when to apply each approach.
• Your communication skills allow you to engage technical and non-technical stakeholders equally well, from a client's CTO to a junior engineer on your team.
• Lead Client Data Engagements: Serve as the senior technical lead on client projects. Own the architecture, guide the build, manage delivery risk, and ensure the solution shipped matches what was promised.
• Build and Productionize Data Solutions: Design and implement scalable, reliable data pipelines and lakehouse architectures on Databricks and cloud platforms. You're hands-on keyboard, you write code, review code, and set the engineering standard for the engagement.
• Architect for Scale and Reliability: Translate complex client requirements into robust technical designs, reference architectures, and data models built to last in production.
• Drive Technical Delivery: Manage technical scope and timelines, identify blockers early, and partner with project managers and client stakeholders to keep engagements on track.
• Mentor Data Engineers: Coach junior and mid-level engineers through hands-on pairing, code review, and direct feedback, raising the floor for everyone around you.
• Promote Knowledge Sharing: Contribute technical blogs, reference architectures, and internal guides that reflect hard-won lessons from real client work.
• Champion DataOps Practices: Establish and enforce modern data engineering standards across engagements, automated testing, pipeline observability, version control, CI/CD, and documentation.