Data Engineer (GCP)

About the role

As a Data Engineer, you will design and scale high-performance data pipelines that power large-scale financial data processing and analytics on GCP. Leveraging Python, SQL, BigQuery, and tools like Dataproc and Cloud Composer, you’ll build distributed, fault-tolerant systems and implement advanced data models for regulatory-grade datasets. This role offers strong ownership, cross-functional collaboration, and the opportunity to integrate AI-driven capabilities using Vertex AI within modern data platforms.

What you will do

• Build and maintain scalable, distributed, fault-tolerant data pipelines on GCP;
• Develop and manage lakehouse layers and Delta Lake workflows using BigQuery and Dataproc;
• Collaborate with stakeholders across data engineering, compliance, and business teams;
• Design and implement pipelines to acquire, normalise, transform, and release large volumes of financial data;
• Design and implement bitemporal data models on BigQuery for regulatory-grade time-series datasets;
• Build and maintain testing frameworks for data pipelines and transformation logic;
• Own end-to-end solutions including ingestion pipelines, QA workflows, correction management, and audit trails;
• Contribute to shared platform services in a collaborative environment;
• Support implementation of AI solutions including data ingestion, anomaly detection, and semantic search using Vertex AI.

Must haves

• 6–8 years of experience in data engineering;
• Proficiency in Python for data pipelines, transformation logic, and automation;
• Proficiency in SQL with hands-on experience in BigQuery including partitioning, clustering, and time-series queries;
• Experience with Cloud Composer (Apache Airflow) for pipeline orchestration;
• Working knowledge of Dataproc (Apache Spark) for batch ingestion and incremental processing;
• Experience with AI-assisted development tools such as GitHub Copilot or similar;
• Experience with Git version control and collaboration workflows;
• Familiarity with REST APIs for integrations;
• Familiarity with GCP technologies (Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, VPC Service Controls);
• Understanding of financial data concepts related to equities and other asset classes;
• Upper-intermediate English level.

Nice to haves

• Knowledge of data libraries such as pandas or PySpark;
• Experience with columnar storage and time-series analytics tools such as ClickHouse;
• Familiarity with Dataplex for data governance and lineage;
• Understanding of Change Data Capture (CDC) using Datastream;
• Understanding of bitemporal data modeling concepts;
• Knowledge of financial reference data such as equities, fixed income, or corporate actions;
• Experience with BigQuery cost management techniques;
• Experience with CI/CD pipelines and Terraform for infrastructure as code;
• Exposure to LLMs and Agentic AI using Vertex AI for data-related use cases.