WithMiraRevisando roles remotos de TI

Senior Data Engineer (Modern Data Platform & AI) (all genders)| Berlin, hybrid | WithMira

Resumen del rol

Senior Data Engineer (Modern Data Platform & AI) (all genders)| Berlin, hybrid

Requisitos y responsabilidades

Contenido del rol extraído en secciones para revisar más rápido.

Data Modeling & Transformation:Design necessary data models and transformations to curate raw data. Develop, optimize and maintain existing data models, pipelines, and transformations to support analytics, reporting, and AI use cases such as but not limited to curating, transforming, annotating and modeling data.
Design necessary data models and transformations to curate raw data.
Develop, optimize and maintain existing data models, pipelines, and transformations to support analytics, reporting, and AI use cases such as but not limited to curating, transforming, annotating and modeling data.
Data Platform Architecture:Architect and contribute in implementing a scalable, modern data platform, including data lakehouse or warehouse, to support real-time/near-real-time data flows from Kafka to downstream consumers. Optimize ETL/ELT pipelines using tools like DBT, Spark, or Airflow, bridging upstream (e.g. Debezium, MSK) and downstream processes. Evaluate and integrate new technologies to support hybrid monolith-microservices architecture and ML and AI enablement. Ensure seamless migrations and minimal disruptions during platform evolution.
Architect and contribute in implementing a scalable, modern data platform, including data lakehouse or warehouse, to support real-time/near-real-time data flows from Kafka to downstream consumers.
Optimize ETL/ELT pipelines using tools like DBT, Spark, or Airflow, bridging upstream (e.g. Debezium, MSK) and downstream processes.
Evaluate and integrate new technologies to support hybrid monolith-microservices architecture and ML and AI enablement.
Ensure seamless migrations and minimal disruptions during platform evolution.
Real-Time Data Integration: Build and optimize real-time data pipelines using Kafka, Spark, and Delta Live Tables.
Data Governance & Quality: Support the team lead in establishing and enforcing data governance frameworks, including data lineage, quality standards, catalogue, metadata management, SSOT for business glossaries/CBC terms, and policies to ensure reliable reporting. Ensure the existence of, or adaptation to, full Data Life Cycle Management (DLCM) and end-to-end testing.
Support the team lead in establishing and enforcing data governance frameworks, including data lineage, quality standards, catalogue, metadata management, SSOT for business glossaries/CBC terms, and policies to ensure reliable reporting.
Ensure the existence of, or adaptation to, full Data Life Cycle Management (DLCM) and end-to-end testing.
AI/ML Enablement: Collaborate with the team to integrate AI/ML capabilities, such as feature engineering and model serving, to accelerate data products for market penetration and operational efficiency, as well as operationalizing ML models and integrate AI into business processes.
Knowledge Sharing: Mentor the team on best practices, modern tools (e.g., Databricks, Snowflake, AI adaptation and integrations like Cursor/CodeRabbit), and cloud-native scalability. And last but not least foster a culture of innovation and continuous improvement.
Stakeholder Collaboration: Collaborate with Product Analytics, domain teams, and business to deliver data solutions that drive value and are aligned with business needs.

Details

Design necessary data models and transformations to curate raw data.
Develop, optimize and maintain existing data models, pipelines, and transformations to support analytics, reporting, and AI use cases such as but not limited to curating, transforming, annotating and modeling data.
Architect and contribute in implementing a scalable, modern data platform, including data lakehouse or warehouse, to support real-time/near-real-time data flows from Kafka to downstream consumers.
Optimize ETL/ELT pipelines using tools like DBT, Spark, or Airflow, bridging upstream (e.g. Debezium, MSK) and downstream processes.
Evaluate and integrate new technologies to support hybrid monolith-microservices architecture and ML and AI enablement.
Ensure seamless migrations and minimal disruptions during platform evolution.
Support the team lead in establishing and enforcing data governance frameworks, including data lineage, quality standards, catalogue, metadata management, SSOT for business glossaries/CBC terms, and policies to ensure reliable reporting.
Ensure the existence of, or adaptation to, full Data Life Cycle Management (DLCM) and end-to-end testing.

Skills

Master's degree in Computer Science, Data Engineering, or related field (or equivalent experience)
10+ years of experience in data engineering, with 5+ years in senior roles focused on modern architectures.
Excellent communication and collaboration skills, the ability to drive change and influence stakeholders, and a passion for mentoring, coaching, and sharing knowledge
Proven expertise in designing, developing & maintaining data lakehouses/DWH (e.g., Databricks Delta Lake, Snowflake) and transformations (e.g., DBT, SQL/Python/Spark).
Strong experience with cloud platforms such as AWS services (S3, Athena, MSK/Kafka, Terraform) and real-time streaming (e.g., Kafka, Spark Structured Streaming, Flink).
Hands-on knowledge of data governance tools (e.g., Unity Catalog, Collibra) for lineage, quality, catalogs, and SSOT.
Familiarity in AI/ML pipelines and MLOps (e.g., MLflow, feature stores) and complex system integration within modern data technologies.
Proficiency in CI/CD for data, and tools like Git, Airflow, or dbt Cloud.
Experience with large-scale data modeling (DataVault, dimensional, schema-on-read) and optimizing for self-service analytics.

Roles similares