Senior Data Engineer

Details

• Develop and automate large scale, high-performance data processing systems (batch and/or streaming)
• Translate business needs into data models: implement data strategies, build data flows and develop conceptual data models.
• Implement deployment and infrastructure automation strategies.
• Build data support for our experimentation efforts, solving problems from statistical test automation to building real-time M/L pipelines
• Automate manual processes, optimize data delivery, and re-design infrastructure for greater scalability.
• Promote data modeling standardization, define, and drive adoption of the standards.
• Build analytics and reporting tools that utilize the data pipeline to provide results and business value to make data-driven decisions.
• Maintain and improve upon our existing ETL pipeline
• Perform routine maintenance tasks and upgrades of our server architecture
• Work with stakeholders to assist with data-related technical issues and support data infrastructure needs.
• Apply best practices for testing and deployment in an agile environment.
• Contribute to shared Data Engineering tooling & standards to improve engineering productivity across the company
• Expert-level understanding of relational databases (columnar and row-based)
• Expertise in SQL is a must.
• Prior experience working with Tier-1 technology consulting firms is a huge plus
• Proven hands-on experience building complex ETLs in a business environment with large-scale, complex datasets
• Scripting experience in python and bash shell required
• Fully proficient in pipeline building and job automations
• Proficiency with Git
• Experience building RESTful APIs
• Experience with error handling and data validation
• Experience working in GCP or other cloud platforms
• Bachelors in Computer Science, Engineering or similar
• Must be eligible to work in the US.