Senior Data Engineer

We are looking for a Senior Data Engineer to join our growing data platform team. We need a skilled engineer who will help us develop and maintain data architectures and processing solutions that power our data scientists, BI analytics and many other internal data consumers. One major role of the team is to prepare and maintain self-serve data infrastructure, and ensure data availability, security and scalability. Another one is to build and maintain pipelines for integrating internal and third-party data while ensuring high data integrity and quality. The engineer will work in a dynamic, fast-changing environment with plenty of interaction with other teams.

Responsibilities

• Design, build, maintain and manage the data platform components.
• Manage data related projects, coordinate work of involved data engineers.
• Lead data modelling and data architecture activities.
• Create, maintain and monitor pipelines for regular data processing while ensuring data reliability, quality and efficiency.
• Maintain the data serving layer, manage permissions and automate processes around the data catalog.
• Develop integrations with internal and external data sources.
• Cross-functional cooperation with data scientists, data engineers, devops, architects, developers, QA and other parties. Collaborate to bring new features and services into production.
• Develop and improve operational practices and procedures.

Requirements

• Experience in Databricks - creating notebooks, developing integration code in Python/PySpark, working with delta tables (Delta Lakehouse),
• Experience in managing projects, data related initiatives, designing data architecture and data modelling.
• Expertise with SQL, database design/structures, ETL/ELT design patterns, and DataMart structures (star, snowflake schemas, etc.)
• Experience in Azure storage solution, particularly Azure Data Lake Storage and working with Parquet files and partitions
• Experience in Azure Data Factory (ADF) - creating pipelines and activities in ADF for full and incremental data loads into Azure Data Lake Store
• Experience with Azure DevOps / Git
• Azure Data Engineer certification (DP-203) or Databricks Data Engineer Associate / Professional would be a plus
• Proficiency in English