Senior Azure Data Engineer

Overview

The Data Engineer participates in the design and build of modern data products that comprise of raw data stores (data lakes) and cleansed data repositories, populated by batch or streaming data pipelines. The Data Engineer works with a team to create a robust, sustainable and flexible design and leads the technical delivery using Agile delivery frameworks like Scrum or Kanban.

Responsibilities

• Developing and managing data processes to ensure that data is available and usable
• Creation and automation of data pipelines and platforms
• Managing and monitoring data quality via automated testing frameworks (Data Driven Testing, TDD, etc.)
• Working closely with Architects, Data Scientists, and DevOps to design, build, test, deliver, and maintain sustainable and highly scalable data solutions
• Researching data acquisition and evaluating suitability
• Integration of data management solutions into client environment
• Actively managing risks to data and ensuring there is a data recovery plan
• Building data repositories such as: data warehouses, data lakes, and operational data stores, etc.

Qualifications

• 4+ years relevant professional work experience.
• Experience and expertise in the following:
  • Creating robust and extensible data pipelines for production systems
  • Experience using Azure data services (e.g., Azure Data Factory, Azure Data Lake Storage, Azure Synapse, Azure Blob Storage)
  • Experience using Azure Databricks, including data ingestion, transformation, and analytics using Databricks notebooks and clusters
  • Experience using continuous integration and continuous deployment (CI/CD) practices and tools (e.g., Azure DevOps, Jenkins)
  • Experience using scripting languages (e.g., Python, Bash, PowerShell) for automating data pipeline deployments and management
  • Creating secure, performant, and well-modeled data stores
  • Common analytical platform architectural patterns (Star Schema, data integration patterns, ABAC, data quality frameworks, etc.
  • Data lake design patterns and technology options (schema on read, metadata capture, search framework
  • Familiarity with NoSQL databases
  • Source code version control management using git
  • Project experience using the Scrum or Kanban framework
  • Professionalism: to include written and oral communication-the ability to communicate collaboratively in front of a whiteboard. An ability to understand your audience and adjust your communication style to fit
  • Aptitude and desire for learning new technologies
  • Technically savvy, entrepreneurial spirit who thrives in environments that reward self-initiative and resourcefulness
  • Understanding of DevOps Research and Assessment (DORA) and the capabilities within the DORA capability catalog is encouraged
  • Microsoft Certified: Azure Data Engineer Associate (DP 203) certification (preferred)