Senior Big Data Engineer

In this role, you will work with our Business Analytics & Insights (BAI) team, collaborating with cross-functional teams to deliver high-quality data solutions in areas such as Supply Chain, Finance, Operations, Customer Experience, HR, Risk Management, and Global IT.

Responsibilities:

• Lead the technical planning for data migration, including data ingestion, transformation, storage, and access control within Azure Data Factory and Azure Data Lake.
• Design and implement scalable, efficient data pipelines to ensure smooth data movement from multiple sources using Azure Databricks.
• Develop reusable frameworks for the ingestion of large datasets.
• Ensure data quality and integrity by implementing robust validation and cleansing mechanisms throughout the data pipeline.
• Work with event-based/streaming technologies to ingest and process data in real-time.
• Provide technical support to the team, resolving challenges during the migration and post-migration phases.
• Stay current with the latest advancements in cloud computing, data engineering, and analytics technologies; recommend best practices and industry standards for data lake solutions.

Requirements

• 5+ years of IT experience.
• Minimum of 4 years working with Azure Databricks.
• Proficiency in Data Modeling and Source System Analysis.
• Strong knowledge of PySpark and SQL.
• Experience with Azure components: Data Factory, Data Lake, SQL Data Warehouse (DW), and Azure SQL.
• Experience with Python for data engineering purposes.
• Ability to conduct data profiling, cataloging, and mapping for technical design and construction of data flows.
• Familiarity with data visualization/exploration tools.

Additional information

• Technical leadership skills with the ability to guide and mentor teams.
• Strong problem-solving capabilities, able to translate business requirements into data science solutions.
• Excellent communication skills, with the ability to explain complex concepts to both technical and non-technical stakeholders.
• Strong team player with excellent interpersonal and collaboration skills.
• Ability to deliver high-quality results within specified timelines.