Lead Data Analytics Engineer

Prudential’s purpose is to be partners for every life and protectors for every future. Our purpose encourages everything we do by creating a culture in which diversity is celebrated and inclusion assured, for our people, customers, and partners. We provide a platform for our people to do their best work and make an impact to the business, and we support our people’s career ambitions. We pledge to make Prudential a place where you can Connect, Grow, and Succeed.

Job Summary:

We are seeking a highly skilled Senior Data Engineer with over 8 years of hands-on experience in Spark, Azure components, and Databricks within the insurance industry. The ideal candidate will possess a deep understanding of data modelling, proficiency in the modern data stack, familiarity with machine learning, a proactive approach to staying updated on cutting-edge technologies, and a proven track record of collaborating with diverse stakeholders in an agile environment.

Key Responsibilities:

• Build pipelines to ingest a wide variety of data from multiple sources within the organization and from external sources.
• Optimize existing pipelines
• Testing data structures to ensure that they are fit for use for data analysts and scientists.
• Prepare and maintain environments for secure prototyping, development, testing and data manipulation for data scientists.
• Design and implement effective data storage solutions and models.
• Assist data models and AI/ML deployment.
• Assess database implementation procedures to ensure they comply with internal and external regulations.
• Prepare accurate database design and architecture reports for management and executive teams.
• Oversee the migration of data from legacy systems to new solutions on Cloud infrastructure.
• Monitor the system performance by performing regular tests, troubleshooting, and integrating new features.
• Automate low value tasks

Minimum Experience, Behaviors and Education Requirements:

• Experience with at least one major Cloud Infrastructure provider (Azure/AWS/GCP)
• Experience building data pipelines using batch processing with Apache Spark (Spark SQL, DataSet / Dataframe API) or Hive query language (HQL)
• Knowledge of Big Data ETL processing tools
• Experience in Data Modelling, Data mapping for Data Warehouse and Data Marts solutions
• Experience with Hive and Hadoop file formats (Avro / Parquet / ORC)
• Basic knowledge of scripting (shell / bash)
• Experience of working with multiple data sources including relational databases (SQL Server / Oracle / DB2 / Netezza), NoSQL / document databases, flat files
• Understanding of CICD tools such as Jenkins, JIRA, Bitbucket, Artifactory, Bamboo and Azure Dev-ops.
• DevOps practices using Git version control
• An interest in staying up to date with industry standards and technological advancements that will improve the quality of your outputs.
• Ability to debug, fine tune and optimize large scale data processing jobs
• Highly capable in Python or Scala
• SQL
• DataBricks
• Knowledge in
• Azure Data Factory
• Google Big Query
• Azure DevOps
• BitBucket or GitHub
• Machine learning
• MLFlow
• Knowledge of life insurance industry preferred.

Prudential is an equal opportunity employer. We provide equality of opportunity of benefits for all who apply and who perform work for our organisation irrespective of sex, race, age, ethnic origin, educational, social and cultural background, marital status, pregnancy and maternity, religion or belief, disability or part-time / fixed-term work, or any other status protected by applicable law. We encourage the same standards from our recruitment and third-party suppliers taking into account the context of grade, job and location. We also allow for reasonable adjustments to support people with individual physical or mental health requirements.