Data Engineer II

About the team

Do you like, love, sip, breathe, sit, sleep and drink data? If yes, then you’ll fit in this team like a pen in a cap. Consuming the ML model’s outputs, understanding those outputs, using them to make decisions, and the rationale for the decisions - all a part of the Data team’s skill set.

About the role

In this role, you will have the opportunity to create a significant impact on our business & most importantly our customers through your technical expertise on data as we take on challenges that can reshape the financial experience for the next generation. If you are a highly motivated team player with a knack for problem solving through technology, then we have a perfect job for you.

What You will do

• Strong engineering mindset - build self serve capabilities, automated monitoring / alerting, self healing (restartability/graceful failures), autoscaling, cost efficient features.
• Good understanding and hands on experience on managing open source data tech stack like Apache Trino, Apache Ranger, Kafka, Superset.
• Build platform features leveraging your deep expertise on AWS stack e.g. S3, EMR, Glue, RDS, EKS, ECS, Redshift, EC2, Terraform, Helm charts, Git.
• Build self serve capabilities for various personas (analysts, data scientists, operations, product managers) to ingest, process and consume the data.
• Continuously monitor and improve performance, capacity, and security of the data platform and regularly perform server tuning and maintenance activities.
• Identify, design, and implement internal process improvements; optimizing self serve, data observability, data quality, governance and security, infrastructure and tech stack enhancements
• You’ll be responsible for guiding the data team towards the company’s goals.

Data Projects You’ll be working on

• Build data platform capabilities to:
  • ingest data from products, services, applications in streaming manner while ensuring data integrity, quality, security (handling PII)
  • Enable users to use a data platform to discover and transform data into actionable insights. Build capabilities using Spark, Trino, Airflow, EMR to enable self serve compute
  • Enable consumption for analytics, data science, operations, CX use cases through dashboard, ad-hoc analysis, systemic (application) consumption
• Data Governance: slice being Fintech and deals with sensitive data, security and compliance are very critical aspects of data management. Data platform needs to ensure security for data in motion, data at rest, authentication and authorization, audit trail, reporting for compliance to regulatory authorities. This required building robust features in the data platform through the life cycle of data.
• Setting up, managing and optimizing open source tech stack
• Hands on experience on managing massive scale data infrastructure on Kubernetes (EKS)
• Observability: Measure, monitor and continuously optimize everything: Infrastructure, workloads, access pattern, resource usage, success rates for workloads (dashboards, ad-hoc queries, ML models, data pipelines, ingestion streams), data quality.
• Cutting edge AI tools for improving dev productivity like copilot

Requirements

What you will need

• 3+ years of experience in working as a Data Engineer with hands-on experience in software engineering practices (CI/CD).
• 2+ years of experience with Spark and Hadoop technologies.
• Hands on programming experience with Python, PySpark and scala (preferably)
• Experience with AWS big data tools EMR, AWS Glue, S3. EKS, Git. Terraform, Helm Charts
• Experience on Graph DB (Neo4j and/or OrientDB) AWS Neptune and Search DB (Elastic Search) is an added advantage.