Senior Spark Engineer

You’ll be working within an international group of teams which span from India to Europe and US

As a Senior Spark/Lead Engineer, you will be working along and guiding a team of diverse engineers. This includes DevOps, Data, Backend and Front End engineers. Should be able to work independently and guide junior engineers. Should be passionate and driven to learn, suggest and adapt to new technologies

Responsibilities

• Discuss the Cost of Change (= code quality) with your team members continuously
• Write unit tests, integration tests and API tests
• Should be able to support the application 24/7 based on team on call rotations.
• Write clean code (mindful about coupling, separation of concerns, etc.)
• You would be expected to write code 90% of the time, with an emphasis on Test Drive Development
• Spend at least 10% of your time learning and improving existing application architecture
• Should be open to learn and adapt to new technology architectures and patterns
• High knowledge of Hadoop or any distributed architectures
• Fair amount of experience with building and creating CI/CD pipelines
• Should be able to conduct code reviews and design discussions
• You would be expected to code and also analyze the impact of the change on data
• You should be able to troubleshoot and solve complex problems in production.
• You should be able to collaborate and coordinate and present ideas with different stakeholders including product, datascience, account managers etc
• You should be able to diagnose AWS infrastructure issues related to the application
• Follow Agile principles and feature related discussions and participate in grooming and planning sessions to effectively translate business requirements to Agile stories.
• Follow best practices on 24/7 application monitoring and orchestration and performance optimization
• Implement and Practice DevOps and SecOps for continuous incremental delivery and quality product with the guidance of senior engineers

Key Skills

• Required

• Bachelor or Master’s degree in computer science or related discipline (field), or equivalent work experience

• 4-8 years of Experience with Spark using Scala, Spark SQL and Presto. Upgrading, Maintaining and performance tuning large Spark/Scala applications is required.

• 4-8 years of advanced experience with Java and Relational databases is required.

• 4+ years of experience working with AWS services (RDS, S3) is required.

• 4+ Hands-on experience with Airflow DAGs and Python is required

• 2+ Experience using monitoring and alert orchestration tools such as Prometheus, Grafana, OpsGenie/PagerDuty tools is a must.

• 2+ Experience with building CICD pipelines in GitLab for applications running on Kubernetes (EKS) using Docker is required.

• Ability to debug, performance tune large scale Java and BigData applications is needed.

• Fluent in English, both spoken and written, with a large vocabulary (C1 english level)

• Understand and implement basic object oriented principles and functional programming principles. Implements good coding practices with good unit and integration testing, with an emphasis on Test Driven Development.

• Commitment to following best practices for security, scalability, and performance.

• Excellent problem-solving skills and ability to troubleshoot complex technical issues in production environments.

• Strong communication skills to collaborate effectively with cross-functional teams, stakeholders, and third-party vendors.

• Continuous improvement mindset to identify opportunities for automation, optimization, and efficiency gains in infrastructure and deployment processes.

• Ability to document processes, procedures, and technical architectures for knowledge sharing and future reference.

• Preferred

• Leadership qualities and the ability to inspire and motivate a team

• At least 1 year of experience with Test-driven development

• Familiarity working with large scale enterprise Java/BigData systems using Agile, TDD & DevOps methodologies.

• Proven track record on delivering enterprise software solutions using Agile scrum methodology.