Data Engineer

• At least 8+ years of extensive experience in data warehousing using both RDBMS and non-RDBMS databases
• Minimum of 5 years of recent hands-on professional experience as a data engineer (backend software engineer also considered), actively involved in coding
• Professional experience in an agile, dynamic, and customer-facing environment is required
• Strong understanding of distributed systems and cloud technologies, with preference given to experience with AWS
• Preferably experienced in data streaming and scalable data processing
• Experience with large-scale datasets, data lake, and data warehouse technologies such as AWS Redshift, Google Big Query, and strongly preferred is Snowflake
• Minimum of 2+ years of experience in Amazon RDS, Amazon Kinesis, Amazon Lambda, Apache Airflows, Amazon Step Functions
• Demonstrated expertise in scripting languages like Python, UNIX shell, and Spark is required
• Understanding of RDBMS, data ingestion, data flows, and data integrations, among others
• Technical proficiency in data models, data mining, and segmentation techniques
• Experience with full SDLC lifecycle and Lean or Agile development methodologies
• Knowledge of CI/CD and GIT deployments
• Ability to work effectively in a diverse, multiple stakeholder environment
• Ability to effectively communicate complex technology solutions to various teams, including technical, business, and management teams

Requirements

• Work with stakeholders to understand needs for data structure, availability, scalability, and accessibility.
• Develop tools to improve data flows between internal/external systems and the data lake/warehouse.
• Build robust and reproducible data ingest pipelines to collect, clean, harmonize, merge, and consolidate data sources.
• Understanding existing data applications and infrastructure architecture
• Build and support new data feeds for various Data Management layers and Data Lakes
• Evaluate business needs and requirements
• Support migration of existing data transformation jobs in Oracle, and MS-SQL to Snowflake.
• Ability to write oracle sql, PLSQL scripts and should have demonstrated working experience in managing the oracle scripts.
• Should have a good hands on with linux scriptings.
• Able to document the processes and steps.
• Develop and maintain datasets.
• Improve data quality and efficiency.
• Lead Business requirements and deliver accordingly.