Azure Big Data Specialist

Overall, 6 to 10 years of experience in IT Industry.

Min 4 years of experience working on Data Engineering using Azure Databricks, Synapse, ADF.

At least 4 Project experience in Building and maintaining ETL / ELT pipelines for large data sets, complex data processing, transformations, business logics, cost monitoring & performance optimization, and feature engineering processes.

Must Have skills:

• Extensive experience with Azure Databricks (ADB), Azure Data Lake Storage (ADLS), Azure Data factory (ADF), Azure SQL Database (SQL DB), SQL, ELT / ETL Pipeline Development in Spark based environment.
• Experience in Real-Time Data Processing using Apache Kafka/Eventhub/IoT Hub, Structured Streaming and Stream analytics.
• Experience with SparkCore, Spark Streaming, PySpark, Python, SparkSQL, Scala, Azure Blob Storage.
• Experience with CI/CD tools like GitHub, Azure DevOps.

Good to have:

• Experience with Apache Airflow
• Experience with infrastructure management (e.g. Terraform)
• Experience with Version control tools, building CI/CD pipelines
• Experience with Azure cloud platform.
• Experience / Knowledge on Containerizations - Docker, Kubernetes
• Experience wo...