Data Engineering Manager - Streaming Specialist

We are seeking a hands-on Data Engineering Manager to lead a small team in developing and maintaining advanced streaming data solutions, while also bridging on-premises and cloud environments. This role combines deep technical expertise in streaming technologies with team leadership, focusing on building robust, scalable data pipelines and real-time analytics systems. Work closely with cross-functional teams, predominantly architecture as well as product and software engineering teams, to implement high-quality data solutions aimed at achieving the company's business goals.

Key Responsibilities

• Manage and mentor a small team of data engineers specializing in streaming technologies
• Design and implement high-throughput, low-latency streaming data pipelines using Kafka
• Develop and optimize stream processing applications using Apache Spark or Apache Flink
• Implement real-time data transformations and analytics using KSQL
• Design and maintain scalable data storage solutions with ClickHouse for fast analytics on streaming data
• Bridge on-premises streaming solutions with cloud services like BigQuery and Looker
• Oversee the implementation of reporting solutions using Superset
• Ensure best practices in data engineering, including data quality, security, and performance
• Collaborate with stakeholders to define streaming data requirements and deliverables
• Stay current with emerging streaming and cloud technologies and evaluate their potential application

Requirements

• 5+ years of experience in data engineering, with at least 2 years focusing on streaming technologies
• At least 2 years in a leadership or management role.
• Strong hands-on experience with Apache Kafka, including cluster management and tuning
• Proficiency in developing stream processing applications using either Apache Spark Structured Streaming or Apache Flink
• Experience with KSQL for stream processing and real-time analytics
• Solid understanding of ClickHouse for high-performance analytics on streaming data
• Familiarity with Superset for data visualization and exploration
• Excellent programming skills in Java, Scala, or Python, with a focus on stream processing
• Experience leading and mentoring a team of data engineers
• Experience with Kafka Connect for data integration and Kafka Streams for stateful processing
• Familiarity with stream processing patterns like windowing, watermarking, and exactly-once processing
• Knowledge of data modeling and optimization techniques for ClickHouse and BigQuery
• Experience integrating on-premises streaming solutions with cloud-based analytics platforms
• Proficiency in developing and managing data pipelines that span on-premises and cloud environments
• Experience with cloud services, particularly BigQuery and Looker
• Experience with real-time machine learning model serving in streaming environments
• Experience with monitoring and observability tools for streaming systems across on-premises and cloud environments

Benefits

• Bi-Monthly company wide social and team building activities
• Lunch Vouchers
• Health Premium
• Training and Development opportunities
• Hybrid & Remote work arrangements
• Flexible working hours

At Logifuture we embrace equality and diversity. We do not discriminate and our values of delivering results together support this statement. We are determined to foster an inclusive environment where everyone feels valued, heard, and empowered to contribute their unique skills and experiences.