Lead Data Engineer

Key roles and responsibilities:
• Design and development of scalable data pipelines and ETL processes to process and
transform large volumes of data efficiently.
• Define and implement data integration solutions to consolidate data from multiple
sources into a unified view, ensuring data consistency and accuracy.
• Optimize data delivery and performance by implementing caching mechanisms,
indexing strategies, and other advanced techniques.
• Collaborate with Data Analysts, Data Architects, and other stakeholders to understand
data requirements and design data solutions aligned with business needs.
• Skilled in high data quality standards by implementing data validation and monitoring
procedures.
• Continuously research and stay up to date with emerging data engineering technologies
and industry best practices.
• Participate in code reviews, mentor, and guide junior data engineers, and contribute to
the development of data engineering standards and best practices.
• Work closely with the product development teams to seamlessly integrate data
engineering solutions into our products and services.

Qualifications:
• Bachelor’s or master’s degree in computer science, Information Technology, or a related
field.
• 4-7+ years of hands-on work experience and a proven track record in data engineering,
data architecture, or a related role.
• Proficiency in data modelling, data warehousing concepts, ETL processes and data processing techniques to transform and cleanse raw data.
• Programming skills in Python, SQL, Scala, Java, or any other programming languages
used in data engineering.
• Extensive hands-on experience with data storage technologies such as Google Data
Fusion, and Apache Data Flow, SQL databases (e.g., MySQL, PostgreSQL) and NoSQL
databases (e.g. Cassandra, MongoDB etc.)
• Hands-on experience with cloud-based data solutions, such as AWS, Azure, or Google
Cloud Platform for deploying and managing big data infrastructure. Internal
• Possessing a working knowledge of data store and visualization tools such as Google
Looker, Power BI, Tableau is an advantage.
• Proven expertise in Data Management/Architecture Practices, including Data
Integration, Data Governance, Data Quality, Metadata Management, Data Security, and
Data Encryption.
• Excellent problem-solving and analytical skills with strong communication and
collaboration abilities.
• Demonstrated ability to lead and mentor a team of data engineers and drive projects to
successful completion.

Technical skills:
• Big Data Technologies: Proficiency in working with big data technologies like Hive,
Hadoop, Apache Spark, Apache Flink, Apache Kafka, Apache HBase, Apache Air flow,
Flink, Pig.
• Data Ingestion: Familiarity with tools and frameworks used to ingest data from different
sources, such as Apache Nifi, Apache Sqoop, Apache Flume, or custom ingestion
solutions.
• Data Modelling: Understanding of data modelling techniques to design efficient and
scalable data structures.
• Distributed Computing: Knowledge of distributed computing concepts and frameworks
to handle large-scale data processing. Elastic Search, Cassandra, MapReduce, Dataproc,
DataBricks etc.
• Data Security: Understanding of data security principles and methods to ensure data
protection and compliance.
• Data Quality Management: Expertise in data quality assessment, validation, and
cleansing techniques to maintain high-quality data.