Lead Data Engineer

Company Name: Celebal Technologies

Job Type: Full-Time (Permanent)

Experience: 10+ Years

Work Authorization: We are looking for candidates who are Green Card holders or US Citizens (USC) only.

Work Mode: Remote (Must be open to travel to client locations based on project requirements)

About Celebal Technologies

Celebal Technologies is a leading Data and AI company specializing in Databricks, cloud technologies, data engineering, and enterprise-scale digital transformation. We help organizations modernize their data platforms and accelerate business outcomes through innovative data and AI solutions.

Role Overview

We are looking for an experienced Lead Data Engineer with 10+ years of overall experience and strong expertise in Databricks, cloud-based data engineering, and modern data architecture.

The ideal candidate should have hands-on experience designing and implementing scalable data pipelines using Databricks, Medallion Architecture, Delta Live Tables (DLT), and Auto Loader. The candidate will lead data engineering initiatives, drive technical architecture decisions, and collaborate with cross-functional teams to deliver enterprise-grade data solutions on Azure or AWS.

Key Responsibilities

• Lead the design, development, and implementation of scalable and high-performance data engineering solutions using Databricks.
• Design and implement end-to-end data pipelines based on Medallion Architecture (Bronze, Silver, and Gold layers).
• Develop and optimize data ingestion and transformation pipelines using Delta Live Tables (DLT) and Auto Loader.
• Build and manage data lakehouse solutions using Delta Lake and Apache Spark.
• Design, develop, and optimize ETL/ELT workflows using PySpark, Python, and SQL.
• Work with Azure or AWS cloud services to build and manage scalable data platforms.
• Implement data quality, validation, governance, and monitoring frameworks.
• Optimize Spark workloads, cluster configurations, and query performance.
• Establish best practices for data engineering, coding standards, CI/CD, and deployment.
• Lead technical discussions, mentor data engineers, and provide technical guidance to the team.
• Collaborate with architects, business stakeholders, and other technical teams to translate business requirements into technical solutions.

Required Technical Skills

• 10+ years of overall experience in Data Engineering, Big Data, and Data Platform development.
• Strong hands-on experience with Databricks and its Lakehouse Platform.
• Extensive experience implementing Medallion Architecture (Bronze, Silver, Gold).
• Strong experience with Delta Live Tables (DLT).
• Hands-on experience with Databricks Auto Loader for incremental and continuous data ingestion.
• Strong programming experience in PySpark, Python, and SQL.
• Experience with Delta Lake, Apache Spark, and distributed data processing.
• Strong cloud experience in either Microsoft Azure or AWS.
• Experience designing and implementing scalable ETL/ELT pipelines.
• Good understanding of data modeling, data governance, security, and data quality frameworks.

Preferred Qualifications

• Databricks certifications are an added advantage.
• Experience with Unity Catalog and data governance.
• Knowledge of CI/CD, Git, and DevOps practices.
• Experience with Infrastructure as Code (Terraform).
• Experience leading teams and managing end-to-end data engineering projects.
• Strong communication, stakeholder management, and problem-solving skills.

What We Are Looking For

• A strong technical leader with excellent hands-on data engineering capabilities.
• Ability to independently drive complex data engineering initiatives.
• Experience designing scalable, reliable, and cost-effective data solutions.
• Willingness to travel to client locations based on project requirements.