Avatar for Celebal Technologies
Creating Intelligent Enterprises
  • B2B
  • Scale Stage
    Rapidly increasing operations
  • Recently funded
    Raised funding in the past six months

Lead Data Engineer - Databricks

Posted: 1 week ago
Job Location
Remote Work Policy

In office - WFH flexibility

Visa Sponsorship

Not Available

RelocationAllowed
Skills
Python
SQL
Security
ETL
Data Governance
Apache Spark
Elt
Pyspark
databricks
Azure Databricks
Delta Lake
AWS Databricks
Medallion Architecture
Lakehouse
Data Pipeline Optimization
Scalable Architecture Design
Databricks Auto Loader
Production-Grade Data Engineering

About the job

Job Title: Lead Data Engineer

Location: Grand Rapid, MI

Work Mode: Hybrid

Job Type: Contract / Full-Time

Certification: Databricks Certification preferred

Job Summary

We are seeking an experienced Lead Data Engineer to design and implement scalable, secure, and high-performance data lakehouse solutions. The ideal candidate should have strong expertise in Databricks, Medallion Architecture, Azure/AWS cloud platforms, SQL, PySpark, Python, and Databricks Auto Loader.

Key Responsibilities

  • Design and implement enterprise-scale data architectures using Databricks.
  • Develop scalable data pipelines using SQL, PySpark, and Python.
  • Implement Medallion Architecture, including Bronze, Silver, and Gold layers.
  • Build incremental and streaming data ingestion pipelines using Databricks Auto Loader.
  • Integrate Databricks with Azure or AWS cloud services.
  • Optimize Spark workloads, SQL queries, cluster configurations, and data processing performance.
  • Implement Delta Lake solutions for reliable data storage and processing.
  • Establish data governance, security, and access control best practices.
  • Provide technical leadership and architectural guidance to data engineering teams.
  • Support production deployments, troubleshooting, and performance optimization.

Required Skills and Qualifications

  • Strong experience with Databricks architecture and Lakehouse solutions.
  • Hands-on expertise in Medallion Architecture.
  • Proficiency in SQL, PySpark, and Python.
  • Practical experience with Databricks Auto Loader.
  • Experience with Azure Databricks or AWS Databricks.
  • Strong understanding of Apache Spark, Delta Lake, and ETL/ELT pipelines.
  • Experience in data pipeline optimization and production-grade data engineering.
  • Knowledge of data governance, security, and scalable architecture design.
  • Strong communication, analytical, and problem-solving skills.

Preferred Qualifications

  • Databricks certification preferred.
  • Experience with Unity Catalog and data governance.
  • Knowledge of Delta Live Tables (DLT) or Lakeflow Declarative Pipelines.
  • Experience with CI/CD, Git, and infrastructure-as-code tools.
  • Experience with cloud migration and enterprise data platform modernization.
  • Previous experience leading technical teams or defining data architecture standards.

Ideal Candidate

The ideal candidate will be a hands-on technical person who can design end-to-end Databricks solutions, guide engineering teams, and deliver scalable data platforms using modern cloud and lakehouse technologies.

Similar Jobs

Archesys company logo
Archesys
Improving the government services that impact everyday lives
Archesys company logo
Archesys
Improving the government services that impact everyday lives