Technical Lead – Azure Databricks

Technical Lead – Azure Databricks

Location: Grand Rapids, MI (Hybrid)

Employment Type: Full-time

Experience Level: 8–15 Years

Department: Data & Analytics

About the Role

We are seeking an experienced and hands-on Technical Lead – Azure Databricks to architect and drive large-scale, enterprise data platform migrations. In this role, you will lead the technical strategy, design, and execution of migrating legacy data warehouses and processing systems (e.g., on-premises Hadoop, Teradata, Netezza, or legacy SQL) to a modern, scalable Lakehouse architecture powered by Azure Databricks.

You will act as the bridge between enterprise architecture, data engineering teams, and key stakeholders, ensuring robust delivery, performance optimization, and adherence to cloud security standards.

Key Responsibilities

• Migration Strategy & Execution: Lead end-to-end design, planning, and implementation of legacy platform migrations to Azure Databricks using the Medallion Architecture (Bronze/Silver/Gold).
• Architecture & Pipeline Engineering: Design highly scalable batch and streaming data pipelines using PySpark/Scala, Spark SQL, Delta Lake, and Delta Live Tables (DLT).
• Data Integration & Orchestration: Build robust orchestration workflows using Azure Data Factory (ADF), Databricks Workflows, and Event Hubs/Kafka.
• Performance Tuning: Optimize Databricks clusters, query performance, compute utilization, partition pruning, and Delta Lake optimizations (Z-Ordering, compaction, liquid clustering).
• Governance & Security: Implement Unity Catalog for fine-grained access control, data lineage, auditing, and data masking across Azure environments.
• Technical Leadership & Mentorship: Guide, mentor, and conduct code/architecture reviews for a team of 5–10 data engineers, establishing best practices in CI/CD, unit testing, and Databricks Asset Bundles (DABs).
• Stakeholder Management: Collaborate with project managers, solution architects, and business stakeholders to identify technical blockers, scope sprints, and deliver business value on schedule.

Required Technical Skills & Qualifications

• Experience: 8–15 years of hands-on data engineering experience, with at least 3+ years leading technical teams on enterprise Azure Databricks implementations.
• Core Databricks Expertise: Deep knowledge of Apache Spark internals, Delta Lake, Delta Live Tables, Unity Catalog, and cluster performance tuning.
• Azure Data Ecosystem: Strong expertise in Azure Data Factory, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Key Vault, Azure Synapse, and Azure DevOps (CI/CD pipelines).
• Migration Experience: Proven track record of delivering at least one end-to-end large-scale migration project (e.g., Hadoop/Cloudera, Teradata, Oracle, or SQL Server to Azure Databricks Lakehouse).
• Programming Languages: Proficiency in Python / PySpark and advanced SQL (experience with Scala is a strong plus).
• Data Modeling: Solid foundation in dimensional modeling (Star/Snowflake schema), modern Lakehouse modeling, and historical data reconciliation strategies.
• DevOps & Automation: Experience setting up CI/CD using Azure DevOps or GitHub Actions, Infrastructure as Code (Terraform/Bicep), and automated testing frameworks.

Preferred Qualifications & Certifications

Certifications:

• Databricks Certified Data Engineer Professional
• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Experience with automated schema migration, data reconciliation, and dual-run validation tooling.
• Familiarity with data observability, monitoring (Databricks System Tables, Azure Monitor/Log Analytics), and cost optimization frameworks (FinOps).