Databricks Architect

Job Title: Databricks Architect

Location: United States

Work Mode: Remote

Job Type: Full-Time

Certification: Databricks Certification preferred

Job Summary

We are looking for an experienced Databricks Architect with strong expertise in the Databricks ecosystem, Medallion Architecture, cloud platforms (Azure/AWS), SQL, PySpark, and Python. The ideal candidate will be responsible for designing and implementing scalable, secure, and high-performance data architectures using Databricks, with hands-on experience in Auto Loader and modern data engineering practices.

Key Responsibilities

• Design and implement scalable data lakehouse architectures using Databricks.
• Develop and maintain data pipelines using PySpark, Python, and SQL.
• Design and implement Medallion Architecture (Bronze, Silver, and Gold layers) for data ingestion, transformation, and consumption.
• Implement incremental and near-real-time data ingestion pipelines using Databricks Auto Loader.
• Integrate Databricks with Azure or AWS cloud services and data sources.
• Optimize Spark workloads, SQL queries, cluster configurations, and data processing performance.
• Implement Delta Lake solutions for reliable data storage, processing, and management.
• Establish data governance, security, access controls, and best practices for enterprise data platforms.
• Collaborate with data engineers, solution architects, and business stakeholders to translate requirements into technical solutions.
• Provide technical leadership, architectural guidance, and support for production deployments.

Required Skills and Qualifications

• Strong hands-on experience with Databricks architecture and the Lakehouse Platform.
• Extensive experience designing and implementing Medallion Architecture.
• Strong expertise in SQL, PySpark, and Python.
• Hands-on experience with Databricks Auto Loader for scalable data ingestion.
• Experience with either Microsoft Azure or AWS cloud services and their integration with Databricks.
• Strong understanding of Apache Spark, Delta Lake, data modeling, and ETL/ELT pipelines.
• Experience in performance tuning, troubleshooting, and optimizing large-scale data workloads.
• Understanding of data governance, security, and production data pipeline best practices.
• Ability to design scalable, reliable, and cost-effective data solutions.

Preferred Qualifications

• Databricks Certified professional certification preferred.
• Experience with Unity Catalog and data governance.
• Knowledge of Delta Live Tables (DLT) / Lakeflow Declarative Pipelines.
• Experience with CI/CD, Git, and infrastructure-as-code practices.
• Experience with enterprise-scale data platform modernization and cloud migration.
• Strong communication, problem-solving, and technical leadership skills.

Ideal Candidate Profile

The ideal candidate should combine strong architectural expertise with hands-on implementation skills, particularly in Databricks, Medallion Architecture, Auto Loader, SQL, PySpark, Python, and Azure/AWS cloud environments.