Databricks Architect

Senior Databricks Architect

Company: Jade Business Services (JBS)

Location: Houston, TX (Hybrid)

Employment Type: Full-Time

Experience Level: Senior / Principal (10+ Years)

Role Summary

Jade Business Services (JBS) is seeking an experienced, visionary Senior Databricks Architect / Data & AI Architect to spearhead enterprise-scale Data and AI transformations. In this role, you will define end-to-end Lakehouse architectures, set technical strategy, drive presales/RFP technical responses, and engage directly with client executive leadership (VP/C-suite).

The ideal candidate blends deep, hands-on architectural expertise across the modern Databricks ecosystem—including real-time streaming, Lakeflow pipelines, Unity Catalog governance, and agentic AI architectures—with proven experience in cloud platforms (Azure, GCP), data warehousing (BigQuery), and strategic consulting.

Key Responsibilities

Solution Architecture & Design

• Architect, design, and validate enterprise-grade Databricks Lakehouse, Real-Time Streaming, Data Engineering, and Generative/Agentic AI solutions.
• Design robust data pipelines leveraging Lakeflow Designer, Lakeflow Connect, and Lakeflow Declarative Pipelines.
• Establish unified enterprise governance, access controls, auditability, and data lineage across catalogs using Unity Catalog (RBAC/ABAC).
• Evaluate architectural trade-offs across performance, scalability, security posture, and cloud compute/storage costs (FinOps).
• Define interoperability strategies across Delta Lake and Apache Iceberg open table formats.

Presales, Consulting & Technical Leadership

• Lead the architecture stream for major RFP, RFI, and client proposal responses, delivering high-impact technical narratives and architectural diagrams.
• Serve as the trusted technical advisor in high-stakes discussions with client executives (CTOs, CDOs, VPs).
• Translate complex business problems into modular, cloud-agnostic, and future-proof Data & AI blueprints.
• Conduct architecture reviews, gap analyses, and proof-of-concept (PoC) implementations for strategic enterprise clients.

Required Technical Competencies

Core Databricks & AI Platform

• Pipeline Orchestration & Ingestion: In-depth hands-on experience with Databricks Lakeflow, Lakeflow Declarative Pipelines, Lakeflow Designer, and Lakeflow Connect.
• Storage & Interoperability: Delta Lake, Delta UniForm, and Apache Iceberg interoperability.
• Unified Governance: Advanced Unity Catalog implementation, fine-grained access control (RBAC/ABAC), dynamic masking, row-level filtering, and system tables/lineage.
• AI & Agentic Systems: Databricks Vector Search / AI Search, Unity AI Gateway, Databricks Genie / Genie Code, Managed Agent Memory, and Agent Bricks.
• Core Compute & Frameworks: PySpark, Apache Spark Structured Streaming, MLflow, Retrieval-Augmented Generation (RAG) frameworks, and agentic AI orchestration.

Cloud & Data Ecosystem

• Cloud Platforms: Strong architectural experience across Microsoft Azure (ADLS Gen2, Azure Synapse, AKS) and Google Cloud Platform (GCP) (GCS, Dataproc, BigQuery).
• Cross-Platform Analytics: Hands-on experience integrating Databricks with Google BigQuery, cross-cloud querying, and data virtualization.
• DevOps & CI/CD: Terraform for Databricks/Cloud provisioning, Databricks Asset Bundles (DABs), and automated CI/CD pipeline deployments.

Qualifications & Experience

• Experience: 10+ years in Data Engineering, Data Platform Architecture, Cloud Architecture, or Enterprise Solution Architecture.
• Databricks Expertise: 5+ years of specialized Databricks architecture experience, demonstrating mastery of the latest platform features and architecture patterns.
• Presales & Advisory: Proven track record in RFP/RFI authoring, presales solution design, client-facing consulting, and executive-level presentations.
• Education: Bachelor’s or Master’s degree in Computer Science, Information Systems, Software Engineering, or related field (or equivalent professional experience).
• Certifications (Preferred):
• Databricks Certified Solutions Architect (Professional)
• Databricks Certified Generative AI Engineer / Machine Learning Professional
• Microsoft Certified: Azure Solutions Architect Expert / GCP Professional Data Engineer