Insights to improve healthcare value chain

Databricks Architect

Houston
|10 years of exp
|Full Time

Posted: 3 days ago

Job Location

Houston

Remote Work Policy

In office

Visa Sponsorship

Not Available

RelocationAllowed

Skills

Python

SQL

Scim

Pyspark

Azure Data Factory

CI/CD

MLFlow

Azure Databricks

Azure Key Vault

Delta Lake

Apache Iceberg

Vector Search

Unity Catalog

Microsoft Entra ID

DATABRICKS PLATFORM

Azure Data Lake Storage Gen2

Private Link

Databricks Asset Bundles

Lakeflow Connect

Lakeflow Declarative Pipelines

Apache Spark Structured Streaming

Databricks Genie

VNet Injection

Genie Code

Agent Bricks

Unity AI Gateway

LakeFlow Designer

Managed Agent Memory

About the job

Senior Azure Databricks Architect/ Data & AI Architect

Company: Jade Business Services (JBS)

Job Summary

We are seeking a Senior Azure Databricks Architect / Data & AI Architect to lead enterprise Data and AI platform modernization. You will design scalable Lakehouse architectures, engineer real-time pipelines, establish data governance, and deploy modern AI solutions natively on Microsoft Azure.

Key Responsibilities

Lakehouse Architecture: Design scalable Azure Databricks Lakehouse, streaming, and data engineering architectures using Delta Lake and Apache Iceberg interoperability.
Pipeline Modernization: Build declarative data pipelines utilizing Lakeflow Designer, Lakeflow Connect, and Lakeflow Declarative Pipelines.
Governance & Security: Implement enterprise governance, automated lineage, and RBAC/ABAC access controls via Unity Catalog across Azure storage.
AI & Analytics: Architect GenAI, RAG, and agentic workflows using Unity AI Gateway, Vector Search, Databricks Genie / Genie Code, and Agent Bricks.
Architecture Governance: Lead technical design reviews, optimize Azure compute/storage costs (FinOps), and establish engineering standards.

Required Technical Skills

Databricks Platform: Lakeflow (Connect / Designer / Declarative Pipelines), Unity Catalog, Delta Lake / Iceberg, Genie Code, Agent Bricks, Managed Agent Memory, Unity AI Gateway, Vector Search, MLflow.
Azure Cloud: Azure Databricks, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory (ADF), Microsoft Entra ID (SCIM), Azure Key Vault, VNet injection / Private Link.
Engineering & Streaming: PySpark, Apache Spark Structured Streaming, Python, SQL, Databricks Asset Bundles (DABs), CI/CD.

Qualifications

10+ years in Data Engineering, Cloud Architecture, or Enterprise Solution Architecture.
5+ years hands-on Databricks architecture experience, including modern platform features.
Strong production experience building and securing platforms natively on Microsoft Azure.
Relevant certifications preferred (Databricks Certified Solutions Architect / Data Engineer Professional, Azure Solutions Architect Expert).