Databricks Developer – Medicare

Databricks Developer – Medicare

• Advantage (Payer)
• Work Model: Remote (U.S.-based candidates only)
• Level: Senior
• Domain: Healthcare Payer / Medicare Advantage

Overview

• Our client, a leading Medicare Advantage health plan, is seeking an experienced Databricks Developer to build and
• operate data solutions on the Databricks lakehouse platform, supporting claims, eligibility, and member/provider
• data used across Medicare Advantage reporting and compliance. This is a Databricks platform-focused role: the
• ideal candidate lives inside Databricks day to day — notebooks, Delta Live Tables, Workflows, Unity Catalog, and
• cluster/job tuning — rather than a general data engineer who happens to use Databricks as one of several tools.
• Candidates should be comfortable working independently in a remote setup and available to start promptly.

Key Responsibilities

• Build, deploy, and maintain data pipelines natively within Databricks, using notebooks, Delta Live Tables
• (DLT), and Databricks Workflows/Jobs for orchestration.
• Develop and optimize Spark (PySpark/Scala/SQL) logic for processing large-scale Medicare Advantage
• claims (EDI 837/835), eligibility, and provider data within the Databricks environment.
• Manage and evolve the Delta Lake medallion architecture (bronze/silver/gold) to ensure clean, reliable,
• well-governed data.
• Configure and maintain Unity Catalog for data governance, lineage, and access control across PHI/HIPAA-
• regulated datasets.
• Tune cluster configurations, job performance, and cost efficiency (autoscaling, cluster policies, Photon,
• job/task-level optimization) within Databricks.
• Use Databricks SQL and SQL warehouses to support ad hoc analysis, validation, and reconciliation of
• Medicare Advantage claims and eligibility data.
• Monitor pipeline health, troubleshoot job failures, and resolve performance issues directly within the
• Databricks platform (Jobs UI, cluster logs, DLT event logs).
• Manage code and deployments via Databricks Repos and CI/CD integration (Databricks Asset Bundles, Azure DevOps/Git).
• Apply PHI/HIPAA-compliant data handling and role-based access practices throughout all Databricks workspace and pipeline configurations.
• Document Databricks pipeline design, workspace structure, and job configurations.

Required Experience & Skills

• 5+ years of hands-on Databricks experience — this is a Databricks-first role; deep platform fluency is the primary requirement.
• Strong command of Spark (PySpark/Scala) and Databricks SQL for pipeline development.
• Hands-on experience building and managing Delta Live Tables (DLT) pipelines.
• Practical experience with Unity Catalog for data governance, access control, and lineage.
• Experience with Databricks Workflows/Jobs for orchestration and scheduling.
• Demonstrated ability to tune cluster and job performance (autoscaling, cluster policies, Photon, cost optimization) within Databricks.
• Experience with Databricks Repos and Git-based CI/CD for notebook/pipeline deployment.
• 3–10+ years of healthcare payer domain experience, ideally with Medicare Advantage data (claims ingestion, eligibility logic, EDI 837/835, CMS-aligned reporting).
• Working knowledge of PHI/HIPAA-regulated data handling and role-based security within a data platform.
• Comfortable working within an Azure or equivalent cloud environment (ADLS, workspace/networking basics) as the underlying infrastructure for Databricks.
• Strong problem-solving skills and ability to work independently in a remote environment.
• U.S.-based, remote-capable (U.S. resident, citizen, or Green Card holder).

Nice to Have

• Databricks Certified Data Engineer (Associate or Professional).
• Experience with MLflow or Databricks' ML/AI tooling.
• Exposure to Azure Data Factory, Power BI, or Snowflake in a supporting/downstream capacity.
• Experience migrating legacy ETL workloads into Databricks.
• Prior experience in a client-facing or consulting environment.
• Experience with EHR-adjacent systems (e.g., Epic Clarity) or provider-side data in addition to payer data.

Ideal Candidate Profile

"Senior Databricks Developer who lives in the platform day to day — Delta Live Tables, Unity Catalog, Workflows,

and Spark performance tuning — with real Medicare Advantage claims and eligibility experience, and a track

record of running governed, production-grade lakehouse pipelines end to end within Databricks.