Jade Business Services
Actively Hiring
Insights to improve healthcare value chain
Databricks Engineer
- Raritan
- |8 years of exp
- |Contract
Posted: 4 days ago• Recruiter recently active
Job Location
Raritan
Remote Work Policy
In office
Visa Sponsorship
Not Available
RelocationAllowed
Skills
SQL
Git
Azure
Auditing
Data Governance
Data Modeling
AWS
Gold
Spark SQL
Pyspark
GCP
databricks
Silver
CI/CD
Delta Lake
Unity Catalog
Medallion Architecture
Access Controls
Databricks Workflows
Bronze
About the job
Job Summary
We are seeking a Senior Databricks Data Architect with strong expertise in Databricks, Unity Catalog, Medallion Architecture, Delta Lake, and cloud data platforms. The ideal candidate will be responsible for designing scalable, secure, and governed data platforms and implementing modern data architectures using Databricks.
Key Responsibilities
- Design and implement scalable data platforms using Databricks and Delta Lake.
- Develop and maintain Medallion Architecture (Bronze, Silver, Gold) for enterprise data pipelines.
- Design and implement Unity Catalog for centralized data governance, access control, data discovery, lineage, and auditing.
- Develop scalable batch and streaming data pipelines using PySpark, Spark SQL, and Databricks Workflows.
- Design data ingestion, transformation, and integration solutions across multiple data sources.
- Implement data quality, validation, monitoring, and reconciliation processes.
- Establish data governance, security, RBAC, cataloging, and compliance standards using Unity Catalog.
- Implement Delta Live Tables / Lakeflow Declarative Pipelines where applicable.
- Optimize Databricks workloads, Spark jobs, Delta tables, clusters, and SQL queries for performance and cost.
- Work with cloud storage and services such as Azure Data Lake Storage, AWS S3, or Google Cloud Storage.
- Design data models and curated datasets for analytics, reporting, and downstream applications.
- Implement CI/CD and infrastructure automation using tools such as Azure DevOps, GitHub, Terraform, or Databricks Asset Bundles.
- Collaborate with Data Engineers, Data Scientists, Analysts, Architects, and business stakeholders to define data requirements and architecture.
- Provide technical leadership, architectural guidance, and best practices for enterprise data engineering initiatives.
Required Skills
- 8+ years of experience in Data Engineering / Data Architecture.
- Strong hands-on experience with Databricks.
- Strong expertise in Unity Catalog and Databricks data governance.
- Strong understanding of Medallion Architecture – Bronze, Silver, and Gold layers.
- Strong experience with Delta Lake / Delta Tables.
- Advanced PySpark and Spark SQL skills.
- Experience designing and implementing enterprise-scale data pipelines.
- Strong SQL and data modeling experience.
- Experience with cloud data platforms such as Azure, AWS, or GCP.
- Experience with data security, access controls, governance, lineage, and auditing.
- Experience with Databricks Workflows and orchestration.
- Experience with Git and CI/CD practices.
Preferred Skills
- Experience with Azure Data Factory, AWS Glue, Airflow, or similar orchestration tools.
- Experience with Delta Live Tables / Lakeflow.
- Experience with Terraform / Infrastructure as Code.
- Experience with Databricks SQL and SQL Warehouses.
- Experience with structured streaming and real-time data processing.
- Knowledge of data mesh and enterprise data governance concepts.
- Experience with Kafka or other streaming technologies.
- Experience with Power BI, Tableau, or other analytics platforms.
Education
Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related field, or equivalent professional experience.
