Lead Data Platform Engineer (Databricks & Data Governance)

Job Title: Lead Data Platform Engineer (Databricks & Data Governance)
Location: United States (Remote / Hybrid — adjust as needed)
Employment Type: Full-time

About the Role
We are looking for an experienced Lead Data Platform Engineer to spearhead the design, implementation, and rollout of our Test Data Management (TDM) and Data Governance frameworks on Databricks. In this role, you will build scalable pipelines for synthetic data generation and data masking, establish enterprise-wide security controls, and enable self-service access to compliant datasets across engineering and QA teams.
Key Responsibilities
• TDM Framework Rollout: Lead the adoption and implementation of Test Data Management (TDM) frameworks across multiple projects on Databricks.
• Data Governance & Security: Establish strict governance controls for sensitive data in non-production environments, implementing Role-Based Access Control (RBAC), metadata management, data classification, and lineage.
• Pipeline Engineering: Design and build scalable, automated pipelines for ingestion, transformation, test data generation, data masking, and environment refreshes.
• Synthetic Data & Self-Service: Architect synthetic data strategies for lower environments and develop reusable data provisioning frameworks that allow teams to access compliant test datasets via self-service.
• Cross-Functional Collaboration: Partner closely with QA, application development, data governance, and information security teams to drive platform enablement and best practices.

Must-Have Qualifications
• Experience: 8+ years of progressive experience in Data Engineering, Data Architecture, or Data Platform roles.
• Databricks Ecosystem: 3+ years of hands-on experience with Databricks (Lakehouse, Delta Lake, Unity Catalog, and PySpark).
• Architecture & Governance: Proven experience implementing Medallion Architecture (Bronze/Silver/Gold) and rolling out Unity Catalog or enterprise data governance solutions.
• Programming: Strong software engineering skills in Python and PySpark.
• Cloud Platforms: Hands-on experience with Azure (ADF, Azure DevOps, Synapse) and/or AWS (S3, Lambda, Glue, Redshift, MWAA).

Preferred Qualifications (Highly Desired)
• Hands-on experience with Test Data Management (TDM) techniques, including data masking, de-identification, and synthetic data generation.
• Familiarity with AI-powered data discovery/classification and Generative AI tools (such as Databricks Mosaic AI).
• Experience implementing data observability and monitoring frameworks.
• Track record of rolling out cross-team frameworks and driving technical enablement.

Nice to Have
• Proficiency in CI/CD pipelines and Infrastructure as Code (Terraform, Azure DevOps, GitHub Actions).
• Working knowledge of regulatory compliance standards regarding sensitive data (PII, GDPR, CCPA, HIPAA).

What We Offer
• Engagement Model: Independent Contractor (1099 / Corp-to-Corp).
• Contract Duration: 1-year initial contract with strong potential for extension based on performance and project needs.
• Competitive Rate: Up to $60 USD/hour, determined by technical assessment and level of experience.
• High-Impact Environment: Opportunity to architect enterprise-level data governance and AI-ready frameworks from the ground up.