Senior Data Engineer – Databricks

Senior Data Engineer – Adobe Experience Platform (AEP) & DatabricksLocation: 100% RemoteJob Type: ContractJob SummaryWe are seeking an experienced Senior Data Engineer with strong expertise in Adobe Experience Platform (AEP), Databricks, enterprise data modeling, and marketing data architecture. The ideal candidate will lead the design, development, and governance of scalable data ingestion and transformation pipelines that support customer data platform (CDP), marketing, sales, product-led growth (PLG), and customer retention use cases.This role requires hands-on technical expertise combined with architecture leadership, data quality management, stakeholder collaboration, and oversight of implementation partners. The candidate will establish reusable data standards, improve pipeline reliability, and ensure high-quality, governed data is available for downstream segmentation, analytics, and activation.Key ResponsibilitiesOwn end-to-end architecture and technical delivery for Adobe Experience Platform implementations, from data modeling and ingestion through downstream activation.Design, develop, and maintain Databricks-based data ingestion and transformation pipelines for marketing and enterprise data assets.Define and maintain enterprise data models for AEP, including XDM schema design, identity resolution strategies, dataset structures, and activation logic.Develop production-ready curated datasets within the enterprise data lake to support marketing, sales, customer retention, and PLG use cases.Establish enterprise standards for schema design, data integration, transformation logic, and reusable data architecture patterns.Lead data quality initiatives by implementing automated validation, monitoring, reconciliation, and remediation frameworks.Identify and resolve recurring data quality issues, identity mismatches, schema inconsistencies, and pipeline failures.Monitor pipeline performance, freshness, reliability, and service-level agreements (SLAs).Collaborate with business stakeholders, marketing teams, product owners, and engineering teams to translate business requirements into technical solutions.Lead technical discussions, architecture reviews, sprint planning, and implementation decisions across cross-functional teams.Govern implementation partner delivery, including reviewing technical designs, validating deliverables, and enforcing enterprise architecture standards.Create and maintain comprehensive technical documentation covering data lineage, transformation logic, schema standards, integrations, and governance policies.Lead knowledge transfer and technical enablement for internal engineering teams and implementation partners.Ensure data governance, privacy, security, and compliance requirements are incorporated throughout the data lifecycle.Required Skills and QualificationsStrong experience in data engineering, enterprise data architecture, and large-scale data integration.Hands-on experience with Adobe Experience Platform (AEP), including XDM schemas, datasets, segmentation, and data activation.Strong experience developing data pipelines using Databricks.Proficiency in SQL and Python for data transformation, analysis, validation, and pipeline development.Experience designing enterprise data models, data ingestion frameworks, and transformation workflows.Experience integrating marketing and customer data from platforms such as Marketo, web analytics, CRM, and other enterprise systems.Knowledge of Customer Data Platforms (CDPs), customer identity resolution, and B2B marketing data architecture.Experience implementing automated data quality checks, pipeline monitoring, and data reconciliation processes.Understanding of data governance, metadata management, data lineage, and enterprise data standards.Experience translating business requirements into scalable technical architecture and implementation plans.Strong communication, technical leadership, documentation, and stakeholder management skills.Preferred QualificationsExperience with Adobe Customer Journey Analytics (CJA) and Adobe Journey Optimizer (AJO).Experience with Marketo and enterprise CRM platforms.Familiarity with Azure cloud services and enterprise data lake architectures.Experience managing external implementation partners such as Infosys, Accenture, or other systems integrators.Experience supporting B2B marketing, demand generation, customer retention, and sales activation initiatives.