Senior Data Engineer

We're Hiring: Senior Data Engineer – Dynamics 365 Analytics (Databricks & Microsoft Fabric)

About the Role

Our client is building a modern analytics platform that turns our Dynamics 365 Finance & Operations data into trusted insights for the business. We're looking for a Senior Data Engineer to lead the build on Databricks and Microsoft Fabric, using AI-assisted tools like Copilot to deliver faster and more efficiently.

Key Responsibilities

• Build and maintain ingestion pipelines that bring D365 F&O and other source data into the lakehouse in near real time

• Develop incremental load and change data capture (CDC) logic, including MERGE/upsert patterns and handling of deleted and late-arriving records

• Orchestrate and schedule pipelines using Databricks Workflows and Fabric Data Factory, with dependency management, retries, and alerting

• Optimize Delta Lake tables and Spark jobs for performance and cost (partitioning, Z-ordering, file compaction, cluster sizing)

• Build automated data quality checks and reconciliation against source system totals

• Publish curated tables to Microsoft Fabric and support BI developers building Power BI semantic models

• Apply CI/CD practices: version control, code reviews, automated testing, and deployments across dev, test, and production

• Monitor pipeline health, troubleshoot failures, and own production support for the data platform

• Use GitHub Copilot, Databricks Assistant, and Copilot in Fabric to accelerate coding, testing, and documentation, while reviewing all AI-generated code

• Document pipelines, data lineage, and table definitions for the wider team

Required Skills

• 5+ years of experience in data engineering, including 2+ years on Azure data platforms

• Azure Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Databricks Workflows, Delta Live Tables, Databricks SQL Warehouse

• Microsoft Fabric: Lakehouse, Data Warehouse, OneLake (shortcuts and mirroring), Data Factory pipelines, Dataflows Gen2, Fabric notebooks

• Power BI: semantic modeling, DAX, Direct Lake and Import modes, row-level security, Power BI Service administration

• Dynamics 365 F&O data integration: Azure Synapse Link for Dataverse, Fabric Link, Dataverse, F&O data entities and table structures

• Azure services: ADLS Gen2, Azure Data Factory, Azure Key Vault, Microsoft Entra ID (managed identities, service principals), Azure Monitor

• AI and Copilot: GitHub Copilot, Copilot in Fabric and Power BI, Databricks Assistant, Fabric data agents; Azure OpenAI is a plus

• Languages: advanced SQL (T-SQL / Spark SQL), Python

• Data modeling: dimensional modeling (Kimball star schema), medallion architecture (bronze/silver/gold)

• DevOps: Git, Azure DevOps or GitHub Actions, Fabric and Databricks Git integration, Databricks Asset Bundles, Terraform or Bicep

• Data quality and governance: data validation frameworks, Microsoft Purview, lineage and access control

Nice to Have

• Microsoft (DP-600 / DP-700) or Databricks certifications

• Exposure to machine learning or generative AI solutions

• Finance or supply chain domain knowledge