Avatar for Soroc Technology
IT solutions and services for infrastructure, staffing, and managed services

Lead Data Engineer

Posted: 1 week ago
Job Location
Remote Work Policy

In office - WFH flexibility

Visa Sponsorship

Not Available

RelocationAllowed
Skills
Python
Machine Learning
SQL
Data Integration
Data Architecture
ETL
Oracle SQL
Supply Chain Analytics
Data Modeling
Data Transformation
Jobs
Pyspark
Notebooks
databricks
Unity Catalog
SQL Warehouses
Lakeflow
Orchestration Patterns
ELT Pipeline Development
Genie Code
LakeFlow Designer

About the job

Role: Databricks Lead (Lead Data Engineer)

Experience Required: 6-8 Years

Location: Edmonton / Calgary / Toronto (Any of these location)

Work Model: Hybrid (3 days per week in office: Monday, Tuesday, Thursday)

Contract Duration: 12 months

Role Summary

Seeking a Databricks Lead with strong expertise in Databricks to design, build, and operationalize scalable Supply Chain data products that deliver measurable business value. This role combines advanced analytics, data pipeline development, and product delivery to transform complex Supply Chain Management (SCM) data into trusted, governed, and actionable insights.

Key Responsibilities

Develop analytics solutions that address Supply Chain business challenges, leveraging knowledge of oil & gas operations, procurement, supplier performance, spend analytics, and enterprise value drivers.

Design, build, test, and optimize scalable ETL/ELT pipelines in Databricks using Python, PySpark, SQL, notebooks, jobs, and orchestration patterns to support reliable Silver and Gold data products.

Transform and model data from ERP and enterprise platforms, including Oracle Cloud, SAP, ADW, Microsoft Dataflow, Dataverse, and other approved data sources.

Build and evaluate machine learning, statistical, and AI-assisted solutions, including:

  • Classification

  • Forecasting

  • Anomaly Detection

  • Text Analytics

  • Natural Language Processing (NLP)

Utilize Databricks platform capabilities such as:

  • Notebooks

  • Jobs

  • Workflows

  • Unity Catalog

  • SQL Warehouses

  • Genie Code

  • Lakeflow / Lakeflow Designer

Maintain project documentation, including idea assessments, requirements, design documents, and weekly status reports.

Communicate technical concepts, requirements, solution designs, and product details effectively to business stakeholders in clear, non-technical language.

Required Skills:

Databricks

Python

PySpark

SQL / Oracle SQL

Data Architecture & Data Modeling

ETL / ELT Pipeline Development

Machine Learning

Supply Chain Analytics

Data Integration and Data Transformation

Preferred Domain Experience:

Supply Chain Management (SCM)

Oil & Gas Industry

Procurement Analytics

Supplier Performance Analytics

Spend Analytics