Senior Data Engineer – dbt, Databricks, AWS & Python

Senior Data Engineer – dbt | Databricks | AWS | Python

Experience: 5+ Years
Role: Senior Data Engineer

About the Role

We are looking for a skilled Senior Data Engineer with strong hands-on experience in dbt, Databricks, AWS, Python, and SQL to design, develop, and maintain scalable data engineering solutions.

The ideal candidate should have practical experience implementing Medallion Architecture (Bronze, Silver, Gold), building production-grade data pipelines, and improving existing data workflows by refactoring legacy code into reliable, scalable, and maintainable solutions.

Key Responsibilities

1. Data Engineering & Pipeline Development

Design and implement scalable data pipelines using Databricks, dbt, Python, and SQL.

Build and maintain Medallion Architecture with Bronze, Silver, and Gold data layers.

Develop dbt models, tests, documentation, and data transformation workflows.

Optimize data processing jobs for performance, reliability, and scalability.

2. Pipeline Optimization & Architecture

Review existing data pipelines and refactor legacy or poorly structured code.

Identify and resolve technical debt, performance bottlenecks, and data quality issues.

Improve pipeline reliability, scalability, maintainability, and cost efficiency.

Implement data validation, monitoring, logging, and error-handling mechanisms.

3. Python & SQL Development

Develop production-grade data engineering applications and automation using Python.

Build reusable Python libraries and utilities for data ingestion and processing.

Write complex SQL queries for data transformation and optimization.

Implement unit testing and data quality checks.

4. AWS Cloud Data Engineering

Work with AWS services such as S3, Glue, Lambda, Redshift, and CloudWatch.

Build and manage cloud-based data processing and storage solutions.

Ensure secure data access, monitoring, and operational reliability.

5. Collaboration & Ownership

Collaborate with data engineers, software engineers, and business stakeholders.

Take ownership of data engineering initiatives from design to production deployment.

Provide ongoing pipeline maintenance, troubleshooting, and performance improvements.

Mandatory Skills

5+ years of experience in Data Engineering or a related field.

Strong hands-on experience with dbt (models, tests, and documentation).

Production experience with Databricks.

Practical experience with AWS S3, Glue, Lambda, or Redshift.

Strong proficiency in Python and SQL.

Hands-on experience implementing Medallion Architecture (Bronze, Silver, Gold).

Experience building, optimizing, and maintaining production data pipelines.

Ability to understand and refactor legacy code.

Strong understanding of data quality, testing, monitoring, and pipeline reliability.

Preferred Skills

Experience with cloud-based data platforms.

Knowledge of CI/CD and data pipeline deployment practices.

Experience with Delta Lake and Spark.

Familiarity with data governance and security best practices.

Ideal Candidate Profile

We are looking for a hands-on Data Engineer who can independently design data pipelines, work with dbt and Databricks, develop Python-based solutions, and improve existing data architecture.

Important: Candidates must have practical experience with dbt, Databricks, AWS, Python, and Medallion Architecture. Knowledge of these tools without hands-on project experience will not be sufficient.