Senior Data Engineer (DBT+AWS)

About the Role
We are seeking a highly skilled Senior Data Engineer with strong hands-on experience in Google Cloud Platform (GCP) or AWS, cloud infrastructure, data ingestion, and Python. The ideal candidate is also strong in dbt, Databricks, and AWS, and has experience implementing Medallion architecture. The ideal candidate will have a proven track record of designing and implementing scalable data platforms, organizing GCP or AWS environments to meet data sovereignty requirements, and building secure and reliable data ingestion and transfer solutions. They should be comfortable working with existing, imperfect codebases and improving them.

Key Responsibilities

1. GCP and AWS Architecture & Infrastructure

• Design, develop, and maintain scalable GCP infrastructure for enterprise data workloads.
• Architect and organize GCP environments to support data sovereignty and data residency requirements.
• Design appropriate GCP projects, environments, networking, storage, IAM, and security configurations for data platforms.
• Establish scalable and reliable cloud infrastructure patterns for data processing and storage.
• Work across GCP and AWS environments, applying consistent infrastructure and security patterns.

2. Data Ingestion & Pipeline Engineering

• Design and implement robust data ingestion pipelines for structured and unstructured data sources.
• Build production-grade ingestion frameworks to reliably process and move data into GCP or AWS, or similar storage.
• Develop scalable and fault-tolerant data pipelines with appropriate validation, monitoring, and error handling.
• Optimize data ingestion workflows for performance, reliability, and cost.
• Ensure data pipelines maintain data quality, integrity, and traceability throughout the ingestion process.
• Design and implement Medallion architecture (Bronze, Silver, Gold layers) to deliver clean, reliable, analytics-ready data.
• Build and maintain transformation workflows using dbt, including modeling, testing, and documentation.
• Develop and optimize data processing workloads on Databricks.

3. Pipeline Architecture Review & Refinement

• Assess existing data pipelines, including messy or poorly documented legacy code, and refactor them into clean, maintainable, well-tested solutions.
• Refine and improve pipeline architecture to increase scalability, reliability, performance, and cost efficiency.
• Identify technical debt and drive improvements without disrupting production workloads.

4. Python Development & Automation

• Develop production-grade data engineering applications and automation using Python.
• Build reusable libraries, services, and utilities to support data ingestion and processing workflows.
• Automate infrastructure and data engineering processes to improve operational efficiency.
• Implement appropriate logging, monitoring, error handling, and testing for Python-based services and pipelines.

5. Collaboration & Ownership

• Work closely with Data Engineers, Software Engineers, Cloud/Infrastructure Engineers, Security teams, and business stakeholders.
• Take ownership of technical initiatives from architecture and design through production deployment and ongoing support.

Requirements

• 5+ years of experience in Data Engineering, Cloud Engineering, or a related field.
• Strong hands-on experience with dbt.
• Good to have experience in Databricks.
• Proven experience designing and implementing Medallion architecture.
• Proven experience designing and implementing GCP architecture and infrastructure for data workloads.
• Good understanding of GCP services such as Google Cloud Storage (GCS), BigQuery, Pub/Sub, Dataflow, Cloud Run/GKE, VPC, IAM, and Cloud Monitoring.
• Working knowledge of AWS equivalents (S3, Redshift, Glue, Lambda, CloudWatch) is an added advantage.
• Experience with cloud networking, security, authentication, authorization, and access control.
• Experience with infrastructure automation and Infrastructure as Code (IaC), preferably Terraform.
• Strong understanding of data quality, monitoring, observability, and production operational best practices.
• Experience organizing GCP environments to meet data sovereignty and data residency requirements.
• Strong experience building data ingestion pipelines and data processing workflows.
• Ability to work with messy or legacy code and refine pipeline architecture.
• Hands-on experience designing and implementing secure data transfer and proxy solutions.
• Strong proficiency in Python, with experience building production-grade applications and data pipelines.
• Excellent communication skills and ability to work independently with minimal supervision.

Work Location: Ahmedabad/Pune