Data Engineer - Lead

About the Role

We are seeking a highly skilled Senior Data Engineer with strong hands-on experience in Google Cloud Platform (GCP), cloud infrastructure, data ingestion, and Python. The ideal candidate will have a proven track record of designing and implementing scalable data platforms, organizing GCP environments to meet data sovereignty requirements, and building secure and reliable data ingestion and transfer solutions.

Key Responsibilities

1. GCP Architecture & Infrastructure
2. Design, develop, and maintain scalable GCP infrastructure for enterprise data workloads.
3. Architect and organize GCP environments to support data sovereignty and data residency requirements.
4. Design appropriate GCP projects, environments, networking, storage, IAM, and security configurations for data platforms.
5. Establish scalable and reliable cloud infrastructure patterns for data processing and storage.
6. Data Ingestion & Pipeline Engineering
7. Design and implement robust data ingestion pipelines for structured and unstructured data sources.
8. Build production-grade ingestion frameworks to reliably process and move data into GCP or similar storage.
9. Develop scalable and fault-tolerant data pipelines with appropriate validation, monitoring, and error handling.
10. Optimize data ingestion workflows for performance, reliability, and cost.
11. Ensure data pipelines maintain data quality, integrity, and traceability throughout the ingestion process.
12. Python Development & Automation
13. Develop production-grade data engineering applications and automation using Python.
14. Build reusable libraries, services, and utilities to support data ingestion and processing workflows.
15. Automate infrastructure and data engineering processes to improve operational efficiency.
16. Implement appropriate logging, monitoring, error handling, and testing for Python-based services and pipelines.
17. Collaboration & Ownership
18. Work closely with Data Engineers, Software Engineers, Cloud/Infrastructure Engineers, Security teams, and business stakeholders.
19. Take ownership of technical initiatives from architecture and design through production deployment and ongoing support. Requirements
20. 5+ years of experience in Data Engineering, Cloud Engineering, or a related field.
21. Strong hands-on experience with Google Cloud Platform (GCP).
22. Proven experience designing and implementing GCP architecture and infrastructure for data workloads.
23. Good understanding of GCP services such as Google Cloud Storage (GCS), BigQuery, Pub/Sub, Dataflow, Cloud Run/GKE, VPC, IAM, and Cloud Monitoring.
24. Experience with cloud networking, security, authentication, authorization, and access control.
25. Experience with infrastructure automation and Infrastructure as Code (IaC), preferably Terraform.
26. Strong understanding of data quality, monitoring, observability, and production operational best practices.
27. Experience organizing GCP environments to meet data sovereignty and data residency requirements.
28. Strong experience building data ingestion pipelines and data processing workflows.
29. Hands-on experience designing and implementing secure data transfer and proxy solutions.
30. Strong proficiency in Python and experience building production-grade applications and data pipelines.
31. Excellent communication skills and ability to work independently with minimal supervision.

Work Location: Ahmedabad/Pune