In office
Not Available
About the job
We are looking for a Senior Data Engineer to build and operate real-time data pipelines powering a large-scale Customer Data Platform (CDP) supporting 100M+ customer profiles. The role focuses on delivering real-time customer insights, near-real-time signals, and low-latency API access through Google Cloud Platform (GCP).The engineer will develop streaming pipelines using Google Pub/Sub, Dataflow (Apache Beam), BigQuery, and Cloud Spanner, while supporting migration and reuse of existing Azure-based ETL solutions involving Databricks, Snowflake, Azure Data Factory, Event Hubs, and Cosmos DB.Key ResponsibilitiesBuild and operate real-time and near-real-time ingestion pipelines using Google Pub/Sub and Dataflow (Apache Beam).Develop data pipelines feeding Cloud Spanner, BigQuery, and customer knowledge graphs.Implement CDC, SCD Type 2, idempotent ETL, clickstream ingestion, schema evolution, replay, and data reconciliation.Optimize Cloud Spanner schemas, indexes, batching, and data loads to support sub-50ms read performance.Design and implement BigQuery partitioning, clustering, cost optimization, and BigQuery-to-Spanner synchronization.Support Azure-to-GCP migration by reusing existing Databricks, Snowflake, and Azure Data Factory pipelines.Manage migration or retirement of Azure services, including Event Hubs, Cosmos DB, Redis, and Functions, with dual-run, rollback, and reconciliation strategies.Build data feeds for GraphQL APIs, REST services, customer knowledge graphs, and MCP servers supporting AI agents.Implement data quality, lineage, access controls, monitoring, alerting, SLOs, and cost optimization.Apply Terraform, CI/CD, Cloud Composer/Airflow, Cloud Run, and Dataproc for deployment and platform operations.Collaborate with product, analytics, cloud engineering, and AI/ML teams to deliver scalable customer data solutions.Conduct design reviews, troubleshoot production issues, perform root-cause analysis, mentor junior engineers, and maintain technical documentation.Required Qualifications5+ years of experience in Data Engineering, including 2+ years building production-grade real-time data pipelines.3+ years of experience with Google Cloud Platform (GCP) data engineering.3+ years of hands-on experience with Google Pub/Sub and Dataflow (Apache Beam).3+ years of experience with BigQuery, including partitioning, clustering, federation, and cost optimization.1+ year of experience with Cloud Spanner, including schema design, primary keys, indexes, change streams, and hotspot optimization.3+ years of experience with Spark Streaming, Kafka, or equivalent streaming technologies.Strong expertise in SQL, Python, and PySpark, including CDC, SCD Type 2, idempotent ETL/ELT, and data modeling.3+ years of experience with Databricks, Delta Lake, and Spark performance tuning.2+ years of experience with Snowflake and Azure services, including Azure Data Factory, Event Hubs, Cosmos DB, Redis, and Functions.3+ years of experience with Terraform/IaC, CI/CD, and Cloud Composer or Airflow.Experience with IAM, VPC Service Controls, observability, monitoring, data lineage, and data quality.Experience building data feeds for REST, gRPC, or GraphQL APIs with low-latency performance targets.Strong understanding of dimensional modeling, Data Vault, medallion architecture, testing, reconciliation, and production support.Bachelor's degree in Computer Science or a related field, or equivalent experience.Strong problem-solving, communication, collaboration, and technical documentation skills.Preferred QualificationsExperience with MCP servers, LLM/AI agents, or Vertex AI data access.Experience with Cloud Run, Dataproc, Dataplex, Unity Catalog, and Delta Live Tables.Knowledge of knowledge graphs, entity resolution, identity resolution, and Customer 360 platforms.Experience with exactly-once processing, watermarking, schema evolution, dead-letter handling, and streaming replay.Experience managing hybrid-cloud data migrations and large-scale customer data platforms.Ability to mentor engineers, lead technical discussions, review architecture, and coordinate delivery across cross-functional teams.ABOUT BRICKRED SYSTEMSBrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.
About the company
Similar Jobs









