Avatar for BrickRed Systems
IT consulting firm providing business intelligence, security, and quality assurance

Senior Data Engineer

  • |5 years of exp
  • |Contract
Posted: 1 month ago
Job Location
Remote Work Policy

In office

Visa Sponsorship

Not Available

RelocationAllowed
Skills
Python
SQL
Redis
REST
Snowflake
Kafka
Big Query
Terraform
IAM
Grpc
GraphQL
Pyspark
Spark Streaming
Airflow
Apache Beam
Azure Data Factory
databricks
CI/CD
Cosmos DB
Delta Lake
Google Pub/Sub
Dataflow
Event Hubs
Functions
Cloud Spanner
Cloud Composer
VPC Service Controls

About the job

We are looking for a Senior Data Engineer to build and operate real-time data pipelines powering a large-scale Customer Data Platform (CDP) supporting 100M+ customer profiles. The role focuses on delivering real-time customer insights, near-real-time signals, and low-latency API access through Google Cloud Platform (GCP).The engineer will develop streaming pipelines using Google Pub/Sub, Dataflow (Apache Beam), BigQuery, and Cloud Spanner, while supporting migration and reuse of existing Azure-based ETL solutions involving Databricks, Snowflake, Azure Data Factory, Event Hubs, and Cosmos DB.Key ResponsibilitiesBuild and operate real-time and near-real-time ingestion pipelines using Google Pub/Sub and Dataflow (Apache Beam).Develop data pipelines feeding Cloud Spanner, BigQuery, and customer knowledge graphs.Implement CDC, SCD Type 2, idempotent ETL, clickstream ingestion, schema evolution, replay, and data reconciliation.Optimize Cloud Spanner schemas, indexes, batching, and data loads to support sub-50ms read performance.Design and implement BigQuery partitioning, clustering, cost optimization, and BigQuery-to-Spanner synchronization.Support Azure-to-GCP migration by reusing existing Databricks, Snowflake, and Azure Data Factory pipelines.Manage migration or retirement of Azure services, including Event Hubs, Cosmos DB, Redis, and Functions, with dual-run, rollback, and reconciliation strategies.Build data feeds for GraphQL APIs, REST services, customer knowledge graphs, and MCP servers supporting AI agents.Implement data quality, lineage, access controls, monitoring, alerting, SLOs, and cost optimization.Apply Terraform, CI/CD, Cloud Composer/Airflow, Cloud Run, and Dataproc for deployment and platform operations.Collaborate with product, analytics, cloud engineering, and AI/ML teams to deliver scalable customer data solutions.Conduct design reviews, troubleshoot production issues, perform root-cause analysis, mentor junior engineers, and maintain technical documentation.Required Qualifications5+ years of experience in Data Engineering, including 2+ years building production-grade real-time data pipelines.3+ years of experience with Google Cloud Platform (GCP) data engineering.3+ years of hands-on experience with Google Pub/Sub and Dataflow (Apache Beam).3+ years of experience with BigQuery, including partitioning, clustering, federation, and cost optimization.1+ year of experience with Cloud Spanner, including schema design, primary keys, indexes, change streams, and hotspot optimization.3+ years of experience with Spark Streaming, Kafka, or equivalent streaming technologies.Strong expertise in SQL, Python, and PySpark, including CDC, SCD Type 2, idempotent ETL/ELT, and data modeling.3+ years of experience with Databricks, Delta Lake, and Spark performance tuning.2+ years of experience with Snowflake and Azure services, including Azure Data Factory, Event Hubs, Cosmos DB, Redis, and Functions.3+ years of experience with Terraform/IaC, CI/CD, and Cloud Composer or Airflow.Experience with IAM, VPC Service Controls, observability, monitoring, data lineage, and data quality.Experience building data feeds for REST, gRPC, or GraphQL APIs with low-latency performance targets.Strong understanding of dimensional modeling, Data Vault, medallion architecture, testing, reconciliation, and production support.Bachelor's degree in Computer Science or a related field, or equivalent experience.Strong problem-solving, communication, collaboration, and technical documentation skills.Preferred QualificationsExperience with MCP servers, LLM/AI agents, or Vertex AI data access.Experience with Cloud Run, Dataproc, Dataplex, Unity Catalog, and Delta Live Tables.Knowledge of knowledge graphs, entity resolution, identity resolution, and Customer 360 platforms.Experience with exactly-once processing, watermarking, schema evolution, dead-letter handling, and streaming replay.Experience managing hybrid-cloud data migrations and large-scale customer data platforms.Ability to mentor engineers, lead technical discussions, review architecture, and coordinate delivery across cross-functional teams.ABOUT BRICKRED SYSTEMSBrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.

Similar Jobs

Archesys company logo
Archesys
Improving the government services that impact everyday lives
tribe.ai company logo
tribe.ai
We embed elite AI engineers to ship real, production-grade AI for enterprises
Neuralink company logo
Neuralink
Ultra-high bandwidth brain-machine interfaces to connect humans and computers
OnTarget Communications company logo
OnTarget Communications
From recruitment and placement to distinct documentation projects, we are always OnTarget
Asgard Aerospace company logo
Asgard Aerospace
Building the system that coordinates aerospace manufacturing production across the globe
Asgard Aerospace company logo
Asgard Aerospace
Building the system that coordinates aerospace manufacturing production across the globe
Asgard Aerospace company logo
Asgard Aerospace
Building the system that coordinates aerospace manufacturing production across the globe
Take-Two Interactive Software company logo
Take-Two Interactive Software
Game development needs creativity, but creativity needs an environment to be nurtured in