Data Engineer

We are seeking an experienced Data Engineer to join a data engineering team supporting an Azure-native third-party data enrichment platform. The role focuses on building reliable, scalable, governed, and cost-efficient data pipelines using Databricks, Spark, Snowflake, Azure Data Factory (ADF), Python, and SQL.

The platform enriches first-party datasets with external identity and attribute data to support analytics, customer activation, research, and Customer Data Platform (CDP) use cases. The ideal candidate will have strong hands-on data engineering experience, excellent Spark troubleshooting skills, and a strong understanding of data quality, privacy, governance, and performance optimization.

Candidates with backgrounds in FAANG, product-based, or Tier-1 technology companies are preferred.

Key Responsibilities

Data Ingestion & Pipeline Development

• Design, develop, and enhance scalable data ingestion pipelines supporting large-volume batch and event-driven workloads.
• Build robust ETL/ELT pipelines using PySpark, Python, SQL, Databricks, and Azure Data Factory.
• Integrate data from third-party enrichment vendors, including large-scale identity and attribute datasets.
• Integrate digital platform data through Conversion API (CAPI) and middleware-based integrations.
• Integrate data from rewards and promotions systems, including offer issuance, redemption, and consumption data.
• Develop scalable and reusable data engineering frameworks and components.

Data Quality, Reliability & Operations

• Implement strong data validation, deduplication, auditability, and data quality controls.
• Design and implement idempotency, replay, backfill, and recovery strategies to maintain pipeline reliability.
• Build and maintain monitoring, alerting, dashboards, and operational readiness capabilities.
• Troubleshoot data pipeline failures using root-cause analysis rather than simply rerunning failed jobs.
• Analyze Spark logs and diagnose issues related to shuffle, skew, partitioning, memory, and performance.
• Improve pipeline stability, reliability, and SLA adherence.

Databricks & Spark Engineering

• Develop and optimize large-scale data processing solutions using Databricks and Apache Spark.
• Apply Spark fundamentals such as partitioning, caching, shuffle optimization, and workload tuning.
• Troubleshoot complex Spark failures and performance bottlenecks.
• Implement modern Databricks and Delta Lake patterns, including Medallion Architecture.
• Contribute to Delta Live Tables (DLT) and other Databricks-based data engineering workflows where applicable.

Snowflake & Data Warehousing

• Develop and support data solutions using Snowflake for analytics and warehousing workloads.
• Apply effective data modeling and query optimization techniques.
• Ensure data structures are scalable, maintainable, and optimized for downstream analytical consumption.

Data Governance, Privacy & Compliance

• Apply data privacy, security, governance, and compliance requirements throughout the data lifecycle.
• Work with Unity Catalog and other governance frameworks for access control, lineage, and data management.
• Implement appropriate controls for PII and non-PII data.
• Maintain documentation for tables, schemas, catalogs, pipelines, and cluster usage.
• Support data lineage, auditability, and governance standards across the platform.

Cost & Performance Optimization

• Design data pipelines with cost efficiency, scalability, and performance in mind.
• Optimize cluster sizing, compute utilization, storage, and workload configurations.
• Identify and resolve performance bottlenecks across Spark, Databricks, Snowflake, and Azure services.
• Balance cost, quality, reliability, and SLA requirements when making technical decisions.

Required Skills & Qualifications

• 5+ years of hands-on Data Engineering experience.
• Strong hands-on development experience with PySpark and SQL.
• Strong experience with Python for data engineering and pipeline development.
• 5+ years of experience with Databricks, ETL, and Azure.
• Strong experience with Azure Data Factory (ADF) for orchestration and data integration.
• Strong understanding of Apache Spark fundamentals, including:
• Partitioning
• Shuffle
• Data skew
• Performance tuning
• Spark troubleshooting
• Cluster optimization
• Experience with Snowflake and analytics/warehouse workloads.
• Experience designing and implementing scalable ETL/ELT pipelines.
• Strong understanding of data engineering reliability patterns, including:
• Data validation
• Idempotency
• Replay and backfills
• Deduplication
• Auditability
• Experience with data governance, lineage, access controls, and PII handling.
• Strong analytical and problem-solving skills with the ability to perform detailed root-cause analysis.
• Ability to work independently and take ownership of technical deliverables with minimal supervision.
• Strong understanding of software development lifecycle, coding standards, testing, documentation, and Agile methodologies.

Preferred / Nice-to-Have Skills

• Experience with event-driven or real-time/streaming data ingestion.
• Experience with Delta Lake and Delta Live Tables (DLT).
• Experience building configuration-driven data pipelines and reusable frameworks.
• Experience with Azure-native data services and integrations.
• Experience supporting Customer Data Platforms (CDP) or customer/marketing data ecosystems.
• Experience working with third-party data enrichment, identity, or customer attribute datasets.
• Knowledge of data privacy, compliance, and enterprise data governance.
• Experience working in FAANG, product-based, or Tier-1 technology organizations.

Architecture & Engineering Responsibilities

• Contribute to HLD, LLD, solution architecture, and data model design.
• Evaluate technical options and select appropriate design patterns and reusable components.
• Design solutions that optimize performance, scalability, maintainability, quality, and cost.
• Translate technical and business requirements into scalable data engineering solutions.
• Participate in design reviews, code reviews, and technical discussions.
• Identify opportunities to improve existing architectures, frameworks, and engineering practices.

Testing, Documentation & Quality

• Develop, review, and execute unit and integration test cases.
• Validate solutions against technical specifications and business requirements.
• Perform defect analysis, root-cause analysis, and remediation.
• Maintain technical documentation, data models, pipeline documentation, standards, and operational procedures.
• Follow established coding standards, development processes, templates, and quality guidelines.
• Support release activities and ensure production readiness.

Collaboration & Communication

• Collaborate with Data Engineers, Architects, Product teams, Analytics teams, and business stakeholders.
• Clarify requirements and provide technical guidance to development and cross-functional teams.
• Communicate technical solutions and design decisions effectively to both technical and non-technical stakeholders.
• Manage multiple priorities, dependencies, risks, and deliverables in a fast-paced environment.
• Proactively identify issues and drive them to resolution.
• Contribute to knowledge sharing, reusable assets, and continuous improvement initiatives.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital, engineering, marketing, analytics, finance, operations, and business transformation domains. With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions.

BrickRed Systems fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers