Senior PySpark & Databricks Data Engineer

Location: Atlanta, Georgia - Hybrid 3 days/onsite

Role Summary :

We are looking for a skilled Data Engineer with strong hands-on experience in PySpark and Databricks to design, build, and maintain scalable data pipelines. The ideal candidate has deep expertise in stream

processing with Apache Kafka, workflow orchestration with Apache Airflow, data warehousing on Amazon Redshift, and building cloud-native solutions on AWS.

Key Responsibilities

• Design and develop scalable ETL/ELT pipelines using PySpark on Databricks

• Build and maintain real-time data streaming pipelines using Apache Kafka

• Orchestrate and schedule data workflows using Apache Airflow (DAG development, monitoring, and troubleshooting)

• Manage and optimize data models and queries in Amazon Redshift

• Architect and deploy data solutions on AWS using services such as S3, Glue, Lambda, EMR, IAM, and CloudWatch

• Collaborate with analysts and platform teams to deliver high-quality data products

• Monitor pipeline performance and implement tuning strategies for large-scale data workloads

• Implement data quality checks, observability, and alerting across pipelines

• Participate in code reviews and contribute to engineering best practices

• Document data flows, architecture decisions, and pipeline logic

Required Skills & Experience

🔹 PySpark — 3+ years

DataFrame API, Spark SQL, query optimizations and performance tuning

🔹 Databricks — 3+ years

Notebooks, Jobs, Delta Lake, Unity Catalog

🔹 Apache Kafka — 2+ years

Producers/consumers, Kafka Streams, schema registry

🔹 Apache Airflow — 2+ years

DAG authoring, task dependencies, operators, scheduling

🔹 Amazon Redshift — 2+ years

Data modeling, query tuning, Redshift Spectrum

🔹 AWS — 3+ years

S3, Glue, Lambda, EMR, IAM, CloudWatch, VPC

🔹 Python — 4+ years

Strong scripting and engineering fundamentals

🔹 SQL — Strong

Complex queries, window functions, performance tuning

Nice to Have

• Experience with Delta Lake

• Familiarity with dbt for transformation layers

• Knowledge of Databricks Workflows alongside Airflow

• Exposure to Confluent Platform or AWS MSK (Managed Kafka)

• Experience with Terraform or AWS CDK for infrastructure-as-code

• Understanding of data governance, security best practices, and lake house architecture

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience)

• 4–7 years of overall experience in data engineering roles

• Strong problem-solving skills and ability to work independently in an agile environment