In office
Not Available
About the job
The Senior Databricks Platform/Admin Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and real-time pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.Key responsibilities:Design, build, and maintain production-grade batch and real-time data pipelines using Databricks, Python, SQL, PySpark, Delta Lake, Apache Kafka, and Spark Structured Streaming.Administer, configure, deploy, monitor, and support Databricks environments on AWS, including workspaces, compute resources, permissions, jobs, and environment-specific configurations.Tune and scale Spark workloads, SQL queries, Delta tables, streaming jobs, and compute resources to meet performance, reliability, latency, and cost objectives.Evaluate serverless, classic, and hybrid compute models, recommending the appropriate approach based on workload characteristics, security, cost, scalability, performance, and operational complexity.Build fault-tolerant streaming pipelines using Kafka and Structured Streaming, including checkpointing, schema evolution, failure recovery, latency monitoring, and streaming-performance optimization.Implement Unity Catalog for centralized governance, metadata management, lineage, data discovery, access control, auditing, and secure data sharing.Automate data ingestion, transformation, testing, deployment, monitoring, and recovery using Databricks Workflows, Lakeflow Jobs, Auto Loader, Git-based CI/CD, and Databricks Asset Bundles where applicable.Establish secure connectivity between source systems, AWS services, Kafka, and Databricks, working with infrastructure, network, cybersecurity, and firewall teams.Configure and troubleshoot AWS networking and security, including VPCs, subnets, routing, security groups, private connectivity, IAM roles and policies, encryption keys, and secrets.Integrate Databricks with AWS services such as Amazon S3, EC2, CloudWatch, KMS, and Secrets Manager, with exposure to Kubernetes and Amazon EKS where applicable.Diagnose and resolve production issues involving source availability, data ingestion, streaming latency, Spark performance, compute capacity, permissions, networking, firewalls, and platform security.Document architecture decisions and technical trade-offs, clearly explaining the impact of design choices on cost, security, performance, availability, scalability, and maintainability.
About the company
- B2B
- Scale StageRapidly increasing operations
Employees joined from
Similar Jobs











