Avatar for Moody's Anlaytics Knowledge Services
Financial Services
  • B2B
  • Scale Stage
    Rapidly increasing operations

Databricks Platform Engineer

Posted: 1 week ago
Job Location
Remote Work Policy

In office

Visa Sponsorship

Not Available

RelocationAllowed
Skills
Python
SQL
Amazon S3
ec2
routing
Apache Kafka
Pyspark
Kms
CloudWatch
databricks
Spark Structured Streaming
Delta Lake
Aws Networking
AWS Security
Secrets Manager
VPCs
Security Groups
IAM Roles
IAM Policies
Subnets
Encryption Keys
Databricks Workflows
Git-Based CI/CD
Secrets
Auto Loader
Databricks Asset Bundles
Private Connectivity
Lakeflow Jobs

About the job

The Senior Databricks Platform/Admin Engineer will design, build, deploy, administer, and optimize enterprise data solutions using the Databricks Data Intelligence Platform on Amazon Web Services. The role combines advanced data engineering with platform administration and requires hands-on ownership of batch and real-time pipelines, platform security, Unity Catalog governance, compute optimization, deployment automation, monitoring, and production support. The engineer will work closely with data engineering, architecture, DevOps, cloud infrastructure, cybersecurity, networking, and firewall teams to deliver secure, scalable, reliable, and cost-effective data platforms.Key responsibilities:Design, build, and maintain production-grade batch and real-time data pipelines using Databricks, Python, SQL, PySpark, Delta Lake, Apache Kafka, and Spark Structured Streaming.Administer, configure, deploy, monitor, and support Databricks environments on AWS, including workspaces, compute resources, permissions, jobs, and environment-specific configurations.Tune and scale Spark workloads, SQL queries, Delta tables, streaming jobs, and compute resources to meet performance, reliability, latency, and cost objectives.Evaluate serverless, classic, and hybrid compute models, recommending the appropriate approach based on workload characteristics, security, cost, scalability, performance, and operational complexity.Build fault-tolerant streaming pipelines using Kafka and Structured Streaming, including checkpointing, schema evolution, failure recovery, latency monitoring, and streaming-performance optimization.Implement Unity Catalog for centralized governance, metadata management, lineage, data discovery, access control, auditing, and secure data sharing.Automate data ingestion, transformation, testing, deployment, monitoring, and recovery using Databricks Workflows, Lakeflow Jobs, Auto Loader, Git-based CI/CD, and Databricks Asset Bundles where applicable.Establish secure connectivity between source systems, AWS services, Kafka, and Databricks, working with infrastructure, network, cybersecurity, and firewall teams.Configure and troubleshoot AWS networking and security, including VPCs, subnets, routing, security groups, private connectivity, IAM roles and policies, encryption keys, and secrets.Integrate Databricks with AWS services such as Amazon S3, EC2, CloudWatch, KMS, and Secrets Manager, with exposure to Kubernetes and Amazon EKS where applicable.Diagnose and resolve production issues involving source availability, data ingestion, streaming latency, Spark performance, compute capacity, permissions, networking, firewalls, and platform security.Document architecture decisions and technical trade-offs, clearly explaining the impact of design choices on cost, security, performance, availability, scalability, and maintainability.

About the company

Moody's Anlaytics Knowledge Services company logo
Financial Services501-1000 Employees
  • B2B
  • Scale Stage
    Rapidly increasing operations

Employees joined from

image
State Farm Ventures
image
KPMG INDIA
Learn more about Moody's Anlaytics Knowledge Services image

Similar Jobs

Archesys company logo
Archesys
Improving the government services that impact everyday lives
Scale AI company logo
Scale AI
Accelerate the development of AI applications
Pulse company logo
Pulse
Transforming healthcare by creating remarkable experiences for doctors and patients
OuterSignal company logo
OuterSignal
OuterSignal is building the customer intelligence layer for commerce
Merciv company logo
Merciv
The Future of Enterprise Intelligence. Read your data’s past. Write your company’s future
Counsel Health company logo
Counsel Health
Counsel offers 24/7 physician-led, AI-enhanced medical advice
Crosby company logo
Crosby
AI-native law firm that combined AI & lawyers to redefine the speed of legal services