Avatar for Cogniify
Cogniify
Actively Hiring
More jobs
Cogniify: Enterprise AI consulting and production system implementation

Databricks Data Engineer

Posted: 1 month ago
Job Location
Remote Work Policy

Onsite or remote

Hires remotely in
Visa Sponsorship

Not Available

RelocationAllowed
Skills
Python
SQL
Azure
Apache Spark
AWS
GCP

About the job

Role Overview

We are looking for an experienced Databricks Data Engineer to support, maintain, and enhance existing Databricks-based data applications and pipelines. The role focuses on ensuring reliability, performance, and scalability of production Databricks workloads rather than building net-new platforms from scratch. You will work closely with data, analytics, and engineering teams to keep critical data applications stable, optimized, and aligned with business needs.

Key Responsibilities

Support and maintain existing Databricks applications, notebooks, jobs, and Delta Lake pipelines in production.
Monitor, troubleshoot, and resolve issues related to job failures, performance degradation, data quality, and cluster utilization.
Optimize existing Spark jobs, SQL queries, and Delta tables for cost, performance, and reliability.
Manage and improve Databricks workspace configurations, including clusters, job scheduling, access controls, and Unity Catalog (where applicable).
Implement and maintain data quality checks, logging, alerting, and basic observability for Databricks workloads.
Collaborate with stakeholders to understand requirements for enhancements or bug fixes on existing applications.
Perform incremental improvements, refactoring, and technical debt reduction on current Databricks solutions.
Ensure adherence to best practices around security, governance, and cost management within the Databricks environment.
Document existing pipelines, dependencies, and operational runbooks.
Participate in on-call or support rotations as needed to maintain production stability

Required Qualifications

6–9 years of overall experience in data engineering, with strong hands-on experience in Databricks.
Solid proficiency in Apache Spark (PySpark and/or Scala) and SQL.
Proven experience supporting and optimizing production Databricks workloads (jobs, notebooks, Delta Lake, workflows).
Strong understanding of Delta Lake concepts (ACID transactions, time travel, optimization techniques such as Z-ordering, vacuum, optimize).
Experience with Databricks Job clusters, Interactive clusters, and performance tuning (partitioning, caching, shuffle optimization, autoscaling).
Familiarity with data modeling, ETL/ELT patterns, and production data pipeline support.
Experience working with cloud platforms (preferably Azure, AWS, or GCP) in the context of Databricks.
Ability to troubleshoot complex Spark and Databricks issues independently.
Strong communication skills and ability to work effectively with EST-aligned team.

Preferred Qualifications

Experience with Unity Catalog, Databricks SQL, or Lakehouse architecture.
Knowledge of CI/CD practices for Databricks (e.g., Databricks Asset Bundles, Git integration, Terraform/ARM templates).
Familiarity with orchestration tools (Airflow, Azure Data Factory, or Databricks Workflows).
Exposure to data quality frameworks, monitoring tools, or cost optimization initiatives on Databricks.
Experience supporting analytics or BI teams consuming Databricks data products.

Why Join UsWhy Join Us?

Join a team of industry veterans from Google, Meta, and top-tier tech companies.
Work on impactful, high-scale projects with leading global clients.
Enjoy a flexible, remote-first culture focused on innovation and excellence.
Competitive salary, equity options, and continuous learning opportunities.
Shape the future of cloud and AI infrastructure at a rapidly growing company.

Perks And Benefits Of Working With Us

Internet allowance
Laptop
PF
Paid PTO
Annual Bonus
Gratuity
Yearly Team building experiences
Mentorship and sponsorship opportunities
Manager resources and support
Life & accidental insurance for additional protection.

We are an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or any other protected characteristic.

Similar Jobs

Carbon Trail company logo
Carbon Trail
AI powered sustainability platform for fashion and retail industry
Bydek company logo
Bydek
Enterprise CDM platform that transforms channel data into clean, actionable insights
Marseer AI company logo
Marseer AI
AI marketing agents for DTC e-commerce brands
Jumeau Capital company logo
Jumeau Capital
Building developer infrastructure in the API economy. Real product. Real revenue. Lean team
AuxoAI company logo
AuxoAI
We help companies—turn their strategies into practical digital and AI solutions
Xohani  Solutions company logo
Xohani Solutions
Central India's top product partner of Salesforce, ServiceNow & Venture Studios