Big Data Engineer

We are seeking a skilled Big Data Engineer with 2–5 years of experience in designing, developing, and maintaining scalable big data processing systems. The ideal candidate should have hands-on experience with Hadoop, Apache Spark, Kafka, and distributed data processing technologies. You will be responsible for building reliable data pipelines, processing large datasets, and supporting real-time data analytics solutions.

Work Locations: New York City, NY | Seattle, WA | Austin, TX
Employment Type: Full-Time
Work Arrangement: On-site – Candidates must be willing to work from the designated office location.

Key Responsibilities

• Design, develop, and maintain scalable big data architectures.
• Build and optimize data processing pipelines using Hadoop and Apache Spark.
• Develop real-time data streaming solutions using Apache Kafka.
• Process and analyze large volumes of structured and unstructured data.
• Develop ETL/ELT workflows for data integration and transformation.
• Optimize distributed data processing jobs for performance and scalability.
• Work with data lakes, data warehouses, and distributed storage systems.
• Implement data validation, quality checks, and monitoring solutions.
• Collaborate with Data Scientists, Data Analysts, and Software Engineers.
• Troubleshoot data pipeline failures and performance issues.
• Ensure data security, reliability, and processing efficiency.

Required Skills

• 2–5 years of experience in Big Data Engineering or a related field.
• Hands-on experience with Hadoop ecosystem technologies.
• Strong knowledge of Apache Spark and PySpark.
• Experience with Apache Kafka and real-time data streaming.
• Programming knowledge in Python, Java, or Scala.
• Strong SQL and database management skills.
• Understanding of distributed computing and parallel processing.
• Experience developing ETL/ELT data pipelines.
• Knowledge of data processing and transformation techniques.
• Strong analytical and problem-solving abilities.

Preferred Qualifications

• Experience with Hive, HDFS, HBase, and YARN.
• Knowledge of Apache Airflow and workflow orchestration.
• Familiarity with AWS, Azure, or Google Cloud Platform.
• Experience with Snowflake, Databricks, or BigQuery.
• Knowledge of Docker and Kubernetes.
• Understanding of data lake and lakehouse architectures.
• Familiarity with CI/CD pipelines and Git.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related technical field, or equivalent practical experience.

Compensation & Benefits

• Annual Salary: $95,000 – $175,000.
• Competitive compensation based on experience and technical expertise.
• Health, dental, and vision insurance.
• Paid vacation and holidays.
• Professional development opportunities.
• Career growth and advancement opportunities.
• Opportunity to work with modern big data technologies.

Application Process

Interested candidates are encouraged to apply with their updated resumes, highlighting their Hadoop, Apache Spark, Kafka, Python/Java/Scala, SQL, and big data engineering experience.