Data Engineer

*About Us: *

BDIPlus, a US-based leading transformation Consulting & Customer Data Platform (CDP) company, is dedicated to delivering cutting-edge capabilities and solutions that foster the development of enduring competitive advantages. Our innovative solutions showcase our unparalleled proficiency in technology and our profound domain expertise within the Financial Services and Insurance sectors. By synergizing our unmatched technical skills with a comprehensive grasp of each client’s institutional landscape and distinctive areas for improvement, we empower them to convert data into actionable and well-organized information. This facilitates precise decision making, increased efficiency, and rampant business growth.

*About the Role *

BDIPlus is seeking a skilled Data Engineer to join our innovative and talented team who have helped numerous clients unlock the potential of their data. As a Data Engineer at BDIPlus, you will play a crucial role in designing, developing, and maintaining our data infrastructure and pipelines. You will work closely with our cross-functional teams to ensure the availability, reliability, and scalability of our data systems. Your expertise in data extraction, transformation and loading (ETL) processes will contribute to the successful implementation of our data-driven solutions.

Roles and Responsibilities

• Design, build and optimize data pipelines and architectures to support efficient ETL processes.
• Develop and maintain data systems, ensuring data quality, integrity, and security.
• Build and maintain shell scripts to automate data workflows, monitoring, and routine operational tasks.
• Work within our Hadoop-based data architecture, developing and optimizing Hive queries and jobs across the broader Hadoop ecosystem.
• Collaborate with data scientists, analysts, and software engineers to understand data requirements and implement effective solutions.
• Monitor and optimize data infrastructure for scalability and efficiency.
• Take ownership of projects end-to-end, clearly articulating technical decisions and trade-offs to both technical and non-technical stakeholders.
• Stay updated with the latest industry trends, tools, and technologies related to data engineering.

Skills and Qualifications

• Bachelor's or Master's Degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer or a similar role, with a strong understanding of data modeling, database design and ETL processes.
• Proficiency in programming languages such as Python, Java or Scala.
• Strong shell/Bash scripting skills for automation and system operations.
• Solid working knowledge of Hive and the Hadoop ecosystem (e.g., HDFS, YARN, MapReduce/Spark on Hadoop).
• Experience with SQL, NoSQL databases, data warehouses, and distributed computing frameworks.
• Proficiency with Google Cloud Platform (GCP); hands-on experience with GCP data services is required.
• Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment.
• A self-starter who takes ownership of their work and can clearly articulate their reasoning and approach to others.
• Attention to detail and a commitment to delivering high-quality results.
• Knowledge of data governance, data privacy and security best practices is a plus