Jobs at Collective[i]

We are seeking an ambitious team player to help the data engineering team with implementing automated processes to support new data sources integrations, expanding data transformation capabilities, enabling to programmatically monitor processes and data quality across multiple systems as well as facilitating data management reporting automation. Responsibilities: Technical implementation and maintenance of automated processes in charge of: Integrating data from various data sources (e.g CRMs, Messaging, Video conference, Telephony, Group Document management, Contract signature/management, etc) Writing in-house data back into source systems Reporting data quality and processes execution alerts Performing data extraction, transformation, cleansing and loading between different data layers and environments Detecting customers settings and metadata at various source and target systems Consolidating and loading relevant data for data quality, metadata management, data consumption, UI usage and processes performance related dashboards Data archival, removal and reprocessing Build scalable data processing pipelines in Spark Debug Spark jobs and do performance tuning Write unit and integration tests for all data processing code Read specs and translate them into code and design documents Perform code reviews and develop processes for improving code quality Technical Requirements: 5+ years of experience developing in object oriented programming languages, preferably Python or Java. Highly Competent with Scala, Spark, the Spark Engine, and the Spark Dataframe API Experience with development best practices under continuous integration, testing and deployment in an AGILE environment involving source control tools such as GIT Familiarity with database tools, Integration Architecture, Data Integration, ETL, Business Intelligence concepts and Big-Data solutions Expertise using SQL for acquiring and transforming data