Data Platform Manager

What you will be doing:

We are seeking a seasoned and dynamic Data Platform Manager to lead the design and implementation of high-performance systems that support financial risk detection. In this role, you will manage a team of talented Data Platform Engineers and Data Architects, collaborate cross-functionally, and be instrumental in architecting solutions using both SQL and NoSQL databases, distributed SQL, Kafka, and Spark-based systems. Your leadership and technical expertise will ensure that our data infrastructure not only supports operational goals but plays a critical role in helping us stay ahead of emerging financial risks, directly impacting the safety and reliability of global financial systems.

What you will be doing:

The role reports to the VP Engineering of Platform and Infrastructure. You’ll be managing a team of engineers - distributed between our London and Lisbon offices - focused on the provision and support of data services for use across the organisation, including classic SQL or NoSQL databases, distributed SQL, Kafka, and Spark-based systems. Your key responsibilities will be:

• Team Leadership: Lead, mentor, and grow a team of Data Infrastructure Engineers responsible for maintaining and improving the data systems across the organisation.
• System Design & Architecture: Design and architect high-performance, low-latency data systems to meet the requirements of real-time financial risk detection and analysis.
• Data Infrastructure Management: Oversee the implementation and management of SQL and NoSQL databases, distributed SQL systems, and stream processing platforms like Kafka and Apache Spark.
• Cross-Functional Collaboration: Work closely with data scientists, software engineers, and risk management teams to understand their data requirements and deliver reliable and efficient solutions.
• Performance Optimisation: Continuously monitor system performance, identify bottlenecks, and implement tuning strategies to ensure systems are performing at optimal levels.
• Innovation & Best Practices: Stay up to date with the latest in data engineering, distributed systems, and database technologies, incorporating best practices into the team's workflows.
• Data Governance & Compliance: Work closely with our Data Governance team to ensure all data systems comply with relevant regulations, privacy standards, and security protocols.
• Project Management: Drive the execution of data infrastructure projects from concept to delivery, managing timelines, resources, and priorities effectively.

Our tech stack:

ComplyAdvantage is fully cloud-based, with a modern Kubernetes-focused tech stack. All compute workloads run in Kubernetes, with clusters in multiple regions to support the needs of our global client base. Our production services are currently based in AWS and GCP, are domain designed and event-driven or event-sourced. As such, Kafka is used widely across teams, and is seen as a core technology in our tech stack along with Yugabyte, our Distributed Database system. We also use other data tooling, such as PostgreSQL, Elasticsearch and Databricks/Spark. To tie in with our multi-cloud plans, we’ll also be developing inter-cloud failover options to be leveraged by our development teams.

Across the wider Technology team, teams build and release containerised applications to support the wide array of activities that our teams are engaged in - from developing low latency client-facing APIs, to machine learning models and data processing pipelines.

Very agile by nature, our teams operated using elaborated CI/CD pipelines. We also make heavy use of Terraform and Helm to define our infrastructure and services, and lean heavily on GitOps paradigms - production and non-production environments are defined in git and changes to these environments (both cloud infrastructure and Kubernetes applications) are managed via git.

About you:

• Experience managing and mentoring engineering teams, with strong leadership and interpersonal skills.
• Expertise in SQL and NoSQL databases (PostgreSQL, Cassandra, MongoDB, etc.) and distributed SQL systems (such as YugabyteDB or CockroachDB).
• Good knowledge of Apache Kafka for real-time data streaming and Apache Spark for distributed data processing.
• A strong understanding of cloud infrastructure (AWS, GCP, or Azure) and its data services, including storage, compute, and networking.
• Experience with monitoring tools and performance optimization techniques for large-scale distributed systems.
• Understanding of data governance, compliance, and security requirements for financial services (e.g., WORM storage, FedRAMP, etc.).
• Hands-on experience with automation, infrastructure-as-code, and CI/CD pipelines.
• Experience with data modelling, sharding and indexing strategies running databases at scale

Nice to haves:

• Experience of working in a start-up/ scale-up environment
• Have experience designing and developing microservices-based architectures
• An understanding of cost and performance considerations when designing data systems architectures
• Experience working with CI/CD tools like CircleCI, Harness, ArgoCD, etc

Benefits:

• Equity as we want you to have a part of what we are building
• We embrace a hybrid approach that requires employees to be in the office for two days a week. We strongly believe that this approach fosters collaboration and enables the building of meaningful relationships
• You will also get a budget to kit out your home office
• Opportunity to work on innovative projects with smart-minded people keen to share their knowledge and continuously improve
• Annual learning budget to drive your performance and career development
• Enhanced parental leave and childcare benefits
• Medical insurance through Bupa with pre-existing conditions covered
• Pension contribution through The Peoples Pension