Lead Data Engineer
Department: Data Engineering Team
Reporting to: Technical Manager
Type of Contract: Permanent
Working Pattern: Full Time
Location: London, Camden. This is not a remote working position, and you must have the right to work in the UK.
FutureLearn is a leading social learning platform formed in December 2012 by The Open University and is now the largest online learning platform in Europe with over eight million people signed up worldwide. FutureLearn uses design, technology and partnerships to create enjoyable, credible and flexible online courses and postgraduate degrees that improve working lives. It partners with a quarter of the world’s top universities, as well as organisations such as Accenture, the British Council, CIPD, Raspberry Pi and the New York Institute of Finance.
Our vision is a global community, where everyone learns together and enjoys access to the education they need to transform their lives. Our award-winning platform helps by provoking conversation around the course material, so that learners and educators learn as much from each other as from the material itself.
We are continuing our expansion as we make this journey from offering short online courses, through micro credentials to full online degrees, and working with employers and governments to reduce skills gaps. The pace of this change is reflected in extremely rapid growth of our revenues.
ABOUT THE ROLE
The Lead Data Engineer provides leadership to the Data Engineering team. They are responsible for establishing the vision and direction the ETL data pipeline and other data services and infrastructure that supports FutureLearn’s business and products and delivering it.
They manage the team’s priorities to ensure that the correct features are being built in the right order and that work supporting the needs of the strategic teams and Data Scientists is balanced with longer term development of our data infrastructure.
This Lead Data Engineer develops the vision and strategy of our data architecture, communicates it and leads the team to deliver it.
To do this, the Lead Data Engineer, collaborates with other product teams to understand and agree priorities and work together on implementing parts of the ETL and other data engineering projects that would enable them to collect and analyse the data they need and build data driven features.
They also collaborate with the Technology team and Site Reliability Engineer to ensure that the work that they are doing is consistent with the overall technical direction of FutureLearn and that the technical dependencies between the application platform and the data platform can be implemented in the most amenable way for both systems.
The Lead Data Engineer is responsible for building and maintaining FutureLearn’s cloud based data pipeline. This involves:
- Architecting, designing and building a bespoke Extract, Transform, Load (ETL) process built predominantly in Ruby
- Architecting, designing and building ancillary services for the ETL, often web based, and using JSON APIs and socket IPC approaches, in both Ruby and Python.
- Managing our Postgres based Data Warehouse including collaborating with Data Scientists to design and enhance the schema using the Kimball model of star schemas.
- Integrating and maintaining big data solutions and recognising where these are appropriate tools to be used - currently we use Amazon Redshift, Amazon ElastiCache and others.
- Working with Data Scientists to build new machine learning and advanced data mining solutions and integrate them reliably into our production architecture.
- Familiarity with the implementation of hybrid content and collaborative filtering based recommendations engines in order to enhance and support our internal Course Recommendations Engine
The Lead Data Engineer also line manages, mentors and support the other Data Engineers in the team.
This person must have excellent knowledge of data platforms and technologies and possess direct experience with some of the data technologies we use. Direct previous experience of working with the ingest, processing and storage of data, ideally in a cloud-based context. You can show understanding of the problems data technologies must solve for an analyst, data scientist or data engineer.
Experience in leading, managing and/or mentoring software engineers as well as leading engineering and/or cross-functional teams is also required
- You will line manage and be responsible for the development of a small team of dedicated data engineers
- You will have demonstrated experience of building software for the cloud, and believe in a DevOps culture
- Strong written and verbal communication skills with proven experience dealing with different levels of an organisation from Executive level down.
- Candidates with direct experience in building data solutions, ETL pipelines, data validation, recommendation engines and machine-learning approaches will be particularly suitable.
- Experience with building Web apps to productionise tools and/or models built by a Data Science team (e.g., Django) would be a bonus.
Technical skills include:
- Amazon Web Services (including EC2, VPC, RDS, IAM, CloudWatch)
- Apache Spark, Amazon Redshift, Airflow and other data technologies
- Confluent Platform (Kafka, Kafka Streams)
- Ruby and Python based API microservices
- Queueing Systems and IPC approaches using ZeroMQ & Sidekiq
- Building ETL pipelines in Ruby/Kiba & PostgreSQL
- Git, GitHub
- Test-driven development (TDD)
- £60k - £80k depending on experience
- 28 days holiday (plus 8 days public holiday)
- Buy & sell up to 5 days holiday
- Charity day (volunteer for a charity of your choice)
- Cycle to work scheme
- Season Ticket loan
- Flexible working environment/hours
- Pension (4% employer / employee contribution)
- OU Staff Fee Course Waiver Programme
- Great coffee, teas, fruit and daily breakfast
FutureLearn is an equal opportunities organisation who value diversity, promote equality and challenge discrimination. We are especially keen to encourage applications from people currently under-represented, including those from the LGBT+ community, neurodiverse people, people with disabilities, and those from a Black, Asian or Minority Ethnic background.
Please contact [email protected], if you require any reasonable adjustments or alterations to be made, to support you through the recruitment process.
We value diversity at FutureLearn, and we do not discriminate on the basis of race, religion, colour, national origin, gender, sexual orientation, age, neurodiversity, marital or disability status.
Do you have a LinkedIn account? Import your resume and save time!