Senior Data Engineer
Requirements
Requires at least five years of production data pipeline experience, strong Python and SQL skills, relational database proficiency, and hands-on Elasticsearch or OpenSearch optimization experience. Candidates should understand batch and streaming patterns, data modeling, and production operations in a tested, reviewed codebase; experience with Kafka, cloud platforms, warehouses, or ML and search systems is beneficial.
Job Description
Consulting and private equity firms come to proSapient when they need to discover knowledge to help them make great decisions and succeed in their goals. It is our mission to support them in their discovery of knowledge.
We help our clients find industry experts who can provide their knowledge via interview or survey; we curate this knowledge in a market-leading software platform; and we help clients surface knowledge they already have through expansive knowledge management.
About the role:
We are looking for a Senior Data Engineer to build and own those pipelines. This is a hands-on, delivery-focused role on a team that ships frequently, working directly with the Lead Search Engineer, an AI/ML Engineer and a PM measured on match quality and CSAT.
Key Duties & Responsibilities:
Pipelines & Data Products
- Build and operate batch and real-time pipelines feeding the matching and search systems.
- Work closely with real-time data streams and search infrastructure using tools like Kafka and Elasticsearch.
- Model the datasets the product layer consumes — candidate features, interaction events, ranking signals — and keep the contracts stable.
- Own the pipelines you build in production: monitoring, alerting, backfills, and the on-call reality of a broken 3am job.
- Instrument recruiter and client behaviour so matching quality can be measured and improved.
- Build the event pipelines behind our CSAT and workflow metrics.
- Conduct and analyse experiments to rapidly iterate on the quality of data processing pipelines and make experiment data queryable without a bespoke analysis every time.
- Add data quality checks and tests at the boundaries that matter and make failures loud instead of silent.
- Reduce cost and latency where it counts, leaving the rest alone.
- Partner with QA to make data correctness a testable property, rather than a hope.
- Identify and provide useful data sources, internal or external, for our structured search and GenAI tools.
Required Skills and Experience:
- 5+ years building production data pipelines.
- Strong Python programming skills, with practical experience writing clean, production-ready code.
- Strong SQL and proficiency with PostgreSQL or other relational databases.
- Strong hands-on experience with Elasticsearch / OpenSearch, including performance optimisation and query tuning.
- Deep understanding of both batch and event/streaming patterns, and a clear sense of when each is appropriate.
- Practical data modelling skills for analytics and for serving application features.
- Experience operating your own pipelines in production (and learned from it), alongside comfort working in a codebase with tests, code review and CI.
- Message brokers like Kafka or RabbitMQ.
- Cloud data warehouse — BigQuery preferred, equivalents fine.
- Docker, Kubernetes, and cloud environments (AWS and/or GCP).
- Experience feeding search, ranking or ML systems specifically.
- Building LLM-based extraction or enrichment pipelines.
- Experience with dbt or a comparable transformation framework.
- Observability tools (e.g. Prometheus, Datadog) and background job processing frameworks.
- Exposure to Elixir services (part of our platform).
What we can offer you:
- Tenure gifts, including vouchers, extra holiday and sabbaticals for each year of employment.
- Health insurance through Vitality.
- Remote working for up to 20 days each year, giving you flexibility and a change of scenery.
- Employee Assistance Programme with personalised health and wellbeing advice from specialist teams.
- Enhanced maternity and paternity pay.
- 25 days' annual leave plus bank holidays, including a week's closure over Christmas.
- MyMindPal app for online mental fitness support.
- Corporate events, from quarterly gatherings to annual winter and summer parties.
Skills
About proSapient
proSapient is a cutting-edge expert network and primary research platform that enables professionals to share insights with clients through short calls, online surveys and long-term engagements. Our clients include many of the world’s largest and most respected consultancy and private equity firms, and we have been featured in the Financial Times and in Sifted as one of the 100 fastest-growing businesses in Europe and the UK. Since our launch in 2017, we have revolutionized expert networking by offering new capabilities including the industry’s first AI-powered platform, enabling smarter, faster decision-making. Headquartered in the UK, proSapient is ISO 27001 accredited and employs over 300 people across four global locations: London, Lisbon, Raleigh, and Toronto. Should an opportunity arise where we think your experience could be relevant for an expert call, one of our Client Services Associates will contact you and help to determine whether there is a suitable fit, agree a rate to compensate you for your time and then submit your profile to our client. Our client then decides whether to work with a given expert and should they wish to move forward, our Associate will assist in scheduling a call at a suitable time. We are continuously reimagining the expert networking landscape and remain at the forefront of technological innovation, committed to building the future of knowledge discovery. In recognition of our technological leadership, we've been recognized as a Future Fifty company by Tech Nation.
View company profile →