AI Evaluation Data Scientist - AI / ML / LLM - (Hybrid (Hybrid) - Barcelona

Overview

AI Evaluation Data Scientist - AI / ML / LLM - Hybrid (Hybrid) - Barcelona

Principal Consultant | Semiconductor, Automotive, Software Engineering

AI Evaluation Data Scientist

A fantastic opportunity for a driven AI Data Scientist to join a leading Quantum AI company, who work on cutting-edge solutions that make AI faster, greener, and more accessible. You’ll be working alongside world-leading experts in quantum computing and AI, with the opportunity to work on challenging projects and shape the future of Generative AI systems.

This is initially a 9 Month Fixed Term Contract, with scope to extend - Hybrid working from sites in Madrid or Barcelona.

Responsibilities

• Design and lead the evaluation strategy for our Agentic AI and RAG systems, turning customer workflows and business needs into measurable metrics and clear success criteria.
• Contribute to the end-to-end design of Agentic AI and RAG systems, injecting a data-and-evaluation perspective into retrieval strategies, orchestration policies, tool usage, and memory to solve complex, real-world problems across industries.
• Develop task-based, multi-step evaluations that reflect how the different components of our systems (retrieval, planning, tool use, memory) perform in real-world scenarios across cloud and edge deployments.
• Develop and refine rigorous evaluation frameworks that reflect real-world performance, going beyond model benchmarks to assess task success, reasoning capabilities, factual consistency, reliability, and user success metrics across diverse problem domains.
• Build and maintain a reproducible evaluation pipeline, including datasets, scenarios, configs, test suites, versioned assets, and automated runs to track regressions and improvements over time.
• Curate and generate high-quality datasets for evaluation, including synthetic and adversarial data, to strengthen coverage and robustness.
• Implement and calibrate LLM-as-a-judge evaluations, aligning automated scoring with human feedback and ensuring fairness, robustness, and representativeness.
• Perform deep error analyses and ablations to uncover failure patterns, maintain a taxonomy of failure modes (reasoning, grounding, hallucinations, tool failures), and provide actionable insights to engineers to improve model and system performance.
• Partner with ML specialists to create a data flywheel, where evaluation continuously informs new dataset creation, improvements on prompts, tool usage, model training, and system refinements, quantifying improvements over time.
• Define and monitor operational metrics (latency, cost, reliability) to ensure evaluations align with production and customer expectations.
• Maintain high engineering standards, including clear documentation, reproducible experiments, robust version control, and well-structured ML pipelines.
• Contribute to team learning and mentorship, guiding junior engineers and sharing expertise in LLM development, evaluation, and deployment best practices.
• Participate in code reviews, offering thoughtful, constructive feedback to maintain code quality, readability, and consistency.

Required minimum Qualifications

By applying to this role, you understand that we may collect your personal data & store & process it on our systems. For more information please see our Privacy Notice (

Note : This posting includes an HTML-only description refined for formatting compliance. No additional roles or unrelated listings are included.

#J-18808-Ljbffr