AI Evaluation Data Scientist - AI / ML / LLM - (Hybrid) - Barcelona
European Tech RecruitCatalonia, EspañaHace 10 horas
Descripción del trabajoDesign and lead the evaluation strategy for our Agentic AI and RAG systems, turning customer workflows and business needs into measurable metrics and clear success criteria. Contribute to the end-to-end design of Agentic AI and RAG systems, injecting a data-and-evaluation perspective into retrieval strategies, orchestration policies, tool usage, and memory to solve complex, real-world problems across industries. Develop task-based, multi-step evaluations that reflect how the different components of our systems (retrieval, planning, tool use, memory) perform in real-world scenarios across cloud and edge deployments. Develop and refine rigorous evaluation frameworks that reflect real-world performance, going beyond model benchmarks to assess task success, reasoning capabilities, factual consistency, reliability, and user success metrics across diverse problem domains. Build and maintain a reproducible evaluation pipeline, including datasets, scenarios, configs, test suites, versioned assets, and automated runs to track regressions and improvements over time. Curate and generate high-quality datasets for evaluation, including synthetic and adversarial data, to strengthen coverage and robustness. Implement and calibrate LLM-as-a-judge evaluations, aligning automated scoring with human feedback and ensuring fairness, robustness, and representativeness. Perform deep error analyses and ablations to uncover failure patterns, maintain a taxonomy of failure modes (reasoning, grounding, hallucinations, tool failures), and provide actionable insights to engineers to improve model and system performance. Partner with ML specialists to create a data flywheel, where evaluation continuously informs new dataset creation, improvements on prompts, tool usage, model training, and system refinements, quantifying improvements over time. Define and monitor operational metrics (latency, cost, reliability) to ensure evaluations align with production and customer expectations. Maintain high engineering standards, including clear documentation, reproducible experiments, robust version control, and well-structured ML pipelines. Contribute to team learning and mentorship, guiding junior engineers and sharing expertise in LLM development, evaluation, and deployment best practices. Participate in code reviews, offering thoughtful, constructive feedback to maintain code quality, readability, and consistency. Master's or Ph.D. in Computer Science, Machine Learning, Data Science, Physics, Engineering, or related technical fields, with relevant industry experience. Solid hands-on experience (3+ years for mid-level, 5+ years for senior) working as a Data Scientist, ML Engineer, or Research Scientist in applied AI / ML projects deployed in production environments. Strong background in evaluation of machine learning systems, ideally with experience in LLMs, RAG pipelines, or multi-agent systems. Proven ability to design and implement evaluation methodologies that go beyond static benchmarks, capturing real-world task success, reasoning, and robustness. Hands-on experience with dataset creation and curation (including synthetic data generation) for training and evaluation. Proven experience with agent-based architectures (task decomposition, tool use, reasoning workflows), RAG architectures (retrievers, vector databases, rerankers), and orchestration frameworks (LangGraph, LlamaIndex). Strong problem-solving skills, with the ability to navigate ambiguity and design practical solutions to open-ended user or business needs. Strong software engineering skills, with proficiency in Python, Docker, Git, and experience building robust, modular, and scalable ML codebases. Familiarity with common ML and data libraries and frameworks (e.g., PyTorch, HuggingFace, LangGraph, LlamaIndex, Pandas, etc.). Experience with cloud platforms (ideally AWS). Fluent in English.
AI Evaluation Data Scientist
A fantastic opportunity for a driven AI Data Scientist to join a leading Quantum AI company, who work on cutting-edge solutions that make AI faster, greener, and more accessible. You’ll be working alongside world-leading experts in quantum computing and AI, with the opportunity to work on challenging projects and shape the future of Generative AI systems.
This is initially a 9 Month Fixed Term Contract, with scope to extend -
- Hybrid working from sites in Madrid or Barcelona.
Responsibilities :
Required minimum Qualifications
By applying to this role, you understand that we may collect your personal data & store & process it on our systems. For more information please see our Privacy Notice (
Crear una alerta de empleo para esta búsqueda
Data Scientist • Catalonia, España
Ofertas relacionadas
- Oferta promocionada
AI / ML Scientist ( GenAi is a plus) - debe residir en España
office people Holdingcatalunya, España Oportunidad de unirte a una gran consultora internacional y trabajar en un proyecto de un gran banco.Excelentes condiciones, beneficios, y plan de carrera!.
AI / ML Scientist / GenAI Engineer - España...Mostrar másÚltima actualización: hace 2 días
- Oferta promocionada
Senior AI / ML Engineer
Keysight TechnologiesCatalonia, España Our ~15,000 employees create world-class solutions in communications, 5G, automotive, energy, quantum, aerospace, defense, and semiconductor markets for customers in over 100 countries.Our technica...Mostrar másÚltima actualización: hace más de 30 días
- Oferta promocionada
Data Scientist / IA / MLOps
Logicalis Spaincatalunya, España Data Scientist / IA / ML / MLOps.El equipo de Data & Analytics de Logicalis trabaja con clientes nacionales e internacionales, llevando a cabo importantes proyectos de ingeniería de datos, intelige...Mostrar másÚltima actualización: hace 4 días
- Oferta promocionada
Beca remunerada Tech AI Strategy
NTT DATA Europe & Latamcatalunya, España En NTT DATA, ¡tenemos todo preparado para acompañarte en tu salto al mundo profesional!.Somos Global Top Employer, un reconocimiento único por las buenas prácticas hacia nuestras personas, prioriza...Mostrar másÚltima actualización: hace más de 30 días
- Oferta promocionada
- Nueva oferta
AI / ML Engineer Lead
MIGx AGCatalonia, España MIGx is a global consulting company with an exclusive focus on the healthcare and life science industries, with their particularly demanding requirements on quality and regulatory aspects.We have b...Mostrar másÚltima actualización: hace 18 horas
- Oferta promocionada
ML & Data Engineer - Barcelona
Oxigent TechnologiesCatalonia, España Te gustaría seguir desarrollando tu carrera como.Cuáles serán tus funciones principales?.Desarrollar y mantener modelos predictivos.
Analizar, estructurar y procesar.Diseñar, limpiar y preparar data...Mostrar másÚltima actualización: hace 2 días
- Oferta promocionada
AI Engineer
Domyncatalunya, España We're looking for a talented Al Engineer to join our team in Madrid, a talent focused on implementing and scaling large language models (LLMs) and generative Al systems.
In this role, you will bridg...Mostrar másÚltima actualización: hace 3 días
- Oferta promocionada
AI / MLOps Engineer
Logicalis Spaincatalunya, España El equipo de Data & Analytics de Logicalis trabaja con clientes nacionales e internacionales, llevando a cabo importantes proyectos de inteligencia artificial y analítica avanzada, estrategia del d...Mostrar másÚltima actualización: hace 4 días
- Oferta promocionada
Data Scientist - Machine Learning & AI
agap2 Españacatalunya, España Esta persona participará en la definición e implementación del marco de soluciones de analítica avanzada e inteligencia artificial, asegurando su impacto en las distintas áreas del negocio y contri...Mostrar másÚltima actualización: hace 4 días
- Oferta promocionada
MLOps Engineer|Databricks| MLflow | Remoto 100%
Cobser Consultingcatalunya, España A en función de experiencia aportada.En Cobser Consulting, tenemos la necesidad de cubrir para cliente bancario el siguiente rol : .
Apoyo en la implementación y operación de pipelines de ML.Diseño y ...Mostrar másÚltima actualización: hace 1 día
- Oferta promocionada
INTERNSHIP : Data Science (Python & ML)
carrotsCatalonia, España Python‑first, statistics‑strong.Data Science Intern (or junior ML Engineer) to partner with our.READ THIS FIRST — Apply only if you meet this : .
Eligibility : You’re currently studying at a university...Mostrar másÚltima actualización: hace 13 días
- Oferta promocionada
AI Product Manager
CapitoleCatalonia, España AI Product Manager con al menos 5 años de experiencia.Conocimientos en prompt engineering e imagen (StableDiffusion, Midjourney, DALL •E).Mostrar másÚltima actualización: hace más de 30 días
- Oferta promocionada
Senior Data Scientist ( AWS / MLOps)
PSS Tecnologías de la Informacióncatalunya, España Te gustaría impulsar tu carrera en el sector IT con un proyecto sólido y un equipo de profesionales excepcionales?.Si valoras la estabilidad y el crecimiento profesional, este es el momento de inco...Mostrar másÚltima actualización: hace 2 días
- Oferta promocionada
Data Science Consultant – Analytics Banking
NTT DATA Europe & LatamCatalonia, España NTT Data somos todas las personas que la formamos.Nuestra misión? Ofrecer soluciones tecnológicas, de negocio, estrategia, desarrollo y mantenimiento de aplicaciones, siendo un referente en consult...Mostrar másÚltima actualización: hace más de 30 días
- Oferta promocionada
AI Data Scientist
European Tech RecruitCatalonia, España A fast-scaling Quantum AI SaaS provider are hiring an AI Evaluation Data Scientist to help shape the future of Generative AI systems.
In this role, you’ll design and lead evaluation frameworks for A...Mostrar másÚltima actualización: hace 25 días
- Oferta promocionada
Machine Learning Engineer - LLM / Fixed Term Contract
European Tech RecruitCatalonia, España Machine Learning Engineer - LLM / Fixed Term Contract.We are currently partnered with a world-leading quantum AI company developing cutting-edge solutions that make artificial intelligence faster, ...Mostrar másÚltima actualización: hace 16 días
- Oferta promocionada
Data Scientist (Hibrido - Barcelona)
Oxigent TechnologiesCatalonia, España Te interesaría seguir desarrollándote como.Multinacional del sector editorial?.Cuáles serán tus funciones principales?.Analizar datos en diferentes etapas del proceso de modelización.Realizar el pr...Mostrar másÚltima actualización: hace 2 días
- Oferta promocionada
Generative AI Engineer
DeepRec.aiCatalonia, Catalonia, SPAIN Founded in 2019, our client had grown into one of Europe's most recognized deep-tech scale-ups, backed by major global strategic investors and EU innovation funds.
Their quantum and AI technologies ...Mostrar másÚltima actualización: hace 4 días