AI Evaluation Engineer

About Hailey 🩷 During the spring of 2020 Hailey HR was founded. We’re a Swedish SaaS HR-tech company that wants to digitalise, simplify and modernise the HR industry with our HR-system - join us on our journey! About the position 💼 As an AI Evaluation Engineer at Hailey HR, you will be a part of our AI development team - working in close collaboration with AI developers, our QA engineer and Product Owners to make sure the AI features we ship are accurate, reliable and trustworthy. We are building a range of AI-driven tools for HR, managers and employees. Building AI features is one thing - knowing that they actually work, and keep working when models and prompts change, is another. That is where you come in. You will own how we evaluate our AI, both before a feature is released and once it is live in production. Our AI features work with sensitive HR data, and protecting our customers' privacy always comes first. That makes evaluating AI quality a real challenge: how do you know an AI feature works well for users without looking at their data? That is the question you will own. If that sounds like a fun problem to solve, keep reading! What you will do

• Define, together with the team, what “good” looks like for each AI feature and turn it into measurable criteria
• Build and maintain test datasets for our AI features, using synthetic data and realistic edge cases
• Develop evaluation pipelines that run in CI
• Set up privacy-friendly monitoring of AI quality in production
• Turn evaluation results into concrete improvement suggestions for the team

What are we looking for?

• Experienced in working with Python, with an interest in AI evaluation methods and frameworks.
• A person with strong analytical abilities and an interest in statistics and measuring things. Someone curious and eager to learn new technologies and AI models, proactive in taking initiative, and thrives in a collaborative team environment.
• Education: A Master’s degree in Computer Science, Data Science, Engineering Physics or equivalent field.
• Experience: Around two years of professional experience in software development, data science or test automation, with some hands-on experience working with LLMs. Experience in evaluating AI systems is a plus.
• Language: Full professional fluency in Swedish and strong English skills, both spoken and written.

We believe you are

• Flexible and thrives in an innovative and dynamic environment, where solutions are always open to discussion.
• Someone who likes to ask “how do we know this works?” - and then builds the answer.
• Enjoys working in a team, pair programming, and sees the value of code reviews because you work closely with your colleagues in the team.
• Loving table tennis! (or at least find it okay, it’s a part of our culture after all 🏓)

More information

• Process: We will have an ongoing selection and as part of the evaluation, you will be asked to submit your academic transcripts - please upload them directly with your application. 👋
• Start: Whenever you’re ready!
• Scope: 100%
• Location: Stockholm, Sweden (on-site)

We’re looking forward to your application! 👋 💌