AI Engineer with Deep Data Science Understanding

emagine is looking for a freelance AI Engineer with Deep Data Science understanding for our partner within the financial sector. The primary goal of the AI Engineer role is to contribute to the development of the AI Skills Registry focusing on building reliable components for AI solutions that operate effectively across various platforms. This role focuses on building essential components that support both AWS AgentCore-based agents and our enterprise AI solution. Main Responsibilities:

• Build components for the AgentCore and/or Amazon Quick and run evaluation frameworks for skills and tools builds.
• Ensure the reliability and accuracy of AI models.
• Help operationalize the Evaluate and Iterate phase of the ADS to ensure robust evaluation practices across various agent builds.
• Define and track quality and cost-per-outcome metrics for production agents.
• Run structured error analysis on agent outputs to identify failure patterns and implement fixes.
• Focus on the development of innovative AI solutions within a dynamic team environment.

Key Requirements:

• Hands-on experience evaluating ML or LLM-based systems in production.
• Working knowledge of evaluation methodology, including error analysis and rubric design.
• Strong Python and data analysis skills.
• Experience with cloud-native ML infrastructure, such as AWS Bedrock.
• Ability to work independently within a growing team.
• Experience in a regulated industry, such as financial services or healthcare.

Nice to Have:

• Familiarity with AWS AgentCore, Strands Agents, or similar agent runtimes.
• Knowledge of GDPR or equivalent data protection regulations.

Other Details: This position is fulltime and requires working on-site.