Manager, AI Evaluation
Key Skills for This Role
Full Job Posting
Requirements
- postgraduate qualifications in a relevant field, or an equivalent combination of education and experience
- demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments
- strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies
- highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making
- experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement
- excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations
- Sponsorship / work rights for Australia
- You must have unrestricted work rights in Australia for the duration of this employment to be eligible to apply. Visa sponsorship is not available for this appointment.
- Pre-employment checks and declarations
- Your employment is conditional upon the successful completion of all pre-employment or background checks required for the role in terms satisfactory to the University. Also, to meet the University’s obligations under the National Higher Education Code to Prevent and Eliminate Gender-Based Violence you will be asked to declare if you have been investigated for, or found to engaged in, sexual harm or gender-based violence in the course of previous employment or in a legal process. Similarly, your ongoing employment is conditional upon the satisfactory maintenance of all relevant clearances and background check requirements. If you do not meet these conditions, the University may take any necessary step, including the termination of your employment.
- EEO statement
- At the University of Sydney, our shared values are trust, accountability and excellence and we strive to be a place where everyone can thrive. We are committed to creating a University community that thrives through diversity and reflects the wider community that we serve. We deliver on this through our commitment to diversity and inclusion, evidenced by our people and culture programs, as well as key strategies to increase participation and support the careers of Aboriginal and Torres Strait Islander People, women, people living with a disability, people from culturally and linguistically diverse backgrounds, and those who identify as LGBTQIA+. We welcome applications from candidates from all backgrounds.
- We are proud to be recognised as an Australian Workplace Equality Index (AWEI) Platinum Employer. Find out more about our work on diversity and inclusion.
- How to apply
- Applications (including a cover letter, CV, and any additional supporting documentation) can be submitted via the Apply button at the top of the page.
- For employees of the University or contingent workers, please login into your Workday account and navigate to the Career icon on your Dashboard. Click on USYD Find Jobs and apply.
- For a confidential discussion about the role, or if you require reasonable adjustment or any documents in alternate formats, please contact Victoria Ford-Rogers Recruitment Operations by email to [email protected]
- © The University of Sydney
- The University reserves the right not to proceed with any appointment.
- Click to view the Position Description for this role.
- Applications Close
Full Job Posting
- Full-time continuing opportunity
- drive the evaluation and quality assurance of generative AI solutions, ensuring robust governance, performance and continuous improvement across the AI Hub
- Base Salary $152,045+ 17% superannuation About the opportunity As the Manager, AI Evaluation, you will lead the assessment, testing and governance of generative AI solutions, ensuring they are safe, effective and fit for deployment and ongoing use. You will establish and oversee a consistent evaluation framework across the AI Hub, driving pre-release testing, governance approvals, performance measurement and continuous monitoring of AI services in production. Working closely with product, delivery and technical teams, as well as key stakeholders across ICT, Legal, Privacy and data stewardship functions, you will play a critical role in embedding best-practice evaluation processes, identifying risks, and ensuring AI solutions continue to deliver reliable and high-quality outcomes across the University. Your key responsibilities will be to:
- lead the development and continuous improvement of AI evaluation frameworks, methodologies and standards to ensure consistent, rigorous assessment across the AI Hub portfolio
- design, curate and govern evaluation datasets, metrics and testing approaches that measure performance, reliability, safety and quality of AI solutions
- oversee pre-release evaluation, testing and assurance activities, providing independent recommendations and evidence to support governance approvals and deployment decisions
- manage ongoing monitoring, regression testing and performance evaluation of production AI services, identifying risks, drift and opportunities for improvement
- partner with technical teams, data stewards, governance bodies and key stakeholders to ensure effective evaluation practices, data integrity and compliance requirements are met
- provide leadership, expert advice and capability development in AI evaluation, communicating findings and driving continuous improvement across the AI Hub and broader university community About you
- postgraduate qualifications in a relevant field, or an equivalent combination of education and experience
- demonstrated experience in data science, machine learning, natural language processing and AI solutions within complex organisational environments
- strong knowledge of generative AI, large language models (LLMs), agentic AI frameworks and modern data science methodologies
- highly developed analytical, problem-solving and data analysis skills, with the ability to interpret complex data, identify patterns and support evidence-based decision-making
- experience developing evaluation, monitoring and training frameworks, with strong attention to detail and a commitment to continuous improvement
- excellent communication, planning and stakeholder engagement skills, with experience working collaboratively in Agile and cross-functional teams within large or complex organisations Sponsorship / work rights for Australia You must have unrestricted work rights in Australia for the duration of this employment to be eligible to apply. Visa sponsorship is not available for this appointment. Pre-employment checks and declarations Your employment is conditional upon the successful completion of all pre-employment or background checks required for the role in terms satisfactory to the University. Also, to meet the University’s obligations under the National Higher Education Code to Prevent and Eliminate Gender-Based Violence you will be asked to declare if you have been investigated for, or found to engaged in, sexual harm or gender-based violence in the course of previous employment or in a legal process. Similarly, your ongoing employment is conditional upon the satisfactory maintenance of all relevant clearances and background check requirements. If you do not meet these conditions, the University may take any necessary step, including the termination of your employment. EEO statement At the University of Sydney, our shared values are trust, accountability and excellence and we strive to be a place where everyone can thrive. We are committed to creating a University community that thrives through diversity and reflects the wider community that we serve. We deliver on this through our commitment to diversity and inclusion, evidenced by our people and culture programs, as well as key strategies to increase participation and support the careers of Aboriginal and Torres Strait Islander People, women, people living with a disability, people from culturally and linguistically diverse backgrounds, and those who identify as LGBTQIA+. We welcome applications from candidates from all backgrounds. We are proud to be recognised as an Australian Workplace Equality Index (AWEI) Platinum Employer. Find out more about our work on diversity and inclusion. How to apply Applications (including a cover letter, CV, and any additional supporting documentation) can be submitted via the Apply button at the top of the page. For employees of the University or contingent workers, please login into your Workday account and navigate to the Career icon on your Dashboard. Click on USYD Find Jobs and apply. For a confidential discussion about the role, or if you require reasonable adjustment or any documents in alternate formats, please contact Victoria Ford-Rogers Recruitment Operations by email to [email protected] © The University of Sydney The University reserves the right not to proceed with any appointment. Click to view the Position Description for this role. Applications Close
About University of Sydney
Public Australian research university providing undergraduate, postgraduate and research education to students in Sydney and worldwide.
Visit company websiteJobs and hiring trendsApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at University of Sydney
Research Fellow- Heat and Health Research Centre
Camperdown, AUS
The University of Sydney seeks a Research Fellow for the Heat and Health Research Centre to advance research on heat and health in a warming world. The role leads original research in climate and environmental physiology
Postdoctoral Research Associate in Frontotemporal Dementia
Camperdown, AUS
Senior Marketing Officer
Camperdown, AUS
The University of Sydney is seeking a senior marketing professional to develop and deliver integrated campaigns and communications, primarily supporting the Faculty of Science. The role requires significant campaign expe
Senior Marketing Officer
Darlington, AUS
The University of Sydney is seeking a Senior Marketing Officer to coordinate marketing and communications for the Sydney Biomedical Accelerator. The role spans digital channels, content, internal communications, events a
Research Fellow- Heat and Health Research Centre
Camperdown, AUS
NHMRC Emerging Leadership Fellow
Camperdown, AUS
DECRA Fellow
Darlington, AUS
Postdoctoral Research Associate in Frontotemporal Dementia
Camperdown, AUS
Research Assistant
Camperdown, AUS
Senior Marketing Officer
Camperdown, AUS
Senior Marketing Officer
Darlington, AUS
Research Fellow
Westmead, AUS