workday
Evaluation Engineer
Freshfields Bruckhaus Deringer
New York City, USA
Mid · 5+ years experience
Onsite
Discovered 1 weeks ago
gitjavajenkinsjirapython
Role Overview
Build and run tests that assess the quality and reliability of the firm’s AI applications.
Help maintain evaluation datasets, investigate failures and provide clear evidence of how AI systems perform on legal and business tasks.
Key Skills for This Role
gitjavajenkinsjirapython
Full Job Posting
Required skills and experience
- Minimum five years’ experience developing software demonstrated through employment, placements or substantial projects using programming languages such as Python, Java or C#.
- Foundational conceptual understanding of large language models, including their capabilities, limitations and common failure modes, and an ability to apply that understanding when developing evaluation criteria, test cases and acceptance thresholds with guidance from senior engineers.
- Ability to use coding agents effectively to support development, testing and analysis, with awareness of best practices for prompting, context management, validation, security, responsible use of generated code and when to seek human review.
- Working knowledge of APIs, Git, automated testing and software development lifecycle practices and CI/CD tools such as Jira and Jenkins.
- Demonstrated curiosity and ability to learn in a rapidly evolving AI landscape, assess new models, tools and evaluation techniques, and apply relevant learning pragmatically with appropriate guidance.
- Foundational understanding of statistics and measurement, including sampling, precision and recall, together with strong analytical skills, attention to detail and the ability to document issues clearly and seek technical or legal guidance when needed.
- Practical experience evaluating AI applications is a plus but is not required, given the newness of the field; relevant academic, internship or project experience will also be considered.
- For individuals assigned and/or hired to work in New York and California, Freshfields is required by law to include a reasonable estimate of the compensation range for this role. This compensation range is specific to the States of New York and California and takes into account the wide range of factors that are considered in making compensation decisions including but not limited to skill sets; experience and training; licensure and certifications; and other business and organizational needs. The disclosed range estimate has not been adjusted for the applicable geographic differential associated with the location at which the position may be filled, and compensation decisions are dependent on the facts and circumstances of each case. A reasonable estimate of the current range is $100,000 to $200,000.
- EEO Statement Freshfields US LLP is proud to be an equal employment employer. Our policies and practices will be free from unlawful discrimination based upon race, color, ethnicity, religion, creed, sex (including pregnancy, childbirth or related medical conditions), national origin, citizenship, immigration status, ancestry, age, marital status, protected veteran status, military service, disability, medical condition, genetic information, sexual orientation, gender identity, or any basis prohibited under federal, state or local law. We strive to promote an atmosphere that encourages equal opportunities and prohibits discriminatory practices, including sexual harassment. Disability Accommodation for Applicants to Freshfields US LLP Freshfields US LLP is an Equal Employment Opportunity employer and provides reasonable accommodation for qualified individuals with disabilities and disabled veterans in job application procedures. If you have any difficulty using our online system and you need an accommodation due to a disability, you may use the alternative email address below to contact us about your interest in employment at [email protected], or you can send your resume to [email protected], or you can call us at +1-212-277-4000.
- EEO Statement
- Freshfields US LLP is proud to be an equal employment employer. Our policies and practices will be free from unlawful discrimination based upon race, color, ethnicity, religion, creed, sex (including pregnancy, childbirth or related medical conditions), national origin, citizenship, immigration status, ancestry, age, marital status, protected veteran status, military service, disability, medical condition, genetic information, sexual orientation, gender identity, or any basis prohibited under federal, state or local law. We strive to promote an atmosphere that encourages equal opportunities and prohibits discriminatory practices, including sexual harassment.
- Disability Accommodation for Applicants to Freshfields US LLP
- Freshfields US LLP is an Equal Employment Opportunity employer and provides reasonable accommodation for qualified individuals with disabilities and disabled veterans in job application procedures. If you have any difficulty using our online system and you need an accommodation due to a disability, you may use the alternative email address below to contact us about your interest in employment at [email protected], or you can send your resume to [email protected], or you can call us at +1-212-277-4000.
Preferred Qualifications
- Practical experience evaluating AI applications is a plus but is not required, given the newness of the field; relevant academic, internship or project experience will also be considered.
Responsibilities
- Implement automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers.
- Prepare and maintain test datasets, source documents and reference examples with legal specialists. Follow requirements for confidentiality, permitted data use and version control.
- Run evaluations of answer accuracy, completeness, citations, retrieval relevance, instruction following and appropriate escalation. Apply agreed scoring criteria and flag cases requiring expert judgement.
- Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs and error handling, in coordination with application QA.
- Review outputs and execution traces to identify failures. Reproduce issues, document supporting evidence and help distinguish data, retrieval, model and application defects.
- Compare results across model, prompt and application versions. Report changes accurately, including uncertainty and limitations, rather than relying solely on headline scores.
- Support the calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.
- Maintain regression suites, contribute to CI/CD integration and turn approved production failure examples into new test cases.
About Freshfields Bruckhaus Deringer
Professional Services5,001-10,000
Freshfields Bruckhaus Deringer is a global law firm providing legal services in areas including corporate transactions, disputes, antitrust, and financial regulation. The firm operates from offices across Europe, Asia, and the Americas.
Visit company websiteApply for this job in 1 click
Skip the repetitive application forms
Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.
Trusted by over 500,000 job seekers on Base Career
More from this employer
More jobs at Freshfields Bruckhaus Deringer
AI Data & Integration Engineer
New York City, USA
Discovered 3 days ago
Financial Controller - MENA
Dubai, UAE
Discovered 5 days ago
MENA Chief Operating Officer & Director of Finance
Dubai, UAE
Discovered 5 days ago
Head of US Tax
New York City, USA
Discovered 1 weeks ago
Legal Trainee
Riyadh, KSA
Discovered 1 weeks ago
Head of HR Tech and AI Transformation
New York City, USA
Discovered 1 weeks ago
Senior Technology Recruiter
New York City, USA
Discovered 3 weeks ago
UX/UI Developer
New York City, USA
Discovered 3 weeks ago