Senior AI Engineer

Additional Information

For Los Angeles County (unincorporated) Candidates:

Qualifications

• 5+ years of experience in software, AI, or ML engineering.
• Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
• Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
• Experience building data pipelines, ETL processes, or streaming data systems.
• Experience with analytics platforms and data warehouses such as Databricks or Snowflake.
• Ability to communicate eval results clearly and influence the teams that build on them.
• Degree in computer science or a related technical field, or equivalent practical experience.
• Additional Information
• 140,960.00 - 193,820.00 USD Annual
• Procore will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable federal, state, and local laws, including the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act.
• A criminal history may have a direct, adverse, and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment: 1. appropriately managing, accessing, and handling confidential information including proprietary and trade secret information, as well as accessing Procore's information technology systems and platforms; 2. interacting with and occasionally having unsupervised contact with internal/external customers, stakeholders, and/or colleagues; and 3. exercising sound judgment.

Preferred Qualifications

• Strong proficiency in Python; experience with TypeScript is a plus.
• This role may also be eligible for Equity Compensation and/or Bonus Incentive Compensation. Procore is committed to offering competitive, fair, and commensurate compensation. Actual compensation will be based on a candidate’s job-related skills, experience, education or training, and location.

Responsibilities

• Improve our online evals: LLM judge accuracy, calibration, and the infrastructure that runs it in production.
• Build and maintain production trace infrastructure with Langfuse and Databricks.
• Build and maintain data pipelines for the goldsets behind our evals, including annotation and quality checks.
• Build agentic loops that mine production traces for failures and turn them into new eval cases and agent improvements.
• Improve the runtime, cost, and reliability of our online and offline eval pipelines.
• Contribute code and architecture, review designs, and mentor other engineers on the team.
• 5+ years of experience in software, AI, or ML engineering.
• Strong proficiency in Python; experience with TypeScript is a plus.
• Experience evaluating LLM systems, such as LLM-as-judge, rubric design, goldset curation, or measuring judge agreement.
• Experience with LLM observability or evaluation tools such as Langfuse, Braintrust, or similar.
• Experience building data pipelines, ETL processes, or streaming data systems.
• Experience with analytics platforms and data warehouses such as Databricks or Snowflake.
• Ability to communicate eval results clearly and influence the teams that build on them.
• Degree in computer science or a related technical field, or equivalent practical experience.
• Base Pay Range:
• 140,960.00 - 193,820.00 USD Annual
• This role may also be eligible for Equity Compensation and/or Bonus Incentive Compensation. Procore is committed to offering competitive, fair, and commensurate compensation. Actual compensation will be based on a candidate’s job-related skills, experience, education or training, and location.
• Procore will consider for employment all qualified applicants, including those with arrest or conviction records, in accordance with the requirements of applicable federal, state, and local laws, including the City of Los Angeles’ Fair Chance Initiative for Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act.
• A criminal history may have a direct, adverse, and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment: 1. appropriately managing, accessing, and handling confidential information including proprietary and trade secret information, as well as accessing Procore's information technology systems and platforms; 2. interacting with and occasionally having unsupervised contact with internal/external customers, stakeholders, and/or colleagues; and 3. exercising sound judgment.