AI Operations Analyst

Company

Opportunity

What you'll contribute

• Review risk-weighted samples of AI chats and Personalized Credit Insights summaries; flag inaccurate, unsupported, off-topic, or unsafe responses.
• Classify member feedback, guardrail blocks, and escalations into a failure-mode taxonomy and track trends.
• Maintain the golden regression dataset with labeled production examples, consistent labels, and documented rubrics.
• Label data to measure automated evaluators, including double-labeling items to assess reviewer agreement.
• Publish weekly quality reports on intent coverage, resolution and containment, refusal and guardrail rates, safety flags, failure modes, and recommended fixes.
• Escalate flagged content to Legal, Compliance, and Trust & Safety, assign severity ratings, and track resolution.
• Use SQL and dashboards to identify member needs, experience gaps, and quality drift.
• Work with Product, Applied AI, and QA colleagues on review tooling, rubrics, sampling, and post-release fix verification.

What we're seeking

• A related bachelor's degree or equivalent experience, plus demonstrated quality, analytical, or operations experience reviewing customer conversations.
• Strong SQL skills and comfort building and reading dashboards in a BI tool such as Tableau.
• Sound judgment, attention to detail, consistent application of written rubrics, and ability to explain decisions.
• Strong written communication for failure descriptions and leadership summaries.
• Discretion and comfort handling sensitive consumer data in a regulated environment.
• Experience working in Agile/Scrum teams.
• Consumer finance, credit, banking, fintech, or other regulated-environment experience; credit report or FICO Score knowledge; and conversational AI, labeling, rubric, taxonomy, Python/Jupyter, or escalation experience are pluses.

Compensation

• Targeted base pay range is $105,000 to $165,000 per year, depending on candidate knowledge, skills, and experience.

Offer

• The company describes competitive compensation, benefits and rewards, professional development, work/life balance, employee resource groups, and social events.

Workplace

• The posting and role location identify the position as work-from-home in the United States.