AI Data Analyst

About Glean

Role overview

You will

• Run ongoing human labeling for priority AI workflows, including response quality, task success, MCP and tool use, and end-to-end workflows.
• Triage bad-query reports, downvotes, escalations, and quality issues; categorize failure modes and send analysis to partner teams.
• Analyze recurring patterns including hallucinations, retrieval failures, tool-use issues, weak grounding, and poor workflow completion.
• Follow and improve labeling guidelines and rubrics for consistent, realistic, useful judgments.
• Maintain labeled datasets, golden sets, and regression slices; monitor coverage, drift, leakage, and difficulty distribution.
• Participate in grader calibration and validate LLM-as-a-judge outputs against human labels.
• Partner on pre- and post-change quality reads, benchmark updates, and release-readiness decisions.
• Report on top failure modes, coverage gaps, quality shifts, and actionable recommendations.

About you

• 3–5 years of experience in data labeling, data analysis, QA, or a related field.
• Strong analytical judgment and attention to detail, with consistent application of nuanced rubrics.
• Experience evaluating AI-generated outputs or working with NLP, search, recommendation, or other ML systems.
• Familiarity with structured qualitative analysis, basic SQL or data retrieval, and spreadsheet workflows.
• Clear written and verbal communication for technical and non-technical audiences.
• Ability to work independently in ambiguity and collaborate effectively across functions.

Location

• Hybrid role requiring four days per week in the office.

Compensation and benefits

• Compensation depends on location, level, job-related knowledge, skills, and experience.
• Certain roles may be eligible for variable compensation, equity, and benefits.

Interview process

• Candidates will complete a brief AI-focused exercise or discussion during the interview process.