AI Quality Analyst (Arabic)

Company Description

Role Overview

Offer Detail

• **Duration of Contra**
• ct: 12 months
• **Pa**
• y: $1500 per month (net/take-home)
• **Mode of wor**
• k: Fully Remote
• **Number of Position**
• s:
• **40Must have Arabic expertise**
• Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred
• BS/BA degree or equivalent experience in a relevant field (e.g., Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field)

Key Qualifications

• **Arabic Profici**
• ency: Ability to read and write in
• the Arabic langug
• age with a high degree of competence,
• as Arabic
• is the focus language for this project
• Personal Account Usage: Willingness to use your primary personal Google account (not a testing account) and enable personal data sources for a genuine assessment
• ent.Schedule Flexibility: Full-time availability in your local time zone is required. We are staffing a global, 24-hour operation
• Exceptional Analytical Thinking: Demonstrate ability to evaluate nuanced and ambiguous AI responses, specifically assessing personalisation quality
• Creative Prompt Engineering: Experience in designing creative, multi-turn starting prompts based on personal context to thoroughly test the model's capabilities
• Strong Evaluation Acumen: Understanding of personalisation concepts, including the ability to identify incorrect personalisation, poor inferences, and forced connections
• Meticulous Attention to Detail: The ability to review Side-by-Side (SxS) model responses and spot subtle differences in naturalness and over-narrat
• ing.Excellent Written Communication: Superior ability to write clear, concise, and structured rationales for model rankings, explicitly referencing specific turn numbers
• Feedback: Ability to provide constructive feedback and detailed annotations
• Communication: Excellent communication and collaboration skills
• Independence: Self-motivated and able to work independently in a remote setting
• Technical Setup: Desktop/Laptop set up with a good internet connection

Descrip

• In this role, you will be part of a dynamic team focused on evaluating the quality of personalised AI interactions. Your day-to-day work will involve
• Designing and executing multi-turn conversational prompts (typically 1-5 turns) that require the AI to utilise your personal information and experience
• Evaluating model responses based on your intent from the starting prompt, checking if the personalisation was appropriately applied
• Analysing responses for Grounding issues, ensuring claims about you are supported by evidence and not flawed inferences or hallucinations
• Assessing Integration quality to ensure personal data is woven naturally into the response without robotic "over-narration "
• Rigorously evaluating and stack-ranking two model responses side-by-side (SxS) to determine which is overall more helpful, easy to use, and enjoyable
• Writing clear, defensible rationales for your comparisons, explicitly referencing where issues or positive aspects occurred in the conversation
• Extracting and verifying "Debug Info" from the model to confirm that chat summaries and data sources were properly utilised
• Maintaining strict data hygiene by deleting evaluation conversations to prevent them from polluting your future chat history

Education & Exper

• BS/BA degree or equivalent experience in a relevant field (e.g., Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field)
• Experience in data annotation, AI quality evaluation, content moderation, or a related role is strongly preferred

Additional D

• Commitments Required: at least 4 hours per day and up to 40 hours per week, with 4 hours of overlap with PST.Engagement type:
• Contractor engagement Length: 12
• months