{bc}
oracle

Principal Software Engineer - IT

Dell Technologies
Karnataka, IND
Full-time
Senior · 8+ years experience
Onsite
Discovered Today
python

Full Job Posting

AI Evaluation Engineer

The Security Resiliency Organization builds the AI-native capabilities that are replacing manual security processes across Dell, and every one of those capabilities has to earn its release through evidence, not demonstration. This team owns the quality bar for that shift: defining what "good enough" means for each workflow and building the harnesses that prove it, before and after release.

Join us as an AI Evaluation Engineer on our Security Resiliency team in Bangalore, India to do the best work of your career and make a profound social impact.

What you’ll achieve

  • You will own the release gate for AI-native security capabilities, deciding what evidence is required before a workflow can go live.
  • You will build evaluation harnesses that measure model and agent behaviour objectively, not by sampling outputs and forming an impression.
  • You will turn evaluation into a repeatable, measurable analytics practice so that every workflow after the first costs less to evaluate than the one before it.

You will:

  • Define and record acceptance criteria for each capability with the functional owner before evaluation begins, and specify the evidence, ground truth and coverage required to support the claim.
  • Build and maintain evaluation harnesses in Python covering dataset construction, test orchestration, scoring and reporting, and integrate them as automated quality gates in CI/CD.
  • Design measurable analytics for models and agentic tools – quantifiable metrics for accuracy, consistency across repeated runs, confidence-versus-accuracy calibration, latency and cost – using tooling such as LangFuse for trace-level evaluation and embedding-based similarity scoring for output comparison.
  • Own the release gate: apply the criteria to the evidence, record the result, and block releases that do not clear the threshold, with thresholds set proportionate to the consequence of an incorrect agent action.
  • Run ongoing regression evaluation after release to detect behavioural drift from model, prompt, data or upstream changes, and report results to engineering and functional stakeholders in decision-ready terms.

Take the first step towards your dream career

Every Dell Technologies team member brings something unique to the table. Here’s what we are looking for with this role:

Essential Requirements

  • 8+ years in software engineering, quality engineering or machine learning evaluation, with systems shipped to production.
  • Hands-on construction of evaluation harnesses in Python: dataset design, test orchestration, scoring implementation and result reporting for non-deterministic systems.
  • Ability to build measurable analytics of models and agentic tools, using LangFuse (or equivalent LLM observability tooling) and embedding-based comparison techniques.
  • Statistical literacy: sample size, variance across runs, and the difference between a real change in behaviour and noise, applied with Python data tooling at volume.
  • Experience owning a quality gate or release decision, including declining a release and holding that position under schedule pressure.

Desirable Requirements

  • Experience evaluating AI systems within a security domain, and familiarity with agent orchestration frameworks such as LangGraph to understand what is being evaluated.
  • Familiarity with the OWASP Top 10 for LLM Applications and the NIST AI Risk Management Framework, ideallyin an audited or regulated environment.

Apply for this job in 1 click

Skip the repetitive application forms

Install the Base Career Chrome Extension and autofill job applications across major job boards with your profile.

Sarah M.James T.Maya R.

Trusted by over 500,000 job seekers on Base Career

Start Free Today

More from this employer

More jobs at Dell Technologies