Customer-Service Agent RL Environment

Mon, 05 Oct 2026 00:00:00 +0000

What it is #

Two LLMs talk to each other. A user simulator plays a customer with a hidden goal and sees only what the agent says. An assistant agent follows a written store policy and uses 13 tools that read and change an in-memory database built from the τ²-bench retail data (25 users, 45 orders, 50 products).

The tools check only data integrity, not policy. So a policy violation changes the database, and the reward can see it.

The reward #

Gate on state. The agent’s database diff must match a correct outcome, and the agent must have made the required lookups. A confident transcript with a wrong write scores zero, and a do-nothing agent can’t pass tasks where the right answer is “no write”.
Efficiency. Fewer wasted tool calls score higher.
Quality, by an LLM judge. The judge checks consent before each write and whether every claim of a finished action matches the tool log. It asks one yes/no question per call and quotes evidence before its verdict. Hard violations cap the reward below the floor of any clean run.

A pytest suite checks that golden runs pass and do-nothing runs fail, with no API calls.

What red-teaming found #

Forged consent. In a hand-written bad conversation, the customer’s text contained fake “AGENT: … CUSTOMER: Yes” lines, and the judge passed the consent check. Feeding the judge speaker-tagged messages fixed it.
Unfair simulator. The user simulator once agreed and sent its stop token in the same message, so a correct agent scored 0.
Against a judge-aware hostile customer, a Sonnet agent scored 1.0 and a Haiku agent scored 0, after handing off to a human following a correct denial.

These come from one or two runs per task, not pass rates.

job-hunt-kit

Wed, 28 Oct 2026 00:00:00 +0000

Code on GitHub

What it is #

A Claude Code plugin with nine skills that cover a job search from start to finish: set up a profile, build master resumes from job-description research, tailor a resume to one posting, draft application answers, find the right person on the team, and write outreach.

Everything lives in plain text and LaTeX files in a job-hunt/ folder that you own. The kit drafts. It never sends an email or submits a form.

How it works #

One source of truth. profile.md holds every fact about you. Resumes, answers and emails may only state what is in it.
Live form schemas. ats.py reads the exact application form from the public Ashby, Greenhouse and Lever APIs, because the pages are rendered client-side and scraping returns only a title. It checks a posting is live by its presence on the job board, since Ashby returns HTTP 200 for any path. It flags field titles worth a human look: inverted sponsorship questions, export-control questions, and decoy prompts addressed to an LLM.
A cheap gate, then a judge. gate.py is a deterministic check for stock phrases, compliance claims (HIPAA, SOC 2 and so on) that the profile never mentions, numbers absent from the profile, and leftover placeholders. judge.py then asks yes/no rubric questions about each email or answer, and returns a fix for every NO.
Claim checking per sentence. The judge splits a draft into sentences and asks, for each one, whether its background facts are supported by the profile and resume text.
Resumable pipeline. pipeline.py tracks roles through sourcing, verification and drafting so a long batch can stop and pick up again.

What’s notable #

The claim-check threshold was set on planted true and false claims: false claims scored at most 0.21 and true ones at least 0.40, so the cutoff sits at 0.3. Each exemption in the deterministic gate is commented with the false positive that caused it.

LLM-as-Judge on Narasimha Karthik J

Customer-Service Agent RL Environment

What it is #

The reward #

What red-teaming found #

job-hunt-kit

What it is #

How it works #

What’s notable #