Agents on Narasimha Karthik J

Customer-Service Agent RL Environment

Mon, 05 Oct 2026 00:00:00 +0000

What it is #

Two LLMs talk to each other. A user simulator plays a customer with a hidden goal and sees only what the agent says. An assistant agent follows a written store policy and uses 13 tools that read and change an in-memory database built from the τ²-bench retail data (25 users, 45 orders, 50 products).

The tools check only data integrity, not policy. So a policy violation changes the database, and the reward can see it.

The reward #

Gate on state. The agent’s database diff must match a correct outcome, and the agent must have made the required lookups. A confident transcript with a wrong write scores zero, and a do-nothing agent can’t pass tasks where the right answer is “no write”.
Efficiency. Fewer wasted tool calls score higher.
Quality, by an LLM judge. The judge checks consent before each write and whether every claim of a finished action matches the tool log. It asks one yes/no question per call and quotes evidence before its verdict. Hard violations cap the reward below the floor of any clean run.

A pytest suite checks that golden runs pass and do-nothing runs fail, with no API calls.

What red-teaming found #

Forged consent. In a hand-written bad conversation, the customer’s text contained fake “AGENT: … CUSTOMER: Yes” lines, and the judge passed the consent check. Feeding the judge speaker-tagged messages fixed it.
Unfair simulator. The user simulator once agreed and sent its stop token in the same message, so a correct agent scored 0.
Against a judge-aware hostile customer, a Sonnet agent scored 1.0 and a Haiku agent scored 0, after handing off to a human following a correct denial.

These come from one or two runs per task, not pass rates.

Spec-Conformance Evals for an Account-Risk Attention Agent

Sun, 13 Sep 2026 00:00:00 +0000

Code on GitHub

What it is #

An AI-engineer assessment from Signal Labs. A fictional company runs an agent that watches 180 customer accounts and decides which risk signals a human should see. The agent’s behavior is defined in a formal spec as a state machine with invariants, timing targets, policy rules and grounding rules. The data is 629 agent dossiers, 32,032 days of telemetry, 3,810 text artefacts, renewal outcomes, and labels from three annotators who disagree.

I built a SignalEvaluator that scores each dossier for quality, risk, whether it deserved attention, and the specific spec rules it broke. On top of it I wrote a violation report, an attention-budget analysis and a writeup.

How it works #

Spec to rules. I transcribed the spec into a table of 29 rules, each with its spec identifier, section, severity class and owning check. Tests parse the LaTeX source of the spec and fail if a rule cites a section that does not exist.
Correct the data first. The evaluator rebuilds telemetry before checking any claim: later backfill rows win over duplicates, a legacy collector’s double-counted API calls are halved, and a latency scale change is undone. Inspection found three more distortions the docs did not list, including zero rows that were collector outages, not dead accounts. Where the data is missing, the check refuses rather than scoring zero.
Source-anchored checks. Each numeric claim is recomputed from corrected telemetry. Each quoted piece of evidence is checked to exist, belong to the same account, and match verbatim. Text is read by rules plus a small local NLI model, with no external API calls inside evaluate().
Tests. A pytest suite includes 25 hand-derived golden dossiers.

Results #

259 of 629 dossiers state a percentage change that does not reproduce on corrected telemetry. In 58 of them the pipeline manufactured the decline; in 121 a real decline was overstated.
Only one rule separated complaints: customer-visible plays on legal-hold or M&A-quiet-period accounts drew complaints at 32% (10 of 31), against 3% for the rest. No rule separated wasted escalations.
On quality scores, the evaluator agreed with each annotator about as well as the annotators agreed with each other (Spearman 0.36 to 0.55, against 0.31 to 0.45 between humans). On “deserved attention” it fell below the human band.
A spec-anchored routing policy under a 5-signals-per-owner-week cap routed 189 signals against the agent’s 294. Its routed signals churned more often (39% against 30%), but it missed 16 accounts with $3.02M of realised loss.
Bursts in signal volume traced to a collector migration, a regional ingest gap, and four public holidays.

job-hunt-kit

Wed, 28 Oct 2026 00:00:00 +0000

Code on GitHub

What it is #

A Claude Code plugin with nine skills that cover a job search from start to finish: set up a profile, build master resumes from job-description research, tailor a resume to one posting, draft application answers, find the right person on the team, and write outreach.

Everything lives in plain text and LaTeX files in a job-hunt/ folder that you own. The kit drafts. It never sends an email or submits a form.

How it works #

One source of truth. profile.md holds every fact about you. Resumes, answers and emails may only state what is in it.
Live form schemas. ats.py reads the exact application form from the public Ashby, Greenhouse and Lever APIs, because the pages are rendered client-side and scraping returns only a title. It checks a posting is live by its presence on the job board, since Ashby returns HTTP 200 for any path. It flags field titles worth a human look: inverted sponsorship questions, export-control questions, and decoy prompts addressed to an LLM.
A cheap gate, then a judge. gate.py is a deterministic check for stock phrases, compliance claims (HIPAA, SOC 2 and so on) that the profile never mentions, numbers absent from the profile, and leftover placeholders. judge.py then asks yes/no rubric questions about each email or answer, and returns a fix for every NO.
Claim checking per sentence. The judge splits a draft into sentences and asks, for each one, whether its background facts are supported by the profile and resume text.
Resumable pipeline. pipeline.py tracks roles through sourcing, verification and drafting so a long batch can stop and pick up again.

What’s notable #

The claim-check threshold was set on planted true and false claims: false claims scored at most 0.21 and true ones at least 0.40, so the cutoff sits at 0.3. Each exemption in the deterministic gate is commented with the false positive that caused it.

Kyn: Multi-Agent Company Brain

Sat, 09 May 2026 00:00:00 +0000

Code on GitHub · Live demo

What it is #

A team build for the Nozomio Hackathon, Track 4 (“The Company Brain”, on Nia and Hyperspell). You describe a work problem, such as “why are payments failing?”, and a supervisor agent decides what context it needs, gathers it, acts, and records what it learned.

I built the agent and backend: the supervisor and subagents, the tools, the seed corpus, the FastAPI server, and the wiring of the run-view UI to the live event stream. A teammate built the design system and the static UI mockup.

How it works #

Supervisor plus four subagents as tools, using the OpenAI Agents SDK. Code context searches an indexed repo through Nia. Company context searches Slack, email, Notion, contracts and tickets through Hyperspell. Memory recalls similar past problems early and writes lessons back before the run ends. Action produces outputs.
Playbooks by intent. The supervisor classifies the problem as an incident, a question, or a briefing. Only incidents open a PR and send messages; questions get a cited answer and no actions.
Real actions. For a code fix, the Action agent clones the target repo, runs Codex CLI with a small, size-capped instruction, commits, pushes, and opens a PR, then drafts a Gmail and posts to Slack. Evidence goes in the PR body, not the Codex prompt.
Seed corpus. A fictional payments company with about six weeks of Slack, email, Notion, contract, PR and ticket data around one incident: a retry bug that amplifies a vendor’s rate limit. An early support ticket hints at the problem, so the agent can connect it where the humans in the story did not.
Server. FastAPI streams run events over SSE, saves each run to disk, and can replay past runs.

What’s notable #

The repo includes artifacts from real runs: RCA skeletons, code-patch proposals and PR drafts for the rate-limit incident.

claude-kit

Thu, 15 Oct 2026 00:00:00 +0000

Code on GitHub

What it is #

A plugin marketplace for Claude Code that bundles the skills, slash commands, agents and workflows I use every day. It has six bundles:

writing: outreach email, LinkedIn and X posts, research blog posts.
research-vault: decode papers into Obsidian notes, extract atomic zettels, build synthesis pages.
explainer-videos: narrated Manim explainer videos from a spec.
netlogo: build and review NetLogo models.
dev-tools: GitHub operations, a timed take-home helper, planning and implementation commands, and coder and tester agents.
life: Vedic astrology charts and PDF invoices.

How it works #

install.sh clones the repo, adds the marketplace, installs the chosen bundles, writes a small env file with the vault paths, and copies the workflows into ~/.claude/workflows/ with those paths filled in. It skips workflows that already exist unless you set FORCE=1.

The seven workflows coordinate several agents. For example, feature-build has a coder implement a task, an implementation-blind tester try to break it, and the coder repair the cited failures, for up to three rounds.

What’s notable #

netlogo-review checks a model in layers ordered by cost: style and config checks, then the headless NetLogo compiler as a blocking gate, then known .nlogox traps. An LLM judge is opt-in, and a live provider call is required when a change touches a provider. It ships with self-test fixtures.
making-explainer-videos includes scripts that check every on-screen word traces back to the approved spec, and falls back to macOS say when no ElevenLabs key is set.
The research-vault skills use zettelkasten-auto for semantic search over notes.