We ate our own Dog Food, using the same specification review engine we ship to you

Photo by Kaboompics.com on Pexels

What happens between a bug report and a pull request

We tell customers that an agent can review a specification more consistently than a room full of engineers, and that it will show its work. That's a real claim, and anyone evaluating us is right to ask whether we trust it enough to run our own house on it. Over the past several months we replaced most of the manual triage between "a customer reported something" and "someone in engineering decided what to do about it" with the same category of system we sell: agents that read, propose, and get overruled when they're wrong.

Every operational issue and every piece of user feedback lands in the same place first: a message queue that feeds a shared pool. It doesn't matter whether it arrived as a support ticket, an in-app feedback form, or a monitoring alert from a Lambda that just threw an unhandled exception. It goes into the same pool, tagged with its source and, where relevant, the customer and document it came from.

Nothing gets read cold from there. Before an agent even opens a ticket, it's checked against what we already know: issues already logged in the same database, and the sentiment already recorded against that customer or feature over time. That triage step runs on the same kind of matching component that powers Sherlock's duplicate-detection layer in WYZER Detective, the same hybrid approach that catches semantically redundant requirements in a specification even when they share no vocabulary. Internally, it does the same job on a support ticket: is this the third report of the same underlying bug worded three different ways, or is it actually new? In WYZER's own validated benchmark runs, across specifications running into the thousands of requirements, including complex tables and images, that matching approach flags roughly 12.8% of requirements as duplicate candidates without a false positive on a clean document. The internal triage pipeline leans on the same logic to collapse near-duplicate tickets before anyone, human or agent, spends time on them twice.

A ticket that reads "export is slow on large documents again" doesn't get treated as a fresh report. Triage checks it against sentiment already logged for that account and finds the same complaint from six weeks earlier, closed at the time as "monitoring," plus two more from unrelated customers hitting the same code path. That history changes what happens next: instead of three separate agents independently investigating the same symptom, one gets assigned the merged ticket with the prior findings attached, and the other two reports get linked to it rather than opened as new work.

Agents watch this pool continuously, not on a schedule and not triggered by someone opening a dashboard. An agent is listening to the queue at 3 a.m. on a Sunday exactly as attentively as it is at 10 a.m. on a Tuesday. A new item lands, triage runs, and if it isn't a known duplicate, an agent picks it up.

Scoped access, and a proposal instead of an answer

Once an agent takes an item, it gets scoped access to the relevant part of the codebase, not the whole repository, just the service or module the issue plausibly touches, and it runs its own analysis. It reads the error, the stack trace where there is one, the relevant code paths, and whatever context the customer gave it.

An agent rarely opens a ticket by summarizing it back to itself the way a person would. It goes straight to the stack trace, pulls the file and line number out of it, opens that file, and reads outward: the function, then whatever calls it, then the test file next to it if one exists. Only after that does it go back and reread the customer's description, checking whether the two stories actually match.

What comes out of that analysis depends on what kind of problem it turns out to be, and the distinction matters more than it sounds like it should. An operational bug, a missing null check on an edge case, a timeout tuned too aggressively for large documents, gets a proposed code fix. A complaint or a question that was never actually a defect, someone confused about how a feature works or unhappy with a result that's technically correct, gets a proposed response message instead. That's the difference. A fix is code. A response is words. The agent picks based on which one actually resolves the ticket, not which one it can generate faster.

Not every part of the codebase gets treated the same way, either. Whether an agent should attempt a code change at all is its own judgment call upstream of the analysis, and for anything touching billing or authentication we keep the scoped access narrower than the ticket alone would justify. Some categories of bug are worth a human reading the ticket before an agent ever opens the file.

The merge request is a proposal, not a deployment

Whatever the agent proposes, a fix or a response, gets packaged the same way: a merge request opens automatically, carrying the analysis, the proposed change, and a plain-language explanation of why. Nothing an agent proposes ships on its own. The operations crew gets notified internally the moment the request is open, and someone has to look at it: approve it as written, refine the wording or the fix, or reject it and say why. That's the same question we ask about accountability for AI-generated content generally, applied to our own house instead of a customer's specification.

This isn't caution for its own sake. Agents are good at matching a fix to a known category of bug and bad at knowing when a fix is technically correct but a bad idea right now, when a customer response is accurate but reads as dismissive, or when a proposed change touches something mid-migration that shouldn't be touched yet. A person catches that in the time it takes to read a diff. An agent generally doesn't, at least not yet, which is the same dependency every AI-powered engineering tool shares whether it's writing a fix or scoring a requirement.

The notification itself is deliberately unglamorous: a message in the team's internal channel, linking the merge request, the original ticket, and a one-line summary of what the agent concluded and why. No dashboard, no separate review tool. Whoever picks it up reads the same diff anyone would review on a normal pull request, checks whether the reasoning holds up against the actual ticket, and either merges, edits, or comments back with what's missing. On a quiet week that turnaround is measured in minutes. On a week where three unrelated services are having a bad day, proposed fixes sit in the queue longer, and that's the honest tradeoff of keeping a human in the loop: throughput is bounded by whoever's on call, not by how fast the agents can generate proposals.

Consent decides what agents learn, not what they see

The distinction we spent the most time getting right sits between visibility and use. An agent can see a piece of feedback the moment it lands in the queue, because triage and response have to happen regardless of whether the customer opted into anything. Seeing it and using it to improve the underlying engine are two different permissions, and the second one is opt-in, checked explicitly per customer, before any of that data touches training or fine-tuning data.

A support ticket gets triaged, analyzed, and answered without ever entering the improvement loop if consent isn't there. A customer who hasn't opted in still gets a human-reviewed response to whatever they reported; it just never becomes a training example for a future version of Sherlock. That line matters because it's easy to conflate the two when you're building one pipeline that touches both, which is why we made the check redundant rather than trust a single flag somewhere upstream. It costs an extra lookup on every item that enters the improvement path. We'd rather pay that cost on every single item than get it wrong once.

Where it still needs a human

The clearest limit shows up at document ingestion. A customer uploads a specification exported from DOORS or Polarion, or sometimes a Word or PDF file that's already been through an AI-based conversion tool before it reaches us, and the converted file is subtly broken: tables collapsed into unreadable text blocks, section numbering that no longer matches the visible headings, images swapped for alt-text placeholders that describe nothing about what was actually there. We've seen this often enough that it isn't really an edge case anymore. It's a recurring failure mode with a specific, identifiable cause.

When that happens, Sherlock can't make sense of the document, and neither can the internal triage agent looking at the resulting error. The ticket still gets flagged, but the proposed action an agent can offer is thin: ask the customer to re-export, or route it for manual review. Not a fix, because there's nothing in our code to fix. The actual damage happened in whatever tool mangled the file before it reached us. That's less a failure of the pipeline than an honest signal of where automated triage runs out of information to act on, and it's exactly the kind of ticket where a person picking up the thread with the customer directly does more good than another layer of automated analysis would.

Running this internally taught us less about agents than about sequencing. The parts that scale well are the parts with a checkable output: matching a new ticket against known issues, reading a stack trace, drafting a fix for a well-understood category of bug. The parts that still need a person are the parts with judgment baked in: whether a fix is right for right now, whether a response reads as reassuring or dismissive, whether a broken document is worth troubleshooting or just needs a different export. Building an agentic product taught us where that line sits for our customers. Running our own operations on the same architecture is where we found out whether we'd drawn it correctly.