Search
How the AI that answers as me on dansabin.com finds the passages it answers from, why it once couldn't walk through my resume, and why I fixed search before the button.
Why couldn't it walk through a resume it had?
Search looked for the words my resume, and my resume is about jobs, not about being a resume. So search came back with four passages from other files and no quote to use. With no quote, the writer isn't called, and the reply is a fixed one that declines rather than guesses.
When one question fails, what do you fix?
Every question about me goes through search before a word of the reply is written. A small model, the router, turns the question into search words. Search returns the passages nearest them by meaning, and the writer writes only from quotes found in those passages. A fix can go around search for the one question that failed, or into search itself, for every question that passes through it.
The resume it couldn't find
I was reading my own copy of the site before it opened, and pressed the button at the foot of my resume that asks the chat about it. It typed "Walk me through your resume." into the box, and I sent it. The row the chat shows above each reply gave what search had looked for, my resume, and where the passages came from: three other pages. The resume wasn't one of them. The reply was "I don't know, and I'd rather not guess."
Nothing logged what search had looked for or what came back, so a miss like this had been invisible outside a live chat. The first fix was one log line per search: its words, the file and score of each passage, and the files the quotes came from.
With that line, the diagnosis showed two places search fell short. The router turned the question into two words, my resume, and those match pages that talk about a resume. And search kept only the four passages nearest those words.
Search first
The diagnosis came with a stack of fixes. The biggest loaded the whole file behind the button and skipped search for it. I asked, are we making things very complex now? and chose to fix search first. My reasons, in my own words:
On the stack of fixes: I was worried about latency first. It was adding too many interim steps. The answer seemed to be focusing on the wrong problem and had the wrong constraint, so the exploration was going the wrong way.
On fixing search first: Search is a tool used in many different paths; it wasn't just a function of bringing the file in. That would be attacking a single symptom, and if the same question was asked not via the button the results would be different. It's fair that from the button the results could have been better, but the user would have seen behavior they didn't understand, and a product that behaves erratically, or is perceived that way, would erode trust.
On the reply itself: The design works. From a trust perspective, I don't know may be fair and at least understandable. If it invented stuff based on limited context the results would be much worse and damage user trust in the worst case, and in any case it could make me look like I didn't know what I was building.
So the button only types the question for the visitor, and the file isn't handed to the agent. A question someone clicks and one they type should behave the same way. We want the agent to actually do the work and not short-circuit any step. We want people to see the action.
What search does now
Before, each search asked the Knowledge Base for the four nearest passages by meaning and kept all four. Now the first search runs on the visitor's own words and on the router's, 25 candidates each. A reranker, a second model that scores how well each passage answers the visitor's words, reads all of them, up to 50, and the five best are kept. If the reranker fails, the five the Knowledge Base scored highest are kept, so the reply still has passages to work from.
What changed in the numbers
The measure is a fixed set of 56 test questions, run the same way before and after a change, with every reply checked: whether it answered, and whether each claim it made is in the passages search returned. The checks run automatically, and a model reads each claim against the passages. I ran each question twice with reranking off and twice with it on. That made 112 runs each way. Runs that passed every check went from about two in three to three in four: 68.8% to 75.0%. Runs where the reply didn't answer went from 7 of 112 to 2. Runs with a claim the passages didn't support went from 9 of 112 to 10.
Now, when the button types "Walk me through your resume.", the resume comes back from search, and the reply has quotes from it.
Want the details?
Ask the window anything this page skips.