Rendered at 12:11:47 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
agentdev001 12 hours ago [-]
Am I wrong to be somewhat peeved by the use of "RAG" in these contexts? I always read things like this, and wonder if instead the author should be saying "Semantic Retrieval" or something something Vector, etc. Retrieval augmented generation captures tool-use, and; semantic search of course is really just a tool under the hood.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
weitendorf 13 minutes ago [-]
To be honest as someone working in this space for the past two years, the problem with the “RAG” and semantic search community is it’s mostly vendors and solutions people selling simple, general stuff to product teams.
If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.
If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.
seanhunter 42 minutes ago [-]
Yes you are wrong. RAG means retrieval-augmented generation. If you’re generating something and that generation is augmented by some retrieval you’re doing RAG. The retrieval doesn’t need to be from a vector db or even based on semantic similarity.
jillesvangurp 3 hours ago [-]
RAG is a fancy acronym that basically boils down to: let's give ai agents the super power of information retrieval (aka. search) and "augment" the generation with a list of results by adding that to the context.
The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever.
This does not actually work all that well beyond really simple use cases. A well tuned traditional search engine can be surprisingly competitive. And I know people that do pretty complicated things with vector search that usually involve training their own models and spending a lot of effort on testing and validating those are any good.
I've been doing stuff with search for a bit over two decades. AI use cases makes information retrieval more relevant than ever. It's a key ingredient to answering questions for complex, proprietary data. And especially when that data is very complex and unstructured, naive approaches tend to have their limitations. In other words, it can pay off to to sit down and do it properly and think about things like data ingestion pipelines, transforming & enriching data, testing search quality, etc. Most of the success of a good search system usually boils down to getting your data right for indexing and optimizing it for how you are going to query your data.
The good news is that with large context windows, precision (best results are at the top) matters a bit less than recall (the search returns what you need when you search for it) these days. You can compensate for imprecise search by just fetching more results. As long as what you needed appears somewhere in the top 500 or so, you'll be fine. The flip side is of course that you end up adding a lot of noise to your context which might throw the LLM off and in general wastes a lot of tokens. That's why precision is still important.
What the article is proposing is post processing imprecise results to filter out the noise with an LLM to compensate for what is basically not a great search implementation. That can work of course (provided your recall doesn't suck). But it's going to add some cost and latency to searches. And usually, agents do multiple searches.
But if your search is so poor, why bother with vector search at all? Especially dense vector search at scale is not cheap. If you are going to fetch lots of results, just use some cheap lexical searches. Sparse vector search might be a good compromise (higher cost to index but similar performance to lexical search).
apwheele 24 minutes ago [-]
Processing 500 retrieved chunks here (probably with an additional LLM) will also add a bit of latency (not shown in these graphs!).
So for user facing apps, that scenario is probably not feasible (more like filter 10 chunks). Which as the parent of this comment suggests is fine to add in extra context given the current size of context windows.
red_hare 11 hours ago [-]
I agree. We're seeing more variants of "RAG" that aren't semantic at all (e.g. coding agents or simple memory systems that feed summary indexes directly into context).
I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.
10 hours ago [-]
tingletech 12 hours ago [-]
I read “RAG Context” as “the retrieved content injected into the context window”
0x696C6961 8 hours ago [-]
So when an agent does "cat file.txt" that's RAG to you?
kristiandupont 6 hours ago [-]
It is to me. And I agree that the term is losing value because it's becoming ubiquitous but it's the differentiation from the first versions of ChatGPT etc., which were purely user input -> LLM -> output driven.
EagnaIonat 7 hours ago [-]
The answer would be yes. It's about using stored knowledge to increase the accuracy of the answer and evidence surfacing.
It doesn't have to be a vector database.
Kapa is one of the few companies doing RAG right.
petesergeant 2 hours ago [-]
> Am I wrong to be somewhat peeved by the use of "RAG" in these contexts?
Probably? The ship has sailed on this one, like it has on "AI" or "crypto". RAG has almost exclusively been used to describe some variation on vector-search + bm25 pushed into the prompt for the last few years.
teaearlgraycold 5 hours ago [-]
RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hear people talk about RAG much at all. I suppose much of what people were trying to solve with document chunking/embedding pipelines has been solved with bigger models and tool calls. And along with that change in tooling we have left behind the term "RAG", which leaves it attached to the concept of those pipelines.
4 hours ago [-]
Folcon 4 hours ago [-]
I do wonder if the term will make a comeback, Retrieval Augmented Generation as a concept is a fairly fundamental idea, or maybe it's considered too generic which is why tool calling is used in favour if it?
The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval
Also is it just me that doesn't like this sort of wording?
> agent has to hold
I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context
grewil2 3 hours ago [-]
"Three knobs matter:"
I can't help reading articles with the radar on for signs of AI-generation nowadays. I have noticed that Claude sometimes uses the word knob for parameter, so here I get suspicious.
daveKoala 2 hours ago [-]
I am being constantly mocked for using there term 'knob fiddling', but I am a British child of the 70's :-)
petesergeant 2 hours ago [-]
The contribution here is the technique (which looks like a nice alternative to reranking) rather than the article or it being some kind of think piece, so does it really matter?
I was able to skim it and go from "lol they reinvented re-ranking" to "oh, that's something more interesting, I should try this one day", and the writing didn't get in the way.
alansaber 1 hours ago [-]
Would be cool to see a retrieval comparison to IE a claude code agent trace for the same query, even a cherry picked one.
Avery29 2 hours ago [-]
Bad or only loosely related context can make the final answer worse than having less context.
wolvoleo 6 hours ago [-]
I think this is really where energy saving comes into play. Context is so incredibly energy and processing time sensitive.
chimpanzee2 4 hours ago [-]
i wouldn't trust the small llm there.
it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!
8 hours ago [-]
StackOptimist 10 hours ago [-]
The risk in relevance-based pruning is the same one summarization has: it's tuned to drop whatever's rare, and in a lot of domains the rare chunk is the whole answer. A contraindication, an "except when," the single row that contradicts the other forty. Those score low against the query precisely because they're phrased in the exception's language, not the question's, so a similarity cut throws them out first.
The strongest version of the pro-pruning case is "prune by relevance to the query, not blindly," and I still don't trust it where being wrong is expensive, because relevance-to-the-query is exactly the signal that misses the buried caveat. If I ask whether drug A interacts with drug B and the warning is written under B's contraindications without naming A, semantic pruning helps me lose it.
What's worked better for me is pruning by structure instead of by score: keep whole records or whole sections as units and drop by type (this entire category of source isn't relevant) rather than by ranking individual sentences. You give up some of the token savings of aggressive sentence-level pruning, but you stop silently deleting the one clause the answer depended on. If you must prune fine-grained, the honest test is recall on a set where the answer hinges on a rare chunk, not average-case QA where there's lots of redundancy to spare.
lbaltensp 4 hours ago [-]
"relevance-to-the-query is exactly the signal that misses the buried caveat."
You are exactly hitting the nail on the head. The underlaying problem of a pruner is to determine document relevance to a user query. And document relevance comes in different flavours:
1. Direct relevance: a document can be relevant to the user query because it directly answers it.
2. Subquery-Conditional Relevance: a document can be relevant because it answers a decomposed subquery of the user query.
3. Document-Interdependent Relevance: a document is relevant only because another document provides bridge context, domain knowledge, disambiguation, a definition or a constraint.
This means the right question is therefore not "is this document relevant to the original query in isolation?", but:
Given the original user query, the decomposed sub-queries, and all other retrieved documents, does this document contribute to form a complete and sufficient set of documents for answering the original query?
This is exactly why we tuned our pruner at kapa based on recall against this Document Relevance Model.
synapsehire 4 hours ago [-]
[flagged]
tauridev 2 hours ago [-]
[flagged]
tangsoupgallery 10 hours ago [-]
[flagged]
Technical_Plant 12 hours ago [-]
[flagged]
esafak 10 hours ago [-]
tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
siquick 8 hours ago [-]
Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.
petesergeant 2 hours ago [-]
My dissertation used a Likert scale with ChatGPT-3 era LLMs, and it was both internally consistent over 5-6 runs on a given statement, and consistent with human raters. I don't think you can bat it away as simply "doesn't work"
esafak 7 hours ago [-]
That's what the rubric is for; it reduces the problem to NLP, which is the LLM's forte. The more objective you can make your rubric, the better.
rooftopzen 14 hours ago [-]
Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks).
"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.
Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.
wolvoleo 6 hours ago [-]
"RAG is dead" was declared because of ever expanding context windows.
However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.
Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).
Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.
To make an anology, in my mind, this is akin to saying "fuel air mixture system" when referring to direct fuel injection specifically, when of course, a carburetor also lives in that category.
If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.
If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.
The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever.
This does not actually work all that well beyond really simple use cases. A well tuned traditional search engine can be surprisingly competitive. And I know people that do pretty complicated things with vector search that usually involve training their own models and spending a lot of effort on testing and validating those are any good.
I've been doing stuff with search for a bit over two decades. AI use cases makes information retrieval more relevant than ever. It's a key ingredient to answering questions for complex, proprietary data. And especially when that data is very complex and unstructured, naive approaches tend to have their limitations. In other words, it can pay off to to sit down and do it properly and think about things like data ingestion pipelines, transforming & enriching data, testing search quality, etc. Most of the success of a good search system usually boils down to getting your data right for indexing and optimizing it for how you are going to query your data.
The good news is that with large context windows, precision (best results are at the top) matters a bit less than recall (the search returns what you need when you search for it) these days. You can compensate for imprecise search by just fetching more results. As long as what you needed appears somewhere in the top 500 or so, you'll be fine. The flip side is of course that you end up adding a lot of noise to your context which might throw the LLM off and in general wastes a lot of tokens. That's why precision is still important.
What the article is proposing is post processing imprecise results to filter out the noise with an LLM to compensate for what is basically not a great search implementation. That can work of course (provided your recall doesn't suck). But it's going to add some cost and latency to searches. And usually, agents do multiple searches.
But if your search is so poor, why bother with vector search at all? Especially dense vector search at scale is not cheap. If you are going to fetch lots of results, just use some cheap lexical searches. Sparse vector search might be a good compromise (higher cost to index but similar performance to lexical search).
So for user facing apps, that scenario is probably not feasible (more like filter 10 chunks). Which as the parent of this comment suggests is fine to add in extra context given the current size of context windows.
I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.
It doesn't have to be a vector database.
Kapa is one of the few companies doing RAG right.
Probably? The ship has sailed on this one, like it has on "AI" or "crypto". RAG has almost exclusively been used to describe some variation on vector-search + bm25 pushed into the prompt for the last few years.
The problem with tool calling is it's too generic in my mind, maybe RAG will make a return when we get around to having different flavours of it, digging into the rich vein that is Information Retrieval
Also is it just me that doesn't like this sort of wording?
> agent has to hold
I find it very generic, I'd much prefer process or recall or any term that indicates what the agent is doing with tokens in that context
I can't help reading articles with the radar on for signs of AI-generation nowadays. I have noticed that Claude sometimes uses the word knob for parameter, so here I get suspicious.
I was able to skim it and go from "lol they reinvented re-ranking" to "oh, that's something more interesting, I should try this one day", and the writing didn't get in the way.
it will be an intellectual bottleneck when it comes to processing the very information your just arduously fished out of the ocean!!
You are exactly hitting the nail on the head. The underlaying problem of a pruner is to determine document relevance to a user query. And document relevance comes in different flavours:
1. Direct relevance: a document can be relevant to the user query because it directly answers it. 2. Subquery-Conditional Relevance: a document can be relevant because it answers a decomposed subquery of the user query. 3. Document-Interdependent Relevance: a document is relevant only because another document provides bridge context, domain knowledge, disambiguation, a definition or a constraint.
This means the right question is therefore not "is this document relevant to the original query in isolation?", but:
Given the original user query, the decomposed sub-queries, and all other retrieved documents, does this document contribute to form a complete and sufficient set of documents for answering the original query?
This is exactly why we tuned our pruner at kapa based on recall against this Document Relevance Model.
"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.
Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.
However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.
Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).
Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.