Steer AGI

In Cache-Augmented Generation, responses are enhanced by using a local cache to store previously generated or retrieved results, avoiding redundant computations.

• Query Matching: Checks if a similar query exists in the cache.
• Cache Hit: Uses the cached response directly or combines it with retrieved results.
• Cache Miss: Executes a full RAG pipeline and updates the cache.

• Reduced Latency: Avoids redundant computations for repeated queries.
• Efficiency: Improves response times in high-throughput systems.
• Applications: Customer support, FAQs, and other repetitive-query environments.

Explore the difference between RAG and CAG

Watch video: RAG vs. CAG: Solving Knowledge Gaps in AI Models by Martin Keen, IBM

GraphRAG leverages graph-based representations of knowledge to improve retrieval and generation. Instead of linear or flat knowledge bases, it works with knowledge graphs where entities and their relationships are explicitly defined.

• Graph-Based Retrieval: Retrieves relevant nodes and subgraphs.
• Contextual Understanding: Generates responses by understanding relationships between entities.
• Integration with Generative Models: Provides structured graph-based context for generation.

• Semantic Understanding: Captures richer contextual relationships.
• Applications: Complex reasoning tasks, multi-hop question answering, biomedical research.

GraphRAG is a powerful approach that combines the benefits of graph-based representations with the generative capabilities of AI models. It allows for more complex and nuanced understanding of data, making it particularly useful for tasks that require reasoning and inference. Read paper From Local to Global: A Graph RAG Approach to Query-Focused Summarization and Microsoft Research Blog GraphRAG: Unlocking LLM discovery on narrative private data

VectorRAG combines retrieval-augmented generation with vector-based search powered by vector embeddings. This method excels in semantic search and retrieval.

• Embedding Generation: Transforms queries and documents into vector embeddings.
• Vector Search: Uses similarity metrics (like cosine similarity) to retrieve the most relevant results.
• Generative Augmentation: Uses retrieved documents for response generation.

• Semantic Precision: Captures intent and meaning beyond keyword matching.
• Applications: Personalized recommendations, document search, and multilingual applications.

HybridRAG combines dense retrieval (vector-based search) and sparse retrieval (traditional keyword-based search like BM25) for a more robust retrieval system.

• Sparse Retrieval: Matches keywords and retrieves documents using traditional methods.
• Dense Retrieval: Uses vector embeddings for semantic retrieval.
• Hybrid Ranking: Combines results from both approaches and ranks them for relevance.

• Balanced Approach: Combines strengths of keyword and semantic search.
• Resiliency: Handles diverse queries effectively.
• Applications: Enterprise search, e-commerce, legal document search.

Retrieval-Augmented Generation (RAG) comes in several specialized variants, each optimized for different use cases. Standard RAG combines retrieval and generation for general question answering, while CacheRAG prioritizes efficiency for high-throughput systems. GraphRAG leverages knowledge graphs for complex reasoning tasks, and VectorRAG employs vector-based semantic search for personalized experiences. HybridRAG combines multiple retrieval techniques for enterprise applications.

RAG metrics, short for Retrieval-Augmented Generation metrics, are evaluation methods used in systems that combine retrieval-based approaches with generative models to assess the quality and effectiveness of generated responses. These metrics are commonly applied in open-domain question answering and similar applications.

• Retriever: Finds relevant documents or passages from a knowledge base. Evaluated using metrics like Recall@K and MRR.
• Generator: Produces responses using retrieved information as context. Evaluated for fluency, informativeness, and accuracy.

• Recall@K: Measures the relevance of retrieved documents in the top K results.
• Mean Reciprocal Rank (MRR): Evaluates how quickly the first relevant document is retrieved.
• BLEU (Bilingual Evaluation Understudy): Measures the overlap of n-grams between the generated text and a reference answer.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Focuses on recall by measuring how much of the reference text is captured in the generated text.
• Factual Consistency: Ensures that the generated response is factually consistent with the retrieved content.

1. Recall@K

Measures how many of the top K retrieved documents contain relevant information.

Formula:

Example:

If K = 2 and the relevant document appears in the top 2 results:

2. Mean Reciprocal Rank (MRR)

Evaluates how quickly the first relevant document is retrieved.

Formula:

Example:

3. BLEU (Bilingual Evaluation Understudy)

Measures the overlap of n-grams between the generated text and a reference answer.

Formula:

4. ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

Focuses on recall by measuring how much of the reference text is captured in the generated text.

5. Factual Consistency

Ensures that the generated response is factually consistent with the retrieved content. This can be evaluated manually or using automated tools.

Python Code for Recall@K and MRR

def recall_at_k(retrieved_docs, relevant_docs, k):
            retrieved_at_k = retrieved_docs[:k]
            return sum(1 for doc in retrieved_at_k if doc in relevant_docs) / len(relevant_docs)

            def mean_reciprocal_rank(retrieved_docs, relevant_docs_list):
                reciprocal_ranks = []
                for relevant_docs in relevant_docs_list:
                    for rank, doc in enumerate(retrieved_docs, start=1):
                        if doc in relevant_docs:
                            reciprocal_ranks.append(1 / rank)
                            break
                    else:
                        reciprocal_ranks.append(0)
                return sum(reciprocal_ranks) / len(relevant_docs_list)

            retrieved_docs = ["doc1", "doc2", "doc3"]
            relevant_docs_list = [["doc2"], ["doc3"]]

            print("Recall@2:", recall_at_k(retrieved_docs, relevant_docs_list[0], 2))
            print("MRR:", mean_reciprocal_rank(retrieved_docs, relevant_docs_list))

Python Code for BLEU

from nltk.translate.bleu_score import sentence_bleu

            reference = [["OpenAI", "was", "founded", "by", "Elon", "Musk", "and", "Sam", "Altman"]]
            candidate = ["OpenAI", "was", "started", "by", "Sam", "Altman", "and", "Elon", "Musk"]

            score = sentence_bleu(reference, candidate)
            print("BLEU Score:", score)

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external data to improve accuracy and relevance. Microsoft's research categorizes RAG tasks into four levels, each requiring progressively complex reasoning and data integration:

This hierarchical framework aids in selecting appropriate RAG architectures tailored to specific use cases, ensuring alignment with task demands and enhancing the system's effectiveness.

• Simple RAG: A foundational method where the system retrieves relevant documents and feeds them to the language model to generate responses, serving as a baseline for more advanced techniques.
• Fusion Retrieval: Combines multiple retrieval methods (such as keyword search and vector-based search) to aggregate a more accurate set of documents.
• Query Transformation: Modifies or expands the user query to better capture the intended meaning, improving the chances of retrieving all relevant information.
• Semantic Chunking: Divides documents into meaningful segments based on semantic coherence rather than fixed lengths, ensuring that each chunk preserves its contextual relevance.
• Choose Chunk Size: Determines the optimal fixed size for breaking text into chunks, balancing context preservation with retrieval efficiency.
• Context Compression: Reduces the volume of retrieved content by summarizing or compressing text while retaining essential information relevant to the query.
• Context Enrichment: Enhances retrieved data by appending additional contextual details, thereby supplying the language model with richer and more informative content.
• Multi-faceted Filtering: Applies various filtering criteria—such as metadata, relevance scores, and content checks—to ensure that only the most pertinent and high-quality documents are used.
• Intelligent Reranking: Uses advanced scoring mechanisms (often LLM-based) to reorder retrieved documents, ensuring that the most relevant information is prioritized for generation.
• Explainable Retrieval: Provides transparency by explaining why certain documents were selected, which helps build trust and allows users to verify the source of the information.
• Hierarchical Indicies: Organizes data into multi-level indices to enable efficient navigation and retrieval at different granularity levels, enhancing both speed and accuracy.
• Retrieval w/Feedback: Incorporates feedback from users or the system itself to iteratively refine the retrieval process, thereby improving future results.
• Adaptive Retrieval: Dynamically adjusts the retrieval strategy based on the specific query and context, ensuring that the most relevant data is fetched for each situation.
• Iterative Retrieval: Executes multiple rounds of retrieval, where each subsequent round refines and expands upon the previous one to enhance the overall quality of the results.
• Ensemble Retrieval: Combines outputs from several retrieval models or approaches using techniques like voting or weighting to generate a more robust and reliable set of documents.
• Graph RAG: Integrates structured data from knowledge graphs, linking related concepts to provide deeper contextual relevance and richer information.
• Multi-Modal Retrieval: Extends retrieval capabilities beyond text to include other data types (such as images, audio, or video), offering a broader data context for response generation.
• Hypothetical Questions (HyDE): Generates hypothetical questions or answers based on the original query to better align retrieval with the user's intent, thereby enhancing the relevance of the retrieved content.
• RAPTOR: Employs a recursive, tree-based method to process and organize retrieved documents, enabling hierarchical summarization and improved context for complex queries.
• Self RAG: Integrates retrieval and generation in a feedback loop, allowing the model to self-assess and dynamically adjust its retrieval strategy to optimize response quality.
• Corrective RAG: Incorporates additional verification and correction steps—such as cross-checking with external sources—to minimize hallucinations and ensure that responses are factually accurate.
• RAG Architectures: A general category where a retrieval model and a generative model are integrated. The retriever fetches documents from an external knowledge base, and the generator produces contextually rich responses. An optional feedback loop can further fine-tune both components.
• Standard RAG: The foundational architecture that retrieves relevant documents from a knowledge base and directly feeds them into a large language model to generate responses. Simple and effective for basic tasks, it may face challenges with complex queries.
• Self-Reflective RAG: Enhances standard RAG by incorporating metacognitive evaluation to refine document selection and continuously assess response quality. Particularly beneficial for high-stakes applications in legal or medical domains.
• Corrective RAG (Architecture): Focuses on quality control by using a Natural Language Inference (NLI) model to validate and classify retrieved documents as correct, ambiguous, or incorrect. This validation layer helps reduce factual errors.
• Speculative RAG: Utilizes a two-tier approach: a smaller, specialized LLM generates multiple draft answers in parallel, and the primary LLM reviews and selects the most accurate response. This design balances speed with high accuracy.
• Cache-Augmented Generation: Enhances response generation by storing previously generated or retrieved outputs in a local cache. This avoids redundant computations for repeated queries, thereby reducing latency and increasing efficiency.
• VectorRAG: Combines retrieval-augmented generation with vector-based search powered by vector embeddings. It excels at semantic search by capturing the underlying meaning of text, making it ideal for personalized recommendations and multilingual applications.
• HybridRAG: Merges dense (vector-based) and sparse (keyword-based) retrieval techniques to balance semantic precision with traditional matching methods, resulting in a robust system that handles diverse queries effectively.
• Recall@K (Metric): Evaluates the fraction of relevant documents that appear within the top K retrieved results. It measures the effectiveness of the retrieval component in capturing pertinent information early in the ranking.
• Mean Reciprocal Rank (MRR) (Metric): Calculates the average reciprocal rank of the first relevant document across queries. A higher MRR indicates that the system consistently retrieves relevant content at the top of the results.
• BLEU Score (Metric): Assesses the precision of n-grams in the generated text relative to a reference answer, indicating how closely the output matches an ideal response.
• ROUGE Score (Metric): Focuses on recall by measuring the overlap of n-grams between the generated text and reference texts, reflecting how much of the reference content is captured in the output.
• Factual Consistency (Metric): Measures whether the generated response is factually aligned with the retrieved content, ensuring that the output is accurate, verifiable, and free from hallucinations.

Video: 4 Methods of Prompt Engineering by IBM Technology

• RAG Techniques - A collection of RAG techniques and implementations
• Sophisticated Controllable Agent for Complex RAG Tasks - advanced Retrieval-Augmented Generation (RAG) solution for complex question answering. It uses sophisticated graph based algorithm to handle the tasks.
• Advanced RAG Solution Accelerator - Microsoft's advanced RAG solution implementation guide
• Building a Multi-Modal RAG System - Guide to building multi-modal RAG systems with Azure OpenAI and Langchain
• Awesome RAG - A curated list of awesome RAG resources and tools
• Graph RAG - Exploring RAG and GraphRAG: Understanding when and how to use both
• Google Vertex AI RAG Engine - Weaviate on Vertex AI RAG Engine: Building RAG Applications on Google Cloud
• Fed-RAG - Simplified RAG fine-tuning across centralized or federated architectures
• A Practical Guide to Building a RAG-Powered Chatbot - Learn how to create a fast, cost-efficient chatbot using Retrieval-Augmented Generation.
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context - Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
• Larimar: Large Language Models with Episodic Memory Control - Efficient and accurate updating of knowledge stored in Large Language Models (LLMs) is one of the most pressing research challenges today. This paper presents Larimar - a novel, brain-inspired architecture for enhancing LLMs with a distributed episodic memory. Larimar's memory allows for dynamic, one-shot updates of knowledge without the need for computationally expensive re-training or fine-tuning. Experimental results on multiple fact editing benchmarks demonstrate that Larimar attains accuracy comparable to most competitive baselines, even in the challenging sequential editing setup, but also excels in speed - yielding speed-ups of 8-10x depending on the base LLM - as well as flexibility due to the proposed architecture being simple, LLM-agnostic, and hence general. We further provide mechanisms for selective fact forgetting, information leakage prevention, and input context length generalization with Larimar and show their effectiveness.
• From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs - Memory is the process of encoding, storing, and retrieving information, allowing humans to retain experiences, knowledge, skills, and facts over time, and serving as the foundation for growth and effective interaction with the world. It plays a crucial role in shaping our identity, making decisions, learning from past experiences, building relationships, and adapting to changes. In the era of large language models (LLMs), memory refers to the ability of an AI system to retain, recall, and use information from past interactions to improve future responses and interactions. Although previous research and reviews have provided detailed descriptions of memory mechanisms, there is still a lack of a systematic review that summarizes and analyzes the relationship between the memory of LLM-driven AI systems and human memory, as well as how we can be inspired by human memory to construct more powerful memory systems. To achieve this, in this paper, we propose a survey on the memory of LLM-driven AI systems. In particular, we first conduct a detailed analysis of the categories of human memory and relate them to the memory of AI systems. Second, we systematically organize existing memory-related work and propose a categorization method based on three dimensions (object, form, and time) and eight quadrants. Finally, we illustrate some open problems regarding the memory of current AI systems and outline possible future directions for memory in the era of large language models.
• Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions - Memory is a fundamental component of AI systems, underpinning large language models (LLMs)-based agents. While prior surveys have focused on memory applications with LLMs (e.g., enabling personalized memory in conversational agents), they often overlook the atomic operations that underlie memory dynamics. In this survey, we first categorize memory representations into parametric and contextual forms, and then introduce six fundamental memory operations: Consolidation, Updating, Indexing, Forgetting, Retrieval, and Compression. We map these operations to the most relevant research topics across long-term, long-context, parametric modification, and multi-source memory. By reframing memory systems through the lens of atomic operations and representation types, this survey provides a structured and dynamic perspective on research, benchmark datasets, and tools related to memory in AI, clarifying the functional interplay in LLMs based agents while outlining promising directions for future research
• Neural Turing Machines - We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.
• When Context Leads but Parametric Memory Follows in Large Language Models - Large language models (LLMs) have demonstrated remarkable progress in leveraging diverse knowledge sources. This study investigates how nine widely used LLMs allocate knowledge between local context and global parameters when answering open-ended questions in knowledge-consistent scenarios. We introduce a novel dataset, WikiAtomic, and systematically vary context sizes to analyze how LLMs prioritize and utilize the provided information and their parametric knowledge in knowledge-consistent scenarios. Additionally, we also study their tendency to hallucinate under varying context sizes. Our findings reveal consistent patterns across models, including a consistent reliance on both contextual (around 70%) and parametric (around 30%) knowledge, and a decrease in hallucinations with increasing context. These insights highlight the importance of more effective context organization and developing models that use input more deterministically for robust performance.
• A Survey on the Memory Mechanism of Large Language Model based Agents - Large language model (LLM) based agents have recently attracted much attention from the research and industry communities. Compared with original LLMs, LLM-based agents are featured in their self-evolving capability, which is the basis for solving real-world problems that need long-term and complex agent-environment interactions. The key component to support agent-environment interactions is the memory of the agents. While previous studies have proposed many promising memory mechanisms, they are scattered in different papers, and there lacks a systematical review to summarize and compare these works from a holistic perspective, failing to abstract common and effective designing patterns for inspiring future studies. To bridge this gap, in this paper, we propose a survey on the memory mechanism of LLM-based agents. In specific, we first discuss ''what is'' and ''why do we need'' the memory in LLM-based agents. Then, we systematically review previous studies on how to design and evaluate the memory module. In addition, we also present many agent applications, where the memory module plays an important role. At last, we analyze the limitations of existing work and show important future directions.