Skip to content
NTPU AI Learn

AI 103 · Module 3

RAG & Knowledge Retrieval

When an agent does not know the answer, should it guess or go and look?

Start Section 1
  • Duration55 min
  • LevelCore
  • PrerequisiteAI 102 recommended, plus Modules 1 and 2 of this course
0 / 6 completed
Sync my progressJoin a session to keep progress across devices and share it with your instructor

Info

After you join, your completed lesson numbers are sent to your instructor together with your learner code, so they can see your progress. Use the codes your instructor gave you, and do not enter your name, company secrets or other personal data. Progress is kept with the rest of that session's data.

Learning objectives

  • Explain the difference between internal knowledge and external knowledge, and why RAG (Retrieval-Augmented Generation) is used
  • Describe the RAG flow: Question, Retrieve Relevant Knowledge, Build Context, Generate Answer, Verify Evidence
  • Decide whether a piece of information should be remembered, retrieved, or accessed through a tool
  • Explain grounding, and what an agent should do when retrieval finds no reliable evidence

The Problem

8 min

Guess, or Go and Look?

A colleague asks the agent, "How many days of parental leave do employees get?" The agent has a fluent answer ready. But where does it come from?

Work scenario

The agent answers "usually 8 weeks" from what it learned in training. Your organization's current policy, revised this year, says something different. The answer sounded right, and nobody could see where it came from.

  • The model's internal knowledge stops at some point in time, so it may be out of date
  • It knows general patterns, not your organization's own documents
  • It usually cannot show which document its answer is based on
  • When it does not know, it may still produce a confident-sounding answer

Warning

For questions that depend on an organization's own or current information, the safe behavior is to look up a trusted source, not to guess. That behavior has to be designed in.

Key takeaway: An agent that guesses sounds the same as one that knows. Design it to look up trusted sources and to show what it relied on.

RAG and Knowledge Sources

9 min

Internal versus External Knowledge, and Why RAG

Internal Knowledge (內部知識) is what the model absorbed during training. External Knowledge (外部知識) lives outside the model: documents, databases, websites and enterprise knowledge bases. Retrieval-Augmented Generation, or RAG (檢索增強生成), lets the agent fetch relevant external knowledge at question time and answer from it.

Internal Knowledge

  • Learned in training, fixed at a point in time
  • Broad and general
  • No specific source to show
  • Cannot include your private or newer documents

External Knowledge (via RAG)

  • Stays at its source and can be kept up to date
  • Specific to your organization or field
  • Can point back to a document and version
  • Access can be controlled at the source
Common knowledge sources (illustrative examples)
SourceExampleAsk before using it
DocumentsThe employee handbook, product manuals, SOPsWhich version is current, and who owns it?
DatabasesA course roster or a supplier registerWho may see which records?
WebsitesA public regulator's pageIs it an authoritative source, and how recent?
Enterprise Knowledge BasesA department wikiIs it maintained, or has it drifted?
Check yourself

An agent must answer a question about your organization's current travel reimbursement rules, which were updated last month. What is the best approach?

An agent must answer a question about your organization's current travel reimbursement rules, which were updated last month. What is the best approach?

Key takeaway: Internal knowledge is general and fixed in time. External knowledge, fetched through RAG, is specific, current and traceable.

10 min

The RAG Flow and Grounding

RAG is a short pipeline. Each step is a place where quality can be won or lost.

The RAG flow
  1. Question: the user asks something
  2. Retrieve Relevant Knowledge: search the sources and find the passages that matter
  3. Build Context: place the retrieved passages, with their sources, into the context
  4. Generate Answer: the AI answers from that context
  5. Verify Evidence: check that the answer is supported by the passages, then a person reviews what matters
  1. Search and Retrieval
    Finding the right passage is the hardest part. If the wrong passages come back, even a good model will answer wrongly. Modules 4 and 5 show how to prepare knowledge so this works.
  2. Generation
    The model writes an answer from the retrieved context. It should stay inside what the evidence says.
  3. Grounding (證據依據)
    A grounded answer is tied to evidence: it cites the source, and each claim can be checked against it.
Ungrounded versus grounded (illustrative example)
AnswerWhat it looks likeCan you check it?
Ungrounded"Employees get 8 weeks of parental leave."No source; it is a guess
Grounded"Employees get 12 weeks of parental leave (Employee Handbook 2026, section 4.2, retrieved today)."Yes; open section 4.2 and compare
Grounded, but nothing found"I could not find a parental leave section in the sources I searched. Please check with HR."Yes; it admits the gap instead of guessing
Check yourself

The agent retrieves no relevant passage for a policy question. Which response best matches grounded behavior?

The agent retrieves no relevant passage for a policy question. Which response best matches grounded behavior?

Key takeaway: RAG is Question, Retrieve, Build Context, Generate, Verify. A grounded answer shows its evidence and admits when there is none.

Remember, Retrieve or Tool

10 min

Remember, Retrieve, or Use a Tool?

There are three ways for information to reach an agent beyond the current context. Choosing the wrong one is one of the most common design mistakes.

Remember versus Retrieve versus Tool
InformationMemoryRAGTool / API
User preferenceYes
Employee handbookYes
Current inventoryYes
Previous task resultYes
Product manualYes

How to tell them apart

  • Remember: small, stable, about the user or ongoing work, and appropriate to keep
  • Retrieve: large or changing documents that belong to a source; fetch the relevant part when needed
  • Tool: live, exact or changing values; ask the system that owns the data at the moment of need

Questions that help

  • Does it change while the agent is idle? Do not copy it into memory.
  • Is it a whole document? Retrieve the relevant part, not all of it.
  • Does it have to be exactly right now? Use a tool that reads the source.
  • Is it only for this step? Put it in current context.

Info

How to connect an agent to tools and live data safely is the subject of AI 201. Here you only need to recognize when a tool, and not memory or retrieval, is the right source.
Check yourself

An agent needs to tell a customer how many units of a product are in stock right now. Where should the number come from?

An agent needs to tell a customer how many units of a product are in stock right now. Where should the number come from?

Key takeaway: Remember small and stable, retrieve large or changing documents, use a tool for live and exact values.

Practice

10 min

Practice: Context, Memory, RAG or Tool?

Below are 8 illustrative pieces of information an agent might need. For each, choose one: Context, Memory, RAG or Tool. Then discuss with a peer or the instructor. Some cases can reasonably be read two ways; what matters is that you can explain your reasoning. Do not enter real company secrets, client personal data or other unauthorized sensitive information in this exercise.

8 pieces of information (decide for yourself)
CaseInformationThink about
1The user's preferred summary lengthSmall, stable, and about the user
2A 40-page employee handbookShould the agent carry all of it every time?
3The number of units of item A-17 in the warehouse right nowDoes it change while the agent is idle?
4The result of the supplier comparison the agent produced yesterdayReusable, but is it still true?
5The paragraph the user has just pasted and asked to shortenIs it only for this step?
6The product manual for model X200A large document with an owner
7Today's exchange rateMust it be exact, and who owns it?
8The draft the user is editing at this momentNeeded for the current step only
  • Context: needed for this step only
  • Memory: small, stable, appropriate to keep
  • RAG: large or changing documents, retrieved when needed
  • Tool: live or exact values from the system that owns them
Check yourself

Case 4: the agent produced a supplier comparison yesterday and stored it as a task memory. Today the user asks it to reuse the result. What should the agent do?

Case 4: the agent produced a supplier comparison yesterday and stored it as a task memory. Today the user asks it to reuse the result. What should the agent do?

Key takeaway: For each piece of information, ask: is it for this step, worth keeping, a document to retrieve, or a live value to look up?

Recap

8 min

Recap: From Memory to Knowledge

Done

When the agent does not know, does it guess, or does it retrieve trusted evidence and show what it relied on?

Module 2: Memory

  • What the agent keeps over time
  • A lifecycle and a policy
  • Small, stable and appropriate information

Module 3: Knowledge

  • What the agent looks up when needed
  • The RAG flow and grounding
  • Documents to retrieve, live values to fetch through tools

Human, AI, Human: people choose the trusted sources, the agent retrieves and answers from them, and people verify the evidence for anything that matters. Retrieval only works if the knowledge was prepared well. Next, you will look at how to break documents into meaningful pieces and describe them so the right passage can be found.

Key takeaway: Remember what is small and stable, retrieve what is large or changing, use tools for live values, and always show the evidence.

Save as PDF

What next?