RAG vs Fine-Tuning for Business AI
Choose between retrieval-augmented generation and fine-tuning with examples, evaluation methods and a pilot for business documents.

Quick answer
Start with retrieval when the assistant needs current facts from documents and you can control access to those documents. Consider fine-tuning when you have repeated examples of the behavior or output you need. Establish a prompt-only baseline and evaluate each approach against the same jobs.
## Three small tests for a warranty assistant
Create a fictional warranty policy in two editions. In the newer edition, basic accessories are covered for a different period and refunds require a distinct form. In the older edition, a similar form has a different identifier. Set aside three questions before building: one with an exact serial-like identifier, one that paraphrases the conditions, and one involving a product outside the policy. The prompt-only system can be expected to decline if it lacks evidence. Retrieval should find the effective edition and appropriate clause without being distracted by the older wording. Incorrect answers on the first and second questions can reveal retrieval gaps, while answering the last with a guess suggests the prompt or overall guardrails need attention.
Separate this from testing whether an answer follows a format. A support system might require status, cited clause and review flag as structured fields. Feed one known passage into the model for each formatting trial. Include mixed cases where the answer field is intentionally empty. If a clearer instruction and validation already give dependable output, this is your baseline. If repeated formatter failures remain with enough representative examples, tuning might be worth a controlled comparison. Teams sometimes use retrieval and tuning together, but their datasets and release schedules differ. Changing a policy should be testable on its own, even when unrelated response behavior is adjusted in a different release.
Identify the real gap
Example situation. Improve answers about a changing shipping policy.
If the missing information lives in a current internal document, give the model access to the appropriate verified passage first. Give the test enough room to show the original problem. Use the same entry point that someone with the issue would use. If several variables change between attempts, note which ones cannot be held constant and avoid assigning a cause yet. You can make a smaller experiment after getting that first picture.
A model recalls an attractive but outdated delivery rule from a training example. It helps to document the specific contrast as soon as you see it. Save an innocuous screenshot, a small sample or a short time record if appropriate, with identifying details removed when they are unnecessary. The next person will need a result that can be interpreted without guessing what happened immediately before you captured it.
Inspect the outcome. Set aside questions about rules that changed last month and score the effective date. If you need to hand this to support or a maintainer, include the expected outcome, the first unexpected observation and the last known working state.
A behavior problem such as emitting fixed fields deserves a separate baseline before selecting a solution. Write down whether the follow-up confirms your original explanation or exposes another factor. Confirmed evidence supports a specific change. If the behavior still varies without a known reason, retain the last working settings and focus the next test on that variation.
Prepare retrieval documents
Turn a ten-page operations manual into searchable passages. Decide what should stay intact while you try the change. For a small trial this might be the original input, a visible result another person already depends on, or access from a second supported device. Keep an appropriate copy or note of the earlier state. Tests that are easy to reverse make it possible to learn even when the first idea is wrong.
Preserve headings, version dates and ownership. Keep clauses together when their exceptions are on the next line.
Watch for this: The top search result contains the normal rule but loses the exception in the next chunk. Consider the exact point where you would stop and revert. If the issue appears, reproduce it without repeating a consequential action unnecessarily. A disposable example or read-only observation can give you the needed evidence. In a shared environment, tell others handling the test what state you expect to see before the next step.
Ask for a case that is only correct if the exception is visible in the retrieved passage. Verify the ordinary case that should have remained intact as well as the intended improvement. If one passes and the other fails, keep both observations and examine the relationship between them.
Try shorter chunks for lists and larger context around clauses with dependencies. Use the result to write a one-sentence acceptance condition. Make it specific enough that another person would know which setup to use and when to stop.
Measure search separately
Diagnose why a support answer about warranty length is wrong.
Log retrieval IDs, order and relevant passages for a synthetic sample. Annotate which passage should answer each question. Start with conditions you can readily repeat. Ask if your normal day includes a meaningful variation, such as a busy period, a second account or input from a different source. Select the variation based on your real use instead of trying every theoretical case.
A model is blamed for missing a rule that never reached its input. Record whether you found a recurring failure, a one-time event or an unresolved observation. The three have different implications. A repeating mismatch is easier to verify and to pass along for repair. A one-time interruption can still matter, but document the context and look for a separate observation before committing to an explanation.
Check whether the correct passage is retrieved before looking at wording or answer style. A simple check for the result usually beats a complicated proxy. If a reading looks surprising, return to the user-visible outcome and trace it back to the earliest relevant event you can observe. Comparing that observation to the expected rule may show exactly what additional information to collect.
Use terms customers actually use, including paraphrases that differ from the language in the rulebook. Use a representative size or repetition count when it affects the result. If a small test and a normal-sized workload behave differently, include both in the handoff so the earlier test can be understood in context.
Add access filters
Use this case to review the plan. Build a knowledge base with separate staff and external help documents.
Apply access constraints before retrieving context and check that excluded documents cannot appear in snippets or citations. Write down who owns the setting, file, service or decision before considering the trial finished. In a personal setup that may simply mean keeping the needed information somewhere you can find it later. In a shared setup it may determine whether the next person is able to continue during an interruption. Give that person enough context to determine if the result still applies.
Failure that matters: The answer does not quote restricted text, but the citation exposes its title to an unauthorized identity.
Run a test that asks for an internal policy from a public account and inspect retrieval results. A successful check has a concrete observable result. Confirm it from the normal user path, then put the test input and the observation in your notes. If cleanup is needed, remove test artifacts carefully and retain the benign evidence needed to repeat the test later.
Check revoked accounts and role changes as part of ordinary content refresh. Check any agreement about when to rerun the trial. An annual review can miss changes that happen at a specific release or after access is revoked. Conversely, some checks make sense on a fixed calendar because no individual event will remind you to try them. Choose the trigger that fits the use case and record a known starting point.
Design tuning examples
Try it when you need to: Teach a helper to return a consistent structured handoff.
Use examples with varied inputs, accepted schemas and expected handling of missing information.
Establish the expected result before starting. A useful way to make this concrete is to describe the one thing you plan to observe and the condition under which it should hold. Save a baseline using the same input or conditions you will use after the change. If another person helps, have them record the expected result independently. That can surface a difference in interpretation before you spend time debugging an apparent failure.
An easy mistake to miss: Training data includes only tidy requests, so the tuned model guesses when a field is missing.
Evaluate separate examples with ambiguity and contradictory input and score whether it asks for clarification. Preserve the comparison in a form you can come back to: write down the input, what you expected and what you actually saw. In a decision with several options, keep one record per option rather than changing the definition of success for each trial.
Preserve a baseline using clear instructions and validation to see what tuning actually adds. Think about whether this follow-up calls for revising the original rule or handling a separate case. If it is separate, keep the original test available and check that both cases still succeed. This is also useful when you need to explain why something that looked fine in a demonstration behaves differently in ordinary use.
Build a realistic benchmark
Evaluate finance-free examples from an internal support desk. This is a good place to watch where the result starts to differ from what you expect. Note the entry conditions and any settings or data that you have assumed. Take an initial reading, then do the work with those same conditions recorded. Avoid resetting everything at the first sign of trouble. A change that accidentally hides the symptom gives you less information about what caused it.
Mix straightforward questions, boundary cases, changing facts, and cases that require declining or escalating. Keep test items out of tuning examples.
Inspect this failure: Every test is an easy verbatim lookup, so the model appears production-ready prematurely. Make the first follow-up test small enough to repeat without changing unrelated parts of the setup. Start at the first boundary you can observe, then check the next one only if the initial result was right. Write down the observations in order. A later symptom can look like the root problem if you check the later step first and do not have an earlier reading.
Use different reviewers on the results and resolve disagreement against source rules. When the check is successful, repeat the ordinary action from its normal starting point. This catches results that only worked because the original test left a session, a temporary setting or a sample file in place. If somebody else follows the directions later, their starting conditions might be different.
Keep one metric for factual grounding, another for format compliance and another for correct escalation.
Budget for maintenance
Compare the ownership costs of a document index and a tuning workflow.
Begin with a simple case where you can tell if things went wrong even before looking at any reports. Keep a copy of the starting inputs and confirm the result that ought to follow from them. Ask what the user will see and what the underlying system ought to do. Those are worth checking separately when a screen, message or summary can still look correct even though a change did not reach the place you intended.
Estimate ingestion, metadata cleanup, access reviews, inference and re-evaluation over several releases.
The initial prototype price is low but no one owns expired documents or labels defective training examples. Include this possibility when you define a failure. Try a known good case beside the suspected one. If both fail in the same way, examine what they share. If only the particular case fails, save the difference that makes it a useful test.
How to verify: Create a small table of recurring workload and name someone for each input stream.
Test a deletion request and a policy update locally to see how quickly the system reflects a change. On the next pass, use a second representative input or condition to make sure the first result still applies. A narrow test can be decisive if it measures the thing you need. A wider sample is useful when your real work includes several distinct patterns and one result cannot stand for all of them.
Combine approaches deliberately
Improve a support system that needs both current policy and an exact reply format. Before putting time or money into this step, agree on a concrete criterion. Treat whatever is essential to complete the job as a mandatory requirement. Keep convenience, speed and ease of handoff in a separate part of your notes. This helps you avoid promoting something that looks easier in a trial but cannot complete a critical part of the task.
Start here. Test retrieval plus a structured prompt first. Add tuning only if repeated, measured behavior gaps remain.
Two components change together, making it hard to find the source of a new defect. Think through how a customer or colleague would discover this error. A clear, visible failure is easier to resolve than a result that looks successful and leaves a problem for the next person. If it would be hard to notice, add one direct inspection that examines the outcome itself. Record evidence for the result instead of relying entirely on a successful status label.
Confirm the result. Run side-by-side comparisons using the same retrieved passages and held-out questions. Work through this step before expanding the change. One realistic example is enough to reveal a category of failures that an ideal example misses. A trial is also a useful time to learn what to do when the result is ambiguous, especially if different people would have to resolve disagreement.
Keep an escape path when there is insufficient evidence to answer or the model cannot follow the format. Capture the decision and the reason in a short note so it can be revisited if the workload, policy or software changes.
A short verification checklist
| What to check | How to tell |
|---|---|
| Identify the real gap | Set aside questions about rules that changed last month and score the effective date. |
| Prepare retrieval documents | Ask for a case that is only correct if the exception is visible in the retrieved passage. |
| Measure search separately | Check whether the correct passage is retrieved before looking at wording or answer style. |
| Add access filters | Run a test that asks for an internal policy from a public account and inspect retrieval results. |
## Retrieval design choices you can test
A keyword search can be a strong baseline for exact product identifiers and standard section titles. Semantic search can help when customer wording differs from the manual, such as “returned parcel” where the rulebook says “return shipment.” A combined system might rank both types of results and remove duplicate passages. Compare which question families improve and whether other families regress. Store the applicable version and access scope with documents and test the exact update path after a policy change. Use a small annotated group of relevant passages rather than measuring only how many documents the index contains. A retrieval system that delivers the right passage alongside a more relevant-looking stale clause can still send an answer in the wrong direction. Evaluate what the model saw, then what it asserted.
Related reading
Official documentation describes current details and requirements. Related FixHubLab guide or tool covers an adjacent part of the workflow. Use the FixHubLab blog to find the rest of the guide series.
