Making knowledge findable with AI

In progress. Target: deflect 15–20% of incidents.

Role: Architect and team lead · Platform: ServiceNow · Status: in testing

AI knowledge rollout: before and the design in testing Before: knowledge was spread across the knowledge base, file servers and shared document sites, which meant extra tickets and help desk calls, extra agent cost and unsatisfactory answers. Options weighed: hosted versus local models, file types, re-ranking and out-of-the-box connectors. The design, now in testing: knowledge governance first with rules and fewer allowed sources, then a vector database built from approved sources, hybrid semantic and keyword search, and ServiceNow connected for retrieval. Target, not yet a result: deflect 15 to 20 percent of incidents. BEFORE Knowledge scattered across three places Knowledge base File servers Shared document sites Employee looking for an answer • Extra tickets and help desk calls • Extra agent cost • Answers that didn’t satisfy HOW Weighed the options before building Hosted vs local models File types Re-ranking Out-of-the-box connectors THE DESIGN In testing Governance first, then retrieval 1 Knowledge governance Rules, and fewer allowed sources 2 Vector database Built only from approved sources 3 Hybrid search Semantic plus keyword 4 Connected to ServiceNow Retrieves from the approved knowledge 15–20% of incidents deflected Target only. Real numbers aren’t in yet.

Context

Answers lived in three places: the knowledge base, file servers and shared document sites. There was no single place to look.

Problem

When people couldn't find an answer, they called the help desk. That meant extra tickets, extra agent time, and answers that weren't always right.

Decision

We tested a lot before settling on an approach: large hosted models against local ones, different file types, ways to re-rank search results, and out-of-the-box connectors. No single strategy won. A mix worked best for our use case, and we're still refining it.

The first fix wasn't AI at all. It was the same thinking as our developer guidelines: fix the bucket before adding water.

What we're building

  • Governance first: rules for how knowledge is written, and fewer places where it's allowed to live.
  • A vector database built from those approved sources.
  • Hybrid search that combines meaning-based and keyword matching.
  • ServiceNow connected to that search, so answers reach people where they already ask for help.

Status

In testing. We're aiming to deflect 15–20% of incidents. Real numbers aren't in yet, and I'll update this page when they are.

What I've learned so far

There's no single right way to do RAG; the best approach depends on the use case. The more time you spend planning the setup and defining the goal, the better the AI works for you.