From Knowledge Discovery to Knowledge Governance: What I Learned About Maintaining GraphRAG

Exploring ontology evolution, deterministic graph updates, AI-assisted relationship discovery, and the challenges of maintaining knowledge over time.

Don’t want to read? Just listen to the podcast on YouTube.

1. Introduction: Building a Knowledge Graph Is Only the Beginning
2. Two Different Perspectives on GraphRAG
   1. Explorative GraphRAG: Knowledge Discovery
   2. Enterprise GraphRAG: Knowledge Governance
3. Why I Think an Ontology Should Come First
4. My New Local GraphRAG Experiment
   1. Why deterministic rules?
5. Twelve Labs: From Initial Graph to Ontology Evolution
6. Lab 9: Breaking Ontology Changes
7. Lab 10: Understanding the Impact of an Ontology Change
8. Lab 11: Can AI Discover Missing Relationships?
9. Lab 12: Why Regression Testing Matters
10. Another Important Finding: Retrieval Is Not the Same as a Correct Answer
11. What Would Be Required for Enterprise GraphRAG?
    1. Ontology Governance
    2. AI-Assisted Semantic Analysis
    3. Deterministic Graph Maintenance
12. What I Learned
13. Final Thoughts: From Discovery to Governance
14. GitHub Repository and Hands-on Labs
15. References

1. Introduction: Building a Knowledge Graph Is Only the Beginning

Some time ago, I published a blog post about building a local GraphRAG solution using Neo4j, LangGraph, and watsonx.ai.

You can find the original post here [1]:

How to Build a Knowledge Graph RAG Agent Locally with Neo4j, LangGraph, and watsonx.ai

At that time, my main interest was knowledge discovery.

I wanted to understand how AI could extract entities and relationships from documents, build a Knowledge Graph, and use this information to answer questions.

The idea was straightforward:

1. Take existing documents.
2. Extract entities and relationships using AI.
3. Build a Knowledge Graph.
4. Search and explore the relationships.
5. Use GraphRAG to answer questions.

This approach is useful when we want to discover connections that are not fully known beforehand. But recently, while preparing another demonstration, I started thinking about a different problem.

What happens when the information changes?

For example:

• A product receives a new technical specification.
• A regulation changes.
• A previously valid relationship is no longer valid.
• New information creates relationships that did not exist before.
• The ontology itself needs to change.

This sounds like a normal data update at first.

But when I thought about the dependencies between entities, I realized that maintaining a Knowledge Graph could be much more complicated than creating one.

That was the starting point for my new experiment.

2. Two Different Perspectives on GraphRAG

The first important realization was that we need to distinguish between two different scenarios.

Explorative GraphRAG: Knowledge Discovery

In an exploratory scenario, we start with documents and try to understand what is inside them. AI can help us identify entities, relationships, and possible graph structures. We may not know the complete ontology beforehand. The ontology can be proposed and refined during the exploration process.

The important question is:

What relationships can we discover from our information?

This was the main direction of my previous GraphRAG example.

Of course, AI-generated relationships should be validated against the original documents. An LLM can produce plausible relationships that are not actually supported by the source.

Enterprise GraphRAG: Knowledge Governance

In an enterprise scenario, the objective changes. Now we want to maintain reliable information over time. We have existing entities, relationships, business rules, and applications that depend on the graph. The ontology becomes an important part of the architecture. It must be defined, validated, controlled, and versioned.

The important question becomes:

How can we update a Knowledge Graph without losing consistency or changing the meaning of existing information?

For me, this is the transition from Knowledge Discovery to Knowledge Governance.

Both approaches are useful, but they solve different problems.

3. Why I Think an Ontology Should Come First

For an enterprise-oriented Knowledge Graph, I would start with an explicit ontology.

An ontology describes the concepts in a domain, their relationships, and the rules that define how they are connected. In my experiment, I use a lightweight ontology that defines the allowed entity types, relationship types, and their connections. It does not implement a complete OWL-based ontology with formal logical reasoning. This is sufficient for my demonstration because I want to investigate how changes to the ontology affect existing knowledge and graph maintenance. For a more formal ontology approach, including logical semantics and reasoning, see the W3C OWL 2 specifications [7, 8]. The repository explains how this lightweight ontology differs from a plain data model [4].

For example, in my demonstration, the first ontology contains three entity types:

• Company
• Service
• Destination

It also contains two relationships:

• OFFERS: Company → Service
• VISITS: Service → Destination

A company offers a service, and a service visits a destination. This is a small ontology, but it is enough to demonstrate the challenge. I believe AI can help us create the initial ontology proposal by analyzing existing information. However, this proposal should be reviewed before it becomes the controlled model for a production system. After approval, the ontology acts as a contract for the graph. New information must be checked against this contract before it is imported. An important distinction is that a new relationship does not always require a new ontology.

• If two existing entities can be connected using an already defined relationship type, we only need to update the graph data.
• If we need a new concept or relationship type, we may need to evolve the ontology.

This distinction became a central part of my new demonstration.

4. My New Local GraphRAG Experiment

To investigate these questions, I created a new GitHub repository [2]:

• graphrag-ontology-evolution-example

The example uses the fictional company Galaxium Travels. It runs locally with:

• Neo4j for graph storage and vector retrieval.
• Ollama with qwen3:8b for the answering agent and AI experiments.
• nomic-embed-text for document embeddings.
• LangChain for the answering agent.
• FastMCP for read-only access to retrieval tools.
• Python for ingestion, validation, maintenance, and experiments.

The setup does not require a cloud API key. The initial installation still needs downloads of dependencies, container images, and models.

One important architecture decision is the separation between reading information and updating information.

The answering agent uses read-only MCP tools. Graph updates go through a separate, deterministic maintenance process. This prevents the answering agent from directly changing the ontology or graph data through its MCP interface. The demonstration is a learning environment, not a production-ready enterprise platform. For example, the read-only tools and the importer still share a Neo4j credential [3].

Why deterministic rules?

I wanted to avoid a situation where an LLM could independently decide which graph structures should exist and immediately write them into the database.

Instead, the current ingestion process uses explicit rules to validate the information before it is stored.

The source preparation is also deterministic.

It is important to mention that the current implementation does not use an LLM to generate the entire ontology or automatically extract all graph facts from arbitrary documents. AI-assisted discovery is introduced later, in a controlled experiment.

This separation helps us identify which results come from deterministic code and which depend on the LLM.

5. Twelve Labs: From Initial Graph to Ontology Evolution

I structured the repository into 12 hands-on labs.

The first eight labs demonstrate the core maintenance workflow.

The main example introduces a training requirement for Lunar Excursions. The initial ontology cannot represent this requirement properly.

Therefore, ontology v2 adds:

• Training as a new entity type.
• REQUIRES: Service → Training as a new relationship.

This is an additive ontology change. The migration checks the stored data and changes the active ontology from v1 to v2. Existing facts remain valid, and the new training relationship can be imported.

So far, this is relatively manageable.

But what happens when an ontology change is not additive?

This is where the next four labs become very interesting.

6. Lab 9: Breaking Ontology Changes

Link to Lab 09 – Breaking Ontology Change

In this experiment, I introduce a proposed ontology v3. Two things change:

First, VISITS is replaced with TRAVELS_TO.

Second, the meaning of REQUIRES changes.

Previously:

Service → REQUIRES → Training

In the proposed ontology:

Destination → REQUIRES → Training

At first, both changes might look like modifications to the model.

But they are not equivalent.

Renaming VISITS to TRAVELS_TO can be a controlled mapping if both relationships have the same intended meaning. It still needs a review and an approval.

Changing REQUIRES is different. The existing source states that Lunar Excursions requires safety training. It does not state that the Moon itself requires the training. Automatically changing the relationship would therefore create a statement that is not supported by the source. In the repository, the migration tool refuses the complete v3 proposal, because it only supports additive changes. Removed or changed definitions need a rebuild.

The lab also checks the six existing facts against the proposed ontology.

Result: Four facts remain valid, and two become structurally invalid.

This demonstrates an important point:

An ontology change can invalidate existing knowledge even when the original source documents have not changed.

And there is another limitation. A structural comparison can detect changes to relationship types and their endpoints. It cannot necessarily detect that the business meaning of an existing relationship has changed while its structure remains identical.

That requires semantic review.

7. Lab 10: Understanding the Impact of an Ontology Change

Link to Lab 10 – Ontology Impact Analysis

This lab was especially important for my original question. I imagined a Knowledge Graph like a spider web.

If I change one part of the web, how do I know which other parts are affected?

A single relationship may connect information from different documents. Changing this relationship can affect queries and information that are not directly connected to the changed source.

In Lab 10, the impact analysis evaluates the proposed ontology v3.

The recorded results identify:

The important observation is that even the company overview document is affected.

It does not own an invalid fact.

But it shares the Lunar Excursions entity with other documents. Through this entity, the graph can retrieve relationships that become invalid under the new ontology. The impact analysis also identifies a code reference that still generates the old VISITS relationship. This means updating the stored graph alone would not be enough. The next ingestion could recreate the old information.

An ontology change can affect the data, the ingestion logic, and the applications using the graph.

The current implementation identifies structural impacts and possible exposure through retrieval. It does not prove that every flagged question produces a wrong answer, and it cannot guarantee that every possible semantic dependency has been found.

That is an important limitation.

8. Lab 11: Can AI Discover Missing Relationships?

Link to Lab 11 – AI-Assisted Relationship Discovery

The next problem is even more interesting.

Until now, we mainly looked at relationships that already exist in the graph.

But what about a relationship that should exist and has never been created? For example, one document describes a service, and another document contains information about a destination. A connection between them may be possible, even if no relationship has been stored. A graph traversal alone cannot find a relationship that is missing. Therefore, I introduced an AI-assisted discovery experiment.

The process has several steps:

1. Deterministic code generates possible relationship candidates from the existing ontology and entities.
2. The LLM evaluates these candidates using the stored document content.
3. The LLM must provide an exact source quotation for each supported proposal.
4. Code checks the candidates and whether the quotations exist in the cited documents.
5. A human reviews the remaining proposals.
6. Only approved information is imported through the normal ingestion process.

The recorded experiment produced a surprising result.

All three proposals passed the technical checks. The source quotations existed, and the relationship structures were allowed. But that did not mean the proposed relationships were supported by the evidence. All three quotes came from the text about Lunar Excursions, not from the service the LLM linked them to. The LLM found the right topic, but connected it to the wrong service.

I rejected two of the three proposals. The one accepted fact, “Lunar Photography Tours VISITS Moon”, needed a better quote from the tour’s own description. Even then, it is a judgment call: “lunar photography” could also mean taking photos of the Moon from orbit. For this synthetic example, I accepted the interpretation that the tour takes customers to the Moon.

For me, this is one of the most important findings of the experiment.

Deterministic validation can check whether data follows the rules. It cannot automatically guarantee that the information is true.

AI can help identify possible missing relationships, but the final decision still requires reliable evidence and, in this example, human review.

Another limitation is that the experiment starts with known entities and allowed relationship types. It does not automatically discover every possible new entity, relationship type, or ontology gap.

The reviewed fact is imported as its own document. This keeps the discovered assertion separate from the mapped source facts. After this lab, the graph contains seven facts across four documents.

The recorded results and the review decisions are documented in Lab 11 of the repository.

9. Lab 12: Why Regression Testing Matters

Link to Lab 12 – Validation and Regression

The last lab addresses another problem. How can we verify that incremental updates produce a consistent graph?

I compare three things:

• The incrementally maintained Knowledge Graph.
• A full rebuild from the sources and reviewed files.
• A manually reviewed ground truth.

The expected final graph contains seven facts across four documents under ontology v2. The lab then deliberately introduces a regression. An older version of the company overview is imported. As a result, the relationship stating that Galaxium Travels offers Lunar Photography Tours disappears. However, the separate document from Lab 11 still contains the relationship stating that Lunar Photography Tours visits the Moon.

The graph now contains a service with a destination relationship, but the corresponding company offering has disappeared. Each document is structurally valid and follows the ontology rules. However, the resulting Knowledge Graph no longer matches the reviewed expected state. This demonstrates that structural validity alone does not guarantee that the complete graph contains the expected knowledge. The regression check detects the problem.

The recorded results show failures in:

• Document revision comparison
• Fact comparison
• Retrieval validation for Q2 (“Which services does Galaxium Travels offer?”)

Interestingly, the retrieval check for Q4 (“Which services take customers to the Moon?”) still passes. The graph now contains a service that visits the Moon, but the company no longer offers this service. Only the comparison against the expected complete state finds this.

After a full rebuild using the intended sources, the checks pass again. The final graph content matches the expected state, even though the historical revision paths differ.

This distinction matters.

The current graph state and the history of how it was created are not the same thing.

A rebuild can reproduce the current facts without reproducing every historical step. The ground truth in this experiment intentionally compares the expected current state, not the full revision history. This is documented in Lab 12 of the repository.

This is a valuable lesson for the design of validation and audit processes.

10. Another Important Finding: Retrieval Is Not the Same as a Correct Answer

The experiment also revealed a problem unrelated to ontology migration, but highly relevant to GraphRAG.

In the recorded runs, deterministic retrieval successfully found the relationship between Lunar Excursions and the Moon across two documents. The search tool already returned this cross-document fact to the answering agent.

Even so, the agent sometimes mentioned unsupported destinations, such as “orbital space hotels” or “lunar orbit”. In one run, it listed the services themselves as destinations. The agent also never called the separate graph_context tool, and the answers varied between runs even at temperature 0.

This is documented in the repository’s Experiment Results [5]. This means we must distinguish between:

• Correct information stored in the graph
• Correct retrieval of that information
• Correct interpretation of the information by the LLM
• Correct final answers

These are different quality dimensions.

In Lab 12, the recorded final answers passed the required-term checks, but those checks are not a complete semantic correctness evaluation.

A Knowledge Graph can contain the correct facts, and the retrieval process can successfully find them, while the answering agent still produces an incomplete or misleading answer.

This is why I believe GraphRAG testing should validate the complete workflow, including stored facts, retrieved context, tool usage, and final generated answers.

11. What Would Be Required for Enterprise GraphRAG?

After these experiments, I would separate an enterprise-oriented GraphRAG maintenance architecture into three responsibilities.

Ontology Governance

• The ontology is explicitly defined, reviewed, approved, and versioned.
• Changes should include an impact analysis before they are applied.
• Breaking changes need a migration plan and potentially a rebuild of affected information.

AI-Assisted Semantic Analysis

• AI can help extract information, identify relationships, and detect possible gaps in the ontology.
• But these outputs should be treated as proposals, not automatically accepted facts.
• Source evidence and semantic validation are essential.

Deterministic Graph Maintenance

• Controlled code handles schema checks, versioning, ingestion, and migration.
• Updates should be reproducible, traceable, and tested against expected states.

The repository demonstrates several parts of this approach, including versioned ontology definitions, deterministic validation, incremental imports, additive migration, impact reporting, and regression checks.

However, it does not implement a complete enterprise governance workflow. For example, persisted approval history, fully automated cross-document impact discovery, unattended semantic extraction, and controlled breaking-change migration remain outside the current implementation.

This topic is also arriving at the platform level. In November 2026, Neo4j announced the Neo4j Ontology Graph Specification, a format and library for defining, validating and migrating ontologies [9, 10]. For me, this shows that ontology governance is becoming a topic for graph platforms, not only for individual projects. However, validating an ontology definition is not the same as proving that all graph facts are semantically correct.

12. What I Learned

My main learning is that building a Knowledge Graph and maintaining a Knowledge Graph are two very different engineering challenges.

Creating an initial graph can be a useful experiment in knowledge discovery. Maintaining a graph introduces additional questions.

• How do we know that a changed document does not invalidate another relationship?
• How do we find relationships that have not been created yet?
• How do we prevent an ontology change from changing the meaning of existing information?
• And how can we verify that the resulting graph remains consistent?

My 12-lab experiment does not solve all these questions. But it makes several of the problems visible and provides concrete ways to investigate them.

The most important findings for me are:

• An ontology should be treated as a controlled and versioned model in enterprise scenarios.
• New graph data and ontology evolution require different update processes.
• Breaking ontology changes can invalidate existing facts.
• Impact analysis must include source documents, graph relationships, ingestion code, and dependent questions.
• AI-generated relationships can pass technical validation and still be wrong.
• Incremental updates need regression tests against an expected graph state.
• Correct retrieval does not automatically result in correct LLM answers.

I also learned that validation must happen at different levels.

Structural correctness, semantic correctness, and graph-wide consistency are related, but they are not interchangeable. An exact quotation proves that the quoted text exists in a document. It does not prove that the proposed relationship is supported by that quotation.

13. Final Thoughts: From Discovery to Governance

When I started working with GraphRAG, I mainly wanted to discover relationships hidden inside documents. Now I see another important challenge.

Knowledge is not static.

Documents change, products evolve, business rules are updated, and regulations may introduce new requirements. A Knowledge Graph must be able to handle these changes without silently losing information or creating unsupported relationships.

For enterprise scenarios, I therefore believe we need more than an ingestion pipeline.

We need a controlled knowledge lifecycle, including ontology governance, semantic analysis, deterministic updates, provenance [6], impact analysis, and regression testing.

This is my current architectural conclusion based on a small local experiment, not a claim that the same design has been proven at enterprise scale. The next challenge is to investigate how these approaches behave with more documents, more dependencies, and more complex ontology changes.

The important question is no longer just:

How can AI help us build a Knowledge Graph?

It is also:

How can we keep that Knowledge Graph trustworthy when the world around it changes?

14. GitHub Repository and Hands-on Labs

The complete source code, setup instructions, sample data, and 12 labs are available here:

GitHub: GraphRAG Ontology Evolution Example

You can start with Lab 01 – Initial Setup and follow the exercises step by step. The first eight labs cover the core workflow. Labs 9–12 explore breaking changes, impact analysis, AI-assisted discovery, and regression testing.

15. References

1. Thomas Suedbroecker — How to Build a Knowledge Graph RAG Agent Locally with Neo4j, LangGraph, and watsonx.ai.
2. Thomas Suedbroecker — GraphRAG Ontology Evolution Example – GitHub Repository.
3. Repository documentation — Architecture.
4. Repository documentation — Ontology vs. Data Model.
5. Repository documentation — Experiment Results.
6. Repository documentation — Data Lineage.
7. W3C — OWL 2 Web Ontology Language: Structural Specification and Functional-Style Syntax (Second Edition).
8. W3C — OWL 2 Web Ontology Language: Document Overview (Second Edition).
9. Neo4j — Neo4j Ontology Graph Specification (documentation).
10. Neo4j — The Neo4j Ontology Graph Specification: A pragmatic, open, AI-first approach (blog post, November 5, 2026).

The recorded results are from my local run on November 13, 2026 (Apple M4 Max, Ollama with qwen3:8b). Model-dependent results may vary between runs. All opinions and conclusions in this post are my own and are based on my personal learning and engineering experiments.

Note: This post reflects my own ideas and experience. AI was used as a writing and thinking aid to structure and check the arguments, not to define them.

GraphRAG, Neo4j, Knowledge Graph, Ontology, Ollama, MCP, Agents