Document question answering with dated facts
Document question answering gets the wrong answer when facts change over time and the system treats every sentence as equally current. A temporal knowledge graph stores each fact with the dates it was true, so “who held this role in March 2025?” has one correct answer, with a file and page to prove it.
Why does ordinary document search give wrong answers?
Ordinary document search gives wrong answers because it finds text that looks similar to the question and has no idea which statement is current. If one report says a project manager was promoted in 2024 and a later report says she became a programme director in 2026, a plain search may return either, or both, and leave you to work out which is right.
Most “chat with your documents” tools work the same way. They cut each file into passages, turn each passage into a vector, and at question time fetch the passages closest in meaning. A language model then reads those passages and writes an answer. This is called retrieval-augmented generation, and it works well for stable questions such as “what is our refund process?”.
It struggles in four situations that come up constantly in real document sets:
- Facts change. Titles, headcounts, prices, owners and statuses are different in different documents.
- Documents disagree. One file says 27 percent, a chart in another says 34 percent, and both are about the same result.
- Spelling varies. The same person or company appears as “Meera K.”, “Meera Kulkarni” and “M. Kulkarni”.
- The answer spans documents. “Who worked on both of these projects?” needs facts from two files joined together.
Passage retrieval can handle none of these reliably, because the passages never say which one wins. The fix is to extract the facts first, with their dates, and then answer from the facts.
What is a temporal knowledge graph?
A temporal knowledge graph is a network of entities (people, companies, projects, documents) connected by facts, where each fact carries the dates it was true and the place in the source where it was found.
A normal knowledge graph might say “Asha works at Northwind”. A temporal one says “Asha held the title Analyst from 2023, and the title Team Lead from January 2026, as stated on page 4 of the quarterly review”. When a newer fact replaces an older one of the same kind, the older one gets an end date. It is not deleted, so you can still ask what was true last year.
Each fact keeps four things:
- The statement, in a structured form: who, what relation, what value.
- The period: valid from, valid to, and when we recorded it.
- The source: the file, the page and the section.
- The other places it appeared, so a fact that three documents repeat is one fact with three sources, not three competing facts.
That last point matters. Repetition is not disagreement, and treating it as new information is how a graph fills with duplicates.
How do you turn documents into dated facts?
You turn documents into dated facts in a pipeline: convert the files to clean text, split them into structured chunks, extract typed facts with dates from each chunk, then merge duplicates and resolve conflicts.
Conversion. PDFs come in two kinds. Some have a text layer. Others are scans, which are only pictures of pages. The system detects which kind each page is. For text PDFs, a document parser keeps headings and tables as tables. For scans, a vision model reads the page image and transcribes it. Charts and diagrams are described by the same vision model, so a number that appears only in a bar chart still becomes searchable text.
Chunking. The text is split along its own structure, by section and table, into pieces of about 600 tokens. Each piece is embedded for similarity search, and the piece remembers which file, page and section it came from.
Extraction. A language model reads each chunk and returns structured facts, not prose. Each fact has a type, a relation, a value, and optional qualifiers such as role, amount, unit and period. Documents are processed oldest first, so that when a later document changes a fact, the graph already holds the earlier version.
Resolution and conflicts. Two steps keep the graph clean. We describe both below, because they decide whether the answers can be trusted.
How does the graph handle spelling variants and duplicates?
The graph handles variants with a three-step order: merge obvious spelling and case variants directly, ask a small decision model about same-type candidates that remain, and never merge job titles by meaning.
First, exact and near-exact variants merge without any model. “Meera Kulkarni” and “meera kulkarni”, or a one-letter typo, are the same entity. The most-used spelling becomes the display name, and the other spellings are kept as aliases so a search for any of them still finds the entity.
Second, for names that are different but might refer to the same thing, a small decision model is shown a short profile of each candidate and asked to choose. This kind of model returns probabilities for fixed options rather than free text, so it is fast and cheap, and its output is easy to check. It is used only for same-type candidates. A person is never compared to a company.
Third, titles are protected. “Analyst” and “Senior Analyst” are close in meaning, and merging them would erase exactly the promotion you want to track. So titles are matched exactly and never merged by similarity.
When a fact is stated again in another document, it becomes an extra source on the existing fact. When a new fact conflicts with an old one, a yes or no decision settles whether the new one ends the old one. Some relations are declared “one at a time” in the schema: a person has one current title, a metric has one current value. For those, a new value closes the previous one. Other relations, such as “worked on this project”, can hold many values at once.
How does it answer a question?
It answers in two routes: a fast route that reads the graph with no large model, and a fallback route that uses one language model call when the question needs explanation.
The fast route starts by reading the question. One batched call to the decision model works out the type of thing being asked for, the relation involved, and the time for each relation: now, in the past, at a specific date, or not relevant. It also notes extra details requested (“and their current titles”) and whether any part asks for a description. A date parser handles phrases such as “mid 2023” that the decision model cannot output.
Vector search plus the decision model then pick the starting entity. At each hop, the model picks which relation, at which time, to follow. Candidates reached at the end are checked one by one. The answers come back with the linking fact, including its role, value and period, and the requested details read straight from the graph. No large model is involved, so a typical question takes a few seconds and costs about $0.001.
The fallback route is used when the question asks how, why or to describe something, when the fast route finds nothing, or when confidence is low. One language model call receives the facts the graph found and the closest text chunks, and writes a cited answer. It may ask the decision model one precise follow-up question, at most twice. That costs more, but still only a few tenths of a cent.
In both routes, the answer carries citations to the file and page. A reader can open the source and check.
What does this look like on a real question?
Take a neutral example. A company uploads annual reviews, project plans and org charts. Someone asks: “What was the title of the project lead for the onboarding redesign in February 2025?”
A plain search finds the passages that mention the onboarding redesign and a lead. One passage, from a 2024 plan, says “Senior Analyst”. A later passage says “Manager”. The language model must guess which is newer and may pick the wrong one, or blend them.
The graph route does it differently. It finds the project entity, follows the “led by” relation valid in February 2025 to one person, then follows that person’s “has title” relation at the same date. The title valid then is returned, with a link to the page where it was stated and the date it took effect. If the person was promoted in March 2025, a question about April returns the new title, and a question about February returns the old one.
Now a harder example: “Which people worked on both the onboarding redesign and the pricing review?” This needs two lookups and an intersection. In passage retrieval, the language model must hold both sets in its head. In the graph, it is a query that returns a list, each name with its source in both projects.
Which approach should you choose?
Choose by how much your facts change and how much a wrong answer costs. The table compares four common approaches.
| Approach | Handles changing facts | Cites file and page | Cross-document joins | Cost per question |
|---|---|---|---|---|
| Keyword search | No | Sometimes | No | Very low |
| Vector search plus language model | Weakly | Often | Weakly | Low to moderate |
| Long-context model reading everything | Weakly | Rarely | Weakly | High, grows with corpus |
| Temporal knowledge graph | Yes, by date | Yes, always | Yes | Low for most questions |
Vector search plus a model is the right start for a small, stable document set: a policy handbook, a product manual. A temporal graph pays for itself when documents accumulate over years, contain roles, numbers or statuses that change, and when people need answers they can defend. Think HR records, contract histories, project archives and regulatory files.
The graph also costs more to build. Ingest takes longer than embedding alone, because every chunk goes through extraction, and the schema (which entity and relation types matter) needs thought. For a fresh customer, we propose the schema from a sample of their documents and let a person review it. We do not hand-craft prompts per customer, since that does not scale.
What are the limits?
The limits are extraction quality, ambiguous source documents and loosely worded questions, and it is better to know them than to be surprised.
- Extraction mistakes happen. A model can read a relation backwards (“the team leads Ravi” instead of “Ravi leads the team”) or store one project under two names. Checks on direction and name merging reduce this. They do not remove it, so the source citation stays on every answer.
- Org charts and diagrams can be ambiguous. A box with an unlabelled line may mean reports-to or collaborates-with. The vision model describes what it sees, and a human reviewer should check charts that drive important answers.
- Questions can be vague. “Where is she involved?” is read literally. A good interface asks a clarifying question or shows the route it took.
- Placeholders and pasted text exist. Real documents contain template text copied from other files. Extraction needs to recognise placeholders and skip them.
We handle these with review screens and visible citations. The aim is an answer you can verify in seconds, not an answer you must take on trust.
Where have we built this?
We built this for a client who uploads scanned and digital documents and needs dated, cited answers, and the details are on the document intelligence case study. The client is unnamed. The system reads scanned PDFs and charts with a vision model, merges name variants, and cites file and page in every answer. It sits within our wider AI agents service.
If your documents live in folders, shared drives or an email archive, and the same questions keep coming back, the document intelligence industry page explains the kinds of problems this suits. For an example of a different agent with strict safety limits, see our article on building a read-only analytics agent. To discuss your own documents, use the contact page. We will start with a small sample and show you what the graph finds before you commit to anything.