top of page

How AI work with policies

Nov 3
3 min read

Our research team conducted research on using AI for policy and rules. Link: https://arxiv.org/abs/2608.09028


Close-up of printed policy pages with highlighted sentences
Policy rules often begin as plain text in long documents.

What PolicyKG does


Our tool is called PolicyKG.


It reads a policy PDF in four steps.


  • It splits the PDF into sentences.

  • It labels each sentence.

  • It turns rules into formal logic.

  • It produces SHACL checks.


Before the labels, here is the meaning. Obligation, permission and prohibition mean must, may, and must not. A sentence can also be marked as not a rule.


Before the logic step, here is the meaning. First-order logic means formal if-then statements. PolicyKG uses this as a middle step when the rule can go through that path.


PolicyKG uses Mistral 7B run locally, with no fine-tuning. That means we did not train the model again on labeled examples for this run.


What we tested


We tested PolicyKG on 1,663 sentences from university policies. These sentences gave 443 rules.


The rule type labels were right 86.9 percent of the time.


Permissions were weakest. PolicyKG got 49 of 75 permissions correct. The reason is simple. The word “may” can mean different things in policy text.


A simple trained model scored 90.3 percent on the same data. But that model needs labeled examples. Our LLM needs none.


For the next step, 79.2 percent of rules, or 351 of 443, went through the logic path. The rest used a direct fallback path.


For SHACL check quality, the F1 score was .866 on 69 checks. Again, the reference answers were made inside our own project. So this is not an independent test.


We also checked whether rules needed higher-order logic. We found no rule that needed it, 0 of 443. This is an audit of one corpus. It is not a proof.


Eye-level view of index cards arranged into must may and must not groups
Grouping sentences by rule type is a key step before checks can be generated.

What changed when we changed the text and the vocabulary method


We also tested changes outside the main university policy corpus.


First, we changed the vocabulary method.


Corpus Adapter means a fixed list of approved names the AI must pick from. On 15 GDPR rules, exact name matches rose from 1 of 15 to 11 of 15 with the Corpus Adapter.


RAG means letting the AI look up examples before answering. Simple retrieval, or RAG, got 9 of 15. The gap between 11 and 9 is not statistically significant.


The adapter’s advantage is not that it clearly beat retrieval. Its advantage is that it is a plain file. That makes output more repeatable and easier to audit.


Next, we changed the type of text.


On 200 lease-contract clauses, Macro F1 fell from .810 to .370. Permission F1 was .038. Lease contracts often say “shall be entitled” for permission. That wording hurt the permission label.


Here is the short version of what we changed and measured.


  • What we changed - Type of text - Vocabulary method


  • What we measured - Label accuracy - Name matches


A full run took 34.9 minutes on an 8 GB consumer GPU. Repeat runs gave identical output on the same machine. A different GPU gave slightly different counts.


What we would use from this work


The main lesson is practical. Written rules can be turned into checks, but the path must be auditable.


Keep the source sentence close to the generated check. Treat permissions as a special risk. Do not trust one score on one corpus as a general claim.


End with these actions.


  1. Keep every check linked to its source sentence.

    This makes review possible when the generated check looks wrong.


  2. Give the AI an approved vocabulary list instead of letting it invent names, and try simple retrieval first.

    The Corpus Adapter gave repeatable, auditable output. RAG was close in this test.


  3. Test permission labels separately.

    Permission wording caused errors in both policy and lease text.


  4. Test on your own documents before trusting any number.

    Our numbers come from this setup, not from other models or other kinds of policy.


  5. Add a pre-filter and a fallback path, and keep human review for rules that fail.

    The human review part is our suggestion. We did not test it.


PolicyKG gave us a clear result from one English university corpus. The useful part is not a promise that every policy will work. The useful part is a method: split the text, label the rule type, control the names, generate checks, and keep the source sentence attached.


bottom of page