Building a Grounded Legal AI Assistant: A Worked Case Study in RAG, Hybrid Search, and EU Data Sovereignty

A grounded legal AI assistant is a vertical retrieval-augmented system that refuses to answer from a language model's memory and instead pulls every claim from a curated, dated corpus of statutes and case law — so each sentence is bound to a citable source the lawyer can verify in one click. The build below documents one such system for the Italian invoicing-and-practice-management platform fattura-avvocato.it: hybrid Postgres search over pgvector + BM25, Reciprocal Rank Fusion, EU-hosted inference, and a monthly all-in cost under $40 for 10,000 queries. The same discipline AEW preaches on the content side — see Answer Engine Optimization explained — is what makes a grounded legal AI honest on the application side.

A lawyer types a question into a general-purpose chatbot and gets back a beautifully composed paragraph citing two Cassazione rulings, an article of the Civil Code, and a paragraph from a 2024 doctrinal note. The paragraph is wrong. One of the rulings has been superseded. One of the articles does not say what the model says it does. The doctrinal note does not exist. The lawyer, if she is unlucky, files the brief. This is not a tooling problem — it is an architecture problem. The numbers, the topology, and the compliance constraints below are real. The pattern generalizes to any vertical where 'looks plausible' is not the same as 'is correct'.

Why is a raw LLM wrong for civil-law legal work?

A general-purpose large language model reasons probabilistically over the corpus it was trained on. In a common-law setting that is closer to how the law itself is built up — case by case, precedent on precedent — so the model's failure modes are at least the right shape. In an Italian-style civil-law setting the gap is wider. The Italian system runs on codified statute, but the working interpretation of a statute — what practitioners call the diritto vivente — is set by the nomofilattica function of the Corte Suprema di Cassazione. A lawyer in Milan does not deduce the meaning of Art. 1218 from the text alone. She looks at how the Cassazione has been applying it, and she looks at what the latest decision changed.

A vanilla LLM does not. It pattern-matches across whatever was in its pre-training set, which may be three years stale, may conflate the Italian Codice Civile with adjacent civil codes, and which gives the model no way to know that a 2022 reading has been overturned in 2024. The output looks impeccable — perfect grammar, plausible structure, the correct register — and is logically coherent. It is also, with non-trivial probability, jurisprudentially wrong. We call this a logical hallucination, and in a regulated profession it is unshippable.

The grounded-RAG architecture in one mental model

Retrieval-Augmented Generation (RAG) is the textbook answer, but textbook RAG fails on legal text. A naïve chunker that splits by token count will sever an article of code from its exceptions, or a principle of law from the facts that bound it. A pure vector search will hallucinate semantic neighbours of 'Art. 1218 c.c.' that are not the actual citation the lawyer needs. The build below addresses both — structurally aware chunking on ingestion, hybrid search on retrieval, and a thin orchestration layer that turns one search box into a deterministic, citable answer.

1. Ingestion pipeline — turn the scraped corpus of rulings, decisions and statutes into structured, metadata-rich, GDPR-sanitized fragments stored in PostgreSQL with vector and lexical indexes.
2. Hybrid retrieval — for every query, run a semantic vector search and a BM25 lexical search in parallel, then fuse the rankings with Reciprocal Rank Fusion (RRF).
3. Grounded generation — pass the top-ranked fragments to a small, cheap, EU-hosted LLM with hard instructions to answer only from the provided context and to cite every claim back to its fragment.

Pipeline 1 — turning a scraped legal corpus into queryable knowledge

The platform already had years of scraped public legal text — Cassazione rulings, lower-court decisions, statutory texts — sitting as raw PDFs and HTML. That corpus is an asset, but only once it has been transformed into something a retriever can actually use. The transformation has four steps, and the order matters.

Pipeline 2 — hybrid search inside Postgres, with no separate cluster

The temptation here is to stand up a dedicated search cluster — Elasticsearch, OpenSearch, a managed vector DB. For a vertical platform that already runs Postgres, that is overspend and overengineering. The same machine, with two open-source extensions, can do both the semantic and the lexical job.

The build uses pgvector for cosine-similarity vector search and a BM25 lexical index (either via pg_textsearch tuning or the BM25 implementation in ParadeDB) on the same text column. The two are run in parallel against every query, and their result lists are fused. Vector-only is wrong for legal work because the queries lawyers actually type fall into two very different shapes.

• Concept queries — 'contractual liability for supplier breach', 'when does force majeure suspend payment obligations'. These need semantic similarity. Token overlap is not enough.
• Citation queries — 'Art. 1218 c.c.', 'Cass. 1234/2024'. These need exact lexical match. A vector retriever will return semantic neighbours of the citation, which is not what the lawyer asked for.

Reciprocal Rank Fusion in one line

Each retriever returns a ranked list of candidates. Reciprocal Rank Fusion combines them by summing the reciprocal of each document's rank across the lists, with a damping constant k (conventionally 60) that prevents any single retriever from dominating the top slot.

RRF(d) = sum over m in M of 1 / (k + rank_m(d)), with k = 60 by default.— Cormack, Clarke & Büttcher, SIGIR 2009

The flow is short. Take the top 20 from the vector retriever and the top 20 from BM25. Compute RRF over the union. Take the top 5. Pass those 5 fragments — and only those 5 — to the generator.

Pipeline 3 — the invisible orchestration that lets the UI be one search bar

Professional AI tools fail in the market not because they cannot answer the question, but because they make the lawyer learn three menus, two toggles, and a query syntax before she can ask it. The build above hides all of that behind a single text input. The orchestration that makes that possible has four stages and the user sees none of them.

What does a grounded legal AI actually cost to run?

The standard objection to verticalized AI in the Italian legal market is the price tag of the incumbents — roughly €1,000 per seat per year for the established commercial offerings. That number was set when frontier-model API calls were two orders of magnitude more expensive than they are in 2026. Today, with intelligent model routing — cheap models for the 95% of routine extraction and classification, premium models only for genuinely hard doctrinal questions — the unit economics flip.

The simulation below assumes 10,000 complex searches per month across the platform's user base, with each search consuming roughly 5,000 input tokens (the retrieved fragments plus prompt scaffolding) and emitting 1,000 output tokens (a cited answer).

EU data sovereignty: the compliance layer is not optional

An Italian legal AI lives under three overlapping regimes: the GDPR, Italy's Law 132/2025 (the recent statute on AI use in professional services, with its Art. 13 client-disclosure duty), and the Codice Deontologico Forense (specifically arts. 13 on confidentiality, 14 on professional competence, 28 on unauthorized delegation, and 55-bis on diligent use of technology). Each one has teeth. GDPR penalties top out at 4% of global turnover or €20m — whichever is higher — and the Italian deontological regime can suspend a lawyer from practice.

The single decision that absorbs most of the compliance risk is where the model runs. A US-hosted inference endpoint — vanilla OpenAI, vanilla Anthropic, without an EU-localization contract — moves personal client data to servers that fall under the US CLOUD Act, which is, for a European lawyer's confidentiality obligation, a non-starter. The build above resolves this in one of three ways.

1. EU-native inference. Mistral AI — French-domiciled, Paris-region data centres, contractually guaranteed not to train on customer data — is the default. The legal status is clean: no extraterritorial reach, processing stays inside the EU.
2. EU-boundary inference on a hyperscaler. Azure OpenAI deployments configured with the EU Data Boundary keep logs and telemetry within the EU; usable for clients with existing Azure commitments.
3. On-prem or private cloud. For studios that need the strictest possible confidentiality posture, open-weights models such as Mistral Nemo 12B run on infrastructure the studio controls. No data ever leaves the perimeter.

Deontology constrains use as well as deployment. Under arts. 13, 14, 28 and 55-bis of the Codice Deontologico, AI output is strictly instrumental. The lawyer's critical review is non-delegable, and personal professional liability for the final filing is undiluted. The tool exists to compress the search-and-draft loop, not to substitute the lawyer.

What an AEO operator should take from this build

The build above is, in its bones, the same pattern AEW preaches on the other side of the wire. Answer engines reward content that is specific, dated, structured, machine-checkable, and easy to extract — exactly the criteria AEW uses when measuring AI citations on the analytics side. The legal AI build wins for exactly the same reasons — its corpus is specific, dated, structured, machine-checkable, and easy to retrieve. The discipline is symmetric: the engine prefers content shaped like this; the application can only be honest if it consumes content shaped like this.

The corollary for an agency is direct. A client's content strategy, an answer engine's retrieval strategy, and a vertical AI's grounding strategy all converge on the same artefact — a page that states a verifiable claim, attaches it to a source, and exposes structure a machine can navigate. That artefact is what compounds into entity authority for AI search over time. Build it, and you are simultaneously optimizing for citation, retrieval, and grounded generation. Skip it, and no amount of model choice fixes the result.

The shipping order

1. Stand up Postgres with pgvector and a BM25 lexical index (pg_textsearch tuning or ParadeDB).
2. Run the ingestion script: structured chunking, metadata enrichment, GDPR sanitization, vector generation via Mistral Embed.
3. Implement the hybrid retrieval SQL function with RRF fusion, returning the top 5 fragments per query.
4. Wire the orchestrator (Node or Python) — parse query, translate filters, retrieve, compose grounded prompt, call the EU-hosted generator.
5. Build the single-input UI with side-panel source verification and per-user audit logging for AI Act and GDPR traceability.
6. Publish the updated terms and ship the model client-disclosure clause required by Art. 13 of Law 132/2025.

None of these steps is exotic. The novelty is in refusing to skip any of them. Skipping the structured chunking gives a fluent system that loses cases. Skipping the EU-localization clause gives a system that ships fast and then gets a regulator letter. Skipping the citation binding gives a chatbot indistinguishable from the one the lawyer can already use for free — and which she will not pay for. The point of a vertical AI is that every one of these constraints is met at once, and the user sees a single uncluttered search box on top.

Frequently Asked Questions

What is a grounded legal AI assistant?

A grounded legal AI assistant is a domain-specific retrieval-augmented system that answers questions only from a controlled corpus of statutes and case law, citing each claim back to a dated, verifiable source. Unlike a general-purpose chatbot, it is instructed to refuse the question when the retrieved context does not support an answer, which structurally eliminates the most dangerous class of legal hallucinations.

Why does naive RAG fail on legal text?

Naive RAG fails for two reasons. First, token-count chunkers sever the legal unit — they will split an article of code from its exceptions, leaving the retriever to surface half a rule. Second, pure vector search hallucinates semantic neighbours of citations like 'Art. 1218 c.c.', which is not what a lawyer asked for. The fix is structurally aware chunking on ingestion and hybrid (vector + BM25) search on retrieval, fused with Reciprocal Rank Fusion.

How much does it cost to run a grounded legal AI on a small SaaS budget?

On Gemini 2.0 Flash, 10,000 grounded queries per month — each consuming roughly 5,000 input and 1,000 output tokens — costs about $9 in inference. Add EU-region managed Postgres with pgvector (~$15), logging and monitoring (~$10), and amortized embedding costs (~$1), and the all-in monthly run-rate is roughly $35. That is the difference between a feature that pays for itself in week one and one that needs a six-month payback story.

Is using OpenAI or Anthropic directly GDPR-compliant for an Italian law firm?

Vanilla OpenAI or Anthropic endpoints route personal client data through US infrastructure that falls under the US CLOUD Act, which is a non-starter for a European lawyer's confidentiality obligation. Three deployment paths absorb that risk: EU-native inference such as Mistral AI (Paris-region, contractually guaranteed not to train on customer data), an EU Data Boundary Azure OpenAI deployment, or open-weights models like Mistral Nemo 12B running on infrastructure the studio controls.

What does Italy's Law 132/2025 require lawyers to do when using AI?

Art. 13 of Law 132/2025 obliges the lawyer to inform the client, in writing inside the mandate, that AI tools may be used in the engagement. The model clause typically states the categories of AI use, the human-review obligation, and the data-handling posture. A platform that supplies this clause turns a regulatory burden into a feature, since individual practitioners would otherwise have to draft and maintain the language themselves.

How does pgvector + BM25 compare to running Elasticsearch or a managed vector DB?

For a vertical SaaS that already runs Postgres, pgvector for semantic search plus a BM25 lexical index (via pg_textsearch tuning or ParadeDB) on the same text column delivers production-grade hybrid retrieval without a second cluster. Adding the pgvectorscale extension's StreamingDiskANN index pushes the similarity graph to disk and drops database instance cost by an order of magnitude. The result is one machine, one backup story, one operational surface — versus a separate search cluster that doubles the on-call burden.

How is grounded legal AI different from optimizing content for AI search?

They are mirror images of the same discipline. A grounded legal AI consumes content that is specific, dated, structured and citable. Answer Engine Optimization produces content that is specific, dated, structured and citable so AI search engines will retrieve and cite it. If you want to be retrieved on the public web, the AEW playbook applies — see how to rank in Perplexity and the broader off-page authority triangle. If you want to build the application, the architecture above applies. The artefact each side rewards is the same.