Semantic search for company documents, on your own servers

Semantic search AI lets your staff type a question in their own words and find the right page in thousands of PDFs, contracts, SOPs and emails, even when the document uses different wording. You build it by turning documents into text, cutting them into passages, storing a meaning vector for each passage next to a normal keyword index, and filtering every result by what the person searching is allowed to open. All of it can run on servers you control.

This guide is for the owner or IT lead of a CA firm, a law office, a manufacturer with shelves of manuals, a clinic or a school. For definitions, see our glossary pages on semantic search and the vector database. This page is about building one.

Why the search box on your shared drive fails

Keyword search finds the words you type, but staff remember what a document says, not how it says it. A partner looks for "the client who wanted quarterly billing" and the engagement letter says "fees payable every three months". A service engineer types the symptom, and the manual lists it under a fault code. Scanned files are worse: a scan with no text layer is an image, so no search finds anything in it.

From a folder of files to a working search box

1. List the sources and who may see them. Shared drives, the document management system, mailboxes. For each, note the owner, how often it changes and the access groups.
2. Get clean text out. Digital PDFs and Word files give up their text directly; scans need OCR first. OCRmyPDF adds a searchable text layer using the Tesseract engine, whose language data includes Tamil, Hindi, Telugu, Kannada, Malayalam, Bengali, Marathi and Gujarati. Docling reads PDF, DOCX, PPTX, XLSX, HTML and email files, keeps tables and reading order, and runs locally. Our OCR and document intelligence page covers phone photos, stamps and handwriting.
3. Cut documents along their structure. Split at headings and clauses, keep tables whole, and carry the file name and page number with every passage. If you split by size, Microsoft's Azure AI Search guidance suggests starting at 512 tokens with 25% overlap, and adding the document title to chunks from the middle of long documents. Tune from there on your own files.
4. Embed each passage with an embedding model you can run yourself (see the table below).
5. Store vectors, text and metadata together: document ID, version, date, client, department and access group.
6. Search two ways at once, by meaning and by keyword, and merge the results.
7. Rerank the top results with a cross-encoder model that reads the question and each passage together.

Embedding models you can run on your own servers

All of these are published on Hugging Face with their weights. Licences, sizes and limits are from each model card.

Input length matters: a model that stops at 512 tokens silently cuts a long clause. And a published language list says nothing about your documents. If your files mix Tamil and English, or Hindi typed in Latin script, put a few dozen real questions through two or three candidates before you choose. Changing models later means re-embedding everything.

Where to store the vectors

You probably already run a database that can hold them.

• PostgreSQL with pgvector (v0.8.7). Its indexes take up to 2,000 dimensions for the vector type and 4,000 for halfvec, so a 4,096 dimension Qwen3-Embedding-8B vector must be shortened first. Full-text search and row-level security sit in the same database.
• MariaDB 11.8 LTS has a built-in VECTOR type and vector index (introduced in 11.7), using a modified HNSW graph and cosine or Euclidean distance.
• MySQL has had a VECTOR type since 9.0 (1 July 2024), but the manual says DISTANCE() is available only on MySQL HeatWave on OCI and MySQL AI, not in the Community or Commercial distributions. On Community you can store vectors but not rank by similarity in SQL.
• Qdrant is a dedicated vector database with payload filters and built-in hybrid queries.
• Milvus handles large collections and computes BM25 keyword scores itself.
• OpenSearch runs a hybrid query (keyword and vector in one request) and, from version 2.19, merges them with reciprocal rank fusion.

Keep the keyword index: hybrid search

Meaning search is weak on exact strings: a GSTIN, a case number, a part number, "Section 43B". Keyword search (BM25) is strong on exactly those. Run both and merge the two ranked lists with reciprocal rank fusion (RRF): each passage scores 1/(k + rank) in each list, and the scores are added. The 2009 SIGIR paper that introduced RRF used k = 60. OpenSearch also defaults to 60; Qdrant's default is 2. Because RRF uses ranks, not raw scores, it merges lists that score on different scales.

Then rerank. A cross-encoder such as bge-reranker-v2-m3 or Qwen3-Reranker (both Apache 2.0, both multilingual) scores each shortlisted passage against the question and reorders them. It is slower than vector search, so it runs only on the shortlist.

Permissions: people find only what they may open

This is the step demos skip. If a junior associate cannot open the partners' fee notes in the drive, search must not show a line from them. OWASP's 2025 list of LLM application risks (LLM08, vector and embedding weaknesses) asks for permission-aware vector stores, data partitioned between groups, and immutable logs of what was retrieved.

In practice:

• copy each file's access groups onto every passage at indexing time, and refresh them when permissions change;
• pass the user's groups into the query as a filter, so restricted passages never enter the result list (filtering after ranking leaks titles and snippets);
• in PostgreSQL, row-level security can enforce the same rule inside the database;
• when a file is deleted or a client leaves, delete its passages and vectors too.

One trap: the pgvector README warns that approximate indexes apply the filter after the scan. If a condition matches 10% of rows, with HNSW and the default ef_search of 40, only about 4 rows come back, so a user with narrow permissions gets thin results. Iterative index scans (pgvector 0.8.0 and later) fix this.

Measure it before you trust it

Write a test set: real questions from staff emails and help requests, each with the document and page that answers it. Run every question and check whether the right passage appears in the top results. Recall at k is the share of the correct passages that appear in the top k; with one right answer per question, it is simply how often the answer is in the top 5 or top 10.

Score your current keyword search first as the baseline, then change one thing at a time (chunk size, model, reranker) and rerun. Include questions with no answer in your documents, and run some as users with limited access to prove nothing leaks.

From search box to a chatbot that cites its sources

Once retrieval passes the test set, adding answers is the smaller step. A language model gets the question and the top passages, writes a short answer from them only, and links each sentence to its file and page. When retrieval finds nothing relevant, it says the documents do not cover the question. That is retrieval-augmented generation, and its quality depends almost entirely on the search underneath it.

Every part can stay in-house: OCR, the embedding model, the database, the reranker and an open-weight language model served on your own GPU server. Nothing leaves your network. Our private AI deployment page covers model serving and hardware sizing.

What we build

Our engineers in Chennai build document search and RAG assistants on PostgreSQL with pgvector, Qdrant, or a store you already run. We combine vector and keyword search and re-rank the results, filter by the user's permissions before any model sees the text, and score every change against 50 to 200 real questions written with your team. Models can be hosted APIs or open-weight models on your own hardware.

FAQ

Is semantic search the same as a chatbot? No. Semantic search returns passages and documents. A RAG chatbot adds a language model that writes an answer from those passages. Build and test the search first.

Do we need a GPU to run semantic search on our own server? Not always. Embedding models are much smaller than chat models: Granite Embedding 97M has 97 million parameters and EmbeddingGemma about 300 million. A GPU helps when you embed a large archive in one go or add a local language model for answers.

Can it search scanned PDFs and Tamil or Hindi documents? Yes, once OCR has added a text layer. Tesseract has Tamil, Hindi and other Indian language data, and several embedding models list 100 or more languages. Test them on your own files before choosing.

We already run MySQL. Can we use it? You can store vectors in MySQL 9 Community, but similarity search with DISTANCE() is limited to HeatWave on OCI and MySQL AI. MariaDB 11.8 or PostgreSQL with pgvector can do the search inside the database.

If your team spends its mornings hunting for the right contract, SOP or manual page, we can look at where your documents live and suggest a first search you can test with real users. Book a consultation.