In short
A RAG system reaches production when its document pipeline, retrieval and evaluation get as much engineering as its prompts. The model is rarely the weak part; the work between your documents and the model is.
Enterprise RAG reaches production by solving five engineering problems in order: the document pipeline, the embedding model, the vector store, the retrieval strategy and evaluation. Gartner predicted in 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, weak risk controls and rising costs.
A RAG prototype is easy to build. Take a few clean PDFs, split them into chunks, embed them with a general model, load them into a vector database and connect a language model. In a demo it works, because the documents are curated and the questions are expected.
Enterprise documents are scanned contracts with OCR errors, old Word files with broken formatting, reports that mix Russian, Kazakh and English on one page, and tables inside presentations. Naive chunking loses context, generic embeddings miss domain vocabulary, and vector search alone returns passages that look relevant and are wrong. Without evaluation the team sees the answers that look right and misses the ones where the model quietly invents.
The ingestion layer that turns scans, legacy formats and mixed-language files into clean chunks that keep their structure and context.
Pick the model by testing it on your own documents, languages and vocabulary. Public benchmarks reflect neither your data nor your questions.
Storage and querying at scale: the metadata schema, indexing, separation between tenants and access control.
Semantic search, keyword search, re-ranking and query rewriting, combined to find the passage that answers the question. A similar passage is not enough.
Measuring retrieval relevance, answer faithfulness and end-to-end quality, so every change to the pipeline is checked against evidence.
An enterprise RAG project builds its advantage in the document pipeline, or collects debt there that no later component can repay. The pipeline handles three things at once: many formats, several languages and the structure of each document.
Scanned documents need production-grade OCR, and tables and figures should come out as structured data. Text that mixes Russian, Kazakh and English needs language-aware chunking that does not cut a sentence where the language switches. Section headings, parent-child links between chunks and metadata on every chunk keep the context intact. The pipeline is a running system, maintained for as long as new documents keep arriving.
The model at the top of the MTEB leaderboard may do poorly on your regulatory filings, internal memos and bilingual technical reports. Start with a test set: real user questions, each paired with the passage that should answer it, drawn from the documents the system will serve.
Run candidate models against that set and measure precision in the top five and top ten results. If people ask in Russian and the source is in English, test cross-language retrieval directly. Larger embeddings capture more nuance and cost more in storage and latency. Fine-tune on domain data only once the pipeline is stable, otherwise every result measures two changes at once.
The vector store is a long-lived infrastructure decision: scale, latency, metadata filtering, separation between tenants and operational maturity all matter. At millions of chunks, dedicated vector databases such as Weaviate, Qdrant, Milvus or Pinecone offer purpose-built indexes, native filtering and horizontal scaling.
The metadata schema matters as much as the vectors. Each chunk carries its source document, its place in the section hierarchy, date, language, access tags and parsing confidence. Filters on those fields narrow a search to the right documents and the right reader. When several business units share one system, isolation is designed in from the start, so a query never reaches data its author may not see.
Semantic search matches meaning and misses exact strings: regulation numbers, product codes, names, precise values. Hybrid search adds keyword retrieval (BM25) to vector retrieval and merges the two result lists, usually with reciprocal rank fusion. The system then catches both the meaning and the exact wording.
A cross-encoder re-ranker reads each candidate passage together with the question and scores relevance more accurately than the first pass. It is expensive, so it runs only on the top candidates. Query rewriting splits compound questions, expands abbreviations and resolves ambiguity. The gain is largest on exact-match and mixed-language questions, where vector search alone fails most often.
Without measurement, every change to the pipeline is a guess: a new chunking strategy, another embedding model, an extra metadata filter. The team can neither confirm an improvement nor see a regression.
A minimum framework measures three things. Retrieval relevance: did the system find the right passages. Faithfulness: did the model use them without inventing. Completeness: did the answer cover the whole question. Automated metrics such as RAGAS run continuously, and people review a rotating sample to catch what the metrics miss. In production, track retrieval latency, throughput and user feedback, or quality will fall and users will be the first to notice.
It depends on document volume, query frequency and where the system runs. At scale, model inference usually costs more than storage. Caching, removing duplicate queries and sending simple questions to smaller models bring it down. We estimate the cost after looking at your sources and expected load.
Retrieval quality falls as documents and questions become more varied. The system returns passages that sound right and are wrong, the model answers confidently from them, and users stop trusting it. The defence is evaluation from day one: automated relevance scoring, human review of samples and dashboards that show a regression before users report it.
Yes, with deliberate work at every layer. Chunking has to handle language switches inside a document, and the embedding model has to retrieve across languages, so a Russian question finds an English source. Multilingual models such as multilingual-e5-large or BGE-M3 do this well, but test them on your own language mix. Kazakh text in Latin and Cyrillic script may need extra preprocessing.
It depends on the state of the documents, the number of systems to connect and the access rules. A prototype comes quickly. Most of the time goes into the document pipeline, evaluation, monitoring, access control and testing with users. A prototype released as a production system usually costs more time later, in fixing retrieval quality.
Start with a general multilingual model and measure it on your evaluation set. If its precision is good enough for your use, fine-tuning may not pay off; if not, train it on query-passage pairs from your own corpus. Fine-tune only after the pipeline and the evaluation are stable, or the results will mislead you.
A RAG demo works on a few clean documents. A system people rely on depends on parsing scans and mixed languages, retrieval tuned to your vocabulary, access control and evaluation that catches errors before users do. opengate designs and builds these systems end to end on your own data. Start with one workflow, and we measure what it gives back.
