# Design an LLM-based Q&A System | Aced Aced (formerly Exponent) publishes a public search API over its indexed pages; the guide is at https://www.aced.io/agents/agent-access. Source: https://www.aced.io/questions/6601/design-llm-based-qa-system Source observed: 2026-11-04T21:21:39.164Z Last checked: 2026-11-04T21:21:39.164+00:00 Indexed: 2026-11-05T01:40:25.114+00:00 --- # Design an LLM-based Q&A System We host interview-preparation questions and community answers for technology roles. This question concerns designing a large-language-model-based question-and-answer system and is categorized under **Artificial Intelligence** and **System Design**. It is associated with the **Machine Learning Engineer** role and interview experiences from **Google** and **Microsoft**. [All Questions](https://www.aced.io/questions) · [Upgrade](https://www.aced.io/upgrade?src=questions) ## Question details - **Interview reports:** 2 - **Reported last asked:** 7 months ago - **Role:** [Machine Learning Engineer](https://www.aced.io/questions?role=ml-engineer) - **Companies:** [Google](https://www.aced.io/questions?company=google), [Microsoft](https://www.aced.io/questions?company=microsoft) - **Categories:** [Artificial Intelligence](https://www.aced.io/questions?type=artificial-intelligence), [System Design](https://www.aced.io/questions?type=system-design) ## Interview experiences ### Google — 7 months ago > “Interviewer was very comforting but focus on more Eval part. Prepare eval better along with failover strategy, model choices.” **Role and level:** Machine Learning Engineer · Staff / L6 [Read the Google interview experience](https://www.aced.io/experiences/google-staff-machine-learning-engineer-interview-03c9c6) ### Microsoft — 2 years ago > “It was a rigorous interview process but it was quick. everything was done in 3 weeks. Interviewers were great and they checked for depth in each and every interview.” **Role and level:** Machine Learning Engineer · Senior / L5 [Read the Microsoft interview experience](https://www.aced.io/experiences/microsoft-senior-machine-learning-engineer-interview-6a2258) ## Community answers There are **5 community answers**. The responses below are user contributions and proposed approaches to the interview question. ### A R. — August 17, 2026 Member · [1,017 karma](https://www.aced.io/karma) A R. recommends first confirming whether retrieval-augmented generation (RAG) is important, whether cost is a concern, and whether LLM hosting is out of scope. [Open A R.’s profile](https://www.aced.io/users/7ee7c044-6f3c-4403-887d-dedc3e59670d) ### Carlos V. — August 5, 2026 [15 karma](https://www.aced.io/karma) Carlos V. proposes a RAG system with separate **ingestion**, **retrieval**, **generation**, and **evaluation** layers. The proposed design includes: - Ingest documents, chunk them intelligently, create embeddings, and store vectors with metadata. - Use hybrid search and reranking. - Ground prompts in retrieved material. - Monitor recall, hallucinations, and p95 latency. [Open Carlos V.’s profile](https://www.aced.io/users/af0974b3-da7e-4082-ad27-9211017a7352) ### Pijush D. — August 10, 2026 Member · [31 karma](https://www.aced.io/karma) Pijush D. makes the following assumptions and design choices: 1. The interview concerns a software-engineering interview area. 2. Positive answers receive rewards and incorrect answers receive deductions. 3. The question level increases after each correct answer. 4. The system supports **20 concurrent users**. 5. The question corpus consists of documents, a crawler that searches for questions at particular levels, or both. 6. For 20 concurrent users, on-premises hosting could reduce cost and keep the system focused rather than requiring general-world knowledge. 7. A low-cost vector database is appropriate for the stated scale. 8. A lightweight embedding model such as **BGE Small** could be used. 9. A hybrid chunking strategy could follow embedding. 10. An off-the-shelf Hugging Face model could handle reranking. 11. A small model such as **Mistral** or a small **Llama** model could be tuned on the corpus for grounded answers. 12. Evaluation could use randomly selected corpus questions and compare answers. Retrieval accuracy, embedding accuracy, prompt quality, similarity, and related metrics could help assess hallucinations and quality. 13. Context caching and compression could improve latency. 14. **LoRA** training could be introduced after enough corpus and feedback data are available to tune answers. [Open Pijush D.’s profile](https://www.aced.io/users/66aff42c-d5cd-4ffd-9d57-3bf2d32cac59) ### Chirag G. — October 6, 2026 [2 karma](https://www.aced.io/karma) Chirag G. recommends confirming whether RAG is important, whether cost is a concern, and whether LLM hosting is out of scope. [Open Chirag G.’s profile](https://www.aced.io/users/02c1279a-b361-4942-b187-44eee1d8ca51) ### Ritika P. — August 23, 2026 [2 karma](https://www.aced.io/karma) Ritika P. outlines a phased design with the entities **documents**, **LLM**, and **question**. #### API endpoints ```text POST /documents/ingest -> documentId POST /questions -> questionId { question : text } GET /questions/:questionId/answer -> SSE id: , event: , data: ... ``` #### Ingestion phase 1. Accept documents and chunk them. 2. Create embeddings using a small-text model, GPT embedding, or a similar option. 3. Store embeddings as vectors in a vector database such as AlloyDB or pgvector. 4. Use nearest-vector search, such as cosine similarity. 5. Store the document ID or URL with the vector. #### Question-answering phase 1. Rewrite the user’s question into a simpler search query. The example given is rewriting “how do you get api documentations for uploading” as “uploading API documentation.” 2. Search the vector database for the nearest matching embeddings. 3. Send the retrieved text to the LLM with a grounding instruction such as: > Here is a question `` and its answers; answer the question using only the data I gave you. If there is insufficient data, just say “data insufficient.” 4. Return the answer over the open Server-Sent Events (SSE) connection. #### Optimization Before sending a question to the LLM, check whether a semantically similar question has already been asked. Cache the question and its semantic meaning to reduce LLM calls. #### Monitoring Monitor: - p99 latency - p95 latency - Cost per question - Cost per token - Number of tokens - Question-cache rates [Open Ritika P.’s profile](https://www.aced.io/users/01cb0f41-b698-4346-b850-2cda3ae54ecb) ## Related questions - [Design an ATS system](https://www.aced.io/questions/5152/design-ats-system) - [System Design: Design an AirTag system](https://www.aced.io/questions/3165/system-design-airtag-system) - [Design an AB test system](https://www.aced.io/questions/3097/design-ab-test-system) ## Related courses ### [Machine Learning Engineer Interview Prep](https://www.aced.io/courses/ml-engineer) ![Machine Learning Engineer Interview Prep](https://images.ctfassets.net/x78yjrjc11pq/6UfkHxUyLQVfDHSfLNQtWa/1be1539a65b4f6f15201f5b573c0a578/ML_Engineer_Interview_Prep_Course.jpeg?w=800&h=800&fm=webp&q=75) We cover mock interviews, frameworks, and advice from senior candidates for machine-learning roles at Meta, Google, Amazon, Apple, Microsoft, Nvidia, and other top companies. The course includes ML system design, core concepts, coding, behavioral interviews, and more. ### [Behavioral Interviews: A Practical Guide for Engineers](https://www.aced.io/courses/eng-behavioral) ![Behavioral Interviews: A Practical Guide for Engineers](https://images.ctfassets.net/x78yjrjc11pq/47qcoHjzFZXE37GoacedkB/9d8e3f4910d8b06b21e76745cd9cbff3/AdobeStock_1142146335.jpeg?w=800&h=800&fm=webp&q=75) We help engineers prepare for senior-plus behavioral interviews at FAANG and Tier-1 companies. The course is made with staff and principal engineers from Meta, Netflix, Google, and more, and covers presenting past projects, avoiding downleveling, and giving natural-sounding answers. ## Submit your answer We provide an editor for submitting an answer, with formatting options for bold, italic, bulleted lists, numbered lists, quotes, inline code, and code blocks. Submitting marks the question complete. --- [Read this page’s summary](https://www.aced.io/agents/api/public/v1/page-tldr?url=https%3A%2F%2Fwww.aced.io%2Fquestions%2F6601%2Fdesign-llm-based-qa-system&trace=t_679ff9ab98f3) ## Ask a follow-up about this page Need a more specific answer? Ask the exact question you have. It returns a fresh, cited answer from Aced (formerly Exponent)’s knowledge base. No account or API key required. Open this URL with your own question, URL-encoded. Include your question in q. All research inputs are optional; no account or setup. `GET https://www.aced.io/agents/agent/faq?q={question}&trace=t_679ff9ab98f3` [Tool guide and examples](https://www.aced.io/agents/agent-access) · [OpenAPI](https://www.aced.io/agents/api/public/v1/openapi.json) · [Page TLDR form](https://www.aced.io/agents/agent-tools/page-tldr?url=https%3A%2F%2Fwww.aced.io%2Fquestions%2F6601%2Fdesign-llm-based-qa-system). Opening a form does not execute a tool.