How do you add an AI chatbot to an existing web app?

What does the architecture look like?

Keep it simple: your frontend sends a message to one backend endpoint; that endpoint owns the LLM call and any retrieval; the response streams back to a chat component. The model provider stays behind your own service so you can add validation, caching, retries, and swap providers without changing the UI.

• ›Frontend chat component — sends messages, renders a streamed reply.
• ›Backend chat service — the one place that talks to the LLM.
• ›Optional retrieval layer — a vector store for answering from your content.
• ›Provider behind the service — never call the model directly from the browser.

Do you need RAG?

Ask one question: does the bot need to answer from your specific content — docs, policies, product data — or only from general knowledge the model already has? If it needs your content, add retrieval-augmented generation so answers are grounded and can cite sources. If it only needs general capabilities (drafting, explaining, formatting), skip retrieval and save the complexity.

When in doubt, ship the simplest version first without RAG, see where it gives vague or wrong answers about your domain, and add retrieval to close that gap.

Building the chat endpoint

The endpoint validates the request, optionally retrieves context, builds the prompt, calls the model, and streams the result. Putting this behind a service layer means the rest of your app depends on a stable interface, not on a specific provider.

@app.post("/api/chat")
async def chat(req: ChatRequest):
    # 1. validate input
    # 2. (optional) retrieve relevant passages for RAG
    context = await retrieve(req.message) if USE_RAG else None
    prompt = build_prompt(req.message, context)
    # 3. call the model behind a service, with a timeout
    return StreamingResponse(stream_completion(prompt), media_type="text/event-stream")

Streaming to the frontend

Stream tokens as they arrive rather than waiting for the full answer. Streaming makes the bot feel fast and lets users start reading immediately, which matters because model responses can take several seconds. On the client, append chunks to the message as they come in and show a typing indicator until the stream closes.

Reliability and cost

The model call is the easy 10%; the reliability around it is the rest. Set a timeout on every call, retry transient failures, and define what happens when the model is unavailable — a fallback message beats a spinner that never resolves. Control cost by caching repeated questions, keeping prompts as small as the task allows, and routing simple requests to cheaper models.

• ›Timeouts and retries on every model call.
• ›A graceful fallback when the provider is down or slow.
• ›Caching for repeated or near-identical questions.
• ›Prompt-size discipline and model routing to manage token spend.

Common pitfalls

1. 01Rewriting the app around AI instead of adding one contained feature.
2. 02Calling the provider from the browser and leaking the API key.
3. 03Shipping without fallbacks, so a model outage breaks the page.
4. 04Skipping evaluation — you cannot improve answer quality you never measure.
5. 05Ignoring cost until the first large bill arrives.