Building AI Chatbots with LLM APIs in Backend Systems

LLM APIs have changed how we build product features. This post covers the backend patterns I've learned from integrating OpenAI and Gemini into production systems.

Streaming Responses

For good UX, stream tokens to the client as they arrive. In Node.js with Express:

app.post('/api/chat', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  const stream = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: req.body.messages,
    stream: true,
  });
  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content || '';
    if (content) {
      res.write(`data: ${JSON.stringify({ content })}
`);
    }
  }
  res.end();
});

Context Window Management

Managing the conversation history to stay within token limits is critical. I use a sliding window approach that preserves the system prompt and recent messages.

Cost Control

LLM APIs are pay-per-token. Production cost control requires: input/output token counting per request, per-user rate limits, caching for repeated queries, and model tiering (use GPT-4o-mini for simple tasks).

The biggest win is always caching — many user queries are semantically similar.