You launched a live chat on your site; clients ask the same questions, and operators drown in routine. We've seen this in dozens of projects: an AI bot handles 70-80% of first-line queries, leaving complex tasks to humans. Integration with Jivo, LiveChat, or Carrot Quest is not just "attaching" an API—it's building a pipeline: from receiving a message to delivering a meaningful response that considers history and business rules. We have 5 years of AI integrations and over 100 projects completed.
Problems We Solve
Context Loss When Transferring to Operator
A client has chatted with the bot for 10 minutes—the bot knows the whole history. When escalating to an operator, the bot sends a brief summary, not the raw log. We implement this with LangChain's ConversationSummaryMemory compressing the dialog into 3-4 sentences. Without this mechanic, the operator spends time reading the log, and NPS drops by 15-20%.
Latency p99 Above 3 Seconds
If the model takes longer than 2.5 seconds to respond, the client leaves. The solution—quantization to INT4 and vLLM with continuous batching. In tests with LLaMA 3 70B, p99 dropped from 4.2s to 1.2s—vLLM provides 3x lower latency than standard inference. For Carrot Quest, we additionally cache frequent requests with Redis (TTL 30 minutes)—covering 40% of dialogues without calling the LLM.
Complex Escalation Rules
You can't just transfer everything. We use a rule system: if a user asks "Where is my order?" three times—immediate escalation, even with high confidence. Or if sentiment is negative (library cardiffnlp/twitter-roberta-base-sentiment), the bot responds politely and offers an operator. Result: the operator gets only 20% of clients, already with context.
How We Integrate AI Bot with Jivo, LiveChat, and Carrot Quest
From our practice: for one online store, we deployed a bot on FastAPI inside Kubernetes. Stack:
- Model: OpenAI GPT-4o (temperature 0.3) with a system prompt describing the product catalog
- Embeddings:
text-embedding-3-small(1536-dim) for RAG over the catalog (ChromaDB) - Escalation: rules based on
langchainwith a confidence threshold of 0.7 and stop words - Deploy: SageMaker with Triton Inference Server running two models (LLM + embedding)
Jivo sends a webhook (see below), FastAPI processes it, invokes the LangChain chain. Response is a JSON with fields event, chat_id, message. Processing time—1.8s at p95.
from fastapi import FastAPI app = FastAPI() @app.post("/jivo-webhook") async def handle_jivo(data: dict): event = data.get("event") if event == "client_message": client_message = data["message"]["text"] chat_id = data["chat_id"] response = await ai_bot.process(client_message) return { "event": "bot_message", "chat_id": chat_id, "message": {"type": "text", "text": response} } return {"event": "ignore"} Jivo supports quick reply buttons, operator transfer (escalation), and typing status. The AI bot works until an escalation condition triggers.
Carrot Quest API
Carrot Quest is a more functional platform with its own no-code bot builder and Python SDK. For custom logic: Webhook Triggers + API for sending messages.
Carrot Quest's feature: rich user data—browsing history, site events. The AI bot can consider: "You've viewed product X several times—want to ask a question?"
AI Modes in Live Chat
| Mode | How It Works | When Suitable |
|---|---|---|
| Bot-first | AI responds first, transfers to operator if needed | High support load (10+ requests/day) |
| Operator-assist | Operator writes response, AI suggests variants | Complex consultations needing product knowledge |
| After-hours bot | AI active only outside working hours | Clients in different time zones, wait time drops |
Transition rule to operator: explicit request + low AI confidence + negative sentiment + VIP client tag from CRM.
Model Comparison for Live Chat
| Model | Response Quality | Latency p99 (average) | Cost per Token |
|---|---|---|---|
| GPT-4o | High | 1.8s | Medium |
| Claude 3.5 | High | 2.0s | Above Medium |
| LLaMA 3 70B (INT4) | Medium | 1.2s | Low |
| Mistral Large | Medium | 1.5s | Low |
Model choice depends on your priorities: maximum quality or data privacy. For confidential data we use open-source models with local deployment.
How the AI Bot Decides When to Transfer to Operator
A scoring system: the model returns response confidence (logprobs). If below 0.7—escalation. Additionally: stop words ("operator", "real person"), negative sentiment (model nlptown/bert-base-multilingual-uncased-sentiment), more than two repeated client questions. All configurable to your rules.
Which LLM Models We Use and Why
Proprietary (GPT-4o, Claude 3.5) for maximum quality. Open-source (LLaMA 3 70B, Mistral Large) for confidential data or cost reduction. In production we deploy via vLLM with INT4 quantization: latency p99 below 1.5s on 70B models. If fine-tuning for a specific domain is needed, we use LoRA for 2-3 epochs. In practice, GPT-4o handles 90% of queries without escalation, while LLaMA 3 only 75%.
How does RAG work in our bot?
RAG (Retrieval-Augmented Generation) is a mechanism that augments the model's response with facts from your knowledge base. We use ChromaDB for vector search: all documents (FAQ, articles, catalog) are split into chunks, each chunk is converted into an embedding (vector) of 1536 dimensions via text-embedding-3-small. The incoming query is also vectorized, the nearest chunks are found (top-k=5), and they are inserted into the prompt along with the question. This gives up-to-date answers without retraining the model.
Work Process
- Analytics & Architecture—we analyze your knowledge base, configure RAG, design LangChain chains.
- Development—write integration code, configure models (fine-tuning or LoRA if needed), test on historical dialogues.
- Testing—A/B test AI vs operators (metrics: resolved dialogues, CSAT, average time).
- Deployment—to your server or our cloud cluster, set up CI/CD pipeline via GitHub Actions.
- Training—instructions for operators on how to take over the chat and how AI changes standard responses.
Timeline Estimates
Basic integration (one channel, one model)—from 2 weeks. If you need RAG over catalog, two models, multiple channels—up to 4 weeks. Cost is calculated individually, based on complexity and data volume. We provide a code warranty (30 days free support) and a certificate for the solutions used, and guarantee SLA (response time p99 < 2s, availability 99.9%).
What's Included
- Architectural scheme (PDF)
- Source code of the bot with comments
- Deployment instructions (Docker Compose, Kubernetes)
- Metrics dashboard (Grafana + Prometheus)—handled/escalated/average time
- 30 days of support after launch
Want to automate your support? Contact us for a consultation and a rough project estimate. Order a pilot on one channel in 2 weeks.