Developing an AI Agent Trained on a Departing Employee's Data

A Key Employee Leaves — Expertise Leaves with Them

The knowledge base remains, but finding answers is a problem. A standard LLM doesn't know your terminology and processes. An AI agent trained on that employee's data can answer as they would. We develop such agents turnkey: from data collection to production.

Why a Standard LLM Falls Short

A general model gives generic answers from the internet. It doesn't know that in your company "contract approval" goes through three levels. It doesn't know Confluence has a report template and Jira has a history of similar tasks. Accuracy on corporate processes without fine-tuning is about 67%. An agent trained on your data raises it to 91%.

What Problems We Solve

• Loss of expertise when an employee leaves. Their unique knowledge of processes, decisions, and contacts stays in their heads. An AI agent captures it in the model and RAG index.
• Unstructured data. 80% of corporate knowledge is in Confluence, Jira, email. We collect, clean, and index it into a vector database (Qdrant, pgvector).
• Slow answer retrieval. Instead of 15 minutes searching Confluence — one query to the agent. Support ticket reduction by 34%.

Which AI Agent Training Approach to Choose?

The hybrid approach is the only one that ensures both style accuracy and freshness. We use it in all production projects. Learn more about RAG on Wikipedia.

How We Collect and Prepare Data

Data sources: Confluence (pages), Jira (resolved tickets), email correspondence (anonymized), corporate files (PDF, DOCX).

Preparation process:

1. Data collection via API (Confluence REST, Jira API, IMAP for email).
2. Cleaning: html-to-text, deduplication, quality filtering (remove answers shorter than 50 tokens).
3. Synthetic Q&A generation from documents — we use GPT-4o-mini to create up to 10 pairs per document.
4. Format conversion: OpenAI messages format (system/user/assistant).

Example data collector code:

from pathlib import Path from typing import Generator import json class CorporateDataCollector: """Collects data from corporate sources""" async def collect_from_confluence(self, space_keys: list[str]) -> list[dict]: """Confluence pages""" docs = [] for space in space_keys: pages = await confluence_client.get_all_pages(space) for page in pages: content = await confluence_client.get_page_content(page["id"]) docs.append({ "source": "confluence", "id": page["id"], "title": page["title"], "content": html_to_text(content), "updated_at": page["version"]["when"], "labels": page.get("labels", []), "space": space, }) return docs async def collect_from_email_threads( self, email_accounts: list[str], filter_subjects: list[str] = None, anonymize_pii: bool = True, ) -> list[dict]: """Email threads as conversation training data""" threads = [] for account in email_accounts: emails = await gmail_client.get_threads(account, filter_subjects) for thread in emails: if len(thread["messages"]) >= 2: # Convert thread to dialog format dialog = self.format_as_dialog(thread["messages"]) if anonymize_pii: dialog = await self.anonymize_pii(dialog) threads.append(dialog) return threads async def collect_from_tickets( self, jira_project: str, status: str = "Done", limit: int = 5000, ) -> list[dict]: """Resolved tickets as Q&A pairs""" tickets = await jira_client.get_issues( jql=f"project={jira_project} AND status={status}", fields=["summary", "description", "comments", "resolution"], limit=limit, ) qa_pairs = [] for ticket in tickets: if ticket.get("comments"): qa_pairs.append({ "question": f"{ticket['summary']}\n{ticket.get('description', '')[:500]}", "answer": self.extract_resolution(ticket), "source": "jira", "ticket_id": ticket["id"], }) return qa_pairs 

class FinetuningDatasetBuilder: async def build_instruction_dataset( self, raw_docs: list[dict], qa_pairs: list[dict], target_format: str = "openai", # "openai", "alpaca", "sharegpt" ) -> list[dict]: dataset = [] # From documents — generate Q&A via LLM for doc in raw_docs: qa_from_doc = await self.generate_qa_from_document(doc["content"]) for qa in qa_from_doc: if target_format == "openai": dataset.append({ "messages": [ {"role": "system", "content": "You are a corporate assistant for the company. Answer employee questions."}, {"role": "user", "content": qa["question"]}, {"role": "assistant", "content": qa["answer"]}, ] }) # From tickets — ready pairs for qa in qa_pairs: if target_format == "openai": dataset.append({ "messages": [ {"role": "system", "content": "You are a technical support assistant."}, {"role": "user", "content": qa["question"]}, {"role": "assistant", "content": qa["answer"]}, ] }) # Deduplication and filtering dataset = self.deduplicate(dataset) dataset = self.filter_quality(dataset, min_answer_length=50) return dataset async def generate_qa_from_document(self, document_text: str) -> list[dict]: """Generate Q&A pairs from a document""" response = await openai_client.chat.completions.create( model="gpt-4o-mini", messages=[{ "role": "user", "content": f"""Create 5-10 questions and answers from the following document. Questions should be as real employees would ask. Answers should be complete and accurate. Document: {document_text[:3000]} Return JSON: [{{"question": "...", "answer": "..."}}]""" }], ) return json.loads(response.choices[0].message.content) def filter_quality(self, dataset: list[dict], min_answer_length: int) -> list[dict]: """Filter low-quality data""" filtered = [] for item in dataset: messages = item.get("messages", []) assistant_msg = next((m for m in messages if m["role"] == "assistant"), None) if assistant_msg and len(assistant_msg["content"]) >= min_answer_length: filtered.append(item) return filtered 

How the Hybrid Architecture Works: fine-tune + RAG

The fine-tuned model knows the company's style and terminology. RAG adds current documents. We combine them in one agent:

from sentence_transformers import SentenceTransformer from openai import OpenAI from qdrant_client import QdrantClient class HybridCorporateAgent: """Combines a fine-tuned model with company style and RAG with current knowledge""" def __init__(self): # Fine-tuned model knows style and terminology self.finetuned_client = OpenAI(base_url="http://vllm-server:8000/v1") self.finetuned_model = "company-assistant-ft-v2" # RAG for current documents self.embed_model = SentenceTransformer("BAAI/bge-m3") self.vector_db = QdrantClient(host="qdrant-server") async def answer(self, question: str, user_context: dict = None) -> dict: # Step 1: Retrieve relevant documents query_embedding = self.embed_model.encode(question) relevant_docs = self.vector_db.search( collection_name="corporate_docs", query_vector=query_embedding, limit=5, score_threshold=0.6, query_filter=self.build_access_filter(user_context), # Access permissions ) # Step 2: Build context context = "\n\n".join([ f"[{doc.payload['title']}]: {doc.payload['content']}" for doc in relevant_docs ]) # Step 3: Respond with fine-tuned model and RAG context response = self.finetuned_client.chat.completions.create( model=self.finetuned_model, messages=[{ "role": "system", "content": f"You are a corporate assistant. Use the documents as the source of truth.\n\nDocuments:\n{context}" }, { "role": "user", "content": question, }], temperature=0.1, ) return { "answer": response.choices[0].message.content, "sources": [{"title": d.payload["title"], "score": d.score} for d in relevant_docs], } def build_access_filter(self, user_context: dict): """Access permission filtering — employee sees only their documents""" if not user_context: return None department = user_context.get("department", "all") clearance = user_context.get("clearance", "public") return { "must": [ {"key": "access_level", "match": {"any": [clearance, "public"]}}, {"key": "departments", "match": {"any": [department, "all"]}}, ] } 

Case Study: IT Company, 300 Employees

A client — an IT company with 300 employees — had a senior developer leaving who managed key processes. Over 5 years he accumulated 8,000 Confluence pages and participated in solving 12,000 Jira tickets. We collected and cleaned data in 3 weeks, generated 45,000 synthetic Q&A pairs, fine-tuned GPT-4o-mini on 60,000 examples (3 epochs), and loaded all documents into Qdrant. Results: accuracy on corporate processes 91% vs. 67% for base GPT-4o, correct terminology 97% vs. 43%, support ticket reduction by 34%.

What's Included in the Work

How Long Does Development Take?

From 8 to 13 weeks depending on data volume and quality. Most time goes to collection and cleaning — you can't speed it up without losing quality. Cost is determined individually based on the number of sources and required accuracy.

Typical Mistakes When Training an AI Agent

• Ignoring access rights. Without filtering, an employee may see documents from another department. Our architecture includes build_access_filter that checks department and clearance level.
• Using only RAG. Without fine-tuning, the model doesn't learn the corporate style — answers sound like from the internet, not a colleague.
• Weak quality filtering. If bad answers (shorter than 50 tokens or without facts) get into the dataset, quality drops. We filter by minimum length and semantic similarity.
• No testing on real questions. Synthetic tests don't reveal gaps. We test on 500+ questions from future users.

Why Order Development from Us?

5 years of experience in NLP and MLOps, 30+ deployed AI agents. We guarantee the agent will answer with at least 85% accuracy on specified processes. We use only proven tools: Hugging Face, Qdrant, vLLM, ONNX Runtime. After delivery, we provide support and retraining as new data accumulates.

Get a consultation — we will analyze your data and suggest the optimal approach. Turnkey development from scratch or integration into existing infrastructure. Order AI agent development — preserve your team's expertise.