One of the most compelling use cases for enterprise AI is building intelligent chatbots that can assist employees, answer customer queries, or automate internal processes. In this hands-on guide based on Practical RHEL AI, weâll walk through creating a production-ready chatbot using RHEL AIâs integrated toolchain.
Traditional chatbot frameworks often require cobbling together disparate tools, managing complex dependencies, and hoping everything plays nicely together. RHEL AI changes this paradigm by providing:
Before starting, ensure you have:
First, set up your InstructLab environment:
# Create a dedicated workspace
mkdir -p ~/chatbot-project && cd ~/chatbot-project
# Initialize InstructLab
ilab config init
# Download the base Granite model
ilab model download --model-name granite-7b-labInstructLab will configure the necessary directories and download the Granite foundation model, which serves as an excellent starting point for enterprise chatbots.
For an enterprise chatbot, youâll want to inject domain-specific knowledge. Create a taxonomy structure:
# Create taxonomy directories
mkdir -p taxonomy/knowledge/company/policies
mkdir -p taxonomy/knowledge/company/productsAdd your knowledge files in YAML format:
# taxonomy/knowledge/company/policies/qna.yaml
created_by: your-username
version: 3
domain: company_policies
seed_examples:
- context: |
Our company offers 25 days of annual leave for all full-time employees.
Leave requests must be submitted at least 2 weeks in advance through
the HR portal. Manager approval is required for requests longer than
5 consecutive days.
questions_and_answers:
- question: How many days of annual leave do I get?
answer: Full-time employees receive 25 days of annual leave per year.
- question: How do I request time off?
answer: Submit your leave request through the HR portal at least 2 weeks in advance.
- question: Do I need approval for vacation?
answer: Manager approval is required for leave requests longer than 5 consecutive days.InstructLabâs Synthetic Data Generation (SDG) pipeline creates high-quality training examples from your knowledge base:
# Generate synthetic data from your taxonomy
ilab data generate --model granite-7b-lab \
--num-instructions 100 \
--output-dir ./generated-data
# Review the generated data
ls -la ./generated-data/The SDG pipeline uses the teacher model to create diverse question-answer pairs, ensuring comprehensive coverage of your knowledge domain.
With synthetic data ready, fine-tune your chatbot model:
# Start the fine-tuning process
ilab model train \
--model-name granite-7b-lab \
--data-dir ./generated-data \
--output-dir ./fine-tuned-model \
--num-epochs 3 \
--effective-batch-size 16For multi-GPU setups, InstructLab automatically leverages DeepSpeed for distributed training:
# Multi-GPU training with DeepSpeed ZeRO-3
ilab model train \
--model-name granite-7b-lab \
--data-dir ./generated-data \
--output-dir ./fine-tuned-model \
--distributed \
--deepspeed-config zero3Once training completes, deploy your chatbot using vLLM for optimized inference:
# Start the vLLM server with your fine-tuned model
ilab model serve \
--model-path ./fine-tuned-model \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1Your chatbot is now accessible via an OpenAI-compatible API at http://localhost:8000.
Create a simple Python client to interact with your chatbot:
#!/usr/bin/env python3
"""Simple chatbot client for RHEL AI"""
import requests
import json
API_URL = "http://localhost:8000/v1/chat/completions"
def chat(message: str, history: list = None) -> str:
"""Send a message to the chatbot and get a response."""
if history is None:
history = []
messages = history + [{"role": "user", "content": message}]
response = requests.post(
API_URL,
json={
"model": "fine-tuned-model",
"messages": messages,
"temperature": 0.7,
"max_tokens": 512
},
headers={"Content-Type": "application/json"}
)
result = response.json()
return result["choices"][0]["message"]["content"]
def main():
"""Interactive chat loop."""
print("Enterprise Chatbot (powered by RHEL AI)")
print("Type 'quit' to exit\n")
history = []
system_prompt = {
"role": "system",
"content": "You are a helpful enterprise assistant. Answer questions accurately and professionally."
}
history.append(system_prompt)
while True:
user_input = input("You: ").strip()
if user_input.lower() in ['quit', 'exit', 'q']:
break
response = chat(user_input, history)
print(f"Bot: {response}\n")
history.append({"role": "user", "content": user_input})
history.append({"role": "assistant", "content": response})
if __name__ == "__main__":
main()For production deployment, containerize your chatbot:
# Containerfile for RHEL AI Chatbot
FROM registry.redhat.io/rhel-ai/rhel-ai-nvidia-runtime:latest
WORKDIR /app
# Copy fine-tuned model
COPY ./fine-tuned-model /app/model
# Copy application code
COPY ./chatbot_client.py /app/
# Expose the API port
EXPOSE 8000
# Start vLLM server
CMD ["ilab", "model", "serve", "--model-path", "/app/model", "--host", "0.0.0.0", "--port", "8000"]Build and run with Podman:
# Build the container
podman build -t enterprise-chatbot:v1 .
# Run with GPU support
podman run --device nvidia.com/gpu=all \
-p 8000:8000 \
enterprise-chatbot:v1When deploying to production, consider these best practices from Practical RHEL AI:
Building an enterprise chatbot with RHEL AI combines the power of modern LLMs with the stability and security of Red Hat Enterprise Linux. The integrated toolchainâInstructLab for fine-tuning, vLLM for inference, and Podman for containerizationâstreamlines the path from prototype to production.
For deeper coverage of chatbot architectures, multi-turn conversation handling, and RAG integration, refer to Chapters 7-9 of Practical RHEL AI.
Get Practical RHEL AI on Amazon