The Problem: 400+ Support Tickets Per Day, 3 Agents
One of our e-commerce clients was drowning in support requests. Their 3-person support team was handling 400+ tickets daily — mostly repetitive questions about order status, returns, and shipping. Response times had ballooned to 18 hours, CSAT scores were dropping, and hiring more agents wasn't in the budget.
They came to Axomble with a clear ask: build an AI chatbot that handles the repetitive stuff so our team can focus on complex issues.
Our Approach: GPT-4 + RAG on Their Knowledge Base
We didn't want to build a rule-based bot with decision trees — those break the moment a customer phrases something differently. Instead, we built a Retrieval-Augmented Generation (RAG) system:
- Knowledge base ingestion: We indexed their 200+ help articles, return policies, shipping docs, and product FAQs into a vector database (Pinecone)
- GPT-4 for reasoning: The chatbot uses GPT-4 to understand customer intent, retrieve relevant context, and generate accurate, natural-language responses
- Order API integration: We connected the bot to their Shopify API so it could look up real order status, tracking numbers, and return eligibility in real-time
- Escalation logic: If the bot's confidence score drops below 0.7 or the customer asks for a human, it transfers to a live agent with full conversation context
The Tech Stack
Here's what we used:
- LLM: OpenAI GPT-4 via API (with function calling for structured actions)
- Vector DB: Pinecone for knowledge base embeddings
- Backend: Node.js with Express, deployed on AWS ECS
- Widget: Custom React chat widget embedded on their Shopify storefront
- Monitoring: Custom dashboard tracking resolution rate, escalation rate, and cost per conversation
The Build Process: 4 Weeks, Start to Finish
Week 1: Discovery and knowledge base preparation. We cleaned and chunked their help articles, set up the vector database, and built the initial RAG pipeline.
Week 2: Core chatbot development. Built the conversation engine, connected the Shopify API, and implemented escalation logic.
Week 3: Testing with real conversations. We replayed 500 historical support tickets through the bot and measured accuracy. Initial accuracy was 82% — we tuned prompts and retrieval parameters to push it to 91%.
Week 4: Deployment and soft launch. We deployed to 20% of traffic first, monitored for 3 days, then rolled out to 100%.
Results After 90 Days
The numbers speak for themselves:
- 70% of tickets resolved without human intervention (up from 0%)
- Average response time dropped from 18 hours to 8 seconds
- CSAT score improved from 3.2 to 4.6 (out of 5)
- Support team reduced from 3 to 1 agent who handles only complex escalations
- Cost per conversation: $0.12 vs $4.50 for human agent handling
What We Learned
RAG quality depends on knowledge base quality. We spent almost as much time cleaning and structuring the help articles as we did building the bot. Garbage in, garbage out — even with GPT-4.
Confidence scoring is essential. The bot needs to know when it doesn't know. Our 0.7 threshold catches most hallucinations before they reach customers.
Start with a narrow scope. We launched with only order status, returns, and shipping queries — not every possible question. This kept accuracy high and gave us room to expand.
Want to Build Something Similar?
If your support team is spending most of their time on repetitive queries, an AI chatbot can free them up within weeks. Book a free strategy call and we'll map out what's possible for your specific use case.
