# Developing a Llama-Based Chatbot for CI/CD Question Answering: A Case Study at Ericsson

**Authors:** Daksh Chaudhary, Sri Lakshmi Vadlamani, Dimple Thomas, Shiva Nejati, Mehrdad Sabetzadeh  
**arXiv:** [2408.09277](https://arxiv.org/abs/2408.09277)  
**Date:** 2024-08-17  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2408.09277v1/

## Key points

- Authors developed a llama-based chatbot for CI/CD question answering at Ericsson
- Chatbot uses retrieval-augmented generation (RAG) model for accuracy and relevance
- Ensemble retriever with BM25 and embedding retrievers showed superior performance
- Chatbot provided fully correct responses for 61.11% of questions, partially correct answers for 26.39%, and incorrect answers for 12.50%
- Error analysis conducted to identify causes of inaccuracies
- Lessons learned during development process reflected upon
- Future directions proposed to enhance chatbot accuracy
- Research accepted for presentation at ICSME 2024

## Summary

In their paper titled "Developing a Llama-Based Chatbot for CI/CD Question Answering: A Case Study at Ericsson," authors Daksh Chaudhary, Sri Lakshmi Vadlamani, Dimple Thomas, Shiva Nejati, and Mehrdad Sabetzadeh present their experience in creating a chatbot tailored for continuous integration and continuous delivery (CI/CD) inquiries within the context of Ericsson, a prominent telecommunications company. The chatbot is specifically designed to navigate the intricacies of CI/CD documentation at Ericsson by utilizing a retrieval-augmented generation (RAG) model to enhance both accuracy and relevance. Through empirical evaluation on industrial CI/CD-related queries, the authors found that an ensemble retriever incorporating BM25 and embedding retrievers demonstrated superior performance. When compared against a set of 72 ground truth CI/CD questions and answers from Ericsson, the most accurate configuration of the chatbot provided fully correct responses for 61.11% of the questions, partially correct answers for 26.39%, and incorrect answers for 12.50%. The paper delves into an error analysis of the partially correct and incorrect responses to identify underlying causes of inaccuracies, offering valuable insights for further refinement. Additionally, the authors reflect on lessons learned during the development process and propose future directions aimed at enhancing the accuracy of their chatbot. This research has been accepted for presentation at the 40th IEEE International Conference on Software Maintenance and Evolution (ICSME 2024), showcasing its significance in advancing knowledge within the field of AI-driven question answering systems in industrial settings like Ericsson.