# Analysis of Thompson Sampling for Partially Observable Contextual Multi-Armed Bandits

**Authors:** Hongju Park, Mohamad Kazem Shirani Faradonbeh  
**arXiv:** [2110.12175](https://arxiv.org/abs/2110.12175)  
**Date:** 2021-10-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.12175v1/

## Key points

- The paper focuses on reinforcement learning policy for partially observable contextual multi-armed bandits.
- Contextual multi-armed bandits are models in reinforcement learning for sequential decision-making associated with individual information.
- Thompson Sampling is a widely-used policy for bandits, but little is known about problems where contexts are not fully observed.
- The authors propose a modified version of Thompson Sampling that leverages Bayesian methods to estimate unobserved contexts based on output observations while balancing exploration and exploitation.
- The presented algorithm establishes theoretical performance guarantees, showing that the regret scales logarithmically with time and the number of arms, and linearly with dimension.
- Numerical analyses were conducted by repeating simulations 50 times for each case, reporting two quantities: ||bµ(t)−µ∗|| and Regret(t).
- Parameter estimates converge fast to truth and errors decrease at appropriate rates over time.
- This paper provides an important contribution to reinforcement learning policies for partially observable contextual multi-armed bandits.
- Future studies could explore leveraging other techniques such as adversarial training or incorporating expert knowledge into the model.

## Summary

This paper focuses on the design and analysis of a reinforcement learning policy for partially observable contextual multi-armed bandits. Contextual multi-armed bandits are classical models in reinforcement learning for sequential decision-making associated with individual information. A widely-used policy for bandits is Thompson Sampling, where samples from a data-driven probabilistic belief about unknown parameters are used to select control actions. However, little is known about problems where contexts are not fully observed. The authors propose a modified version of Thompson Sampling that leverages Bayesian methods for balancing exploration and exploitation and estimates unobserved contexts based on the sequence of output observations. The presented algorithm establishes theoretical performance guarantees, showing that the regret scales logarithmically with time and the number of arms, and linearly with dimension. To validate their approach, the authors conduct numerical analyses by repeating simulations 50 times for each case, reporting two quantities: ||bµ(t)−µ∗|| and Regret(t). They show that parameter estimates converge fast to truth and errors decrease at appropriate rates over time. Overall, this paper provides an important contribution to reinforcement learning policies for partially observable contextual multi-armed bandits by introducing a modified version of Thompson Sampling which leverages Bayesian methods to balance exploration and exploitation while estimating unobserved contexts based on output observations. Future studies could explore leveraging other techniques such as adversarial training or incorporating expert knowledge into the model.