Search Results for « LLM-based Multi-Agent Reinforcement Learning »

Communication is an effective mechanism for coordinating the behaviors of multiple agents, broadening their views of the environment, and to support their collaborations. In the field of multi-agent deep reinforcement learning (MADRL), agents can improve the overall learning performance and achieve their objectives by communication. Agents can communicate various types of messages, either to all agents or to specific agent groups, or conditioned on specific constraints. With the growing body of research work in MADRL with communication (Comm-MADRL), there is a lack of a systematic and structural approach to distinguish and classify existing Comm-MADRL approaches. In this paper, we survey recent works in the Comm-MADRL field and consider various aspects of communication that can play a role in designing and developing multi-agent reinforcement learning systems. With these aspects in mind, we propose 9 dimensions along which Comm-MADRL approaches can be analyzed, developed, and compared. By projecting existing works into the multi-dimensional space, we discover interesting trends. We also propose some novel directions for designing future Comm-MADRL systems through exploring possible combinations of the dimensions.

Fleets of physical components, deployed artificial-intelligence (AI) models, and same-role AI agents degrade over time and require replacement or retraining. Each intervention has a cost, while each batch adds shared setup, validation, or release costs. Evidence is buried in work orders, transcripts, and deployment logs. Existing decompositions have unmeasured optimality gaps, reinforcement learning provides no certificate, and both require a new solve per fleet. Language-model studies emphasize extraction accuracy rather than maintenance costs from reading errors. We introduce a large-language-model (LLM)-observed multi-agent maintenance framework with an amortized reinforcement-learning policy. A zero-shot LLM converts unit records into structured, auditable events describing actions, replacement or retraining status, confidence, and abstention. A tracker maps events to fleet health counts, while an exact count-based model generates optimal actions to train a reusable policy that transfers zero-shot to new fleets. We prove the state reduction and optimal renewal ordering, and certify each solved action against the Bellman equation. Across nine degradation datasets, the policy adds 1.00% cost. On logs, the reader matches supervised baselines with scarce labels. In 42,000 agent episodes, coordinated retraining saves 11.9% in graphicsprocessing-unit (GPU) seconds, while a 7-billion-parameter reader cuts text-induced maintenance regret from 56.0% to 31.7% with zero false resets.

We propose a \emph{collaborative} multi-agent reinforcement learning algorithm named variational policy propagation (VPP) to learn a \emph{joint} policy through the interactions over agents. We prove that the joint policy is a Markov Random Field under some mild conditions, which in turn reduces the policy space effectively. We integrate the variational inference as special differentiable layers in policy such that the actions can be efficiently sampled from the Markov Random Field and the overall policy is differentiable. We evaluate our algorithm on several large scale challenging tasks and demonstrate that it outperforms previous state-of-the-arts.

Multi-agent systems (MAS) have gained popularity due to their effectiveness in diverse applications. Among these, decentralized approaches, which rely on inter-agent communication, have emerged as a promising alternative to centralized MAS coordination methods. However, existing communication protocols often learn abstract representations that are difficult for humans to interpret. Also, the limited wireless bandwidth in real-world applications imposes a significant challenge on efficient communication among agents. To address these issues, in this paper, we consider a multi-robot navigation scenario and propose a goal-oriented human-interpretable feature communication protocol, enabling robots to transmit only the most semantically important features. Building on this protocol, we first introduce a Multi-Agent Proximal Policy Optimization (MAPPO)-based Bandwidth-Adaptive Dual-Policy (MAPPO-BDP) framework, which employs a dual-policy architecture to jointly optimize feature selection and navigation under bandwidth constraints. However, as MAPPO-BDP requires retraining for different bandwidth constraints, we further propose a Large Language Model (LLM)-based Bandwidth-Adaptive Dual-Policy (LLM-BDP) framework. By leveraging LLM-based multi-agents, LLM-BDP can adapt to varying bandwidth constraints by simply modifying the input prompt, thereby eliminating the need for retraining. Experimental results show that both MAPPO-BDP and LLM-BDP frameworks successfully complete navigation tasks within a limited number of steps by effectively communicating the most important features under bandwidth constraints. We demonstrate that LLM-BDP achieves better reasoning adaptability by utilizing more advanced LLMs and engineering example prompts. Notably, with human-interpretable feature communication protocol, robots trained with the MAPPO-BDP framework can seamlessly collaborate with robots from the LLM-BDP framework to complete tasks, even without co-training, demonstrating the practical applicability of the proposed communication protocol in new scenarios.

Effective communication is pivotal for addressing complex collaborative tasks in multi-agent reinforcement learning (MARL). Yet, limited communication bandwidth and dynamic, intricate environmental topologies present significant challenges in identifying high-value communication partners. Agents must consequently select collaborators under uncertainty, lacking a priori knowledge of which partners can deliver task-critical information. To this end, we propose Interference-Aware K-Step Reachable Communication (IA-KRC), a novel framework that enhances cooperation via two core components: (1) a K-Step reachability protocol that confines message passing to physically accessible neighbors, and (2) an interference-prediction module that optimizes partner choice by minimizing interference while maximizing utility. Compared to existing methods, IA-KRC enables substantially more persistent and efficient cooperation despite environmental interference. Comprehensive evaluations confirm that IA-KRC achieves superior performance compared to state-of-the-art baselines, while demonstrating enhanced robustness and scalability in complex topological and highly dynamic multi-agent scenarios.

Multi-agent systems (MAS) have gained popularity due to their effectiveness in diverse applications. Among these, decentralized approaches, which rely on inter-agent communication, have emerged as a promising alternative to centralized MAS coordination methods. However, existing communication protocols often learn abstract representations that are difficult for humans to interpret. Also, the limited wireless bandwidth in real-world applications imposes a significant challenge on efficient communication among agents. To address these issues, in this paper, we consider a multi-robot navigation scenario and propose a goal-oriented human-interpretable feature communication protocol, enabling robots to transmit only the most semantically important features. Building on this protocol, we first introduce a Multi-Agent Proximal Policy Optimization (MAPPO)-based Bandwidth-Adaptive Dual-Policy (MAPPO-BDP) framework, which employs a dual-policy architecture to jointly optimize feature selection and navigation under bandwidth constraints. However, as MAPPO-BDP requires retraining for different bandwidth constraints, we further propose a Large Language Model (LLM)-based Bandwidth-Adaptive Dual-Policy (LLM-BDP) framework. By leveraging LLM-based multi-agents, LLM-BDP can adapt to varying bandwidth constraints by simply modifying the input prompt, thereby eliminating the need for retraining. Experimental results show that both MAPPO-BDP and LLM-BDP frameworks successfully complete navigation tasks within a limited number of steps by effectively communicating the most important features under bandwidth constraints. We demonstrate that LLM-BDP achieves better reasoning adaptability by utilizing more advanced LLMs and engineering example prompts. Notably, with human-interpretable feature communication protocol, robots trained with the MAPPO-BDP framework can seamlessly collaborate with robots from the LLM-BDP framework to complete tasks, even without co-training, demonstrating the practical applicability of the proposed communication protocol in new scenarios.

Team formation and the dynamics of team-based learning have drawn significant interest in the context of Multi-Agent Reinforcement Learning (MARL). However, existing studies primarily focus on unilateral groupings, predefined teams, or fixed-population settings, leaving the effects of algorithmic bilateral grouping choices in dynamic populations underexplored. To address this gap, we introduce a framework for learning two-sided team formation in dynamic multi-agent systems. Through this study, we gain insight into what algorithmic properties in bilateral team formation influence policy performance and generalization. We validate our approach using widely adopted multi-agent scenarios, demonstrating competitive performance and improved generalization in most scenarios.

Cross-organization cyber defense must reconcile collaborative learning with privacy and adversarial robustness — yet standard federated learning ships full gradient tensors, leaking sensitive posture and inviting Byzantine manipulation. We present FedMARL-LTI, a federated multi-agent reinforcement learning framework whose architecture answers both pressures with a single decision: organizations share neither raw data nor model weights, only differentially-private 768-dimensional semantic threat embeddings. The contribution is fourfold. (1) Semantic Abstraction (SA) channel: per organization, each round, the local gradient is summarized by an LLM, projected to a 768-dim embedding, L2-clipped, and Gaussian-noised before any numeric quantity leaves the host. The bottleneck reduces the per-element noise scale from O(√(d_model )) to O(√m) with m=768≪dmodel≈3×105. (2) Formal privacy analysis: the SA+DP cascade satisfies (ε,δ)-DP and bounds per-round mutual-information leakage by min{Ttoklog2V, m⁄2 log2 (1+C2/(mσ2))} (Theorem 1), with Rényi composition over T federation rounds (Theorem 2). (3) Byzantine-resilient ClippedClustering aggregator combining L2 clipping with cosine-similarity clustering. (4) Hierarchical MARL policy with threat-profile-aware LLM-IRR reward shaping, wired end-to-end and disclosed honestly (the LLM call is currently stubbed with a deterministic projection for reproducibility). We evaluate on CybORG CAGE-4 with n=5 organizations, 30 federation rounds × 5 episodes × 100 steps per round. The SA channel adds statistical-zero utility cost vs. no-privacy baseline: SA-only Δreward = -0.66 (t=+0.31, NS), dual SA + Weight-DP Δreward = +1.90 (t=-0.71, NS), all N=5 seeds, all |t|< 1.3. A controlled signal/noise probe confirms a 19.58× improvement of SA over Weight-DP at fixed DP budget — matching the predicted √(d/m)≈19.8. Under Byzantine sign_flip at 30% (N=15), ClippedClustering is directionally strongest (F1=0.025 vs FedAvg 0.020, Krum 0.016) but the edge is not statistically significant (CC vs Krum t=+1.59, p=0.15, d=+0.58; the earlier N=5 “3.4×” gap was small-sample optimism, §5.2); its decisive Byzantine win is the harsher random_noise attack, where FedAvg diverges to NaN and Krum collapses to 0.002 while ClippedClustering survives at 0.020 (§5.7, Cohen’s d=+3.77). The cooperative-PPO family (MAPPO, IPPO) outperforms value/actor-critic (QMIX, MADDPG) by ≈20 reward units, p< 0.001. All host-level F1 values stay below 0.05 at the 15K-step training horizon used here; the relative claims of the paper (privacy zero-cost, ClippedClustering’s decisive Byzantine win on the harshest attacks per §5.7, cooperative-PPO dominance) are unaffected by this scope. A 200K-step long-horizon replication (§6.3 L1) lifts F1 above the 15K plateau (to ≈0.044, N=5) — confirming that horizon, not the privacy/Byzantine machinery, gates absolute accuracy — but a finer 60-checkpoint run shows the climb is volatile and non-monotonic and does not reach deployment-grade, an honest stability-not-compute limitation. We release all 141 raw run JSON outputs (Phases 1–3, the L4 backend comparison, and the algorithm/aggregator baselines), the figures, and analysis scripts for replication.

Agreement Technologies refer to open computer systems in which autonomous software agents interact with one another, typically on behalf of humans, in order to come to mutually acceptable agreements. With the advance of AI systems in recent years, it has become apparent that such agreements, in order to be acceptable to the involved parties, must remain aligned with ethical principles and moral values. However, this is notoriously difficult to ensure, especially as different human users (and their software agents) may hold different value systems, i.e. they may differently weigh the importance of individual moral values. Furthermore, it is often hard to specify the precise meaning of a value in a particular context in a computational manner. Methods to estimate value systems based on human-engineered specifications, e.g. based on value surveys, are limited in scale due to the need for intense human moderation. In this article, we propose a novel method to automatically \emph{learn} value systems from observations and human demonstrations. In particular, we propose a formal model of the \emph{value system learning} problem, its instantiation to sequential decision-making domains based on multi-objective Markov decision processes, as well as tailored preference-based and inverse reinforcement learning algorithms to infer value grounding functions and value systems. The approach is illustrated and evaluated by two simulated use cases.