Grounded Language Learning in a Simulated 3D World

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learning

   Evan Zheran Liu, Sahaana Suri, Tong Mu et al. · 2023 · Computation and Language · Not summarized yet

   Abstract

   Whereas machine learning models typically learn language by directly training on language tasks (e.g., next-word prediction), language emerges in human children as a byproduct of solving non-language tasks (e.g., acquiring food). Motivated by this observation, we ask: can embodied reinforcement learning (RL) agents also indirectly learn language from non-language tasks? Learning to associate language with its meaning requires a dynamic environment with varied language. Therefore, we investigate this question in a multi-task environment with language that varies across the different tasks. Specifically, we design an office navigation environment, where the agent's goal is to find a particular…

2. Close Not relevant

   The Vector Grounding Problem

   Dimitri Coelho Mollo, Raphaël Millière · 2023 · Computation and Language · AI summary available

   Abstract

   The remarkable performance of large language models (LLMs) on complex linguistic tasks has sparked a lively debate on the nature of their capabilities. Unlike humans, these models learn language exclusively from textual data, without direct interaction with the real world. Nevertheless, they can generate seemingly meaningful text about a wide range of topics. This impressive accomplishment has rekindled interest in the classical 'Symbol Grounding Problem,' which questioned whether the internal representations and outputs of classical symbolic AI systems could possess intrinsic meaning. Unlike these systems, modern LLMs are artificial neural networks that compute over vectors rather than symb…

3. Close Not relevant

   Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality

   Yang Su · 2023 · Computation and Language · AI summary available

   Abstract

   Large Language Models (LLMs) are trained and aligned to follow natural language instructions with only a handful of examples, and they are prompted as task-driven autonomous agents to adapt to various sources of execution environments. However, deploying agent LLMs in virtual reality (VR) has been challenging due to the lack of efficiency in online interactions and the complex manipulation categories in 3D environments. In this work, we propose Voice2Action, a framework that hierarchically analyzes customized voice signals and textual commands through action and entity extraction and divides the execution tasks into canonical interaction subsets in real-time with error prevention from enviro…

4. Close Not relevant

   A framework for the emergence and analysis of language in social learning agents

   Tobias J. Wieczorek, Tatjana Tchumatchenko, Carlos Wert Carvajal et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Artificial neural networks (ANNs) are increasingly used as research models, but questions remain about their generalizability and representational invariance. Biological neural networks under social constraints evolved to enable communicable representations, demonstrating generalization capabilities. This study proposes a communication protocol between cooperative agents to analyze the formation of individual and shared abstractions and their impact on task performance. This communication protocol aims to mimic language features by encoding high-dimensional information through low-dimensional representation. Using grid-world mazes and reinforcement learning, teacher ANNs pass a compressed me…

5. Close Not relevant

   Role-Play with Large Language Models

   Murray Shanahan, Kyle McDonell, Laria Reynolds · 2023 · Computation and Language · AI summary available

   Abstract

   As dialogue agents become increasingly human-like in their performance, it is imperative that we develop effective ways to describe their behaviour in high-level terms without falling into the trap of anthropomorphism. In this paper, we foreground the concept of role-play. Casting dialogue agent behaviour in terms of role-play allows us to draw on familiar folk psychological terms, without ascribing human characteristics to language models they in fact lack. Two important cases of dialogue agent behaviour are addressed this way, namely (apparent) deception and (apparent) self-awareness.

6. Close Not relevant

   Character-LLM: A Trainable Agent for Role-Playing

   Yunfan Shao, Linyang Li, Junqi Dai et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs) can be used to serve as agents to simulate human behaviors, given the powerful ability to understand human instructions and provide high-quality generated texts. Such ability stimulates us to wonder whether LLMs can simulate a person in a higher form than simple human behaviors. Therefore, we aim to train an agent with the profile, experience, and emotional states of a specific person instead of using limited prompts to instruct ChatGPT API. In this work, we introduce Character-LLM that teach LLMs to act as specific people such as Beethoven, Queen Cleopatra, Julius Caesar, etc. Our method focuses on editing profiles as experiences of a certain character and train…

7. Close Not relevant

   A Zero-Shot Language Agent for Computer Control with Structured Reflection

   Tao Li, Gang Li, Zhiwei Deng et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs) have shown increasing capacity at planning and executing a high-level goal in a live computer environment (e.g. MiniWoB++). To perform a task, recent works often require a model to learn from trace examples of the task via either supervised learning or few/many-shot prompting. Without these trace examples, it remains a challenge how an agent can autonomously learn and improve its control on a computer, which limits the ability of an agent to perform a new task. We approach this problem with a zero-shot agent that requires no given expert traces. Our agent plans for executable actions on a partially observed environment, and iteratively progresses a task by identi…

8. Close Not relevant

   OpenAgents: An Open Platform for Language Agents in the Wild

   Tianbao Xie, Fan Zhou, Zhoujun Cheng et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Language agents show potential in being capable of utilizing natural language for varied and intricate tasks in diverse environments, particularly when built upon large language models (LLMs). Current language agent frameworks aim to facilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-level designs. We present OpenAgents, an open platform for using and hosting language agents in the wild of everyday life. OpenAgents includes three agents: (1) Data Agent for data analysis with Python/SQL and data tools; (2) Plugins Agent with 200+ daily API tools; (3) Web Agent for autonomous web brows…