# Adapting a Foundation Model for Space-based Tasks

**Authors:** Matthew Foutter, Praneet Bhoj, Rohan Sinha, Amine Elhafsi, Somrita Banerjee, Christopher Agia, Justin Kruger, Tommaso Guffanti, Daniele Gammelli, Simone D'Amico, Marco Pavone  
**arXiv:** [2408.05924](https://arxiv.org/abs/2408.05924)  
**Date:** 2024-08-12  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2408.05924v1/

## Key points

- Foundation models, such as large language models, show promise in providing contextual understanding for robots in unstructured environments.
- Core challenges in applying foundation models to space robotics include scalability of ground-in-the-loop operations, generalizing prior knowledge to new environments, and handling multi-modality in tasks and sensor data.
- Preliminary investigation focused on using pretrained multi-modal foundation models in a space robotics scenario where a rover navigates a planetary environment.
- Existing vision-language models lack visual reasoning capabilities for space-based applications but can be significantly improved through fine-tuning on programmatically generated tasks.
- Fine-tuning VLMs with domain-specific data like Martian imagery can enhance decision-making capabilities and efficiency in robotic missions beyond Earth's atmosphere.

## Summary

, , , , 

Foundation models, such as large language models, have shown promise in endowing robots with contextual understanding to navigate complex tasks in unstructured environments. In the realm of space robotics, three core challenges drive the need for adapting foundation models to space-based applications: scalability of ground-in-the-loop operations, generalizing prior knowledge to novel environments, and handling multi-modality in tasks and sensor data. To address these challenges, a preliminary investigation was conducted on the application of pretrained multi-modal foundation models in the space domain. The focus was on a space robotics scenario where a rover navigates a planetary environment. Language annotations were programmatically generated on the AI4Mars image dataset to evaluate vision-language models (VLMs) across spatial reasoning and navigation tasks inspired by scientific interest identification and motion plan validation. The study revealed that existing VLMs lack visual reasoning capabilities in space-based applications. However, fine-tuning a VLM on programmatically generated tasks significantly enhanced its performance across various visual reasoning tasks. Even with a limited training dataset consisting of only a few thousand images reused for different question-answer pairs, the quality of VLM outputs improved notably. Moving forward, pathways were proposed for extending these findings to orbital in-space applications, marking a promising step towards developing generalist models for space exploration. Additionally, related work highlighted recent advancements in vision-language models trained on internet-scale data and emphasized the importance of incorporating foundation models at different levels of autonomy within robotics systems. Overall, this study underscores the potential of leveraging foundation models in space robotics to overcome key challenges and enhance decision-making capabilities in extraterrestrial environments. By fine-tuning existing models with domain-specific data like Martian imagery, researchers can pave the way for more efficient and effective robotic missions beyond Earth's atmosphere.