# Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy

**Authors:** Ioana Ciucă, Yuan-Sen Ting, Sandor Kruk, Kartheik Iyer  
**arXiv:** [2306.11648](https://arxiv.org/abs/2306.11648)  
**Date:** 2023-06-20  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2306.11648v1/

## Key points

- Study explores the application of Large Language Models (LLMs), specifically GPT-4, in Astronomy
- In-context prompting with NASA Astrophysics Data System papers improves GPT-4's performance in hypothesis generation
- Adversarial prompting further enhances GPT-4's ability to extract crucial details and generate meaningful hypotheses
- Astro-GPT workflow involves pre-processing 1000 papers from Galactic Astronomy corpus using langchain library
- Retrieval phase includes similarity search and contextual compression to filter out irrelevant information
- Adversarial experiment designed involving secondary GPT-4 model for critiquing and enhancing generated ideas
- Results show significant improvement in hypothesis quality with adversarial prompting and domain-specific context enrichment
- Experimental setup includes different numbers of papers for hypothesis generation and replication cycles
- Exploration of embeddings and their impact on hypothesis generation is included in the study (results in appendix)

## Summary

, , , , : This study explores the application of Large Language Models (LLMs), specifically GPT-4, in the field of Astronomy. The researchers utilize in-context prompting by supplying GPT-4 with up to 1000 papers from the NASA Astrophysics Data System to investigate how immersing the model in domain-specific literature can improve its performance. The findings indicate a significant enhancement in hypothesis generation when using in-context prompting, and this benefit is further amplified by adversarial prompting. The researchers demonstrate how adversarial prompting empowers GPT-4 to extract crucial details from a vast knowledge base and generate meaningful hypotheses, representing a groundbreaking advancement in utilizing LLMs for scientific research in Astronomy. The study employs an Astro-GPT workflow that involves pre-processing 1000 papers from the Galactic Astronomy corpus using the langchain library. These papers are transformed from PDF to text and segmented into chunks of 1000 tokens each. OpenAI's text-ada-002 embedding model is used to embed these segmented units. The retrieval phase begins with converting chat history and input queries into standalone inputs, which are then embedded. A similarity search is conducted between the embedded query and a vector database. Langchain's contextual compression is utilized to filter out irrelevant information from individual chunks, resulting in final texts that, along with standalone inputs, serve as the foundation for hypothesis formulation by GPT-4. To evaluate the model's capabilities, an adversarial experiment is designed involving a secondary GPT-4 model that critiques generated ideas and suggests potential enhancements. This feedback is reformulated within a feedback-question structure by a third GPT-4 instance and returned to the initial model. The study presents results based on human evaluation of hypotheses and critiques generated by AI models. Adversarial prompting and domain-specific context enrichment significantly enhance the quality of hypothesis generation. The effectiveness of adversarial prompting becomes evident when an extensive context of 1000 papers is provided, leading to substantial improvements in both the quality and consistency of AI judge and AI generator outputs. The experimental setup involves using different numbers of papers (Nk, where k ∈ {1, 10, 100, 1000}) for hypothesis generation by the in-context prompted model. An adversarial response follows from an Adversarial GPT-4 model, which is then reformulated by a moderator GPT-4 instance and fed back to the generator model. This cycle is repeated twice for each Nk and replicated five times in total. The same approach is applied to 1000 papers without resampling, resulting in a total of 60 hypotheses and 40 critiques. The study also includes exploration of embeddings and their impact on hypothesis generation; results are presented in the appendix.