# AXS: A framework for fast astronomical data processing based on Apache Spark

**Authors:** Petar Zečević, Colin T. Slater, Mario Jurić, Andrew J. Connolly, Sven Lončarić, Eric C. Bellm, V. Zach Golkhou, Krzysztof Suberlack  
**arXiv:** [1905.09034](https://arxiv.org/abs/1905.09034)  
**Date:** 2019-05-22  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/1905.09034v1/

## Key points

- Introduction of AXS (Astronomy eXtensions for Spark) framework
- Leveraging Apache Spark for efficient data processing
- Features include online positional cross-matching support and Python library
- Development of distributed ZONES algorithm and data partitioning scheme
- Achievements in fast cross-matching between Gaia DR2 and AllWise datasets
- Facilitation of end-user analyses of large datasets like LSST
- Emphasis on efficiency, scalability, and performance across diverse datasets

## Summary

The paper "AXS: A framework for fast astronomical data processing based on Apache Spark" introduces AXS (Astronomy eXtensions for Spark), an open-source framework designed to analyze vast astronomical catalogs using familiar Python/AstroPy concepts and SQL statements. Leveraging the power of Apache Spark, AXS offers efficient online positional cross-matching support and a Python library for essential operations. The authors also developed a distributed variant of the ZONES algorithm and a sophisticated data partitioning scheme to enhance cross-matching speed and manage data skew. Notable achievements include performing an on-the-fly cross-match between Gaia DR2 and AllWise datasets in just 30 seconds. The paper discusses how AXS can facilitate end-user analyses of large datasets like LSST and highlights its efficiency, scalability, and performance across diverse datasets. In conclusion, AXS is a cutting-edge solution that harnesses the capabilities of Apache Spark to revolutionize astronomical data analysis processes.