# Hypersparse Neural Network Analysis of Large-Scale Internet Traffic

**Authors:** Jeremy Kepner, Kenjiro Cho, KC Claffy, Vijay Gadepally, Peter Michaleas, Lauren Milechin  
**arXiv:** [1904.04396](https://arxiv.org/abs/1904.04396)  
**Date:** 2019-04-08  
**Summary page:** https://www.summarizepaper.com/en/paper/1904.04396v2/

## Key points

- The Internet is reshaping society and it is important to understand Internet traffic quantitatively.
- A large dataset of Internet traffic data consisting of 50 billion packets has been collected and curated.
- A groundbreaking approach using a hypersparse neural network has been developed to analyze the data.
- The analysis focuses on "video" streams within the Internet traffic and utilizes 10,000 processors in the MIT SuperCloud.
- Previously unseen leaf nodes and isolated links in Internet traffic have been discovered through this method.
- A two-parameter modified Zipf-Mandelbrot distribution is used to accurately describe various source/destination statistics.
- The statistical model can analyze moving sample windows ranging from 100,000 to 100,000,000 packets over multiple years and continents.
- Model parameters inferred from the analysis differentiate between different network streams.
- The leaf parameter of the model correlates with the fraction of traffic in different network topologies, providing insights into overall traffic patterns.
- The hypersparse neural network pipeline is adaptable and can incorporate different network statistics and training models by changing image filter functions.
- This adaptability allows for further refinement of analysis and exploration of additional aspects of Internet traffic dynamics.

## Summary

The Internet is a powerful force that is reshaping our society, and as such, it is crucial to have a quantitative understanding of Internet traffic. To achieve this, our team has been diligently collecting and curating the largest publicly available dataset of Internet traffic data which consists of 50 billion packets. To analyze this massive amount of data we have developed a groundbreaking approach using a hypersparse neural network. This analysis focuses specifically on "video" streams within the Internet traffic and utilizes 10,000 processors in the MIT SuperCloud. Through this innovative method we have made an intriguing discovery – the significance of previously unseen leaf nodes and isolated links in Internet traffic. Furthermore, our neural network approach has allowed us to accurately describe various source/destination statistics by employing a two-parameter modified Zipf-Mandelbrot distribution. This statistical model has proven effective in analyzing moving sample windows ranging from 100,000 to 100,000,000 packets over collections spanning multiple years and continents. By inferring model parameters from our analysis we are able to differentiate between different network streams. Notably, the model's leaf parameter exhibits a strong correlation with the fraction of traffic in different underlying network topologies which provides valuable insights into how different components contribute to overall Internet traffic patterns. One notable advantage of our hypersparse neural network pipeline is its adaptability; we can easily incorporate different network statistics and training models by making simple changes to the image filter functions. This flexibility enables us to refine our analysis further and explore additional aspects of Internet traffic dynamics. In conclusion, our research offers an extensive exploration into large-scale Internet traffic analysis using advanced techniques such as hypersparse neural networks. By leveraging this approach along with the vast amount of data we have collected and curated we gain valuable insights into the complex nature of Internet traffic patterns; these findings contribute significantly to our quantitative understanding of how the Internet is transforming our society.