# Identifying nonlinear dynamical systems from multi-modal time series data

**Authors:** Philine Lou Bommer, Daniel Kramer, Carlo Tombolini, Georgia Koppe, Daniel Durstewitz  
**arXiv:** [2111.02922](https://arxiv.org/abs/2111.02922)  
**Date:** 2021-11-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.02922v1/

## Key points

- Growing interest in utilizing machine learning methods for identifying nonlinear dynamical systems from multi-modal time series data
- Proposed framework integrates multiple data modalities to identify and predict nonlinear dynamical systems
- Utilizes dynamically interpretable recurrent neural networks as approximators and modality-specific decoder models from generalized linear models
- Emphasizes capturing invariant properties of the underlying system during model training, such as attractor geometry and power spectrum
- Introduces expectation-maximization (EM) and variational inference (VI) approaches for effective training of the integrated system

## Summary

, , , , 

In the field of identifying nonlinear dynamical systems from multi-modal time series data, there is a growing interest in utilizing machine learning methods to reconstruct latent dynamical systems in a completely data-driven, unsupervised manner. Previous work has focused on using domain knowledge for parameter inference through differential equations. However, this study aims to address cases where the exact form of the equations is unknown or where the complexity of the system requires general purpose equations for approximation. The proposed framework integrates multiple data modalities to identify and predict nonlinear dynamical systems. This approach utilizes dynamically interpretable recurrent neural networks as approximators, coupled with modality-specific decoder models from generalized linear models. By leveraging different types of time series data from various measurement channels, including continuous sensor readings and categorical behavioral responses, researchers can gain insights into how neural activity influences behavior or predict behavioral outcomes based on sensor data. While the integration of multi-modal data into predictive models has been extensively researched in other fields such as healthcare and engineering, it remains relatively unexplored in the realm of dynamical system identification. The primary objective of this work is to bridge this gap by developing novel algorithms for reconstructing latent nonlinear dynamical systems from diverse data domains described by different distributional models. The study emphasizes capturing invariant properties of the underlying system during model training, such as attractor geometry and power spectrum. This helps improve robustness against noisy or missing information in one data channel by leveraging other channels. To effectively train the integrated system, both an expectation-maximization (EM) and variational inference (VI) approach are introduced. Overall, this research contributes to advancing our understanding of complex dynamical systems by leveraging multi-modal data integration and machine learning techniques for improved identification and prediction capabilities.