Unveiling learning trends in convolutional neural networks with dynamic mode decomposition

Abstract

The widespread adoption of pre-trained models in artificial intelligence poses several evaluation challenges, primarily due to limited access to comprehensive information including training data, test data, loss function, and hyperparameter values. Traditional evaluation metrics, like accuracy, precision, and recall, demand the availability of train and test data, making it difficult to determine the expected performance or quality of the deep learning models. This paper addresses this challenge by introducing a comprehensive empirical analysis of Convolutional Neural Networks (CNNs) using Dynamic Mode Decomposition (DMD) theory. Feature maps from the convolutional layers are used to model the trained CNN as a linear dynamical system. By constructing snapshots for DMD analysis, we decompose the system's dynamics into individual modes and eigenvalues, providing a linear approximation of the intricate CNN behavior. We also introduce DMD-based metrics, including entropy of DMD-eigenvalues (EDMD), DMD eigen gap/spectral gap, and zero amplitude DMD modes, to quantify the effectiveness and quality of the learned CNN (well-trained vs poorly-trained) without the need to access the test data. Results indicate that the entropy of DMD-eigenvalues (EDMD) correlates well with the classical evaluation metrics and also distinguishes between well-trained and poorly-trained models. Additionally, we explored the potential of interpreting CNNs through stable DMD modes and their corresponding eigenvalues. Our approach was thoroughly evaluated on both shallow and deep models across various datasets for classification and segmentation tasks, demonstrating its effectiveness in quantifying CNN learning evolution and task performance. We also conducted a comprehensive analysis to examine the correlation between the proposed metrics and classical evaluation metrics on pre-trained models, specifically using ImageNet for classification (20 models) and Pascal VOC for segmentation (3 models) tasks. The code is available at: https://github.com/sikha2552/DMD_XAI.