Deep Learning Innovations in Video Classification: A Survey on Techniques and Dataset Evaluations

Abstract

Video classification has achieved remarkable success in recent years, driven by advanced deep learning models that automatically categorize video content. This paper provides a comprehensive review of video classification techniques and the datasets used in this field. We summarize key findings from recent research, focusing on network architectures, model evaluation metrics, and parallel processing methods that enhance training speed. Our review includes an in-depth analysis of state-of-the-art deep learning models and hybrid architectures, comparing models to traditional approaches and highlighting their advantages and limitations. Critical challenges such as handling large-scale datasets, improving model robustness, and addressing computational constraints are explored. By evaluating performance metrics, we identify areas where current models excel and where improvements are needed. Additionally, we discuss data augmentation techniques designed to enhance dataset accuracy and address specific challenges in video classification tasks. This survey also examines the evolution of convolutional neural networks (CNNs) in image processing and their adaptation to video classification tasks. We propose future research directions and provide a detailed comparison of existing approaches using the UCF-101 dataset, highlighting progress and ongoing challenges in achieving robust video classification.

1. Introduction

In recent years, video classification, despite its challenges, has become successful within deep learning. This area has gained attention following the advent of deep learning models, which have been proven to be practical tools for automatic video classification. The plethora of online video datasets highlights the importance of accurate video classification tasks. The rapid advancement of digital technologies has led to a substantial increase in the generation and consumption of video content globally. Various streaming and live video platforms, such as YouTube, Youku, TikTok, and others, have become essential for daily entertainment, significantly influencing user behavior and generating considerable revenue through user engagement. For example, YouTube alone reports over 122 million active users engaging with its content daily [1].

Additionally, Youku is one of China’s most popular video platforms. Notably, the number of views from external links on Youku is relatively higher than on YouTube [2]. Lastly, TikTok, widely recognized for its short-form videos, has proven to be an exceptionally effective platform for advertising and launching new products [3]. The advancement of deep learning in video classification has further enhanced the capabilities of these platforms, enabling more personalized content delivery and improved user experiences. Based on the description above, video classification tasks play a fundamental role in enhancing accuracy in action recognition, presenting significant challenges and opportunities for video classification, which has become a challenging and successful area within deep learning in recent years.

Video classification tasks are not just for academic research, but also present significant challenges and opportunities for major companies such as Google, Mountain View, CA, USA, (YouTube); Facebook, Menlo Park, CA, USA, (Meta); Amazon, Seattle, WA, USA, (Amazon Prime Video); Microsoft, Redmond, DC, USA; and Apple, Cupertino, CA, USA. These companies invest in competitions and release datasets to advance video classification technologies across various sectors, including entertainment and media, healthcare, security and surveillance, automotive, and retail. By doing so, they drive innovation and improve applications in these diverse fields. For instance, Google has introduced the YouTube-8M dataset [4], a valuable resource for researchers, containing millions of video features and over 3700 labels. Such initiatives underscore the importance of robust video classification models in various applications, both in industry and academic research.

Additionally, advancements in video analysis have seen the introduction of sophisticated models such as GPT-4 on the Azure platform. GPT-4, a state-of-the-art language model, can be fine-tuned for video analysis tasks by leveraging its extensive natural language understanding capabilities [5]. The Azure platform offers scalable and efficient processing, enabling real-time video analysis applications. Integrating such models with other machine learning frameworks enhances the robustness and versatility of video analysis systems.

Our survey aims to fill these gaps by providing an updated and thorough review of deep learning techniques for video classification, incorporating the latest research trends and methodologies. The following contributions are made:

• Comprehensive overview of CNN-based models: We summarize the state-of-the-art CNN models used for image analysis and their applications in video classification, highlighting key architectures such as LeNet-5, AlexNet, VGG-16, VGG-19, Inception, GoogleNet, ResNet, SqueezeNet, Enet, ShuffleNet, and DenseNet. Each model’s features, evaluations, and problem-solving capabilities are detailed to provide a foundational understanding of video classification tasks.
• Deep learning approaches for video classification: We cover the integration of CNNs and recurrent neural networks (RNNs) for video classification. CNNs capture spatial features within video frames, while RNNs model temporal dependencies, making this combination effective for video understanding and exploring the use of transformer models in conjunction with CNNs to enhance spatial and temporal feature modeling.
• Uni-modal and multi-modal fusion frameworks: We compare uni-modal approaches, which utilize a single data modality, with multi-modal approaches that integrate various modalities (text, audio, visual, sensor, and more). Multi-modal methods improve classification accuracy by leveraging the complementary strengths of different data types.
• Feature extraction and representation techniques: Effective feature extraction is critical for video classification. We review techniques such as Scale-Invariant Feature Transform (SIFT) and data augmentation methods like random rotation and shift, which have been shown to improve classification accuracy in video datasets.

The rest of the paper is organized as follows: Section 2 covers background studies on video classification. Section 3 details techniques from papers using deep learning models for video classification. Section 4 overviews benchmark datasets, evaluation metrics, and comparisons. Section 5 provides the research discussion, the challenges in video classification, and future directions. Lastly, Section 6 provides the conclusions. Table 1 provides the abbreviations used in this paper.

2. Background Studies on Video Classification

This section provides a comprehensive description of the background studies on video classification. The review process included summarizing relevant papers, identifying keywords related to the proposed topic, selecting appropriate publishers, analyzing abstract themes, and applying inclusion and exclusion criteria to ensure the quality of the papers in our survey. Figure 1 provides the selection process used in the current research study to create a video classification survey.

2.1. Relevant Surveys

Several review articles have addressed video classification recently, as summarized in Table 2. The existing works often need more coverage of the most recent state-of-the-art advancements, require closer alignment with current research trends, and have limitations. The following discussion highlights the limitations and critical points of these works.

Anushya [6] provides a comprehensive review covering methods for video classification, clustering, tagging, and training. However, the study’s comprehensiveness, conciseness, coverage of the topic, datasets, and analysis of state-of-the-art approaches could be extended. It is important to keep the audience updated with the latest advancements in video classification by incorporating recent state-of-the-art approaches. Additionally, Rani et al. [7] conducted a recent review on video classification topics, covering some recent video classification approaches and summarizing descriptions of recent works. An analysis of recent state-of-the-art approaches could be incorporated for further enhancements.

Li et al. [8] conducted a recent systematic review on live sports video classification, encompassing tools, video interaction features, and feature extraction methods. The paper concentrates on the classification of live sports videos. Zuo et al. [9] introduce fuzzy local spatio-temporal descriptors for video action recognition. They aim to tackle the uncertainty in pixel voting arising from the varying numbers of bins in traditional feature extractors. The study highlights a drawback related to the inherent uncertainty in the process of pixel voting concerning the bins in the histogram, which the proposed fuzzy descriptors seek to resolve. In contrast, Islam et al. [10] introduced models for video classification including deep learning and proposed machine learning techniques. Ullah et al. [11] also conducted a recent systematic review on human activity recognition. Furthermore, Rehman et al. [12] review comprehensive deep learning techniques for video classification. The review focuses on the pre-training and main model techniques in deep learning for video classification.

These existing reviews highlight the evolution of video classification research. However, to update recent studies, significant gaps and new trending paradigms will be addressed in our study, particularly in terms of coverage of the latest state-of-the-art advancements, comprehensive deep learning techniques, and dataset evaluations of recent models.

2.2. Evolution of CNNs in Image Processing

The CNN paradigm has evolved significantly and is extensively applied to image-processing tasks, demonstrating its practical relevance and impact. Its inherent parallelism and local connectivity properties make this architecture highly suitable for analog processing systems. Among these architectures, there are time-series approaches and evolved CNNs for image processing, such as LeNet-5, AlexNet, VGG-16, VGG-19, Inception, GoogleNet, ResNet, YOLO, SqueezeNet, Enet, ShuffleNet, DenseNet, MobileNet, and BowNet. A summary of these popular CNN architectures, consistent with the pattern of leaning on networks, is shown in Figure 2, where the extent of the network increases from the left-most (LeNet-5) to the right-most (BowNet). LeNet-5, one of the earliest CNN architectures, has demonstrated the effectiveness of CNNs for image recognition with its simple yet effective structure, paving the way for deeper networks.

AlexNet significantly advanced the field by introducing Rectified Linear Unit (ReLU) activation functions and dropout for regularization, achieving a breakthrough performance in the ImageNet 2012 competition. VGG-16 and VGG-19 further deepened this network architecture, utilizing small 3 × 3 convolution filters, and achieved high accuracy on large-scale datasets. Inception (GoogleNet) introduced the Inception module to capture multi-scale features efficiently, balancing high performance with reduced computational complexity. ResNet addressed the vanishing gradient problem in deep networks through residual learning, enabling ultra-deep networks to train and set new benchmarks in image classification. SqueezeNet aimed for model compression, achieving AlexNet-level accuracy with 50× fewer parameters, making it ideal for resource-constrained environments. DenseNet, with its dense connectivity, improved gradient flow and feature reuse, and achieves high performance with efficient parameter usage with the different key features and contributions shown in Table 3.

The transition from 2D CNNs to 3D CNNs marked a significant evolution, particularly relevant to video classification. While 2D CNNs process spatial information in images, 3D CNNs extend this capability to temporal dimensions, capturing motion and changes over time, which are crucial for understanding video data. Integrating CNNs with RNNs and transformer models further enhances video classification by combining spatial feature extraction with temporal dependency modeling, improving performance in capturing complex temporal dynamics in video data. These advancements collectively underscore the evolution of CNNs in tackling the challenges of video classification, with each contributing unique strengths in terms of model architecture, computational efficiency, and classification accuracy.

2.3. Fundamental Techniques in CNN-Based Image Processing

The thread that progresses from the formerly proposed architectures toward the recently proposed architectures depends on the network for image processing. In Table 4, numerous existing papers are detailed based on their descriptions of CNN models for image processing, including the authors’ approaches in their works, the models’ features, the models’ names, evaluations, and the problems solved in the papers.

3. Video Classification

This section presents a comprehensive and significant review of deep learning models for video classification tasks. This section also covers all steps and details the processes involved in video classification tasks, including the modalities of video data, parallel processing approaches, breakthroughs in video classification, and recent state-of-the-art deep learning models for video classification tasks.

A significant amount of research has focused on enhancing video classification and object segmentation accuracy and efficiency in surveillance systems. For instance, Zhao et al. [24] proposed a real-time approach for moving object segmentation and classification from HEVC-compressed surveillance videos. Their paper focuses on utilizing the unique features extracted directly from the HEVC-compressed domain for video surveillance tasks. They introduced a method that involves pre-processing steps such as motion vector interpolation and outlier removal, clustering blocks with nonzero motion vectors into connected foreground regions, and applying object region tracking based on temporal consistency to refine the moving object boundaries. Additionally, they developed a person–vehicle classification model using a bag of spatial–temporal HEVC syntax words to accurately classify moving objects as persons or vehicles and provide solid performance in accurately segmenting and classifying moving persons and vehicles in real-time scenarios.

3.1. Fundamental Deep Learning Architecture for Video Classification

Video classification fundamentally derives from image classification, with individual frames combined to form a video. Significant advancements in video classification have been driven by the development of deep learning architectures designed to automatically learn and extract features from raw video data [25]. These architectures typically rely on supervised learning, where models are trained on labeled datasets to recognize patterns and make accurate predictions. CNNs have become a cornerstone of video classification tasks among the various deep learning models. CNNs are particularly effective at capturing spatial features within individual video frames through their layered structure of convolutional filters [26].

Extending CNNs to 3D convolutions or combining them with other models, such as RNNs or attention mechanisms, can also capture temporal dynamics, improving the model’s ability to understand complex motion patterns and video semantic content [27]. Recent advancements have introduced vision transformers (ViTs), which leverage transformer architecture to process video data by treating video frames as sequences of image patches. ViTs have shown promising results in capturing spatial and temporal features, providing a powerful alternative to traditional CNN-based approaches [28]. The second approach is hybrid models, as shown in Figure 3, which integrate CNNs with various advanced techniques such as convolutional recurrent neural networks (CRNNs), two-stream networks, 3D CNNs, attention mechanisms, combinations of CNNs with transformer-based architectures, multi-modal fusion, and ensemble models, as summarized in this paper.

3.2. Parallel Processing in Video Classification

Video classification is computationally intensive due to the large volume of data and the complexity of analyzing both spatial and temporal dimensions. Parallel processing addresses these challenges by breaking down the video classification task into smaller, independent units that can be processed concurrently using multiple threads or computed units [29]. This approach significantly improves efficiency, reduces processing times, and enables effective parallelism in video data’s spatial and temporal domains.

In the context of 3D-CNN architectures, parallel processing enhances performance by decomposing the classification task into multiple sub-tasks. This decomposition is especially beneficial for hybrid models, which combine various neural network architectures such as CRNNs, two-stream networks, attention mechanisms, and combinations of CNNs and transformer-based architectures. These hybrid models also incorporate multi-modal fusion and ensemble methods to improve classification accuracy and robustness.

Each sub-task in these hybrid models is processed independently using 3D-CNNs. For instance, convolutional networks can handle spatial feature extraction, while recurrent networks manage temporal dependencies [30]. Attention mechanisms can further refine the focus on relevant video segments, and transformer-based architectures can capture long-term dependencies across video frames. Table 5 below summarizes several studies that utilize hybrid models for video classification, highlighting their architectures, features, and operational mechanisms.

Figure 4 illustrates the parallel processing framework with hybrid models for video classification, which is particularly effective for large-scale video datasets. The input video is divided into frames and processed in parallel. After frame processing, hybrid models train the data, resulting in class scores or predictions for the objects within each frame.

3.3. The Methods Used in Video Classification

This section describes the difference between the video classification methods used to label a short video clip and the human activity performed in the video clip. We describe the difference between single frame, late fusion, early fusion, and slow fusion, as shown in Figure 5. Single-frame video classification involves breaking down a video into individual frames and treating each frame as an independent image for classification. Features are extracted from each frame using techniques such as CNNs, and then a classifier predicts the category or label of each frame. While this approach simplifies processing and is easy to implement, it overlooks crucial temporal information for understanding video context [26]. It is adequate for tasks with less critical temporal details and limited computational resources, but it needs help with dynamic videos requiring a deeper understanding of temporal relationships.

Late fusion for video classification is a powerful technique that offers distinct advantages. Unlike other methods, it does not fuse information at the feature level. Instead, it aggregates predictions from individual classifiers, and the models are trained on different input modalities, such as visual, audio, or textual data [41]. These individual predictions are combined using techniques like averaging, voting, or weighted averaging to produce a final classification result. Late fusion is advantageous because it allows each modality to be processed independently, enabling the use of specialized models for each modality. This approach is efficient when different modalities contain complementary information about the video content.

Early fusion for video classification is a unique approach that involves combining information from different modalities and sources early in the classification process. Instead of processing each modality independently and combining their predictions later, early fusion merges the features extracted from different modalities into a single representation before passing it through a classifier [35]. This combined representation captures visual, audio, and possibly textual information in a unified feature space. Early fusion is helpful because it allows for the joint learning of features across modalities, potentially enhancing a model’s ability to capture complex relationships in the data.

Slow fusion for video classification involves gradually integrating information from different sources and modalities to improve classification accuracy. Instead of combining information at a single stage, slow fusion aggregates features and predictions from different modalities across multiple time steps or frames. This approach allows the model to capture the video sequence’s temporal dependencies and contextual information. Slow fusion typically involves techniques such as RNN and LSTM networks to integrate information over time [42]. Slow fusion is helpful because of the capability to capture the video’s spatial and temporal characteristics, which enables more robust and nuanced classification.

3.4. Hybrid Models for Video Classification

In this section, we summarize the hybrid models for video classification that leverage various neural network architectures, and the techniques proposed to improve the accuracy of video classification systems, as shown in Figure 6. These hybrid models combine different approaches to effectively capture spatial and temporal features in videos, enhancing the model’s ability to understand complex motion patterns and semantic content. In video classification, the spatial and temporal streams are two different approaches to capturing information from video data. The spatial stream focuses on individual frames, extracting features such as objects and scenes using CNNs. This stream captures the static-appearance information from the video.

On the other hand, the temporal stream analyzes the motion between frames, capturing dynamic information such as movement and actions. This technique uses methods like optical flow or RNNs to understand the temporal evolution of the video. By combining these two streams, models can achieve a more comprehensive understanding of the appearance and motion within the video, leading to improved classification performance.

The choice of model depends on several factors, including the complexity of the classification task, the availability of computational resources, and the characteristics of the input data [35]. By carefully selecting and combining different neural network architectures and techniques, practitioners can tailor hybrid models to suit specific video classification challenges, leading to more accurate and reliable results. Integrating transformer models with CNNs has become a powerful approach to enhance further spatial and temporal feature modeling in video classification tasks. CNNs are adept at extracting detailed spatial features from individual frames, making them highly effective for recognizing objects and identifying features within a static context. However, video classification requires understanding the content of individual frames and capturing the temporal dynamics across sequences of frames.

Transformers excel in modeling long-range dependencies and capturing temporal relationships due to their attention mechanisms, which allow them to focus on relevant parts of the input sequence over time. Combining these strengths, hybrid models leverage CNNs to process the spatial information within each frame and transformers to understand the temporal evolution across frames. This fusion facilitates comprehensive spatio-temporal feature extraction, enabling more accurate and robust video classification. This technique, which includes fine-tuning pre-trained models, is commonly employed to integrate these architectures, leveraging transfer learning to enhance performance and reduce training time. Recent research has shown that this hybrid approach significantly improves the accuracy of video classification models by effectively addressing the complexities of spatial and temporal feature extraction. Details of the differences between the strengths and weaknesses of these models are shown in Table 6.

Abdullah et al. [43] proposed a hybrid deep learning model tailored for video classification tasks, specifically a CNN-RNN system. The proposed approach involves utilizing a single-layer LSTM network to learn temporal features from stacked spatial features extracted by a CNN for each video instance. Notably, instead of employing general-purpose object classifiers, such as pre-trained CNNs trained on ImageNet, they opt to use a CNN model specialized in extracting emotion features. The hybrid model design aims to leverage the strengths of both CNNs and RNNs, enhancing the system’s ability to classify videos accurately by modeling spatial and temporal information and capturing nuanced emotional and motion-related features in video facial expression recognition tasks.

Feichtenhofer et al. [44] introduced CNNs for human action recognition in videos, presenting various methods to effectively integrate appearance and motion information. Despite its advantages, the two-stream architecture has encountered two primary limitations. Firstly, pixel-wise correspondences between spatial and temporal features need learning. Secondly, the architecture faces constraints at a temporal scale, given that spatial CNNs operate solely on individual frames, while temporal CNNs operate on temporal sequences independently. These limitations degrade the model’s ability to capture fine-grained spatio-temporal relationships, potentially impacting its performance in accurately recognizing human actions in videos.

Fan et al. [45] present a video-based emotion-recognition system centered around a hybrid network as a core module. This hybrid network combines RNN and 3D convolutional networks (C3D) in a late-fusion manner. The RNN and C3D are tasked with encoding appearance and motion information differently within the system. Specifically, the RNN receives appearance features extracted by a CNN from individual video frames as input, and subsequently encodes the motion information. On the other hand, C3D simultaneously models both the appearance and motion aspects of the video. This hybrid approach aims to leverage the complementary strengths of RNNs and C3Ds to effectively capture appearance and motion features, enhancing the system’s ability to recognize emotions accurately from video data.

Li et al. [46] proposed a hybrid attention model (HAM) within a deep CNN framework for image classification tasks. The model utilizes an intermediate feature map as its input by integrating a HAM. The HAM operates in two main steps: (1) generating a single-channel attention map and a refined feature map through the channel submodel; (2) then, based on this attention map, the spatial submodel divides the refined feature map into two groups along the channel axis. This division process facilitates the creation of a pair of spatial attention descriptors, providing enhanced focus on specific regions within the image. The HAM aims to improve the discriminative power of the model by dynamically attending to relevant spatial features, thereby enhancing its performance in image classification tasks.

Mekruksavanich et al. [47] introduced a novel approach to human activity recognition using multiple CNNs. The proposed method creates an effective architecture by incorporating a hybrid CNN with a channel attention mechanism. This mechanism enhances the network’s ability to capture intricate spatio-temporal characteristics hierarchically, enabling it to discern between various human movements in daily activities. By leveraging the channel attention mechanism, the model dynamically focuses on relevant features, improving its capacity to recognize and differentiate between different human activities more accurately.

Ullah et al. [48] present an innovative end-to-end hybrid framework for anomaly detection, integrating a CNN with a vision transformer-based architecture. This framework analyzes surveillance videos and identifies anomalous events using spatial and temporal information. The model processes the spatial and temporal features extracted from the surveillance video in the first step. Then, in the second step, these features are passed through the vision transformer-based model to capture the long-term temporal relationships among various complex surveillance events. The features obtained from the backbone model are subsequently inputted into a sequential learning model, where temporal self-attention mechanisms are employed to generate an attention map. This attention map highlights relevant spatio-temporal regions within the video, aiding in accurately detecting anomalous activities. Xu et al. [49] proposed a multi-modal emotional classification framework to capture user emotions in social networks. This framework features a 3D convolutional–LSTM (3DCLSTM) model, a hybrid model for classifying visual emotions, and a CNN-RNN hybrid model for classifying text-based emotions. The hybrid multi-modal fusion approach leverages the benefits of both feature fusion and decision-layer fusion strategies, effectively overcoming their limitations. This approach uses C3D to learn the spatio-temporal characteristics inputted into the convolutional LSTM model during training and classification. This combination enables the model to accurately capture and classify complex emotional cues from visual and textual data.

Jagannathan et al. [50] proposed automatic vehicle classification, which plays a vital role in intelligent transportation and visual traffic surveillance systems. Adaptive histogram equalization and a Gaussian mixture model are implemented to enhance the quality of the collected vehicle images and detect vehicles from denoised images. The extracted features are then used as the input for an ensemble deep learning technique for vehicle classification.

Table 6. Summary of strengths and weaknesses of hybrid models for video classification.

Hybrid Model	Strengths	Weaknesses
CNN-RNN [43]	Combines spatial features with temporal dynamics; specialized CNN for emotional features.	Complexity in training; high computational resources.
Two-stream architecture [44]	Effectively integrates appearance and motion; separate processing for spatial and temporal data.	Pixel-wise correspondences between spatial and temporal features need learning; constraints on temporal scale.
Hybrid network (RNN + 3D CNN) [45]	Late-fusion of RNN and C3D; encodes appearance and motion separately.	Increased model complexity; potential for overfitting with limited data.
HAM [46]	Dynamically attends to relevant spatial features and improves discriminative power.	Requires significant tuning and potentially high computational cost.
Hybrid CNN with channel attention mechanism [47]	Captures intricate spatio-temporal characteristics; dynamic focus on relevant features.	High complexity; requires substantial computational resources.
CNN + vision transformer [48]	Long-term temporal relationship modeling; effective for anomaly detection in surveillance.	High computational and memory requirements; complex architecture.
3DCLSTM + CNN-RNN [48]	Effective multi-modal fusion; captures spatio-temporal and textual emotional cues.	Very high complexity; demanding computational resources.
Ensemble models [50]	Enhanced image quality; accurate vehicle classification.	Requires extensive pre-processing; high computational cost.

Table 6. Summary of strengths and weaknesses of hybrid models for video classification.

Hybrid Model	Strengths	Weaknesses
CNN-RNN [43]	Combines spatial features with temporal dynamics; specialized CNN for emotional features.	Complexity in training; high computational resources.
Two-stream architecture [44]	Effectively integrates appearance and motion; separate processing for spatial and temporal data.	Pixel-wise correspondences between spatial and temporal features need learning; constraints on temporal scale.
Hybrid network (RNN + 3D CNN) [45]	Late-fusion of RNN and C3D; encodes appearance and motion separately.	Increased model complexity; potential for overfitting with limited data.
HAM [46]	Dynamically attends to relevant spatial features and improves discriminative power.	Requires significant tuning and potentially high computational cost.
Hybrid CNN with channel attention mechanism [47]	Captures intricate spatio-temporal characteristics; dynamic focus on relevant features.	High complexity; requires substantial computational resources.
CNN + vision transformer [48]	Long-term temporal relationship modeling; effective for anomaly detection in surveillance.	High computational and memory requirements; complex architecture.
3DCLSTM + CNN-RNN [48]	Effective multi-modal fusion; captures spatio-temporal and textual emotional cues.	Very high complexity; demanding computational resources.
Ensemble models [50]	Enhanced image quality; accurate vehicle classification.	Requires extensive pre-processing; high computational cost.

3.5. Challenges and Future Directions of Video Classification

In this section, we cover the challenges faced in video classification tasks and summarize future directions to improve performance, including advancements in hardware, improved algorithms, and the use of transfer learning models.

3.5.1. Challenges in Video Classification

This section presents several benchmark datasets commonly used for video classification tasks, providing a comprehensive overview of their characteristics and challenges. We present various aspects of these datasets, including data complexity, computational resource requirements, and real-time processing capabilities. The complexity of video data, the substantial computational power needed to process large-scale datasets, and the necessity for efficient algorithms to handle real-time processing are significant challenges in the video classification domain. Table 7 summarizes some notable datasets, detailing the dataset name, the total number of videos, the resolutions of the videos, the number of classes present, and the year of publication. This summary helps illustrate the diverse range of datasets available, each with unique features and applications. These datasets serve as critical benchmarks for developing and evaluating video classification models, providing a standard for comparing performance across different approaches.

3.5.2. Future Directions of Video Classification

In video-processing tasks, hardware components are critical for enhancing time performance during training on large-scale datasets. For example, Kyrkou et al. [51] proposed an SVM cascade architecture implemented on a Spartan-6 field-programmable gate array (FPGA) platform. They evaluated it for object detection on 800 × 600 (super video graphics array) resolution images. The proposed system, enhanced by a neural network capable of processing cascade information, achieves a real-time processing rate of 40 frames per second in the face detection benchmark application. Karpathy et al. [26] proposed large-scale video classification with CNNs. Since CNNs typically take weeks to train on large-scale datasets, even on the fastest available GPUs, this approach to speeding up the networks reduces the number of layers and neurons in each layer and the resolution of images during the training phase. This approach achieved good performance in terms of training time. Pérez et al. [52] present a scalable, low-power, low-resource-utilization accelerator architecture for inference on the MobileNet V2 CNN. The architecture uses a different system with an embedded processor as the central controller, external memory to store network data, and dedicated hardware implemented on reconfigurable logic with scalable processing elements.

Developers increasingly use FPGAs in image-processing applications, especially for real-time tasks like vehicle number plate recognition, video classification, and medical imaging (e.g., X-rays and CT scans). An FPGA embedded with a camera can perform image processing rapidly, as the images are streamed rather than processed as a sequence of individual frames. FPGAs allow for the custom hardware design of algorithms using hardware description languages optimized for performance and resource efficiency [53]. This process entails the utilization of algorithms by deploying a collection of application-specific intellectual property cores sourced from a library. High-level synthesis is employed to transform a C-based representation of the algorithm into synthesizable hardware. Subsequently, the system maps the algorithm onto a parallel array of programmable soft-core processors.

Using the FPGA for video classification begins with capturing video frames using sensors. An acquisition device initially processes these frames and then transfers them to memory, where resources can be shared with the CPU/GPU. Finally, the FPGA processes the data, leveraging its memory for efficient video classification. The image processing method using the FPGA library is shown in Figure 7.

Given these capabilities, how can FPGAs be optimized further for video classification using deep learning techniques? What are this approach’s potential advantages and limitations, and how can it be leveraged to enhance the accuracy and efficiency of real-time video-processing applications? All these questions are open research topics for researchers and industries to consider in future work to improve the parallel processing of large datasets via video classification tasks and real-time video classification applications.

3.6. Overview of Deep Learning Frameworks and Hybrid Models for Video Classification

This section describes two different approaches for video classification: single and hybrid deep learning. Single deep learning entails the utilization of a singular neural network architecture for classification tasks. This method utilizes the potency of deep neural networks to learn features and patterns directly from raw video data autonomously. The primary advantages include simplicity in design and implementation and the capacity to train end-to-end models without extensive manual feature engineering.

In contrast, hybrid deep learning combines multiple neural network architectures and integrates traditional machine learning techniques with deep learning models to enhance performance. This approach capitalizes on the strengths of various models, such as combining CNNs for spatial feature extraction with RNNs for temporal sequence modeling. Hybrid models can further incorporate pre-processing steps, feature extraction techniques, and ensemble methods to augment accuracy and robustness. The principal advantages of hybrid deep learning encompass heightened flexibility, superior performance on intricate tasks, and the potential to reduce labeled data requirements through techniques like transfer learning.

Table 8 summarizes the difference between single deep learning and hybrid deep learning models. These frameworks adeptly manage the complexities of vast volumes of high-dimensional video data. Their flexibility facilitates the creation of custom neural network architectures customized for video classification tasks, empowering researchers and developers to explore various model configurations to achieve optimal results. Within deep learning frameworks, three primary processes contribute to evaluating video classification tasks: (1) data augmentation, (2) the utilization of pre-trained models, and (3) the exploration of diverse model architectures. Each process plays a pivotal role in enhancing performance across video classification tasks.

3.6.1. Data Augmentation

The initial stage of data augmentation is pivotal for video classification tasks, as it marks the first step in building a robust video classification system. This stage is a foundational component for enhancing accuracy as the process progresses. Mao et al. [54] propose various techniques to improve data augmentation. They apply a random rotation of up to 70 degrees to the video frames and a random shift of 224 × 224 pixels. Similarly, Takahashi et al. [55] apply random image cropping and patching for deep CNNs to improve the accuracy of model training. Kim et al. [56] propose using brightness adjustment techniques to enhance training, and involve representative colors using a power function to improve the training process.

Furthermore, Taylor et al. [57] introduce methods to strengthen deep learning through generic data augmentation by applying exposure techniques during image classification training to achieve high performance. On the contrary, Sayed et al. [58] propose an improved method for handling motion blur in online object detection that yields promising results for object detection tasks. Additionally, Kim et al. [59] introduced a data augmentation technique utilizing the adaptive inverse peak signal-to-noise ratio they devised while examining the impact of color attributes within the training images.

Diba et al. [60] proposed a novel noise filtration algorithm for denoising neuromorphic camera data using a graph neural network (GNN)-driven transformer approach. The key innovation in their work is the introduction of the GNN–transformer algorithm, which classifies active event pixels in raw streams into real log-intensity variations or noise. They utilized a message-passing framework called EventConv within the GNN to capture spatiotemporal correlations among events while maintaining their asynchronous nature. Additionally, they introduced the known-object ground-truth labeling (KoGTL) approach to generate labeled datasets under various lighting conditions, including moonlight, to train and test their algorithm. The proposed algorithm outperformed state-of-the-art methods by at least 8.8% in terms of filtration accuracy when tested on unseen datasets. To augment the data for training and testing, they conducted experiments in various lighting conditions, including very good lighting, office lighting, low light, and moonlight, recording scenes with static and moving cameras in different directions to capture dynamic event and noise-generation scenarios.

3.6.2. Pre-Training on Hybrid Models

The visual applications of deep learning include object detection, visual object recognition, image segmentation, and more. The pre-training step involves training a deep neural network on a vast amount of labeled video data, typically employing techniques like supervised learning. This process enables a model to ascertain general features and patterns inherent in various types of videos, such as motion, shapes, and textures. Pre-trained models undergo fine-tuning for specific video classification tasks. Fine-tuning entails adjusting the pre-trained model’s parameters to better align with the characteristics of the target dataset or system. This step often requires less labeled data than training a model from scratch, rendering it more feasible for video classification tasks with limited annotated samples.

Pre-trained models for video classification commonly rely on CNNs. Among the prevalent methods are those based on CNNs, including two primary approaches: (1) RNNs with LSTM networks, and (2) optical flow, as shown in Figure 8. These models incorporate architectures such as 3D CNNs to capture spatio-temporal features from video sequences directly. Ramesh et al. [61] proposed utilizing pre-trained CNNs, such as AlexNet, GoogLeNet, and MobileNet, to enhance video classification accuracy and improve sports training and performance. To construct a custom video-style classifier, Aryal et al. [62] presented several pre-trained models, such as VGG16, InceptionV3, and ResNet50. They utilized Keras, a powerful deep learning library that implements these deep models, to build the classifiers.

Multi-stream models are built on the idea of separating the temporal stream and the spatial stream. The temporal stream in action recognition provides significant advantages by capturing motion information that static images cannot transfer. This stream analyzes the sequence and flow of frames over time, allowing the system to understand the dynamics and transitions of actions. By focusing on temporal changes, the stream can detect subtle movements and patterns, improving the accuracy of recognizing complex activities, such as distinguishing between similar actions that differ in speed or direction. The temporal stream enhances the robustness of action recognition systems in varying conditions, as it can better account for changes in perspective, lighting, and occlusions that might affect single-frame analysis.

The spatial stream in action recognition offers significant advantages by capturing detailed spatial information from individual frames, which is crucial for understanding the context and environment of an action. This stream focuses on the appearance and layout of objects and scenes, allowing the system to identify essential features such as shapes, textures, and edges. By analyzing these spatial characteristics, the spatial stream enhances the system’s ability to recognize and differentiate between various objects and their configurations within a frame. It is particularly beneficial for identifying the context in which an action occurs, such as distinguishing between different sports or activities based on surroundings.

Additionally, another approach to the primary model involves combining CNNs with transformer architectures for video classification on pre-trained models, representing a novel approach that leverages the respective advantages of the architectures [63]. CNNs are adept at extracting spatial features from individual video frames, making them ideal for object recognition and feature extraction tasks. Meanwhile, transformers are excellent at capturing long-range dependencies and temporal relationships within sequential data, leveraging attention mechanisms to focus on the relevant parts of the input sequence. By integrating CNNs for spatial feature extraction and transformers for temporal modeling, the combined model can effectively analyze the visual content of each frame and the temporal dynamics across frames.

This comprehensive understanding of spatial and temporal information enables more accurate and robust video classification. Whether through joint training or fine-tuning pre-trained models, this approach leverages transfer learning to improve performance while reducing training time. Selva et al. [28] introduced a transformer architecture modification to efficiently handle video data, reduce redundancy, reintroduce functional inductive biases, and capture long-term temporal dynamics for video classification tasks. As depicted in Figure 9, the sequence length $M$ depends on the data sample, $N$ is the desired number of sequences, and $FC$ denotes the number of channels.

3.6.3. Hybrid Approach to Training Models on Video Classification

The evolution of models for video classification using the hybrid approach showcases remarkable progression. Table 9 covers various aspects of these hybrid models, including the authors’ proposed approach, the features, the models employed, the datasets used for the experiments, the problems with the approach, the results/findings, and the year of publication. These studies focus on hybrid architectures, particularly those combining CNN with CRNNs, two-stream networks, attention mechanisms, combinations of CNNs and transformer-based architectures, multi-modal fusion, and ensemble models. Classification accuracy is a standard evaluation metric that frequently uses the UCF-101 dataset for experimentation. Multi-classification problems are addressed in almost all cases. The superior performance of 3D convolutions for spatio-temporal feature learning is noted. Various fusion techniques such as average fusion, kernel average fusion, weight fusion, logistic regression fusion, and multiple kernel learning fusion are generally less effective than the multi-stream, multi-class fusion technique using hybrid architectures in video classification tasks.

4. Evaluation Metrics and Comparison of Existing State-of-the-Art Video Classification Tasks

This section covers several benchmark datasets utilized for video classification tasks. The datasets for action recognition have evolved to become highly complex and realistic. The earliest datasets, such as KTH and Weizman, featured a fixed number of actions and minimal action categories. A list of datasets used for action recognition is given in Table 7. The details provided for these datasets include the dataset name, the total number of videos in each dataset, the resolution of the videos, the number of classes present in the dataset, and the year of publication of the dataset.

4.1. Performance Metrics for Evaluation in Video Classification

Throughout this section, we describe how the evaluation of video classification models is mostly achieved using different performance measures. The most common measures to evaluate models include accuracy, precision (precision measures include conducting positive meaning determination), recall (precision tests for the productive detection of the classifier’s positive result), F1 score, micro-F1 score, and K-fold cross-validation. Several recent studies have utilized these measures, as shown in Table 10.

4.2. Comparison of Datasets for Video Classification

Various datasets have been curated for training and evaluating video classification models. Each dataset has unique characteristics, presents distinct challenges, and is suitable for specific tasks. In Table 11, we comprehensively compare these datasets, which serve as benchmarks for evaluating model performance, and give a summary with invaluable resources to train and test the different algorithms across various tasks and scenarios that match the dataset characteristics.

4.3. Comparison of some Existing Approaches on the UCF-101 Dataset

The UCF-101 dataset is a widely utilized benchmark in computer vision and video classification and was published by researchers from the University of Central Florida in 2012 [80]. It comprises 13,320 videos across 101 action categories, ranging from everyday human activities to sports and leisure activities. The videos are typically short, lasting a few seconds, and were captured from YouTube videos. The dataset offers various actions, including running, jumping, playing musical instruments, and more. UCF-101 is a valuable resource for training and evaluating video classification algorithms, aiding researchers in developing robust models for action recognition and related tasks. Table 12 displays existing papers and their comparative results using the UCF-101 dataset.

The progression of research utilizing the UCF-101 dataset, as depicted in Figure 10, offers a compelling narrative of advancement in video classification. This research, a collaborative effort of many in the field, has led to significant breakthroughs. Choutas et al. [125] propose using an interactive three-dimensional (I3D) method combined with Pose moTion (PoTion), achieving an accuracy of 29.3%. Combining I3D with PoTion involves integrating the PoTion representation, which encodes the movement of pose critical points over a video clip, with the I3D architecture for action recognition. These heatmaps are colored based on the frame’s time and encode the probability of each pixel containing a specific joint. The colorized heatmaps are summed over all frames to create the PoTion representation. This representation is then used to train a shallow CNN with six convolutional layers and one fully connected layer for action classification. The PoTion representation is combined with the I3D approach, which uses spatio-temporal convolutions and pooling operators from an image classification network. When combined with I3D on RGB and optical flow streams, low accuracy is observed in some classes, such as tying a bow tie and making sushi. This is due to factors like the poor visibility of human actors, especially in first-person videos, the partial visibility of joints, and videos focusing more on objects than human actors. In these cases, the PoTion representation may not capture relevant information effectively, leading to lower accuracy.

This marked a starting point for subsequent studies to build upon. Varadarajan et al. [38] introduced the omnidirectional vector (OmniVec) model, achieving a remarkable 99.6% accuracy on the UCF-101 dataset. The OmniVec model is a unified learning framework designed to learn embeddings from multiple modalities and tasks using a shared backbone network. Its three main components are modality-specific encoders, a shared backbone network, and task-specific heads. OmniVec trains sequentially on different tasks and modalities, grouping tasks by the extent of information they exploit across modalities. For example, semantic segmentation embeds more local information than classification. Training data involves mixing samples from each modality for a given task, with modality encoders replaced while task heads and the backbone remain the same. OmniVec handles RGB images and videos, using a standard backbone for processing and facilitating knowledge sharing. It infuses cross-domain information, aligning embeddings from different modalities in a shared space. An iterative training mechanism mixes modalities and groups tasks for better learning. OmniVec achieves high generalization on unseen datasets and tasks, performing state-of-the-art work on various images and videos. It excels in cross-modal knowledge transfer, outperforming other methods in tasks like point cloud classification, semantic segmentation, and text summarization. OmniVec achieves 99.6% accuracy on the UCF-101 dataset by learning robust representations from multiple modalities and tasks, leveraging a shared backbone network for meaningful representation extraction and generalization. This dramatic improvement underscores the dataset’s pivotal role as a cornerstone resource for both researchers and industries engaged in exploring video classification tasks.

5. Discussion

The advancements in deep learning for video classification have been substantial, providing significant improvements in accuracy and efficiency. This section delves into the comparative performance of various deep learning approaches that emphasize the distinction between single and hybrid models.

5.1. Single and Hybrid Models

Single models, primarily leveraging CNNs and RNNs, have been foundational in video classification. CNNs are adept at extracting spatial features from video frames, while RNNs excel at capturing temporal dependencies. However, their isolated use often limits their ability to exploit the intricate spatio-temporal dynamics inherent in videos entirely.

Hybrid models, which combine CNNs with RNNs or transformers models, have emerged as a superior alternative. These models benefit from the strengths of CNNs’ spatial feature extraction capabilities and RNNs’ or transformers’ temporal modeling. For instance, integrating CNNs with transformers has shown remarkable success in enhancing spatial and temporal feature modeling [30], as evidenced by improved accuracy metrics across various benchmarks, including the UCF-101 dataset [80].

5.2. Data Augmentation and Pre-Training

Data augmentation and pre-training on large-scale datasets have also played a pivotal role in advancing video classification. Techniques such as random rotation, shift, and other augmentation methods have improved model robustness and performance [54,55,56,57,58,59]. Moreover, pre-trained models, which leverage transfer learning, have significantly improved classification tasks by utilizing learned features from extensive datasets to enhance performance on target tasks [70,71,72,73,74].

5.3. Challenges in Video Classification

Despite these advances, several challenges are present in the domain of video classification:

• Handling large-scale datasets: Efficiently processing and training on large-scale video datasets remains a significant challenge. The computational cost and time required are substantial, necessitating the development of more efficient algorithms and hardware optimizations.
• Generalization capabilities: It is crucial to ensure that models generalize well across diverse video datasets and real-world scenarios. Current models often struggle with overfitting to specific datasets, limiting their applicability.
• Temporal consistency and long-term dependencies: Capturing long-term temporal dependencies and maintaining temporal consistency across frames is another critical area where existing models can improve. While hybrid models have made strides in this direction, there is still room for enhancing effectiveness.
• Reporting and comparing performance metrics: The inconsistent reporting of performance metrics, such as the absence of standard deviation values, poses a significant challenge. Standard deviations are essential for assessing the variability and reliability of the reported accuracies. Future research should ensure the inclusion of these statistics to facilitate a more robust comparison of algorithm performance. This will enhance the reliability and relevance of the reported results and aid in developing more robust models.

5.4. Future Directions

Future research in video classification should focus on addressing these challenges through several promising avenues:

• Efficient model architectures: Developing more efficient model architectures that can handle large-scale datasets without compromising performance is essential. This includes exploring novel neural network designs and leveraging advancements in hardware acceleration.
• Advanced data augmentation techniques: Incorporating more sophisticated data augmentation techniques to simulate real-world variations will improve model robustness and generalization.
• Integration of multimodal data: Utilizing multimodal data, such as combining video with audio and text, can provide a more comprehensive understanding of the content, leading to better classification performance.
• Improved temporal modeling: Enhancing temporal modeling capabilities will be crucial, particularly for long-term dependencies. This will involve developing new types of recurrent units or attention mechanisms explicitly tailored to video data.

6. Conclusions

This paper has comprehensively reviewed deep learning methodologies applied to video classification, offering detailed summaries of significant studies and highlighting key findings. The exploration includes research leveraging the UCF-101 dataset, and clarifies the methodologies and techniques proposed to enhance accuracy. In examining the landscape of video classification tasks, the review covered techniques applied in the image-processing stage, the integration of CNN models with other models for video classification, and open research questions regarding parallel processing techniques for handling large-scale datasets. Additionally, we explained the differences between single-frame, late-fusion, early-fusion, and slow-fusion approaches. We also discussed the challenges and future directions of video classification, evaluation metrics, comparisons from existing papers, and proposed approaches for conducting experiments with large datasets. Our review highlighted the deep learning approaches that outperform other state-of-the-art video classification methods. The integration of CNNs with other models, such as RNNs and transformers, has significantly improved the capture of spatial and temporal features. However, significant gaps still need to be addressed, particularly in handling large-scale datasets efficiently and improving the generalization capabilities of models.