In this paper, we investigate the problem of causal image classification with multi-label learning. As multi-label learning involves a diversity of supervision signals, it is considered a challenging issue to solve. Previous approaches have attempted to improve performance by identifying label-related image areas or exploiting the co-occurrence of labels. However, these methods are often characterized by complicated procedures, tedious computations, and a lack of intuitive interpretations. To overcome these limitations, we propose a novel approach that incorporates the concept of causal inference, which has been shown to be beneficial in other computer vision problems. Our method, called causal multi-label learning (CMLL), enables the selection of multiple objects from the original image through a multi-class attention module. These objects are then subjected to causal intervention to learn the causal relationships between different labels. Our proposed approach is both elegant and effective, with low computational cost and few parameters required for the multi-class causal intervention approach. Extensive tests and ablation studies demonstrate that the proposed method significantly improves prediction performance without a significant increase in training and inference times.

Nowadays, solving time series prediction problems is an open and challenging task. Many solutions are based on the implementation of deep neural architectures, which are able to analyze the structure of the time series and to carry out the prediction. In this work, we present a novel deep learning scheme based on an adaptive embedding mechanism. The latter is exploited to extract a compressed representation of the input time series that is used for the subsequent forecasting. The proposed model is based on a two-layer bidirectional Long Short-Term Memory network, where the first layer performs the adaptive embedding and the second layer acts as a predictor. The performances of the proposed forecasting scheme are compared with several models in two different scenarios, considering both well-known time series and real-life application cases. The experimental results show the accuracy and the flexibility of the proposed approach, which can be used as a prediction tool for any actual application.

Differentiable Architecture Search (DARTS) is a simple yet efficient Neural Architecture Search (NAS) method. During the search stage, DARTS trains a supernet by jointly optimizing architecture parameters and network parameters. During the evaluation stage, DARTS discretizes the supernet to derive the optimal architecture based on architecture parameters. However, recent research has shown that during the training process, the supernet tends to converge towards sharp minima rather than flat minima. This is evidenced by the higher sharpness of the loss landscape of the supernet, which ultimately leads to a performance gap between the supernet and the optimal architecture. In this paper, we propose Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to distill knowledge from previous steps of the supernet to guide its training in the current step, effectively reducing the sharpness of the supernet's loss and bridging the performance gap between the supernet and the optimal architecture. Furthermore, we introduce the concept of voting teachers, where multiple previous supernets are selected as teachers, and their output probabilities are aggregated through voting to obtain the final teacher prediction. Experimental results on real datasets demonstrate the advantages of our novel self-distillation-based NAS method compared to state-of-the-art alternatives.

Deep neural networks have become increasingly significant in our daily lives due to their remarkable performance. The issue of adversarial examples, which are responsible for the vulnerability problem of deep neural networks, has attracted the attention of researchers in the study of robustness of these networks. To address the issues caused by the restricted diversity and precision of adversarial perturbations in neural networks, we introduce a novel technique called Adversarial Boundary Diffusion Probability Modeling (Adv-BDPM). This approach combines boundary analysis and diffusion probability modeling. First, we combined the denoising diffusion probability model with the boundary loss to design the boundary diffusion probability model, which can generate corresponding boundary perturbations for a specific neural network. Then, through the iterative process of boundary perturbations and its corresponding orthogonal perturbations, we proposed a decision boundary search algorithm to generate adversarial samples. The comparison experiments with black-box attacks in ImageNet demonstrate that Adv-BDPM has better attack success rate and perturbation precision. The comparison experiments with white-box attacks in CIFAR-10 and CIFAR-100 demonstrate that Adv-BDPM has better attack success rate, attack diversity for the same sample, and can effectively defend against adversarial training with shorter running time.

Adversarial training is considered one of the most effective methods to improve the adversarial robustness of deep neural networks. Despite the success, it still suffers from unsatisfactory performance and overfitting. Considering the intrinsic mechanism of adversarial training, recent studies adopt the idea of curriculum learning to alleviate overfitting. However, this also introduces new issues, that is, lacking the quantitative criterion for attacks' strength and catastrophic forgetting. To mitigate such issues, we propose the self-paced adversarial training (SPAT), which explicitly builds the learning process of adversarial training based on adversarial examples of the whole dataset. Specifically, our model is first trained with "easy" adversarial examples, and then is continuously enhanced by gradually adding "complex" adversarial examples. This way strengthens the ability to fit "complex" adversarial examples while holding in mind "easy" adversarial samples. To balance adversarial examples between classes, we determine the difficulty of the adversarial examples locally in each class. Notably, this learning paradigm can also be incorporated into other advanced methods for further boosting adversarial robustness. Experimental results show the effectiveness of our proposed model against various attacks on widely-used benchmarks. Especially, on CIFAR100, SPAT provides a boost of 1.7% (relatively 5.4%) in robust accuracy on the PGD10 attack and 3.9% (relatively 7.2%) in natural accuracy for AWP.

Recent research efforts on Few-Shot Learning (FSL) have achieved extensive progress. However, the existing efforts primarily focus on the transductive setting of FSL, which is heavily challenged by the limited quantity of the unlabeled query set. Although a few inductive-based FSL methods have been studied, most of them emphasize learning superb feature extraction networks. As a result, they may ignore the relations between sample-level and class-level representations, which are particularly crucial when labeled samples are scarce. This paper proposes an inductive FSL framework that leverages the Hierarchical Knowledge Propagation and Distillation, named HKPD. To learn more discriminative sample-level representations, HKPD first constructs a sample-level information propagation module that explores pairwise sample relations. Subsequently, a class-level information propagation module is designed to obtain and update the class-level information. Moreover, a self-distillation module is adopted to further improve the learned representations by propagating the obtained knowledge across this hierarchical architecture. Extensive experiments conducted on the commonly used few-shot benchmark datasets demonstrate the superiority of the proposed HKPD method, which outperforms the current state-of-the-art methods.

In this paper, the exponential consensus of leaderless and leader-following multi-agent systems with Lipschitz nonlinear dynamics is illustrated with aperiodic sampled-data control using a two-sided loop-based Lyapunov functional (LBLF). Firstly, applying input delay approach to reformulate the resulting sampled-data system as a continuous system with time-varying delay in the control input. A two-sided LBLF which captures the information on sampled-data pattern is constructed and the symmetry of the Laplacian matrix together with Newton-Leibniz formula have been employed to obtain reduced number of decision variables and decreased LMI dimensions for the exponential sampled-data consensus problem. Subsequently, an aperiodic sampled-data controller was designed to simplify and enhance stability conditions for computation and optimization purposes in the proposed approach. Finally, based on the controller design, simulation examples including the power system are proposed to illustrate the theoretical analysis, moreover, a larger sampled-data interval can be acquired by this method than other literature, thereby conserving bandwidth and reducing communication resources.

This article focuses on the neural adaptive output feedback control study related to nonaffine stochastic multiple-input, multiple-output nonlinear plants. First, a K-filter state observer based on a radial basis function neural network is designed to estimate the remaining unavailable states. Then, a novel adaptive command-filtered backstepping output feedback control framework is established, where an improved command filter with a fractional-order parameter is applied to conquer the calculation size problem. Specifically, the highlight of this work is that it designs a modified error compensation signal and incorporates the concept of deferred constraint to eradicate the negative effect caused by the filter errors. In addition, the network bandwidth resources, control impulse, and control accuracy are synthesized using an amended switching event-triggered mechanism. The theoretical analysis proved that the proposed control approach guarantees that the tracking error can converge to a preassigned region within a user-defined time while the violation of the deferred output constraint can be excluded. Two illustrative studies are provided to demonstrate the validity and superiority of the developed control method.

The performance in restoring compressed multi-view video (MVV) of the existing learning-based methods is limited because they only utilize information of temporally adjacent frames or parallax neighboring views. However, the compression artifacts caused by multi-view coding (MVC) may be related to the reference errors of intra-frame, inter-frame, and inter-view. In this paper, with delicately utilizing the stereo information from both temporal and parallax domains, a motion-parallax complementation network (MPCNet) is proposed to restore the quality of compressed MVV more efficiently. First, we introduce a motion-parallax complementation strategy consisting of a coarse stage and a fine stage. By mutually compensating the feature extracted from multiple domains, useful multi-frame information can be efficiently preserved and aggregated step by step. Second, an attention-based feature filtering and modulation module (AFFM) is proposed, which provides an efficient fusion method for two features by suppressing misleading information. By deploying it in most submodules of the proposed approach, the representational ability of MPCNet can be improved, resulting in a more substantial restoration performance. Experimental results prove the effectiveness of MPCNet by an average increase of 1.978 dB in PSNR, and 0.0282 in MS-SSIM. The BD-rate reduction can reach 47.342% on average. The subjective quality is greatly improved and lots of compression distortions are eliminated. Meanwhile, this work also benefits the accuracy improvement for high-level vision tasks, e.g., mIoU of semantic segmentation and mAP of object detection achieve 0.352 and 51.71, respectively. Quantitative and qualitative analyses demonstrate that MPCNet outperforms state-of-the-art approaches.

Much mathematical effort has been devoted to developing Principal Component Analysis (PCA), which is the most popular feature extraction method. To suppress the negative effect of noise on PCA performance, there have been extensive studies and applications of a large number of robust PCAs achieving outstanding results. However, existing methods suffer from at least two shortcomings: (1) They expressed PCA as a reconstruction model measured by Euclidean distance, which only considers the relationship between the data and its reconstruction and ignores the differences between different data points; (2) They did not consider the class-specificity distribution information contained in the data itself, thus lacking discriminative properties. To overcome the above problems, we propose a Sparse Discriminant Principal Components Analysis (SDPCA) model based on contrastive learning and class-specificity distribution. Specifically, we use contrastive learning to measure the relationship between samples and their reconstructions, which fully takes the discriminative information between data into account in PCA. In order to make the extracted low-dimensional features profoundly reflect the class-specificity distribution of the data, we minimize the squared ℓ1,2-norm of the low-dimensional embedding. In addition, to reduce the effects of redundant features and noise and to improve the interpretability of PCA at the same time, we impose sparsity constraints on the projection matrix using the squared ℓ1,2-norm. Our experimental results on different types of benchmark databases demonstrate that our model has state-of-the-art performance.

This paper investigates the asymptotic stability and synchronization of fractional-order (FO) memristive neural networks with time delays. Based on the FO comparison principle and inverse Laplace transform method, the novel sufficient conditions for the asymptotic stability of a FO nonlinear system are given. Then, based on the above conclusions, the sufficient conditions for the asymptotic stability and synchronization of FO memristive neural networks with time delays are investigated. The results in this paper have a wider coverage of situations and are more practical than the previous related results. Finally, the validity of the results is checked by two examples.

In this paper, a novel parallel learning framework is developed to solve zero-sum games for discrete-time nonlinear systems. Briefly, the purpose of this study is to determine a tentative function according to the prior knowledge of the value iteration (VI) algorithm. The learning process of the parallel controllers can be guided by the tentative function. That is to say, the neighborhood of the optimal cost function can be compressed within a small range via two typical exploration policies. Based on the parallel learning framework, a novel dichotomy VI algorithm is established to accelerate the learning speed. It is shown that the parallel controllers will converge to the optimal policy from contrary initial policies. Finally, two typical systems are used to demonstrate the learning performance of the constructed dichotomy VI algorithm.

This paper is concerned with self-representation subspace learning. It is one of the most representative subspace techniques, which has attracted considerable attention for clustering due to its good performance. Among these methods, low-rank representation (LRR) has achieved impressive results for subspace clustering. However, it only considers the similarity between the data itself, while neglecting the differences with other samples. Besides, it cannot well deal with noise and portray cluster-to-cluster relationships well. To solve these problems, we propose a Contrastive Self-representation model for Clustering (CSC). CSC simultaneously takes into account the similarity/dissimilarity between positive/negative pairs when learning the self-representation coefficient matrix of data while the form of the loss function can reduce the effect of noise on the results. Moreover, We use the ℓ1,2-norm regularizer on the coefficient matrix to achieve its sparsity to better characterize the cluster structure. Thus, the learned self-representation coefficient matrix well encodes both the discriminative information and cluster structure. Extensive experiments on seven benchmark databases indicate the superiority of our proposed method.

Designing efficient and accurate network architectures to support various workloads, from servers to edge devices, is a fundamental problem as the use of Convolutional Neural Networks (ConvNets) becomes increasingly widespread. One simple yet effective method is to scale ConvNets by systematically adjusting the dimensions of the baseline network, including width, depth, and resolution, enabling it to adapt to diverse workloads by varying its computational complexity and representation ability. However, current state-of-the-art (SOTA) scaling methods for neural network architectures overlook the inter-dimensional relationships within the network and the impact of scaling on inference speed, resulting in suboptimal trade-offs between accuracy and inference speed. To overcome those limitations, we propose a scaling method for ConvNets that utilizes dimension relationship and runtime proxy constraints to improve accuracy and inference speed. Specifically, our research notes that higher input resolutions in convolutional layers lead to redundant filters (convolutional width) due to increased similarity between information in different positions, suggesting a potential benefit in reducing filters while increasing input resolution. Based on this observation, the relationship between the width and resolution is empirically quantified in our work, enabling models with higher parametric efficiency to be prioritized through our scaling strategy. Furthermore, we introduce a novel runtime prediction model that focuses on fine-grained layer tasks with different computational properties for more accurate identification of efficient network configurations. Comprehensive experiments show that our method outperforms prior works in creating a set of models with a trade-off between accuracy and inference speed on the ImageNet datasets for various ConvNets.

Graph Neural Networks (GNNs) have been successfully applied to graph-level tasks in various fields such as biology, social networks, computer vision, and natural language processing. For the graph-level representations learning of GNNs, graph pooling plays an essential role. Among many pooling techniques, node drop pooling has garnered significant attention and is considered as a leading approach. However, existing node drop pooling methods, which typically retain the top-k nodes based on their significance scores, often overlook the diversity inherent in node features and graph structures. This limitation leads to suboptimal graph-level representations. To overcome this, we introduce a groundbreaking plug-and-play score scheme, termed MID. MID comprises a Multidimensional score space and two key operations: flIpscore and Dropscore. The multidimensional score space depicts the significance of nodes by multiple criteria; the flipscore process promotes the preservation of distinct node features; the dropscore compels the model to take into account a range of graph structures rather than focusing on local structures. To evaluate the effectiveness of our proposed MID, we have conducted extensive experiments by integrating it with a broad range of recent node drop pooling methods, such as TopKPool, SAGPool, GSAPool, and ASAP. In particular, MID has proven to bring a significant average improvement of approximately 2.8% over the four aforementioned methods when tested on 17 real-world graph classification datasets. Code is available at https://github.com/whuchuang/mid.

We introduce Tadam (Trust region ADAptive Moment estimation), a new optimizer based on the trust region of the second-order approximation of the loss using the Fisher information matrix. Despite the enhanced gradient estimations offered by second-order approximations, their practical implementation requires sizable batch sizes to estimate the second-order approximation matrices and perform matrix inversions. Consequently, integrating second-order approximations entails additional memory consumption and imposes substantial computational demands due to the inversion of large matrices. In light of these challenges, we have devised a second-order approximation algorithm that mitigates these issues by judiciously approximating the pertinent large matrix, requiring only a marginal increase in memory usage while minimizing the computational burden. Tadam approximates the loss up to the second order using the Fisher information matrix. Since estimating the Fisher information matrix is expensive in both memory and time, Tadam approximates the Fisher information matrix and reduces the computational burdens to the O(N) level. Furthermore, Tadam employs an adaptive trust region scheme to reduce approximate errors and guarantee stability. Tadam evaluates how well it minimizes the loss function and uses this information to adjust the trust region dynamically. In addition, Tadam adjusts the learning rate internally, even if we provide the learning rate as a fixed constant. We run several experiments to measure Tadam's performance against Adam, AMSGrad, Radam, and Nadam, which have the same space and time complexity as Tadam. The test results show that Tadam outperforms the benchmarks and finds reasonable solutions fast and stably.

The efficiency of communication across workers is a significant factor that affects the performance of federated learning. Though periodic communication strategy is applied to reduce communication rounds in training, the communication cost is still high when the training data distributions are not independently and identically distributed (non-IID) which is common in federated learning. Recently, some works introduce variance reduction to eliminate the effect caused by non-IID data among workers. Nevertheless the provable optimal communication complexity O(log(ST)) and convergence rate O(1/(ST)) cannot be achieved simultaneously, where S denotes the number of sampled workers in each round and T is the number of iterations. To deal with this dilemma, we propose an optimization algorithm SQUARFA that adopts stagewise training framework coupling with variance reduction and uses a quick-start phase in each loop. Theoretical results show that SQUARFA achieves both optimal convergence rate and communication complexity for both strongly convex objectives and non-convex objectives under PL condition, thus fills the gap mentioned above. Then, a variant of SQUARFA yields the optimal theoretical results for general non-convex objectives. We further extend the technique in SQUARFA to the large batch setting and achieve optimal communication complexity. Experimental results demonstrate the superiority of the proposed algorithms.

This article introduces a novel model of asymmetric neural networks combined with fractional difference memristors, which has both theoretical and practical implications in the rapidly evolving field of computational intelligence. The proposed model includes two types of fractional difference memristor elements: one with hyperbolic tangent memductance and the other with periodic memductance and memristor state described by sine functions. The authenticity of the constructed memristor is confirmed through fingerprint verification. The research extensively investigates the dynamics of a coupled neural network model, analyzing its stability at equilibrium states, studying bifurcation diagrams, and calculating the largest Lyapunov exponents. The results suggest that when incorporating sine memristors, the model demonstrates coexisting state variables depending on the initial conditions, revealing the emergence of multi-layer attractors. The article further demonstrates how the memristor state shifts through numerical simulations with varying memductance values. Notably, the study emphasizes the crucial role of memductance (synaptic weight) in determining the complex dynamical characteristics of neural network systems. To support the analytical results and demonstrate the chaotic response of state variables, the article includes appropriate numerical simulations. These simulations effectively validate the presented findings and provide concrete evidence of the system's chaotic behavior.

This paper considers an optimal control of an affine nonlinear system with unknown system dynamics. A new identifier-critic framework is proposed to solve the optimal control problem. Firstly, a neural network identifier is built to estimate the unknown system dynamics, and a critic NN is constructed to solve the Hamiltonian-Jacobi-Bellman equation associated with the optimal control problem. A dynamic regressor extension and mixing technique is applied to design the weight update laws with relaxed persistence of excitation conditions for the two classes of neural networks. The parameter estimation of the update laws and the stability of the closed-loop system under the adaptive optimal control are analyzed using a Lyapunov function method. Numerical simulation results are presented to demonstrate the effectiveness of the proposed IC learning based optimal control algorithm for the affine nonlinear system.