Résultats de recherche pour « Neural Network Learning »

Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks. Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a `forgetting event' to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Across several benchmark data sets, we find that: (i) certain examples are forgotten with high frequency, and some not at all; (ii) a data set's (un)forgettable examples generalize across neural architectures; and (iii) based on forgetting dynamics, a significant fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

In this paper, we aim at developing scalable neural network-type learning systems. Motivated by the idea of constructive neural networks in approximation theory, we focus on constructing rather than training feed-forward neural networks (FNNs) for learning, and propose a novel FNNs learning system called the constructive FNN (CFN). Theoretically, we prove that the proposed method not only overcomes the classical saturation problem for constructive FNN approximation, but also reaches the optimal learning rate when the regression function is smooth, while the state-of-the-art learning rates established for traditional FNNs are only near optimal (up to a logarithmic factor). A series of numerical simulations are provided to show the efficiency and feasibility of CFN.

Furthermore, the trusted party becomes a single point of failure, thus both data and model privacy could be compromised by data breaches, hacking, leaks, etc.Hence, solutions originating from the cryptographic community replace and emulate the trusted party with a group of computing servers.In particular, to enable privacy-preserving training of NNs, several studies employ multiparty computation (MPC) techniques and operate on the two [83], [28], three [82], [110], [111], or four [26], [27] server models.Such approaches, however, limit the number of parties among which the trust is split, often assume an honest majority among the computing servers, and require parties to communicate (i.e., secret share) their data outside their premises.This might not be acceptable due to the privacy and confidentiality requirements and the strict data protection regulations.Furthermore, the computing servers do not operate on their own data or benefit from the model training; hence, their only incentive is the reputation harm if they are caught, which increases the possibility of malicious behavior.A recently proposed alternative for privacy-preserving training of NNs -without data outsourcing -is federated learning.Instead of bringing the data to the model, the model is brought (via a coordinating server) to the clients, who perform model updates on their local data.The updated models from the parties are averaged to obtain the global NN model [75], [63].Although federated learning retains the sensitive input data locally and eliminates the need for data outsourcing, the model, that might also be sensitive, e.g., due to proprietary reasons, becomes available to the coordinating server, thus placing the latter in a position of power with respect to the remaining parties.Recent research demonstrates that sharing intermediate model updates among the parties or with the server might lead to various privacy attacks, such as extracting parties' inputs [53], [113], [120] or membership inference [78], [86].Consequently, several works employ differential privacy to enable privacy-preserving exchanges of intermediate values and to obtain models that are free from adversarial inferences in federated learning settings [67], [101], [76]. Although differentially private techniques partially limit attacks to federated learning, they decrease the utility of the data and the resulting ML model.Furthermore, training robust and accurate models requires high privacy budgets, and as such, the level of privacy achieved in practice remains unclear [55].Therefore, a distributed privacy-preserving deep learning approach requires strong cryptographic protection of the intermediate model updates during the training, as well as of the final model weights.Recent cryptographic approaches for private distributed learning, e.g., [119], [42], not only have limited ML functionalities, i.e., regularized or generalized linear models, but also employ traditional encryption schemes that make them vulnerable to post-quantum attacks.This should be cautiously considered, as recent advances in quantum computing [47], [87], [105], [116], increase the need for deploying quantum-resilient cryptographic schemes that eliminate Abstract-In this paper, we address the problem of privacypreserving training and evaluation of neural networks in an N-party, federated learning setting.We propose a novel system, POSEIDON, the first of its kind in the regime of privacy-preserving neural network training.It employs multiparty lattice-based cryptography to preserve the confidentiality of the training data, the model, and the evaluation data, under a passive-adversary model and collusions between up to N -1 parties.To efficiently execute the secure backpropagation algorithm for training neural networks, we provide a generic packing approach that enables Single Instruction, Multiple Data (SIMD) operations on encrypted data.We also introduce arbitrary linear transformations within the cryptographic bootstrapping operation, optimizing the costly cryptographic computations over the parties, and we define a constrained optimization problem for choosing the cryptographic parameters.Our experimental results show that POSEIDON achieves accuracy similar to centralized or decentralized non-private approaches and that its computation and communication overhead scales linearly with the number of parties.POSEIDON trains a 3-layer neural network on the MNIST dataset with 784 features and 60K samples distributed among 10 parties in less than 2 hours.

Neural networks offer a promising tool for the analysis of nonlinear economies. In this paper, we derive conditions for the stability of nonlinear rational expectations equilibria under neural network learning. We demonstrate the applicability of the conditions in analytical and numerical examples where the nonlinearity is caused by monetary policy targeting a range, rather than a specific value, of inflation. If shock persistence is high or there is inertia in the structure of the economy, then the only rational expectations equilibria that are learnable may involve inflation spending long periods outside its target range. Neural network learning is also useful for solving and selecting between multiple equilibria and steady states in other settings, such as when there is a zero lower bound on the nominal interest rate.

This article presents a learning method for a special class of belief networks known as noisy-or networks. By extending the Hebbian rule of neural networks, two learning rules are developed to learn the probabilities of nodes and the internode causal strengths, respectively. The latter rule also learns structures of networks because a nonzero causal strength indicates the existence of a causal link. One distinct feature of this method is its ability to work in a sequential or incremental manner in which a network adjusts its parameters upon the arrival of every case description. As a result, this method is capable of not only constructing a causal knowledge base from a fixed set of case data but also dynamically adapting an existing knowledge base to a changing environment. To prove the convergence of learning, a Liapunov function is identified for the dynamic system defined by the learning rules. Computer experiment results show that this method is significantly faster than some existing learning methods for belief networks. © 1996 John Wiley & Sons, Inc.

Backpropagation (BP) algorithm is widely used to solve many real world problems by using the concept of Multilayer Perceptron (MLP).However, major disadvantages of BP are its convergence rate is relatively slow and always being trapped at the local minima.To overcome this problem, Genetic Algorithm (GA) has been used to determine optimal value for BP parameters such as learning and momentum rate and, also for weight optimization.Although GA has successfully improved Backpropagation Neural Network (BPNN) learning, there are still some issues such as longer training time to produce the output and usage of complex functions in selection, crossover and mutation calculation.In this study, Particle Swarm Optimization (PSO) algorithm has been chosen and applied in feedforward neural network to enhance the learning process in terms of convergence rate and classification accuracy.Two experiments have been conducted; Particle Swarm Optimization Feedforward Neural Network (PSONN) and Genetic Algorithm Backpropagation Neural Network (GANN).The results show that PSONN give promising results in terms of convergence rate and classification accuracy compared to GANN.

Impedance control is one of the most effective control methods for the manipulators in contact with their environments. The characteristics of force and motion control, however, is determined by a desired impedance parameter of a manipulator's end-effector that should be carefully designed according to a given task and an environment. The present paper proposes a new method to regulate the impedance parameter of the end-effector through learning of neural networks. Three kinds of the feed-forward networks are prepared corresponding to position, velocity and force control loops of the end-effector before learning. First, the neural networks for position and velocity control are trained using iterative learning of the manipulator during free movements. Then, the neural network for force control is trained for contact movements. During learning of contact movements, a virtual trajectory is also modified to reduce control error. The method can regulate not only stiffness and viscosity but also inertia and virtual trajectory of the end-effector. Computer simulations show that a smooth transition from free to contact movements can be realized by regulating impedance parameters before a contact.