Search Results for « Deep Variational Bayes Filters »

These notes were compiled as lecture notes for a course developed and taught at the University of the Southern California. They should be accessible to a typical engineering graduate student with a strong background in Applied Mathematics. The main objective of these notes is to introduce a student who is familiar with concepts in linear algebra and partial differential equations to select topics in deep learning. These lecture notes exploit the strong connections between deep learning algorithms and the more conventional techniques of computational physics to achieve two goals. First, they use concepts from computational physics to develop an understanding of deep learning algorithms. Not surprisingly, many concepts in deep learning can be connected to similar concepts in computational physics, and one can utilize this connection to better understand these algorithms. Second, several novel deep learning algorithms can be used to solve challenging problems in computational physics. Thus, they offer someone who is interested in modeling a physical phenomena with a complementary set of tools.

Variational Bayes (VB) replaces the posterior with the closest member of a tractable family, maximizing the evidence lower bound. The bound is slack by exactly the Kullback-Leibler divergence between surrogate and posterior-an information gap that is never observed and never closed by computation-while the mean-field surrogate that makes VB tractable removes posterior correlation by construction. We argue this compromise is now an anachronism, because scalable full Bayes is available through two complementary routes. The first is conditional Gaussianization: many likelihoods and priors are mean-scale mixtures of normals, so that conditional on a latent variable every full conditional is exactly Gaussian. Pólya-Gamma augmentation, support-vector-machine augmentation, the quantile asymmetric-Laplace mixture, and global-local shrinkage are instances of one envelopeand-duality construction, with conjugate, parallel, exact Gibbs samplers. The second is the Noise Outsourcing Theorem: the posterior is a measurable map θ = f (y, ε) of independent noise, and f is learned by simulation and supervised deep learning as an implicit quantile network-one forward pass per draw, with all dependence intact, no density, no Markov chain, and likelihood-free. The theorem applies in time, giving a weight-free generative particle filter. Sparsity and MAP estimation follow from the same machinery. Three experiments quantify the contrast: mean-field VB destroys a posterior correlation of-0.92, deflates standard deviations to 38%, opens a 0.93-nat gap, and drops 95% coverage to 58%, while exact inference is calibrated; a learned generative map recovers bounded, skewed, and bimodal posteriors that a Gaussian surrogate cannot represent. 

Deep Neural Networks, often owing to the overparameterization, are shown to be capable of exactly memorizing even randomly labelled data. Empirical studies have also shown that none of the standard regularization techniques mitigate such overfitting. We investigate whether the choice of the loss function can affect this memorization. We empirically show, with benchmark data sets MNIST and CIFAR-10, that a symmetric loss function, as opposed to either cross-entropy or squared error loss, results in significant improvement in the ability of the network to resist such overfitting. We then provide a formal definition for robustness to memorization and provide a theoretical explanation as to why the symmetric losses provide this robustness. Our results clearly bring out the role loss functions alone can play in this phenomenon of memorization.

Deep neural networks and the ENO procedure are both efficient frameworks for approximating rough functions. We prove that at any order, the ENO interpolation procedure can be cast as a deep ReLU neural network. This surprising fact enables the transfer of several desirable properties of the ENO procedure to deep neural networks, including its high-order accuracy at approximating Lipschitz functions. Numerical tests for the resulting neural networks show excellent performance for approximating solutions of nonlinear conservation laws and at data compression.

Evidential deep learning, built upon belief theory and subjective logic, offers a principled and computationally efficient way to turn a deterministic neural network uncertainty-aware. The resultant evidential models can quantify fine-grained uncertainty using the learned evidence. To ensure theoretically sound evidential models, the evidence needs to be non-negative, which requires special activation functions for model training and inference. This constraint often leads to inferior predictive performance compared to standard softmax models, making it challenging to extend them to many large-scale datasets. To unveil the real cause of this undesired behavior, we theoretically investigate evidential models and identify a fundamental limitation that explains the inferior performance: existing evidential activation functions create zero evidence regions, which prevent the model to learn from training samples falling into such regions. A deeper analysis of evidential activation functions based on our theoretical underpinning inspires the design of a novel regularizer that effectively alleviates this fundamental limitation. Extensive experiments over many challenging real-world datasets and settings confirm our theoretical findings and demonstrate the effectiveness of our proposed approach.

We consider the generic deep image enhancement problem where an input image is transformed into a perceptually better-looking image. Recent methods for image enhancement consider the problem by performing style transfer and image restoration. The methods mostly fall into two categories: training data-based and training data-independent (deep internal learning methods). We perform image enhancement in the deep internal learning framework. Our Deep Internal Learning for Image Enhancement framework enhances content features and style features and uses contextual content loss for preserving image context in the enhanced image. We show results on both hazy and noisy image enhancement. To validate the results, we use structure similarity and perceptual error, which is efficient in measuring the unrealistic deformation present in the images. We show that the proposed framework outperforms the relevant state-of-the-art works for image enhancement.