In this talk, we introduce some of our group's recent work on approximate Bayesian inference for computer vision and natural language processing. We start by introducing Dynamic Word Embeddings: a new Bayesian probabilistic language model which allows to track the semantic evolution of individual words over time. The model combines a probabilistic version of word2vec with latent time series models and shows improvements over various baselines in terms of performance and interpretability. We also focus on the advancement of theoretical aspects of approximate inference. Here, we present our recent study on mini-batch diversification using determinantal point processes, a modified stochastic gradient descent algorithm. This scheme allows us to balance an imbalanced data set by actively subsampling diversified mini-batches, resulting in higher predictive likelihoods on held-out data, and stochastic gradients that have a lower variance. Finally, we revisit stochastic gradient descent with constant learning rates and interpret it as an approximate Bayesian inference algorithm with favorable properties.
- Dynamic Word Embeddings via Skip-Gram Filtering. R. Bamler and S. Mandt. ICML 2017.
- Stochastic Learning on Imbalanced Data: Determinantal Point Processes for Mini-batch Diversification. C. Zhang, H. Kjellström, and S. Mandt. ArXiv 05/2017.
- A Variational Analysis of Stochastic Gradient Algorithms. S. Mandt, M. Hoffman, and D. Blei. ICML 2016.
Robert Bamler is a Postdoctoral Associate at Disney Research Pittsburgh. He works on statistical machine learning and approximate Bayesian inference, with a focus on applications to natural language processing. Robert obtained his Ph.D. in theoretical condensed matter physics from University of Cologne, Germany, in 2016.
Cheng Zhang is a Postdoctoral Associate at Disney Research Pittsburgh. She is active within the areas of machine learning and computer vision. Currently, she is working on approximate Bayesian inference. She has received her PhD at the Department of Robotics, Perception and Learning (RPL), KTH Royal Institute of Technology, Sep, 2016. During her PhD studies, she was a visiting student of Prof. Neil Lawrence’ group and was an intern at Microsoft Research Cambridge in the Infer.Net group.
Stephan Mandt is a Research Scientist at Disney Research Pittsburgh, where he leads the statistical machine learning group. He was was a postdoctoral researcher with David Blei, first at Princeton University and then at Columbia University. Stephan holds a Ph.D. in Theoretical Physics from the University of Cologne, Germany.