Deep Learning

Upon successful completion of this course, the students should:

- Understand the underlying concepts and properties related to learning by Deep Neural Networks (DNNs), including its formulation as empirical risk minimization problems and the different training methods.

- Be able to evaluate the performance of common DNN architectures/types, such as deep convolutional/recurrent neural networks, as well as factor models, analyzing also the practical issues.

- Know how to formulate and apply the Deep Learning framework to solve several practical problems in different application domains. 

This course offers an in-depth study on the mathematical and algorithmic foundations of deep neural networks (DNNs).

The course covers the following main topics:

• Part 1: Fundamentals of statistical learning and large-scale optimization for modern machine learning
  • Review of statistical learning theory, empirical risk minimization, estimators.
  • Supervised, unsupervised and reinforcement learning.
  • Limitations of traditional machine learning methods, motivation for deep neural networks (DNNs).
  • Large-scale statistical learning (convex case): batch vs. mini-batch training, stochastic gradient descent, acceleration methods, second-order methods, variance/noise reduction methods.
  • Non-convex optimization for statistical learning: DNN training problem as non-convex problem, successive linearization/convex approximation, block-coordinate descent methods (application to training auto-encoders and factor models), block-successive upper-bound minimization methods (application to training general DNNs), convergence guarantees.
• Part 2: Algorithms and Architectures of Deep Neural Networks
  • Overview of DNN types: feed-forward, convolutional, linear and recurrent.
  • Mathematical models: DNNs as compositions of non-linear operators.
  • Training with back-propagation, the role of regularization, dropout training.
  • Large-scale training of DNNs: challenges (ill-conditioning, inexact gradients, complex loss surface), algorithms for large-scale training with adaptive learning rate (ADAGRAD, ADAM).
  • Common DNN architectures: Deep convolutional neural networks, Deep recurrent neural networks, LTSM networks, Deep reinforcement learning.
  • Factor models: Principal/independent component analysis, Manifold learning. 
• Part 3: Current challenges and Applications of Deep Learning
  • Novel optimization methods for training DNNs (beyond first-order methods).
  • Training DNNs using Alternating Direction Multiplier Methods (ADMMs).
  • Training with convergence guarantees.
  • Adversarial examples and robustness.
  • Applications in data-driven inference and signal processing tasks.
  • Applications in wireless communication and sensor networks.
  • Geometric deep learning and applications to 3D data.
  • Open problems in deep learning research.