Syllabus
Feed-forward neural networks
- Basic architectures and activation functions
- Optimization algorithms for training deep models
Regularization of deep learning models
- Classical regularization based on penalization by parameter norm.
- Dropout
- Batch Normalization
- Multi-task learning
Convolutional neural networks
- Convolution and pooling layers
- Very deep convolutional network architectures
- State-of-the-art models for image recognition, object detection and image segmentation
Recurrent neural networks
- Recurrent neural networks and their training
- Long short-term memory
- Gated recurrent units
- Bidirectional and deep recurrent neural networks
- Encoder-decoder architectures
Practical methodology
- Selection of a suitable architecture
- Selection of hyperparameters
Natural language processing
- Distributed word representation
- Representation of words as sequences of characters
- State-of-the-art algorithms for morphological tagging, named-entity recognition, machine translation, image captioning
Deep generative models
- Variational autoencoders
- Generative adversarial networks
Structured prediction
- CRF layer
- CTC loss and its applications in state-of-the-art speech recognition algorithms.
Introduction to reinforcement learning
Annotation
Neural machine translation recently became a new interesting and successful paradigm. The new paradigm brings new theoretical concepts and new ways of seeing the classic problems of machine translation.
The goal of this seminar is to familiarize the students with the theoretical framework of neural machine translation in such depth that would allow them to study the most recent academic papers on this topic.