CNN

Recent Advances in Convolutional Neural Networks

Posted by c cm on February 9, 2017

1. Introduction

基本结构:

  1. input layer
    • first layer
  2. convolutional layer
    • aims to learn feature representations of the inputs. The activation function introduces nonlinearities to CNN.
  3. pooling layer
    • aims to achieve shift-invariance by reducing the resolution of the feature maps.
    • between convolutional layers,
  4. fully-connected layer
    • aims to perform high-level reasoning.
    • take all neurons in the previous layer and connect them to every single neuron of current layer to generate global semantic information.
  5. output layer
    • last layer

2. Improvements

1. Convolutional Layer

  • Tiled Convolution
  • Transposed Convolution
  • Dilated Convolution
  • Network in Network
  • Inception Module

2. Pooling Layer

  • $L_p$ Pooling
    Average Pooling和Max Pooling的一般形式
  • Mixed Pooling
    Average Pooling和Max Pooling的加权
  • Stochastic Pooling
    拟合multinomial分布,抽样pooling
  • Spectral Pooling
  • Spatial Pyramid Pooling
  • Multi-scale Orderless Pooling

3. Activation Function

  • ReLU(Recitified Linear Unit)
  • Leaky ReLU
  • Parametric ReLU
  • Randomized ReLU
  • ELU(Exponential Linear Unit)
  • Maxout
  • Probout

4. Loss Function

  • Hinge Loss
  • Softmax Loss/Large-Margin Softmax
  • Contrastive Loss
  • Triplet Loss
    anchor/positive/negative pair
  • KL Divergence/JSD

5. Regularization

  • lp-norm
  • Dropout
    随机将neuron的输出置为0
  • DropConnect
    随机将学习到的weight置为0

6. Optimization

  • Data Augmentation
  • Weight Initialization
  • Stochastic Gradient Descent
  • Batch Normalization
  • Shortcut Connections