Relevant literature

Book chapter about the philosophy behind deep architecture model, motivating them in the context of Artificial Intelligence

 

• Scaling Learning Algorithms towards AI |  |
  Bengio, Y. and LeCun, Y. 
  Book chapter in "Large-Scale Kernel Machines"

Introducing Deep Belief Networks as generative models:

 

• A fast learning algorithm for deep belief nets |    | 
  Hinton, G. E., Osindero, S. and Teh, Y. 
  Neural Computation (2006)

Deep Belief Networks as a simple way of initializing a deep feed-forward neural network:

 

• To recognize shapes, first learn to generate images |  | 
  Hinton, G. E. 
  Technical Report (2006) 

General study of the framework of initializing a deep feed-forward neural network using a greedy layer-wise procedure:

 

• Greedy Layer-Wise Training of Deep Networks |   | 
  Bengio, Y., Lamblin, P., Popovici, P., Larochelle, H. 
  NIPS 2006

An application of greedy layer-wise learning of a deep autoassociator for dimensionality reduction:

 

• Reducing the dimensionality of data with neural networks |    | 
  Hinton, G. E. and Salakhutdinov, R. R. 
  Science 2006

A way to use the greedy layer-wise learning procedure to learn a useful embeding for k nearest neighbor classification:

 

• Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure |  | 
  Salakhutdinov, R. R. and Hinton, G. E. 
  AISTATS 2007

Different theoretical results about Restricted Boltzmann Machines (RBMs) and Deep Belief Networks, like the universal approximation property of RBMs:

 

• Representational Power of Restricted Boltzmann Machines and Deep Belief Networks |  | 
  Le Roux, N. and Bengio, Y. 
  Technical Report

A novel way of using greedy layer-wise learning for Convolutional Networks:

 

• Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition |  | 
  Ranzato, M'A, Huang, F-J, Boureau, Y-L, and Le Cun, Y. 
  CVPR 2007

How to generalize Restricted Boltzmann Machines to types of data other than binary using exponential familly distribution:

 

• Exponential Family Harmoniums with an Application to Information Retrieval |   | 
  Welling, M., Rosen-Zvi, M. and Hinton, G. E. 
  NIPS 2004

An evaluation of deep networks on many datasets related to vision:

 

• An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation |   | 
  Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y. 
  ICML 2007

Application of deep learning in the context of information retrieval:

 

• • Semantic Hashing |  | 
    Salakhutdinov, R. R. and Hinton, G. E. 
    IRGM 2007