The way I see it, the two most important features of Neural Networks that make them so powerful are 1. Differentiability and 2. Compositionality. Differentiability enables optimization using gradient descent, which is orders of magnitude faster than most other numerical optimization methods. Compositionality, on the other hand, means that we can make use of the chain rule for differentiation, and break down potentially unwieldy functions into small manageable units that we can handle one at the time. However, in most cases the Neural Network architecture itself needs to be *designed*. Optimizing this design is completely nontrivial, and is in fact where most of the Neural Network research is focused.
On Neural Networks and Gradient Descent
On Neural Networks and Gradient Descent
On Neural Networks and Gradient Descent
The way I see it, the two most important features of Neural Networks that make them so powerful are 1. Differentiability and 2. Compositionality. Differentiability enables optimization using gradient descent, which is orders of magnitude faster than most other numerical optimization methods. Compositionality, on the other hand, means that we can make use of the chain rule for differentiation, and break down potentially unwieldy functions into small manageable units that we can handle one at the time. However, in most cases the Neural Network architecture itself needs to be *designed*. Optimizing this design is completely nontrivial, and is in fact where most of the Neural Network research is focused.