LiNLNet: Gauging required nonlinearity in deep neural networks
Abstract
Feedforward deep neural networks (DNNs) commonly involve layer-wise linear operations and subsequent nonlinear operations, which are repeated through all layers. The nonlinear operations by nonlinear activations in each layer remarkably enhance the expressiveness of DNNs, resulting in the great success in a variety of application domains. Although the necessity of layer-wise nonlinear operations is agreed, the optimal nonlinearity for each layer in a given DNN is not clear. In this regard, we propose an easy-to-use method to layer-wise measure the optimal nonlinearity for a given DNN using its replica termed a linear-nonlinear network (LiNLNet). The key to the LiNLNet is the use of linear-nonlinear units (LiNLUs) whose degree of nonlinearity is parameterized by a trainable parameter p. The parameter p is shared among all LiNLUs in a given layer, thus indicating the layer-wise optimal nonlinearity. This method allows layer-level pruning such that the layers that do not require nonlinearity are merged into the subsequent layers, reducing computational complexity. For proofs of concept, we applied the proposed method to a MLP, AlexNet, VGG16, and ResNet18 on CIFAR-10 and ImageNet. The results commonly indicate the last hidden layer as a linear layer that may be merged into the output layer, reducing memory usage by 27% while maintaining the accuracy for LiNL-AlexNet on ImageNet.
Document Details
- Document Type
- Pub Defense Publication
- Publication Date
- Mar 01, 2023
- Source ID
- 10.1063/5.0134713
Entities
People
- Doo Seok Jeong
- SeongMin Jin
Organizations
- Hanyang University
- National Research Foundation of Korea
- Office of Naval Research Global