Boosted Lasso
Abstract
In this paper, we propose the Boosted Lasso (BLasso) algorithm that is able to produce an approximation to the complete regularization path for general Lasso problems. BLasso is derived as a coordinate descent method with a fixed small step size applied to the general Lasso loss function (L1 penalized convex loss). It consists of both a forward step and a backward step and uses differences of functions instead of gradient. The forward step is similar to Boosting and Forward Stagewise Fitting, but the backward step is new and crucial for BLasso to approximate the Lasso path in all situations. For cases with finite number of base learners, when the step size goes to zero, the BLasso path is shown to converge to the Lasso path. For nonparametric learning problems with a large or an infinite number of base learners, BLasso remains valid since its forward steps are Boosting steps and its backward steps only involve the base learners that are included in the model from previous iterations. Experimental results are also provided to demonstrate the difference between BLasso and Boosting or Forward Stagewise Fitting. In addition, we extend BLasso to the case of a general convex loss penalized by a general convex function and illustrate this extended BLasso with examples.
Document Details
- Document Type
- Technical Report
- Publication Date
- Dec 01, 2004
- Accession Number
- ADA473146
Entities
People
- Bin Yu
- Peng Zhao
Organizations
- University of California, Berkeley