Optimization and Generalization in Boosting
Ling Li, Yaser Abu-Mostafa

Abstract. The superior out-of-sample performance of AdaBoost has been attributed to the fact that it minimizes a cost function based on margin. In order to examine how the cost function, in and of itself, affects the out-of-sample performance, we apply several more sophisticated optimization techniques directly to the cost function. When the AdaBoost exponential cost function is optimized, our methods generally yield much lower cost and training error but higher test error, which implies that the exponential cost is vulnerable to overfitting. With the optimization power gained, we can adopt more "regularized" cost functions that have better out-of-sample performance but are difficult to optimize. Our experiments demonstrate that with suitable cost functions, our methods can have better out-of-sample performance. (full report)

Monotonic Bernoulli Trials
Amrit Pratap, Yaser Abu-Mostafa, Pietro Perona

Abstract. When estimating a number of bernoulli variables which have a certain monotonicity constraint, if the number of samples for each variable is small, then the estimates will not satisfy the monotonicity constraint. Better performance is achieved by endorcing the monotonicity constraint on the estimation procedure. (full report)