Class-Balance Bias in Regularized Regression
Regularized models are often sensitive to the scales of the features in the data and it has therefore become standard practice to normalize (center and scale) the features before fitting the model. But there are many different ways to normalize the features and the choice may have dramatic effects on the resulting model. In spite of this, there has so far been no research on this topic. In this paper, we begin to bridge this knowledge gap by studying normalization in the context of lasso, ridge, and elastic net regression. We focus on normal and binary features and show that the class balances of binary features directly influences the regression coefficients and that this effect depends on the combination of normalization and regularization methods used. We demonstrate that this effect can be mitigated by scaling binary features with their variance in the case of the lasso and standard deviation in the case of ridge regression, but that this comes at the cost of increased variance. For the elastic net, we show that scaling the penalty weights, rather than the features, can achieve the same effect. Finally, we also tackle mixes of binary and normal features as well as interactions and provide some initial results on how to normalize features in these cases.
Citation
@misc{larsson2025,
author = {Larsson, Johan and Wallin, Jonas},
publisher = {arXiv},
title = {Class-Balance Bias in Regularized Regression},
number = {arXiv:2501.03821},
date = {2025-01-08},
url = {http://arxiv.org/abs/2501.03821},
doi = {10.48550/arXiv.2501.03821},
langid = {en},
abstract = {Regularized models are often sensitive to the scales of
the features in the data and it has therefore become standard
practice to normalize (center and scale) the features before fitting
the model. But there are many different ways to normalize the
features and the choice may have dramatic effects on the resulting
model. In spite of this, there has so far been no research on this
topic. In this paper, we begin to bridge this knowledge gap by
studying normalization in the context of lasso, ridge, and elastic
net regression. We focus on normal and binary features and show that
the class balances of binary features directly influences the
regression coefficients and that this effect depends on the
combination of normalization and regularization methods used. We
demonstrate that this effect can be mitigated by scaling binary
features with their variance in the case of the lasso and standard
deviation in the case of ridge regression, but that this comes at
the cost of increased variance. For the elastic net, we show that
scaling the penalty weights, rather than the features, can achieve
the same effect. Finally, we also tackle mixes of binary and normal
features as well as interactions and provide some initial results on
how to normalize features in these cases.}
}