Feature Normalization under Finite Training Budgets
How feature standardization changes the trade-off between learning rare effects and amplifying noise in stochastic gradient descent with limited data.
Feature scaling affects both how quickly stochastic gradient descent learns rare effects and how much noise it accumulates. We compare standardized and unscaled training under a shared observation budget, allowing each method to choose its learning rate and stopping time. For two independent binary features, we characterize the best attainable risk on the rare subgroup as its prevalence decreases. The preferred scaling depends on how the signal is distributed between the common and rare features, as well as on the noise and available budget. Further bounds and experiments examine finite prevalence, reused observations, estimated preprocessing, and fitted intercepts. We also show that selecting a predictor after training can reverse the comparison.
Citation
@misc{larsson2026,
author = {Larsson, Johan},
publisher = {arXiv},
title = {Feature {Normalization} Under {Finite} {Training} {Budgets}},
number = {arXiv:2610.16845},
date = {2026-10-12},
url = {https://arxiv.org/abs/2610.16845},
doi = {10.48550/arXiv.2610.16845},
langid = {en},
abstract = {Feature scaling affects both how quickly stochastic
gradient descent learns rare effects and how much noise it
accumulates. We compare standardized and unscaled training under a
shared observation budget, allowing each method to choose its
learning rate and stopping time. For two independent binary
features, we characterize the best attainable risk on the rare
subgroup as its prevalence decreases. The preferred scaling depends
on how the signal is distributed between the common and rare
features, as well as on the noise and available budget. Further
bounds and experiments examine finite prevalence, reused
observations, estimated preprocessing, and fitted intercepts. We
also show that selecting a predictor after training can reverse the
comparison.}
}