How to Train Your Intercept: A Comparison of Intercept Update Strategies in Coordinate Descent
How intercept updates affect coordinate descent for regularized generalized linear models, and why Newton steps help with imbalanced responses.
The way a coordinate descent solver updates the intercept can determine how quickly it fits a regularized generalized linear model, particularly when responses are imbalanced. We compare updates using a global curvature bound, a safeguarded Newton step, and exact minimization over the intercept. Our analysis explains how conservative updates slow convergence by failing to resolve the coupling between the intercept and the coefficients. Experiments with simulated data, real data, and production solvers show that one Newton step often achieves progress similar to exact minimization at a lower cost. We also examine quadratic approximations and multinomial models, and give recommendations for handling the intercept in each setting.
Citation
@misc{larsson2026,
author = {Larsson, Johan and Fabricius-Bjerre, Frederik},
publisher = {arXiv},
title = {How to {Train} {Your} {Intercept:} {A} {Comparison} of
{Intercept} {Update} {Strategies} in {Coordinate} {Descent}},
number = {arXiv:2610.18043},
date = {2026-10-13},
url = {https://arxiv.org/abs/2610.18043},
doi = {10.48550/arXiv.2610.18043},
langid = {en},
abstract = {The way a coordinate descent solver updates the intercept
can determine how quickly it fits a regularized generalized linear
model, particularly when responses are imbalanced. We compare
updates using a global curvature bound, a safeguarded Newton step,
and exact minimization over the intercept. Our analysis explains how
conservative updates slow convergence by failing to resolve the
coupling between the intercept and the coefficients. Experiments
with simulated data, real data, and production solvers show that one
Newton step often achieves progress similar to exact minimization at
a lower cost. We also examine quadratic approximations and
multinomial models, and give recommendations for handling the
intercept in each setting.}
}