Hyperparameter-free DP optimization (GeN-DP)

Implements HyFreeDP, a hyperparameter-free differentially private optimizer that sets its own learning rate via a privatized generalized-Newton step.

Standard DP training fixes a learning rate schedule by grid search, which is costly and itself leaks information through the data-dependent choice. HyFreeDP removes this by combining automatic per-sample gradient clipping with the generalized Newton (GeN) learning rate, adapted to DP. Each step privatizes both the gradient direction $m_t$ and a few loss probes along that direction; fitting a quadratic to the probed losses yields a privatized slope and curvature whose ratio is the learning rate. The direction can be passed through any base optimizer (momentum, Adam-style preconditioning, weight decay) as DP post-processing.

$$\begin{aligned} m_t &= \frac{1}{B}\Big(\sum_{i} \min\!\big(\tfrac{R_g}{\lVert g_i\rVert},1\big)\,g_i + \sigma_g R_g\, z_g\Big),\qquad z_g\sim\mathcal{N}(0,I)\\ \tilde{L}(\theta) &= \frac{1}{B}\Big(\sum_{i} \min\!\big(\tfrac{R_l}{L_i},1\big)\,L_i + \sigma_l R_l\, z_l\Big),\qquad z_l\sim\mathcal{N}(0,1)\\ (b,a) &= \arg\min_{a,b}\ \sum_{j}\Big|\,\tilde{L}(\theta-\eta_j m_t) - \big(\tilde{L}(\theta) - b\,\eta_j + \tfrac{a}{2}\,\eta_j^{2}\big)\Big|^{2}\\ \eta_t &= \frac{b}{a} = \frac{(G^{\top}m)_{\mathrm{DP}}}{(m^{\top}Hm)_{\mathrm{DP}}}\\ \theta_{t+1} &= \theta_t - \eta_t\, m_t \end{aligned}$$

where $g_i,L_i$ are the per-sample gradient and loss, $R_g,R_l$ the gradient and loss clipping norms, $\sigma_g,\sigma_l$ the noise multipliers, $B$ the batch size, and $z_g,z_l$ Gaussian DP noise; $\tilde{L}$ is the privatized loss probed at offsets $\eta_j$ along $-m_t$, and the least-squares fit returns the privatized directional slope $b=(G^{\top}m)_{\mathrm{DP}}$ and curvature $a=(m^{\top}Hm)_{\mathrm{DP}}$, giving the generalized-Newton step $\eta_t=b/a$.

Reference: Zhiqi Bu, Ruixuan Liu, "Towards hyperparameter-free optimization with differential privacy", ICLR 2025. https://arxiv.org/abs/2503.00703

---

Back to the Canon