This research establishes the first comprehensive set of scaling laws for training language models with differential privacy (DP), enabling more efficient privacy-utility trade-offs.

• DP training significantly impacts model performance, with larger models suffering more pronounced effects
• Compute-optimal models under DP require larger batch sizes and lower learning rates than non-private counterparts
• Privacy budgets (measured by epsilon ε) can be optimized by balancing dataset size and training steps
• Empirical findings demonstrate that parameter scaling alone cannot overcome privacy-induced performance degradation

This research is crucial for organizations that need to train models on sensitive user data while maintaining robust privacy guarantees, providing a principled approach to make privacy-compute-performance tradeoffs.

Scaling Laws for Differentially Private Language Models