TallyTrain: Communication-Efficient Federated Distillation

Summary

arXiv:2607.00173v1 Announce Type: new Abstract: Federated learning is bandwidth-bound on two orthogonal axes: model size, which limits how often parameter-averaging methods can afford to merge, and class count, which makes per-probe soft-label distillation prohibitive at large vocabularies. Both ceilings tighten as modern systems scale. We collapse the class-count axis to $\lceil \log_2 C \rceil$ bits per probe by transmitting only each peer's $\arg\max$ class index, where $C$ is the number of o

SpinGraph analysis pending — check back after processing.

• How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
• Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning
• Token Geometry
• Geometry-Aware R-Structured Kolmogorov-Arnold Networks
• On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
• Conditional Inference Trees and Forests for Feature Selection