Token Geometry

Summary

arXiv:2607.01455v1 Announce Type: new Abstract: Language models learn continuous programs over discrete symbols, with the embedding table and LM-head acting as the read/write interface between them. We show that this interface has gradient geometry distinct from dense hidden weights which can be exploited to improve the Pareto frontier across supervised finetuning, RL, and pretraining, while only utilizing kilobytes of optimizer state. We introduce Ember, a lightweight optimizer for embedding an

SpinGraph analysis pending — check back after processing.

• How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size
• Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning
• Geometry-Aware R-Structured Kolmogorov-Arnold Networks
• On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
• Conditional Inference Trees and Forests for Feature Selection
• The Rollout Infrastructure Tax in Coding-Agent Reinforcement Learning