LLMCMC: Language Models as Metropolis-Hastings Kernels for Probabilistic Program Inference
Lanhao Niu and N. Siddharth
Submitted to NeurIPS 2026 Workshop on Principles of Generative Modeling, 2026
TL;DR: This work introduces LLMCMC, an amortized MCMC framework that uses autoregressive language models to generate proposals from probabilistic program text, enabling posterior geometry to be transferred across programs.
@article{niu2026mcmc,title={LLMCMC: Language Models as Metropolis-Hastings Kernels for Probabilistic Program Inference},author={Niu, Lanhao and Siddharth, N.},journal={Submitted to NeurIPS 2026 Workshop on Principles of Generative Modeling},year={2026},}
ICML 2026
Language Models Represent and Transform Concepts with Shared Geometry
Zhimin Hu, Lanhao Niu, and Sashank Varma
ICML 2026 Workshop on Mechanistic Interpretability, 2026
TL;DR: This work characterizes the geometry of how context structures concept spaces across language models, showing it is semantically organized and shared across models.
@article{hu2026representational,title={Language Models Represent and Transform Concepts with Shared Geometry},author={Hu, Zhimin and Niu, Lanhao and Varma, Sashank},journal={ICML 2026 Workshop on Mechanistic Interpretability},year={2026},}
CogSci 2026
The Representational Geometry of Number
Zhimin Hu, Lanhao Niu, and Sashank Varma
CogSci, 2026
TL;DR: This work finds that language models preserve relational structure between concepts to support generalization, while projecting them into distinct subspaces to support multitasking, using number as a testbed.
@article{hu2026number,title={The Representational Geometry of Number},author={Hu, Zhimin and Niu, Lanhao and Varma, Sashank},journal={CogSci},year={2026},}
LLMCMC: Language Models as Metropolis-Hastings Kernels for Probabilistic Program InferenceSubmitted to NeurIPS 2026 Workshop on Principles of Generative Modeling, 2026TL;DR: This work introduces LLMCMC, an amortized MCMC framework that uses autoregressive language models to generate proposals from probabilistic program text, enabling posterior geometry to be transferred across programs.
Language Models Represent and Transform Concepts with Shared GeometryICML 2026 Workshop on Mechanistic Interpretability, 2026TL;DR: This work characterizes the geometry of how context structures concept spaces across language models, showing it is semantically organized and shared across models.