Journal article · 2026 · Open access
Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning
IEEE Access, 14, 162936–162957 · Published 14 October 2026 · DOI 10.1109/access.2026.3743463
In brief
One loss, the Bisimulation Infomax Loss, ties representation learning and curiosity-driven exploration to the same geometric object, the bisimulation metric, with no hand-tuned trade-off weight. The resulting method, Adaptive Bisimulation Mirror-Descent, cuts steps-to-threshold by up to 33% on MiniGrid and lifts returns by up to 17% on the Distracting Control Suite.
- Journal
- IEEE Access, vol. 14
- Published
- 14 October 2026
- Access
- Open access · CC BY 4.0
- Review
- Peer-reviewed
- Authors
- 7
Abstract
Two long-standing difficulties in reinforcement learning (RL)—learning compact state representations and principled exploration—are usually handled by separate components glued together with a hand-tuned weight. We argue they are not separate: both are manifestations of one geometric object, the bisimulation pseudometric of the underlying Markov decision process (MDP). Our core contribution is a single loss, the Bisimulation Infomax Loss (BIL), removing the conventional λ trade-off coefficient, together with two certificates written in one measurable quantity, the embedding distortion κ*. First, a Dual Near-Optimality Theorem: under stated regularity and model assumptions, an approximate isometry of the bisimulation pseudometric lies simultaneously within O(κ*²) of the InfoNCE bound attained by an exact isometry and within O(κ*) of the corresponding one-step epistemic-information-gain guarantee. Second, a Curiosity Certificate: under explicit reachability and hitting-probability assumptions, the exploration horizon satisfies T_explore = Õ((κ*)^−D), with D the doubling dimension of the bisimulation geometry; the measured slope of −1.96 (R² = 0.96) is consistent with an effective dimension near two. We also bound the error of bootstrapped bisimulation targets and propagate it through every guarantee. We then propose Adaptive Bisimulation Mirror-Descent (ABMD), combining BIL with generalized Gauss–Newton preconditioning, an adaptive bisimulation temperature, and behavior-preserving augmentations. ABMD reduces steps-to-threshold by up to 33% over the strongest baseline on MiniGrid, lifts returns on the Distracting Control Suite by up to 17%, and gains steadily on four Procgen environments, all under one hyperparameter setting. Sensitivity analyses, cross-environment ablations, out-of-distribution stress tests, and compute-normalized comparisons accounting for the 1.9× per-step overhead are reported.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Gupta, D., Sikarwar, A., Dhangar, S., Jha, C. K., Jha, P., Sharma, A., & Abdullayev, V. H. (2026). Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning. IEEE Access, 14, 162936–162957. https://doi.org/10.1109/access.2026.3743463
MLA 9
Gupta, Deepak, et al. “Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning.” IEEE Access, vol. 14, 2026, pp. 162936–162957. https://doi.org/10.1109/access.2026.3743463.
Chicago
Gupta, Deepak, Aditya Sikarwar, Sanidhya Dhangar, Chanakya Kumar Jha, Pallavi Jha, Akhilesh Sharma, and Vugar Hacimahmud Abdullayev. 2026. “Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning.” IEEE Access 14: 162936–162957. https://doi.org/10.1109/access.2026.3743463.
Harvard
Gupta, D. et al. (2026) ‘Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning’, IEEE Access, 14, pp. 162936–162957. doi:10.1109/access.2026.3743463.
IEEE
D. Gupta et al., “Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning,” IEEE Access, vol. 14, pp. 162936–162957, 2026, doi: 10.1109/access.2026.3743463.
BibTeX
@article{gupta2026bisimulation,
title = {{Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning}},
author = {Gupta, Deepak and Sikarwar, Aditya and Dhangar, Sanidhya and Jha, Chanakya Kumar and Jha, Pallavi and Sharma, Akhilesh and Abdullayev, Vugar Hacimahmud},
journal = {IEEE Access},
year = {2026},
volume = {14},
pages = {162936--162957},
doi = {10.1109/access.2026.3743463},
url = {https://doi.org/10.1109/access.2026.3743463}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- representation learning
- exploration
- curiosity-driven learning
- InfoNCE
- embedding distortion
- isometries
- doubling dimension
- mirror descent
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
What is Bisimulation Geometry in reinforcement learning?
It is the view, introduced in this paper, that learning a compact state representation and exploring efficiently are two sides of one geometric object: the bisimulation pseudometric of the Markov decision process, which measures how differently two states behave. An encoder that embeds this metric faithfully is good for both at once.
What is the Bisimulation Infomax Loss (BIL)?
A single training loss that anchors both self-supervised representation learning and curiosity-driven exploration to the bisimulation metric, so the usual hand-tuned λ weight between the two objectives is no longer needed.
What is Adaptive Bisimulation Mirror-Descent (ABMD)?
The paper's algorithm. It combines the Bisimulation Infomax Loss with generalized Gauss–Newton preconditioning, an adaptive bisimulation temperature and behavior-preserving augmentations, and runs under one hyperparameter setting.
How well does ABMD perform?
It reduces steps-to-threshold by up to 33% over the strongest baseline on MiniGrid, lifts returns by up to 17% on the Distracting Control Suite and gains steadily on four Procgen environments, at a measured 1.9× per-step compute overhead.
What guarantees does the paper prove?
Two certificates in one measurable quantity, the embedding distortion κ*: a Dual Near-Optimality Theorem (an approximate isometry is within O(κ*²) of the InfoNCE bound and within O(κ*) of the one-step epistemic-information-gain guarantee) and a Curiosity Certificate bounding the exploration horizon by Õ((κ*)^−D).
How do I cite this paper in APA style?
Gupta, D., Sikarwar, A., Dhangar, S., Jha, C. K., Jha, P., Sharma, A., & Abdullayev, V. H. (2026). Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning. IEEE Access, 14, 162936–162957. https://doi.org/10.1109/access.2026.3743463
Is this paper free to read?
Yes. It is open access under CC BY 4.0: download the published PDF from this page.
Who wrote this paper?
Deepak Gupta, Aditya Sikarwar, Sanidhya Dhangar, Chanakya Kumar Jha, Pallavi Jha, Akhilesh Sharma, Vugar Hacimahmud Abdullayev.
What is the DOI of this paper?
10.1109/access.2026.3743463, permanently at https://doi.org/10.1109/access.2026.3743463.
Bibliographic data from OpenAlex (CC0), last checked 20 November 2026. The PDF is the published version from IEEE Access, open access under CC BY 4.0.