Journal article · 2026 · Open access

Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning

Deepak Gupta, Aditya Sikarwar, Sanidhya Dhangar, Chanakya Kumar Jha, Pallavi Jha, Akhilesh Sharma, Vugar Hacimahmud Abdullayev

IEEE Access, 14, 162936–162957 · Published 14 October 2026 · DOI 10.1109/access.2026.3743463

Download PDFDOI 10.1109/access.2026.3743463

In brief

One loss, the Bisimulation Infomax Loss, ties representation learning and curiosity-driven exploration to the same geometric object, the bisimulation metric, with no hand-tuned trade-off weight. The resulting method, Adaptive Bisimulation Mirror-Descent, cuts steps-to-threshold by up to 33% on MiniGrid and lifts returns by up to 17% on the Distracting Control Suite.

Journal
IEEE Access, vol. 14
Published
14 October 2026
Access
Open access · CC BY 4.0
Review
Peer-reviewed
Authors
7

Abstract

Two long-standing difficulties in reinforcement learning (RL)—learning compact state representations and principled exploration—are usually handled by separate components glued together with a hand-tuned weight. We argue they are not separate: both are manifestations of one geometric object, the bisimulation pseudometric of the underlying Markov decision process (MDP). Our core contribution is a single loss, the Bisimulation Infomax Loss (BIL), removing the conventional λ trade-off coefficient, together with two certificates written in one measurable quantity, the embedding distortion κ*. First, a Dual Near-Optimality Theorem: under stated regularity and model assumptions, an approximate isometry of the bisimulation pseudometric lies simultaneously within O(κ*²) of the InfoNCE bound attained by an exact isometry and within O(κ*) of the corresponding one-step epistemic-information-gain guarantee. Second, a Curiosity Certificate: under explicit reachability and hitting-probability assumptions, the exploration horizon satisfies T_explore = Õ((κ*)^−D), with D the doubling dimension of the bisimulation geometry; the measured slope of −1.96 (R² = 0.96) is consistent with an effective dimension near two. We also bound the error of bootstrapped bisimulation targets and propagate it through every guarantee. We then propose Adaptive Bisimulation Mirror-Descent (ABMD), combining BIL with generalized Gauss–Newton preconditioning, an adaptive bisimulation temperature, and behavior-preserving augmentations. ABMD reduces steps-to-threshold by up to 33% over the strongest baseline on MiniGrid, lifts returns on the Distracting Control Suite by up to 17%, and gains steadily on four Procgen environments, all under one hyperparameter setting. Sensitivity analyses, cross-environment ablations, out-of-distribution stress tests, and compute-normalized comparisons accounting for the 1.9× per-step overhead are reported.

Cite this paper

Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.

APA 7

Gupta, D., Sikarwar, A., Dhangar, S., Jha, C. K., Jha, P., Sharma, A., & Abdullayev, V. H. (2026). Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning. IEEE Access, 14, 162936–162957. https://doi.org/10.1109/access.2026.3743463

MLA 9

Gupta, Deepak, et al. “Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning.” IEEE Access, vol. 14, 2026, pp. 162936–162957. https://doi.org/10.1109/access.2026.3743463.

Chicago

Gupta, Deepak, Aditya Sikarwar, Sanidhya Dhangar, Chanakya Kumar Jha, Pallavi Jha, Akhilesh Sharma, and Vugar Hacimahmud Abdullayev. 2026. “Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning.” IEEE Access 14: 162936–162957. https://doi.org/10.1109/access.2026.3743463.

Harvard

Gupta, D. et al. (2026) ‘Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning’, IEEE Access, 14, pp. 162936–162957. doi:10.1109/access.2026.3743463.

IEEE

D. Gupta et al., “Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning,” IEEE Access, vol. 14, pp. 162936–162957, 2026, doi: 10.1109/access.2026.3743463.

BibTeX

@article{gupta2026bisimulation,
  title     = {{Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning}},
  author    = {Gupta, Deepak and Sikarwar, Aditya and Dhangar, Sanidhya and Jha, Chanakya Kumar and Jha, Pallavi and Sharma, Akhilesh and Abdullayev, Vugar Hacimahmud},
  journal   = {IEEE Access},
  year      = {2026},
  volume    = {14},
  pages     = {162936--162957},
  doi       = {10.1109/access.2026.3743463},
  url       = {https://doi.org/10.1109/access.2026.3743463}
}

Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)

Topic and keywords

Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics

  • representation learning
  • exploration
  • curiosity-driven learning
  • InfoNCE
  • embedding distortion
  • isometries
  • doubling dimension
  • mirror descent

Work with this paper

Most-cited papers in Computer Science

All highly cited Computer Science papers

Questions about this paper

What is Bisimulation Geometry in reinforcement learning?

It is the view, introduced in this paper, that learning a compact state representation and exploring efficiently are two sides of one geometric object: the bisimulation pseudometric of the Markov decision process, which measures how differently two states behave. An encoder that embeds this metric faithfully is good for both at once.

What is the Bisimulation Infomax Loss (BIL)?

A single training loss that anchors both self-supervised representation learning and curiosity-driven exploration to the bisimulation metric, so the usual hand-tuned λ weight between the two objectives is no longer needed.

What is Adaptive Bisimulation Mirror-Descent (ABMD)?

The paper's algorithm. It combines the Bisimulation Infomax Loss with generalized Gauss–Newton preconditioning, an adaptive bisimulation temperature and behavior-preserving augmentations, and runs under one hyperparameter setting.

How well does ABMD perform?

It reduces steps-to-threshold by up to 33% over the strongest baseline on MiniGrid, lifts returns by up to 17% on the Distracting Control Suite and gains steadily on four Procgen environments, at a measured 1.9× per-step compute overhead.

What guarantees does the paper prove?

Two certificates in one measurable quantity, the embedding distortion κ*: a Dual Near-Optimality Theorem (an approximate isometry is within O(κ*²) of the InfoNCE bound and within O(κ*) of the one-step epistemic-information-gain guarantee) and a Curiosity Certificate bounding the exploration horizon by Õ((κ*)^−D).

How do I cite this paper in APA style?

Gupta, D., Sikarwar, A., Dhangar, S., Jha, C. K., Jha, P., Sharma, A., & Abdullayev, V. H. (2026). Bisimulation Geometry: Unifying Representation Learning and Exploration in Reinforcement Learning. IEEE Access, 14, 162936–162957. https://doi.org/10.1109/access.2026.3743463

Is this paper free to read?

Yes. It is open access under CC BY 4.0: download the published PDF from this page.

Who wrote this paper?

Deepak Gupta, Aditya Sikarwar, Sanidhya Dhangar, Chanakya Kumar Jha, Pallavi Jha, Akhilesh Sharma, Vugar Hacimahmud Abdullayev.

What is the DOI of this paper?

10.1109/access.2026.3743463, permanently at https://doi.org/10.1109/access.2026.3743463.

Bibliographic data from OpenAlex (CC0), last checked 20 November 2026. The PDF is the published version from IEEE Access, open access under CC BY 4.0.