Preprint · 2021 · Open access
Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.
arXiv (Cornell University) · Published 1 December 2021
In brief
“Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.” is a 2021 preprint by Archana Bura and 4 co-authors, published in arXiv (Cornell University). OpenAlex counts 4 citations as of 22 November 2026. It is open access: the full text is free to read at arXiv (Cornell University).
- Citations
- 4
- Most cited in
- 2022 (1)
- References
- 50
- Authors
- 5
Abstract
We consider the problem of learning an episodic safe control policy that minimizes an objective function, while satisfying necessary safety constraints -- both during learning and deployment. We formulate this safety constrained reinforcement learning (RL) problem using the framework of a finite-horizon Constrained Markov Decision Process (CMDP) with an unknown transition probability function. Here, we model the safety requirements as constraints on the expected cumulative costs that must be satisfied during all episodes of learning. We propose a model-based safe RL algorithm that we call the Optimistic-Pessimistic Safe Reinforcement Learning (OPSRL) algorithm, and show that it achieves an $\tilde{\mathcal{O}}(S^{2}\sqrt{A H^{7}K}/ (\bar{C} - \bar{C}_{b}))$ cumulative regret without violating the safety constraints during learning, where $S$ is the number of states, $A$ is the number of actions, $H$ is the horizon length, $K$ is the number of learning episodes, and $(\bar{C} - \bar{C}_{b})$ is the safety gap, i.e., the difference between the constraint value and the cost of a known safe baseline policy. The scaling as $\tilde{\mathcal{O}}(\sqrt{K})$ is the same as the traditional approach where constraints may be violated during learning, which means that our algorithm suffers no additional regret in spite of providing a safety guarantee. Our key idea is to use an optimistic exploration approach with pessimistic constraint enforcement for learning the policy. This approach simultaneously incentivizes the exploration of unknown states while imposing a penalty for visiting states that are likely to cause violation of safety constraints. We validate our algorithm by evaluating its performance on benchmark problems against conventional approaches.
Cite this paper
Formatted from the catalogue record. Check names, capitals and page numbers against the paper before you submit.
APA 7
Bura, A., HasanzadeZonuzy, A., Kalathil, D. M., Shakkottai, S., & Chamberland, J. (2021). Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees. arXiv (Cornell University). http://arxiv.org/pdf/2112.00885.pdf
MLA 9
Bura, Archana, et al. “Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.” arXiv (Cornell University), 2021. http://arxiv.org/pdf/2112.00885.pdf.
Chicago
Bura, Archana, Aria HasanzadeZonuzy, Dileep M. Kalathil, Srinivas Shakkottai, and Jean‐François Chamberland. 2021. “Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.” arXiv (Cornell University). http://arxiv.org/pdf/2112.00885.pdf.
Harvard
Bura, A. et al. (2021) ‘Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.’, arXiv (Cornell University). Available at: http://arxiv.org/pdf/2112.00885.pdf.
IEEE
A. Bura, A. HasanzadeZonuzy, D. M. Kalathil, S. Shakkottai, and J. Chamberland, “Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.,” arXiv (Cornell University), 2021.
BibTeX
@misc{bura2021safe,
title = {{Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees.}},
author = {Bura, Archana and HasanzadeZonuzy, Aria and Kalathil, Dileep M. and Shakkottai, Srinivas and Chamberland, Jean‐François},
howpublished = {arXiv (Cornell University)},
year = {2021},
url = {http://arxiv.org/pdf/2112.00885.pdf}
}
Download for a reference manager: BibTeX (.bib) · RIS for Zotero, Mendeley and EndNote (.ris)
Citations by year
| Year | Citations | Share of the best year |
|---|---|---|
| 2022 | 1 | |
| 2023 | 1 | |
| 2024 | 1 | |
| 2026 | 1 |
Topic and keywords
Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- safe reinforcement learning
- constrained Markov decision process
- model-based reinforcement learning
- regret bounds
- cumulative regret
- safety gaps
- unknown transition probabilities
Work with this paper
Most-cited papers in Computer Science
- ImageNet classification with deep convolutional neural networksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · Communications of the ACM · 2017 · 114,726 citations
- Long Short-Term MemorySepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 109,097 citations
- A Mathematical Theory of CommunicationClaude E. Shannon · Bell System Technical Journal · 1948 · 91,736 citations
- Deep learningYann LeCun, Yoshua Bengio, Geoffrey E. Hinton · Nature · 2015 · 90,226 citations
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksShaoqing Ren, Kaiming He, Ross Girshick and 1 more · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2016 · 74,143 citations
Questions about this paper
How do I cite this paper in APA style?
Bura, A., HasanzadeZonuzy, A., Kalathil, D. M., Shakkottai, S., & Chamberland, J. (2021). Safe Exploration for Constrained Reinforcement Learning with Provable Guarantees. arXiv (Cornell University). http://arxiv.org/pdf/2112.00885.pdf
How many times has this paper been cited?
4 times, according to OpenAlex on 22 November 2026. Google Scholar and Scopus count different sets of sources, so their numbers differ.
Is this paper free to read?
Yes. It is open access at arXiv (Cornell University): sign in and open the full text from this page.
Who wrote this paper?
Archana Bura, Aria HasanzadeZonuzy, Dileep M. Kalathil, Srinivas Shakkottai, Jean‐François Chamberland.
Bibliographic data from OpenAlex (CC0), last checked 22 November 2026. ResearchGuru does not host this paper; once you are signed in, the links open it at the publisher or the repository that does.