Remove dirty question overlay when an error occurred (#49217)

* Add reproduction for #12586

* Do not show the dirty overlay when an error occurred

* Fix tests that rely on dirty overlay

* Add issue tag to test case

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

---------

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>