-
- Downloads
Fixing flaky tests related to `inner-query-name-collisions-test` (#32355) (#35612)
It appears the code fails when run in parallel due to the `metabot-settings/enum-cardinality-threshold` value. To demonstrate, I ran: ```clojure (frequencies (pmap (fn [_] (clojure.test/run-test inner-query-name-collisions-test)) (range 1000))) ``` and got: ``` {{:test 1, :pass 12, :fail 0, :error 0, :type :summary} 414, {:test 1, :pass 11, :fail 1, :error 0, :type :summary} 586} ``` Running with a simple sequential map produced no errors. Wrapping the second test in `tu/with-temporary-setting-values [metabot-settings/enum-cardinality-threshold 10]` reduced the error count dramatically (in the teens for 1000 samples), but still wasn't bulletproof. I'm guessing the two `testing` directives in the test are run in parallel and the setting is not thread safe. By adding the explicit threshold to the test and breaking it out into a standalone test the error count drops dramatically. ``` (frequencies (pmap (fn [_] (clojure.test/run-test inner-query-name-collisions-with-joins-test)) (range 1000))) ;=> {{:test 1, :pass 6, :fail 0, :error 0, :type :summary} 998, {:test 1, :pass 5, :fail 1, :error 0, :type :summary} 2} ``` I'm still not sure why this fails (only twice in 1000 runs), but apparently something about this operation is not thread safe. However, given the additional isolation added by this PR we go from a 60ish percent failure rate down to a 0.2% failure rate locally. Hopefully this is adequate enough to prevent any substantial level of future flakes.
Please register or sign in to comment