Sometimes a test passes when run by itself, but fails due to ordering
issues. This is typically caused by another test failing to properly
clean up after itself, leaving the database in a dirty state. For
example, we might change permissions and forget to change them back, so
that another test hits an unexpected permissions failure.

Finding the cause of these has been a source of pain for me, so I wrote
a small snippet to do it for me.

If you have a test that passes when run by itself, but fails when run
along with all the other tests, you can run
`(dev/find-root-test-failure! #'my-ns/my-intermittent-failure-test')`.

It will run *all* tests. After each one, it'll run the test you passed
in. Once that test starts failing, it'll alert you about which test
caused it to fail.

This still throws an exception on trying to directly remove the final
JOIN condition. But if the condition is being removed because a field it
depends on is being removed (eg. a custom column, a breakout, a
previous stage aggregation) then the entire JOIN is also removed, in the
usual cascading fashion.

Fixes #36690.

* Add dev fn to render dashboards with static-viz renderers

Run `(dev.render-png/render-dashboard-to-html 1)` to render the dashboard with id 1 to a handy html file.

This file will render each dashcard in the dashboard 3 ways:
 - as a png, like you would see in Slack or in an email body (if it's a chart)
 - as html/svg, like you'd see in the email body (for tables).. might also be helpful to inspect the svg output from
 the graaljs interpreter
 - 10 row table representing what the .csv attachment for the dashcard would look like

Note: to get this branch working properly, there are a couple lines commented out to pass the linter... it's a work in
progress and will be cleaned up for better use.

Todo:
 - [ ] avoid with-redefs to pass the linter
 - [ ] add a preview endpoint that can be used so that you don't need to run a Clojure repl to get the same result.

* Eliminate with-redefs to allow csv 'render'

* Make a 'preview' ns to power `api/pulse/preview_dashboard/:id`

* Make dev.render-png dashboard preview fns the same as the preview ns

* Dashboard Subscription preview ns now also uses csv attachment code

This change now uses the csv attachment code to include an html table representing what we expect to see from csv exports.

* Add dev fn to render dashboards with static-viz renderers

Run `(dev.render-png/render-dashboard-to-html 1)` to render the dashboard with id 1 to a handy html file.

This file will render each dashcard in the dashboard 3 ways:
 - as a png, like you would see in Slack or in an email body (if it's a chart)
 - as html/svg, like you'd see in the email body (for tables).. might also be helpful to inspect the svg output from
 the graaljs interpreter
 - 10 row table representing what the .csv attachment for the dashcard would look like

Note: to get this branch working properly, there are a couple lines commented out to pass the linter... it's a work in
progress and will be cleaned up for better use.

Todo:
 - [ ] avoid with-redefs to pass the linter
 - [ ] add a preview endpoint that can be used so that you don't need to run a Clojure repl to get the same result.

* Eliminate with-redefs to allow csv 'render'

* Make a 'preview' ns to power `api/pulse/preview_dashboard/:id`

* Make dev.render-png dashboard preview fns the same as the preview ns

* Dashboard Subscription preview ns now also uses csv attachment code

This change now uses the csv attachment code to include an html table representing what we expect to see from csv exports.

* Collect all element styles into a single style tag to use with the server's nonce

This allows the dashboard_preview endpoint to render properly by gathering all of the element styles into a single
style tag, which can then be given the server's nonce value so that CSP doesn't strip the style out.

This is useful for the preview endpoint so that we get an accurate picture of what our tables will look like in
emails, plus it makes the presentation look just a bit more readable.

* dev render_png fns match output used on the endpoint now too.

* Hickory dependency out of dev into regular deps

* Move sample-dataset.edn into test-data.edn

Replace (mt/dataset sample-dataset with (mt/dataset test-data

Replace more references to sample-dataset with test-data

Fix the format of the combined data

Might revert: remove test that doesn't apply, now that there's 1 DB

Make api.database-test pass, though expected could be wrong

* Update unit tests for combined dataset

* Bump Bigquery version from v3_ to v4_

* Bump v3_test-data to v4_test-data

* Remove wrapping with mt/dataset

Alphabetize fk-mappings

* Remove needless whitespace edits

* Fix failed e2e_test by removing reference to sample-dataset

* Fix tests for Mongo, Oracle, Snowflake, and Presto

* Fix unit tests again for Mongo, Presto, and Snowflake

* Will revert: run driver tests on my fork

* Will revert: run driver tests on my draft fork

* Revert "Will revert: run driver tests on my draft fork"

This reverts commit 078c8af1.

* Revert "Will revert: run driver tests on my fork"

This reverts commit e3e2922f.

* Make the postgres driver test pass again by reverting a change

* Apply Case Nelson's patch to fix Athena test

https://github.com/metabase/metabase/pull/36064#issuecomment-1824837705
Props @snoe

* Remove (mt/dataset sample-dataset) from new tests

* Maybe fix Athena driver tests

Revert needless deletions of (mt/dataset test-data
Maybe those deletions caused the
failing Athena driver tests.
But those deletions are out of scope
for this PR either way.

* Remove sample-dataset reference from a test in the master merge

* In card_test.clj, replace sample-dataset with test-data

* Merge in master, resolve conflict in pivot-from-model-test

* Merge in master, resolve conflict in pivot-from-model-test

* Must revert: allow databse creation

To see if it will fix:
https://github.com/metabase/metabase/actions/runs/7120808552/job/19388845720?pr=36064#step:3:441


Athena database creation is disabled: not creating database v2_test_data. Tests will likely fail.

* Revert "Must revert: allow databse creation"

This reverts commit d4c8f129.

* Add back in (mt/dataset where I deleted it

But replace sample-dataset with test-data.
This might be wrong, or maybe it's not needed in
so many places.
But it could show if this is the problem.

* Fix the failed sync test

---------

Co-authored-by: Case Nelson <case@metabase.com>

* Updating honeysql alias in dev ns

This just fixes a broken require to align with the work in #36670.

Fixes #36757

* a more refined first crack at adding perm-graph access for db or group id

* new routes: filter data-perm-graph on db or group

add tests

* remove inline defs

* code review responses

* fix typo

* fix the other tests

* update the tests to use the right malli schema

* reuse private vars in tests

* pull graph checker into test util ns and apply it

* Adding results_metadata to qp middleware

This adds two middlewares to `metabase.query-processor.middleware.results-metadata`:
- `inject-result-metadata`: Adds `result_metadata` from the context, if present, into the query map during preprocessing.
- `merge-existing-metadata`: A post-processing middleware that merges `results_metadata` (if present) from the query data into the metadata.

This has the effect of preserving existing metadata, which is particularly important for user-curated metadata, such as semantic type overrides.

This may also facilitate addressing the following TODO in the same ns:

```
;; 1. Is there some way we could avoid doing this every single time a Card is ran? Perhaps by passing the current Card
;;    metadata as part of the query context so we can compare for changes
```

* Preserving result_metadata in query processor

This PR uses `qp.util/combine-metadata` to simplify the combination of provided `:result_metadata` with computed metadata. It also updates pulse rendering code to use this new code path and adds and updates tests to vet this logic.

* fmt

* Modifying PR to document and use existing fns

The current qp pipeline will correctly propagate custom metadata if a question is derived from a model or if the model is processed like so:

```clojure
(qp/process-query
  (assoc-in dataset_query [:info :metadata/dataset-metadata] result_metadata))
```

This PR fixes rendering unit tests to clarify this and provides a much better working example in the `dev.render-png` ns.

* Updating test

* Adding Pulse w/Metadata Tests

This pr adds a unit test that constructs a dashboard and simulates a pulse email, then checks the resulting HTML for correctness.

A dependency on the [hickory](https://github.com/clj-commons/hickory) library was added to dev. This allows us to easily parse HTML into data and navigate the parse tree.

Note that it does NOT check the attachments for formatting correctness.

Spelling and import consistency changes were also made.

* Adding Percent Metadata Type

This adds the :type/Percentage metadata type to Metabase. It does
not currently do any fingerprinting. It just enables setting of the
type for the FE.

* Making :type/Percentage a semantic type as well as a decimal.

* support Percentage setting in the semantic type editor, infer percentage column formatting default

* Support for static-viz rendering when user metadata is `:type/Percentage`

This PR adds in user-defined metadata in `prep-for-html-rendering` so that `number-formatter` can properly render the column as a percentage. This may actually fix a family of issues as user-defined metadata doesn't appear to be used anywhere in this ns. We still need to consider cases where the formatting is specified by the viz settings, but this one step in the right direction.

* Percent semantic types render as percents in tables

This PR adds logic to properly capture a percent semantic type and renders it as such in a table static viz. Note that the `number-formatter` function in `metabase.pulse.render.common` has logic that renders the percent as a truncated int. IDK if this the desired long term behavior. I'll have to ask product that, but if we just make the change it could have potential impacts on places that expect this behavior. It seems like it would make more sense to be like Excel and allow each column to set its significant digits.

* Revert "support Percentage setting in the semantic type editor, infer percentage column formatting default"

This reverts commit e001e6312e6f0bd5700c1d9eb28c88829d87a70a.

---------

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

Use `(dev.fe-helpers/current-query)` from a CLJS REPL to get the query currently open in the UI.

Thanks to @dpsutton for the help here! I investigated an alternative approach of reusing the machinery that we use in
tests to determine whether the database already exists, but that turned out to rely on a private atom that stored a
list of databases created by the tests. I didn't want to muck around with those internals, and it turned out to be
simpler than expected to make `can-connect?` work (although I did need to catch the exception thrown when we couldn't
connect, which seems a little odd - I would have expected `can-connect?` to catch that and return `false`).

Fixes https://github.com/metabase/metabase/issues/9962

* Fix #33083 (part 2)

* Equality nonsense

* Test fix

* Remove leftover test

* Cljs fix

* Appease Kondo

* Appease Kondo more

* PR feedback

* `index-of-matching-metadata`

* Update some other things to use `index-of-closest-matching-metadata`

* PR feedback

* Don't use metadata provider for ref <=> metadata comparisons

* mark-selected-columns doesn't need metadata provider

* Update some join stuff to match with metadata

* Update more stuff to use `closest-matching-metadata`

* Test fix 

* Test fix

* Take advantage of the QP now using MLv2 metadata providers to simplify test

* Well ok maybe we do need the 3-arity after all =(

* Test fix

* Remove stray NOCOMMIT

* Some cleanup and extra tests

* Hopefully the last few test fixes

* Ok NOW I think I fixed stuff.

* Add a test for binning display name (degrees)

* Fix indentation

* Give up on the Card nonsense for now

* Ok I give up

* Revert changes ...

* QP: use Malli

* Fix schema names

* Fix missing namespaces

* Fix busted stuff

* Simplify the new schema

* Test fix 

* Test fixes 

* MongoDB fixes 

* FAST mu/defn

* Malli registry

* Misc fixes

* Some cleanup

* Misc cleanup and test fixes 

* mu/disable-enforcement

* Test fix

* Fix Cljs version of `disable-enforcement`

* Misc improvements

* Update tests

* Fix Kondo warnings

* Fix Kondo warnings

* Fix busted Malli schema

* PR feedback

* Fix busted schema

* Kondo fixx

* Test fixes 

* Use Malli version of `describe` now that PR was merged

* Work around https://github.com/metosin/malli/issues/924

* Test fixes 

* Improvements

* Test fix

* Squash

* Replace connection-with-timezone usages

* Add deprecation warning.

* Fix docstring

* Database to toucan2

* Table to toucan2

* Field to toucan 2

Before:
```
❯ clj -M:"$ALIASES"
WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.algo.generic.math-functions, being replaced by: #'clojure.algo.generic.math-functions/abs
operator.clj:172 recur arg for primitive local: sum is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: sum
Clojure 1.12.0-alpha2
user=> (do (dev) (start!))
Warning: environ value /Users/dan/.sdkman/candidates/java/current for key :java-home has been overwritten with /Users/dan/.sdkman/candidates/java/17.0.1-zulu/zulu-17.jdk/Contents/Home
2023-05-12 14:26:37,091 INFO metabase.util :: Maximum memory available to JVM: 8.0 GB
2023-05-12 14:26:47,248 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
..instrumented #'metabase.util.malli/with-api-error-message
..instrumented #'metabase.util.honey-sql-2/identifier
..instrumented #'metabase.util.honey-sql-2/normalize-type-info
..instrumented #'metabase.util.honey-sql-2/with-database-type-info
..instrumented #'metabase.util.honey-sql-2/cast
..instrumented #'metabase.util.honey-sql-2/quoted-cast
```

After:

```
❯ clj -M:"$ALIASES"
WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.algo.generic.math-functions, being replaced by: #'clojure.algo.generic.math-functions/abs
operator.clj:172 recur arg for primitive local: sum is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: sum
Clojure 1.12.0-alpha2
user=> (do (dev) (start!))
Warning: environ value /Users/dan/.sdkman/candidates/java/current for key :java-home has been overwritten with /Users/dan/.sdkman/candidates/java/17.0.1-zulu/zulu-17.jdk/Contents/Home
2023-05-12 15:11:21,678 INFO metabase.util :: Maximum memory available to JVM: 8.0 GB
2023-05-12 15:11:30,806 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
util.cljc:346 recur arg for primitive local: cumulative_byte_count is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: cumulative-byte-count
WARNING: infinite? already refers to: #'kixi.stats.math/infinite? in namespace: kixi.stats.core, being replaced by: #'clojure.core/infinite?
2023-05-12 15:11:55,615 INFO driver.impl :: Registered abstract driver :sql  
```

* Tidy database-supports? defmethods

* Fix oracle build

* Fix driver

* Fix typo

* prefer driver/database-supports? instead of driver/supports?

* driver/supports? -> driver/database-supports? in tests

* Fix foreign-key support for sqlite in tests

* supports? -> database-supports?

* `supports?` -> `database-supports?`

* supports? -> database-supports?

* Update supports? docstring

* Fix tests

* Fix test

* Remove unused require

* Add mt/with-everything-store in tests

* fix test

* Fix tests

* Add note in driver changelog

* Don't load metabase.query-processor-test.test-mlv2 until we load QP

* Add schema and tests for `:value`

* Don't skip tests for #29895

* Conversion logic for `:value`

* pMBQL normalization should handle `:effective-type`

* Options should validate `:base-type`, `:effective-type`, etc. if present

* Use `:effective-type` for type checking if present

in query_processor.clj we were requiring

```clojure
(when-not config/is-prod?
  (classloader/require 'metabase.query-processor-test.test-mlv2))
```

And this was blowing up as it was not present in the aliases used for
cypress and the FE crew.

Now user.clj requires it in the same way that it requires
`[metabase.test-runner.assert-exprs]` so we can run tests with
`partial=` and their ilk. And it is required by the test entry namespace
so CI and test runners find it as well.

The `resolve` command in query_processor.clj now works in dev and CI

```clojure
query-processor=> (resolve 'metabase.query-processor-test.test-mlv2/post-processing-middleware)
,#'metabase.query-processor-test.test-mlv2/post-processing-middleware
```
(comma to prevent commit message from eliding as a comment)

so presumably tests that require the middleware tricks from this test
namespace will continue to work.

* Add QP test middleware to validate converted pMBQL queries against MLv2 schema

* Revert 'fix' for #29944

* Fix ups

* Hack

* Fix final test failure

* Remove unused namespace

* Fix Kondo warning

* Skip #29958

* convert db\/insert! to t2

* db/select-one-field -> t2/select-one-fn

* `db/select-field` -> `t2/select-fn-set`

* db/select-ids -> t2/select-pk-set

* db/select-field->field -> t2/select-fn->fn

* db/select-field->id -> t2/select-fn->pk

* `db/select-id->field` -> `t2/select-pk->field`

* add discourage var

* db/select-one -> t2/select-one

* db/select-one-field -> t2/select-one-fn

* add discourage var

* Mind = blown 

* Remove printls

* Simplify things

* Better exampls

* Update dev.clj

Adding a dev-level docker-compose file. Located in /dev and is self-documented.