* Adds a middleware that sets 2 dynamic vars

- These vars will be used to insert context for embedding analytics

* migration adding embedding_client and embedding_version to qe and vl tables

* clarify function api

* headers come in -> dynamic vars -> written to db

When the vars are set we insert their values into query execution and
query_log under embedding_client and embedding_version.

* update update migrations

* add predconditions to migrations

* fix tablenames + dataset_test

* Code Review responses

- remove commented out requires
- clarify test for merge-count-maps

I tightened this up by removing the condition here, thinking that CI
would catch it if it was necessary - but I'm getting failures in
development. Oops - didn't realize that this defmethod wasn't actually
applied during CI runs.

We have N+1 detection for simple hydration methods that checks to see
whether the hydration method made any DB queries at all. This makes
sense, because a simple hydration method is called on each instance
individually - if we make database calls *each time*, we're in trouble.

Where this runs into issues is with cached data, where the simple
hydration method is populating the cache - e.g. `:can_read` or
`:can_write` dereferencing `api/*current-user-permissions-set*`.
Currently we specifically exclude those hydration methods from the
check.

But after running into this again for a different simple hydration
method, I think there's a more robust approach:

- first, run the hydration method without counting DB queries

- then, run it again and make sure we don't make any additional DB
queries.

There is a potential downside here: if someone wrote a simple hydration
method with side effects, they'd potentially be very confused (in
development - this doesn't run in prod!) when those side effects
happened twice for a single hydration. But I *think* that's fairly
unlikely.

* Reuse connection for test data loading & JDBC data loading overhaul

* Revert silly change

* Remove unneeded logic for recording which datasets are loaded or not-yet-loaded

* Ok so I guess we do need the code to detect duplicate dataset loading after all

* B I G  improvements

* Appease Kondo

* Test fixes 

* More test fixes 

* Fix dataset-already-loaded?-test

* Fix test

* Fix Presto JDBC and greatly improve test data load speed for it

* Postgres test fix 

* Fix compilation errors

* MySQL fixes 

* Fix syntax error

* MySQL fixes 

* Implement `dataset-already-loaded?` for Athena and cache the set of existing databases

* Fix Redshift

* Fix Presto JDBC

* Vertica test fix 

* Fix Oracle dataset-already-loaded?

* Test fixes and parallelize some API tests.

* Test fixes 

* Oracle test fix? 

* Another test fix 

* Stop putting random spaces in code bodies this is not JavaScript code

* with-temp-vals-in-db should merge in original value of Database settings

* The most test fixes 

* 3 or 4 test fixes 

* Add schemas around stuff to catch errors

* Appease Kondo

* Implement dataset-already-loaded? for vertica

* Implement dataset-already-loaded? for SQLite and Snowflake

* Debug QP improvements from today's eng demo

* More taps

* Fixes 

* Debug QP 2.0

* Update for the ensure-legacy and ensure-pmbql wrappers

There were two problems here:

- wrong args to `mdb/migrate!` (passing the db-type as well as the data
source), and

- in MySQL, the changelog table name is not `:databasechangelog`. Use
`metabase.db.liquibase/changelog-table-name` to get the actual table name.

* Handles card events for CRU (not D) => recent view

* add tests

* avoid some errors in dev

We expect recent-views/post-process to return nil occasionally (for
permissions reasons, or more), so let's filter those out instead of
printing a noisy unactionable warning

* uncomment and fix test

* fix some cypress tests + docs improvements

* update more cypress tests

* more cypress test updates

* adding a card puts it into your recents now

* fix notebook-data-source

* fix notebook-data-source

* fixing more e2e tests

* fix command palette e2e tests

* run prettier

* fix style issue

* update notebook-data-source tests

* update homepage e2e tests

* Next question is in recents, now

* update command palette: skip the first pagedown, since it is a noop

* check the buttons for their respective names

* ensure that the model is there, and no questions are

* Remove unused `metabase.api.transform`

* Copy the deps graph improvements from my other PR

* New combined X-Rays API namespace

* Fix kondo error

* Squashed commit

* Update config

* Backport the kondo ignore stuff from other PR.

* adds watch alias, which reloads BE on file changes

* watch dev, src, test + every path in `:extra-paths`

Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>
Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: Mark Bastian <markbastian@gmail.com>
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* first move to a db api

* mdb.spec into mdb

* mdb.u/isa -> mdb.query/isa

* mdb.u/qualify -> mdb.query/qualify

* last of mbql.u -> mbql.query and mbql.u

* remove last of the mdb.connection/db-type and use from mdb/db-type

* remove the last non-cmd usages of mdb.connection

had some circularity in

models.interface requires mdb/db-type for sql flavored "now"
mdb requires db/setup
which requires db/custom_migrations to ensure they are on cp
custom migration requires json functions in models.interface

so had to do a dynamic require for the db-type

* get db-type from mdb not mdb.env

* invoke `(mdb/app-db)` and not pass a var. lots of things fail

* mdb.setup/migrate! into mdb with potemkin

* remove get-connection. call .getConnection on the app-db

perhaps invites clumsy usage outside of `with-open` if it's a top level
function

* namespace docstrings

* last few and grab db.env/db-file into api ns

* lint cleanup

* addresses PR comments

* Drop report_card.dataset [WIP] [ci skip]

* Drop report_card.dataset

* Some test fixes

* More test fixes

* All OSS tests should be fixed now?

* Remove check for :dataset key from snake-hating-map

* Remove NOCOMMIT stuff

* Remove unused namespaces

* PR feedback

* Migrate dataset attribute in autocomplete suggestions (#39054)

* FE - Migrate `dataset: true` to enum value in Bookmarks (#39056)

* Fix Card["type"] - Bookmark["card_type"] mapping and add an extra assertion

* Fix card-type check out of raw query

* Fix the test

* Fix tests after merge

* Log body when unexpected response code

* Get logging in CI

* Relax is_upload schema for latest mariadb

---------

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

* QP enterprise middleware should use `defenterprise`

* Test fix? 

* Remove `:rff` from the Query Processor context map

* Sort namespaces

* Simplify QP entrypoints part 1

* Test fixes 

* Test fixes 

* Make sure defenterprise dispatch is done for every QP run

* B I G  QP context overhaul

* WIP

* Remove qp.context.default

* Test fix 

* PR feedback

* QP context overhaul [WIP] [ci skip]

* QP context overhaul [WIP] [ci skip]

* Fix duplicate preprocesses

* Some test fixes.

* More test fixes [ci skip]

* Source card resolution middleware should be preprocessing middleware, not around [ci skip]

* Test fixes [WIP] [ci skip]

* Remove `test-qp-middleware`

* Remove `qp/compile`

* Fix Kondo warnings/cleanup

* Remove `mt/compile`

* Fix most tests

* Mostly everything working?

* 4 failures, 1 error

* Eliminate QP context `reducedf`

* 3 failures, 0 errors <3

* All tests are passing <3

* Remove dead debug-qp code

* Fix test failure

* Remove async context

* Remove raisef

* Forget it, just remove context entirely [ci skip]

* Fix Kondo errors

* Fix Kondo warnings

* Fix Kondo warnings

* Fix Kondo warnings

* Remove empty namespace

* Fix some reflection warnings

* Update dox [ci skip]

* Test fix 

* Fix pivot QP

* Misc improvements

* Test fix 

* Improvements and test fixes

* More test fixes 

* Test fixes 

* Address PR feedback

* Test fixes 

* More test fixes and docstring improvements

* Appease Eastwood

* Test fixes 

* More test fixes

* Test fixes 



* Revert test parallelization in metabase.models.params.chain-filter-test

* Update src/metabase/async/util.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* load analytics only when changed using a checksum

- over audit deserialization files

* update docstrings

* fix testing wording

* fix nil analytics-resource-dir

* in tests, when you delete the audit db, set the checksum back to 0

* add a comment

* WIP hashing ia content from inside a jar

* Do the unzip sooner, still hashes the file system

Co-authored-by: John Swanson <john.swanson@metabase.com>

* cleanup java-time -> java-time.api

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>

Sometimes a test passes when run by itself, but fails due to ordering
issues. This is typically caused by another test failing to properly
clean up after itself, leaving the database in a dirty state. For
example, we might change permissions and forget to change them back, so
that another test hits an unexpected permissions failure.

Finding the cause of these has been a source of pain for me, so I wrote
a small snippet to do it for me.

If you have a test that passes when run by itself, but fails when run
along with all the other tests, you can run
`(dev/find-root-test-failure! #'my-ns/my-intermittent-failure-test')`.

It will run *all* tests. After each one, it'll run the test you passed
in. Once that test starts failing, it'll alert you about which test
caused it to fail.

This still throws an exception on trying to directly remove the final
JOIN condition. But if the condition is being removed because a field it
depends on is being removed (eg. a custom column, a breakout, a
previous stage aggregation) then the entire JOIN is also removed, in the
usual cascading fashion.

Fixes #36690.

* Add dev fn to render dashboards with static-viz renderers

Run `(dev.render-png/render-dashboard-to-html 1)` to render the dashboard with id 1 to a handy html file.

This file will render each dashcard in the dashboard 3 ways:
 - as a png, like you would see in Slack or in an email body (if it's a chart)
 - as html/svg, like you'd see in the email body (for tables).. might also be helpful to inspect the svg output from
 the graaljs interpreter
 - 10 row table representing what the .csv attachment for the dashcard would look like

Note: to get this branch working properly, there are a couple lines commented out to pass the linter... it's a work in
progress and will be cleaned up for better use.

Todo:
 - [ ] avoid with-redefs to pass the linter
 - [ ] add a preview endpoint that can be used so that you don't need to run a Clojure repl to get the same result.

* Eliminate with-redefs to allow csv 'render'

* Make a 'preview' ns to power `api/pulse/preview_dashboard/:id`

* Make dev.render-png dashboard preview fns the same as the preview ns

* Dashboard Subscription preview ns now also uses csv attachment code

This change now uses the csv attachment code to include an html table representing what we expect to see from csv exports.

* Add dev fn to render dashboards with static-viz renderers

Run `(dev.render-png/render-dashboard-to-html 1)` to render the dashboard with id 1 to a handy html file.

This file will render each dashcard in the dashboard 3 ways:
 - as a png, like you would see in Slack or in an email body (if it's a chart)
 - as html/svg, like you'd see in the email body (for tables).. might also be helpful to inspect the svg output from
 the graaljs interpreter
 - 10 row table representing what the .csv attachment for the dashcard would look like

Note: to get this branch working properly, there are a couple lines commented out to pass the linter... it's a work in
progress and will be cleaned up for better use.

Todo:
 - [ ] avoid with-redefs to pass the linter
 - [ ] add a preview endpoint that can be used so that you don't need to run a Clojure repl to get the same result.

* Eliminate with-redefs to allow csv 'render'

* Make a 'preview' ns to power `api/pulse/preview_dashboard/:id`

* Make dev.render-png dashboard preview fns the same as the preview ns

* Dashboard Subscription preview ns now also uses csv attachment code

This change now uses the csv attachment code to include an html table representing what we expect to see from csv exports.

* Collect all element styles into a single style tag to use with the server's nonce

This allows the dashboard_preview endpoint to render properly by gathering all of the element styles into a single
style tag, which can then be given the server's nonce value so that CSP doesn't strip the style out.

This is useful for the preview endpoint so that we get an accurate picture of what our tables will look like in
emails, plus it makes the presentation look just a bit more readable.

* dev render_png fns match output used on the endpoint now too.

* Hickory dependency out of dev into regular deps