* Convert `auto-bucket-datetimes` middleware to MLv2

* New Kondo rule time.

* Test & lint fixes 



* Fix typos

* Apply suggestions from code review

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Support parsing {{variable}}s and such in queries

[Fixes #39954]

[Fixes #39953]

* Legacy MBQL documentation

* Do not commit dox

* Subject the legacy MBQL stuff to the same stricter linting rules as MLv2

* PR feedback

* Add new user-local setting `notebook-native-preview-preferences`

* Update user settings on FE

* Use two distinct setting entries

* Update FE types

* More explicit `notebook-native-preview-sidebar-width` definition

* Add selectors

* Initialize QB uiControls based on the BE settings

* Rename QB store entry

`isNativePreviewSidebarOpen` -> `isShowingNotebookNativePreview`

* Rename QB store entry

`nativePreviewSidebarWidth` -> `notebookNativePreviewSidebarWidth`

* Add Redux actions

* Dispatch user settings on top of Redux store

* Fix test title typo

* Add E2E test to make sure sidebar is resizable

* Expand the E2E test to cover persisting settings across sessions

* Be more explicit that we're resizing along x axis

* Store window breakpoint as a variable

* Calculate handle's left position more explicitly

* Split from #40146: only the filter changes

* Don't include format-rows changes here.

* Test fixes 

* Test fix 

* Misc test fixes 

* Test fix 

* Tweak SQL Server fix

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* SQL Lineage: Create the QueryField model

This links Cards (with native queries) to Fields, showing us which
queries use which Fields.

QueryFields are automatically maintained as part of the Card lifecycle

Cards with stale queries (powered by QueryField info) are exposed via the `card?f=stale` API

* first move to a db api

* mdb.spec into mdb

* mdb.u/isa -> mdb.query/isa

* mdb.u/qualify -> mdb.query/qualify

* last of mbql.u -> mbql.query and mbql.u

* remove last of the mdb.connection/db-type and use from mdb/db-type

* remove the last non-cmd usages of mdb.connection

had some circularity in

models.interface requires mdb/db-type for sql flavored "now"
mdb requires db/setup
which requires db/custom_migrations to ensure they are on cp
custom migration requires json functions in models.interface

so had to do a dynamic require for the db-type

* get db-type from mdb not mdb.env

* invoke `(mdb/app-db)` and not pass a var. lots of things fail

* mdb.setup/migrate! into mdb with potemkin

* remove get-connection. call .getConnection on the app-db

perhaps invites clumsy usage outside of `with-open` if it's a top level
function

* namespace docstrings

* last few and grab db.env/db-file into api ns

* lint cleanup

* addresses PR comments

Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Jerry Huang <jhuang37050@gmail.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>

* [AS+BCM] fix first 3 todos

* [AS+BCM] add issues

* add jdbc proxy issue

* driver/driver-features -> driver/features

* killing more todos and remove `supports?`

* add more issues

* dont let people who dont have snippet perms access snippets

* adding more issues

* replace calls to support? in mongo driver

* fix namespace sorting

* whitespace

* Add new user-specific setting `last-used-database-id`

* Update user settings when user selects a database for a native query

* Remember the last used database in the native query editor

* Fix type error

* Set database id when initializing new native question

* Persist the database only for "New > Native query"

* Persist last used database for native models

* Mark `last-used-database-id` setting as potentially `undefined`

* Reduce diff

* Remove unused prop from QB

* Lift dispatch logic up to the `View` component

* Fix type errors

* Make this setting work for all authenticated users

* Add initial E2E tests

* Add initial backend unit test

* Expand the backend unit test to check for user-local only property

* Fix E2E test and make it more explicit

* Add mongo-related E2E test

* Fix typo

* Add permission-related E2E tests

- Adds reproduction for #39053

* Make sure no redundant requests are made

This commit ensures no redundant requests are made when selecting
the same, i.e. the previously selected, database.

* Add E2E tests related to model actions

* Clean up

* Extract persist database logic into an action

* Update the setting name as suggested in the code review

* Make the setting description a bit more explicit

* Add one more permissions-based E2E test

* Store postgres database name in a variable

* creator sentiment cleanup

* change setting to be survey-enabled

* fix test

* change send-creator-sentiment-emails

* Add new email function and send email to creator

* remove debug stuff

* add tests, address comments

* actually add tests

* Update src/metabase/email/messages.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* address comments

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* QP enterprise middleware should use `defenterprise`

* Test fix? 

* Remove `:rff` from the Query Processor context map

* Sort namespaces

* Simplify QP entrypoints part 1

* Test fixes 

* Test fixes 

* Make sure defenterprise dispatch is done for every QP run

* B I G  QP context overhaul

* WIP

* Remove qp.context.default

* Test fix 

* PR feedback

* QP context overhaul [WIP] [ci skip]

* QP context overhaul [WIP] [ci skip]

* Fix duplicate preprocesses

* Some test fixes.

* More test fixes [ci skip]

* Source card resolution middleware should be preprocessing middleware, not around [ci skip]

* Test fixes [WIP] [ci skip]

* Remove `test-qp-middleware`

* Remove `qp/compile`

* Fix Kondo warnings/cleanup

* Remove `mt/compile`

* Fix most tests

* Mostly everything working?

* 4 failures, 1 error

* Eliminate QP context `reducedf`

* 3 failures, 0 errors <3

* All tests are passing <3

* Remove dead debug-qp code

* Fix test failure

* Remove async context

* Remove raisef

* Forget it, just remove context entirely [ci skip]

* Fix Kondo errors

* Fix Kondo warnings

* Fix Kondo warnings

* Fix Kondo warnings

* Remove empty namespace

* Fix some reflection warnings

* Update dox [ci skip]

* Test fix 

* Fix pivot QP

* Misc improvements

* Test fix 

* Improvements and test fixes

* More test fixes 

* Test fixes 

* Address PR feedback

* Test fixes 

* More test fixes and docstring improvements

* Appease Eastwood

* Test fixes 

* More test fixes

* Test fixes 



* Revert test parallelization in metabase.models.params.chain-filter-test

* Update src/metabase/async/util.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Gotta go fast and gotta stay safe.

We have 413 calls to with-redefs, so I'm not going to try get 'em all yet.

* tmp: patched monger for 4.11.1 mongo java driver

* tmp: Update monger utils

Aggregation probably wont work now, but we are not using those from monger anyway. With this change in place I'm able to load needed namespaces and create test-data dataset successfully.

Commit contains lot of condo errors that should be resolved while porting.

* WIP: Monger removed in favor of java driver wrapper

* Update java driver wrapper

* Update srv test

* Update comments

* Use non keywordized run-command for `dbms-version`

* Fix according to e2e tests

* Cleanup

* Separate `java-driver-wrapper` into multiple namespaces

* Fix semantic type inference for serialized json

* Fix options for run-command

* Cleanup json namespace

* Cleanup conversion namespace

* Cleanup operators

* Update kondo in operators ns

* Cleanup connection namespace

* Cleanup mongo namespace

* Cleanup util namespace

* Add todo

* Move session related code to util

* Cleanup database namespace

* Update docstring for conn string generation

* Update docstrings

* Update tests

* Update linter for with macros

* Update modules/drivers/mongo/src/metabase/driver/mongo/connection.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/mongo/src/metabase/driver/mongo/connection.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Use transient in from-document for building a map

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update can-connect to use let form

- Avoid nested `with-mongo-database` call.
- Avoid creating a set used for searching for database name.

* Remove redundant set use from describe-database

* Change from-document keywordize default to false

* Remove log message translation

* Update maybe-add-ssl-context-to-builder! to always return builder

* Indent from-document

* Remove redundant ConvertToDocument extensions

* Use oredered-map in do-find's sort

* Use ex-info in details-normalized

* Add imports and update comment in execute ns

* Update fixme comment

* Pass opts instead of a selection to from-document in do-find

* Avoid unnecessary double dot call

* Update connection test ns according to review remarks

* Make tests parallel

* Docstring update

* Update docstring

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Closes https://github.com/metabase/metabase/issues/36103

### Description

This change introduces a new `with-dbs` macro which is hygienic and supports an arbitrary number of databases.

For now I am leaving the old `with-source-and-dest-dbs` macro, so we can deal with the noise of upgrading the tests in a later PR. I upgrade a single test so that the usage is clear.

* add user-provisioning code

* sort namespace

* fix linter

* fix linter

* swap to ee

* fix logic

* add tests

* address comments

* fix linter

* update tests, address comments

* Add kondo for missing :export? on defsetting
* And tons of explicit ignores, including some inadvertent test vars, probably
* Disable linter in tests
* Handle multi settings

Ignore another clj-kondo library config missed by https://github.com/metabase/metabase/pull/36889

This was causing dirty changes to be left in the repo after running `bin/kondo.sh`. The script should be clean after this change, I think.

* Kondo: ignore all libraries config

* Add README

* Update .gitignore

* a more refined first crack at adding perm-graph access for db or group id

* new routes: filter data-perm-graph on db or group

add tests

* remove inline defs

* code review responses

* fix typo

* fix the other tests

* update the tests to use the right malli schema

* reuse private vars in tests

* pull graph checker into test util ns and apply it

* Apply column formatting to CSV exports

This PR applies the same formatting logic to CSV exports as it does to pulse bodies (e.g. HTML).

Formerly, there were two related formatting paths:
- `metabase.query-processor.streaming.common/format-value`, which is from a protocol that takes an object and returns a string. It is what was used to export csv data. It applies no actual formatting, only converts objects to strings.
- `metabase.pulse.render.body/get-format`, which builds a formatter from an optional time zone, column metadata, and visualization settings. This formatter takes a string and formats it. It was only used for rendering inline artifacts, such as embedded HTML in emails.

The first function is insufficient to format row data as it is unaware of the column metadata and viz settings. We need to use that data everywhere data is exported in a uniform way.

The solution is to lift `get-format` from `metabase.pulse.render.body` to a more common location (`metabase.pulse.render.common` in this PR step, but needs to be moved out of the pulse code to be a more shared concern) and use it everywhere artifacts are generated.

For csv export, this was achieved as follows in `metabase.query-processor.streaming.csv`:

```clojure
(defmethod qp.si/streaming-results-writer :csv
  [_ ^OutputStream os]
  (let [writer     (BufferedWriter. (OutputStreamWriter. os StandardCharsets/UTF_8))
        formatters (atom {})]
    (reify qp.si/StreamingResultsWriter
      (begin! [_ {{:keys [ordered-cols results_timezone]} :data} viz-settings]
        (swap! formatters (constantly (zipmap
                                        ordered-cols
                                        (map (fn [col]
                                               (p.common/get-format results_timezone col viz-settings))
                                             ordered-cols))))
        (csv/write-csv writer [(map (some-fn :display_name :name) ordered-cols)])
        (.flush writer))

      (write-row! [_ row _row-num cols {:keys [output-order]}]
        (let [[ordered-row
               ordered-cols] (if output-order
                               (let [row-v  (into [] row)
                                     cols-v (into [] cols)]
                                 [(for [i output-order] (row-v i))
                                  (for [i output-order] (cols-v i))])
                               [row cols])]
          (csv/write-csv writer [(map (fn [c r]
                                        (let [formatter (@formatters c)]
                                          (formatter (common/format-value r))))
                                      ordered-cols ordered-row)])
          (.flush writer)))

      (finish! [_ _]
        ;; TODO -- not sure we need to flush both
        (.flush writer)
        (.flush os)
        (.close writer)))))
```

The formatters for each column are build in the `begin!` function and then looked up in each `write-row!`.  The existing `format-value` is used to produce a string then passed into our looked up column formatter.

Note that the new unit tests simulate a pulse and grab the provided temp files as attachments and analyzes those for correctness. This should work in a CI environment so long as the test process has permission to both write attachments to the temp directory and read those attachments back out. Also note that these tests can be slow (but not terribly so).

Primary changes:
- `metabase.email.messages` - fix spelling
- `metabase.pulse.render.body` - move `get-format` out of this ns
- `metabase.pulse.render.common` - move `get-format` into this ns
- `metabase.query-processor.streaming.csv` - new logic to apply pulse renderer formatting to csv exports
- `metabase.pulse.pulse-integration-test` - adding unit tests

One TODO before a final commit of this PR is to move the `get-format` logic out of a pulse ns into something more general. Ultimately, it would be nice if this was a common capability used by both BE and FE.

* Refactoring formatting code for common utility

This PR refactors `metabase.pulse.render.common` to `metabase.formatter` as this is code we want applied to several locations, not just in pulses. It also updates references to these nses and the consistent alias.

A key observation of this formatting code, and reason for the refactor, is that it is "metabase-aware" in that it takes into account metadata columns and visualization settings when building formatters rather than just being a simple generic date or number formatter. This is a common code path that should be used any time we are rendering static assets and could potentially be used as common FE code with future development.

Moves:
- `metabase.pulse.render.common` to `metabase.formatter`
- `metabase.pulse.render.datetime` to `metabase.formatter.datetime`
- `metabase.pulse.render.common-test` to `metabase.formatter-test`
- `metabase.pulse.render.datetime-test` to `metabase.formatter.datetime-test`

* Ordering consistent aliases in kondo config

* Rebase fix to use formatter ns in streaming.csv

* Adding `metabase.formatter` require.

* Updating require alias on new test.

* Adding results_metadata to qp middleware

This adds two middlewares to `metabase.query-processor.middleware.results-metadata`:
- `inject-result-metadata`: Adds `result_metadata` from the context, if present, into the query map during preprocessing.
- `merge-existing-metadata`: A post-processing middleware that merges `results_metadata` (if present) from the query data into the metadata.

This has the effect of preserving existing metadata, which is particularly important for user-curated metadata, such as semantic type overrides.

This may also facilitate addressing the following TODO in the same ns:

```
;; 1. Is there some way we could avoid doing this every single time a Card is ran? Perhaps by passing the current Card
;;    metadata as part of the query context so we can compare for changes
```

* Preserving result_metadata in query processor

This PR uses `qp.util/combine-metadata` to simplify the combination of provided `:result_metadata` with computed metadata. It also updates pulse rendering code to use this new code path and adds and updates tests to vet this logic.

* fmt

* Modifying PR to document and use existing fns

The current qp pipeline will correctly propagate custom metadata if a question is derived from a model or if the model is processed like so:

```clojure
(qp/process-query
  (assoc-in dataset_query [:info :metadata/dataset-metadata] result_metadata))
```

This PR fixes rendering unit tests to clarify this and provides a much better working example in the `dev.render-png` ns.

* Updating test

* Adding Pulse w/Metadata Tests

This pr adds a unit test that constructs a dashboard and simulates a pulse email, then checks the resulting HTML for correctness.

A dependency on the [hickory](https://github.com/clj-commons/hickory) library was added to dev. This allows us to easily parse HTML into data and navigate the parse tree.

Note that it does NOT check the attachments for formatting correctness.

Spelling and import consistency changes were also made.