* Remove unused `metabase.api.transform`

* Copy the deps graph improvements from my other PR

* New combined X-Rays API namespace

* Fix kondo error

* Squashed commit

* Update config

* Backport the kondo ignore stuff from other PR.

* Metabot API namespace

* Squashed commit

* Update config

* Backport the kondo ignore stuff from other PR.

* Cam's module linter 5000

* Get Kondo passing

* Revert changes to usage of mdb.setup/setup-db!

* Check ns shapes

* Add ns :require shape linter; PR feedback

* Linter fixes 

* Fix add-alias-info matching for nominal field literal refs

* Minor cleanup

* Minimize diff

* A bit more cleanup

* Cumulative aggregations w/ multiple breakouts should reset counts (#2862)

* Update dox

* Do the nesting FIRST before adding alias info.

* Remove unused

* FE for Offset()

* Cleanup stray code

* Prettier

* Kondo

* Test fixes 

* Better error message when no breakout

* Support offset as an expression with no breakouts. Still require an ORDER BY tho.

* Fix most stuff

* Hack city??????

* Revert nest-query changes and disable the offset-no-breakout test for now.

* Appease TypeScript

* Test fix 

* Oracle & MySQL test fixes

* Test fixes 

* Remove maxWorkers for FE unit tests

* Convert `auto-bucket-datetimes` middleware to MLv2

* New Kondo rule time.

* Test & lint fixes 



* Fix typos

* Apply suggestions from code review

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Support parsing {{variable}}s and such in queries

[Fixes #39954]

[Fixes #39953]

* Legacy MBQL documentation

* Do not commit dox

* Subject the legacy MBQL stuff to the same stricter linting rules as MLv2

* PR feedback

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* SQL Lineage: Create the QueryField model

This links Cards (with native queries) to Fields, showing us which
queries use which Fields.

QueryFields are automatically maintained as part of the Card lifecycle

Cards with stale queries (powered by QueryField info) are exposed via the `card?f=stale` API

Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Jerry Huang <jhuang37050@gmail.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>

* [AS+BCM] fix first 3 todos

* [AS+BCM] add issues

* add jdbc proxy issue

* driver/driver-features -> driver/features

* killing more todos and remove `supports?`

* add more issues

* dont let people who dont have snippet perms access snippets

* adding more issues

* replace calls to support? in mongo driver

* fix namespace sorting

* whitespace

* QP enterprise middleware should use `defenterprise`

* Test fix? 

* Remove `:rff` from the Query Processor context map

* Sort namespaces

* Simplify QP entrypoints part 1

* Test fixes 

* Test fixes 

* Make sure defenterprise dispatch is done for every QP run

* B I G  QP context overhaul

* WIP

* Remove qp.context.default

* Test fix 

* PR feedback

* QP context overhaul [WIP] [ci skip]

* QP context overhaul [WIP] [ci skip]

* Fix duplicate preprocesses

* Some test fixes.

* More test fixes [ci skip]

* Source card resolution middleware should be preprocessing middleware, not around [ci skip]

* Test fixes [WIP] [ci skip]

* Remove `test-qp-middleware`

* Remove `qp/compile`

* Fix Kondo warnings/cleanup

* Remove `mt/compile`

* Fix most tests

* Mostly everything working?

* 4 failures, 1 error

* Eliminate QP context `reducedf`

...

* tmp: patched monger for 4.11.1 mongo java driver

* tmp: Update monger utils

Aggregation probably wont work now, but we are not using those from monger anyway. With this change in place I'm able to load needed namespaces and create test-data dataset successfully.

Commit contains lot of condo errors that should be resolved while porting.

* WIP: Monger removed in favor of java driver wrapper

* Update java driver wrapper

* Update srv test

* Update comments

* Use non keywordized run-command for `dbms-version`

* Fix according to e2e tests

* Cleanup

* Separate `java-driver-wrapper` into multiple namespaces

* Fix semantic type inference for serialized json

* Fix options for run-command

* Cleanup json namespace

* Cleanup conversion namespace

* Cleanup operators

* Update kondo in operators ns

* Cleanup connection namespace

* Cleanup mongo namespace

* Cleanup util namespace

* Add todo

* Move session related code to util

* Cleanup database namespace

* Update docstring for conn string generation

* Update docstrings

* Update tests

* Update linter for with macros

* Update modules/drivers/mongo/src/metabase/driver/mongo/connection.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/mongo/src/metabase/driver/mongo/connection.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Use transient in from-document for building a map

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update can-connect to use let form

- Avoid nested `with-mongo-database` call.
- Avoid creating a set used for searching for database name.

* Remove redundant set use from describe-database

* Change from-document keywordize default to false

* Remove log message translation

* Update maybe-add-ssl-context-to-builder! to always return builder

* Indent from-document

* Remove redundant ConvertToDocument extensions

* Use oredered-map in do-find's sort

* Use ex-info in details-normalized

* Add imports and update comment in execute ns

* Update fixme comment

* Pass opts instead of a selection to from-document in do-find

* Avoid unnecessary double dot call

* Update connection test ns according to review remarks

* Make tests parallel

* Docstring update

* Update docstring

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Closes https://github.com/metabase/metabase/issues/36103

### Description

This change introduces a new `with-dbs` macro which is hygienic and supports an arbitrary number of databases.

For now I am leaving the old `with-source-and-dest-dbs` macro, so we can deal with the noise of upgrading the tests in a later PR. I upgrade a single test so that the usage is clear.

* Add kondo for missing :export? on defsetting
* And tons of explicit ignores, including some inadvertent test vars, probably
* Disable linter in tests
* Handle multi settings

* a more refined first crack at adding perm-graph access for db or group id

* new routes: filter data-perm-graph on db or group

add tests

* remove inline defs

* code review responses

* fix typo

* fix the other tests

* update the tests to use the right malli schema

* reuse private vars in tests

* pull graph checker into test util ns and apply it

* Apply column formatting to CSV exports

This PR applies the same formatting logic to CSV exports as it does to pulse bodies (e.g. HTML).

Formerly, there were two related formatting paths:
- `metabase.query-processor.streaming.common/format-value`, which is from a protocol that takes an object and returns a string. It is what was used to export csv data. It applies no actual formatting, only converts objects to strings.
- `metabase.pulse.render.body/get-format`, which builds a formatter from an optional time zone, column metadata, and visualization settings. This formatter takes a string and formats it. It was only used for rendering inline artifacts, such as embedded HTML in emails.

The first function is insufficient to format row data as it is unaware of the column metadata and viz settings. We need to use that data everywhere data is exported in a uniform way.

The solution is to lift `get-format` from `metabase.pulse.render.body` to a more common location (`metabase.pulse.render.common` in this PR step, but needs to be moved out of the pulse code to be a more shared concern) and use it everywhere artifacts are generated.

For csv export, this was achieved as follows in `metabase.query-processor.streaming.csv`:

```clojure
(defmethod qp.si/streaming-results-writer :csv
  [_ ^OutputStream os]
  (let [writer     (BufferedWriter. (OutputStreamWriter. os StandardCharsets/UTF_8))
        formatters (atom {})]
    (reify qp.si/StreamingResultsWriter
      (begin! [_ {{:keys [ordered-cols results_timezone]} :data} viz-settings]
        (swap! formatters (constantly (zipmap
                                        ordered-cols
                                        (map (fn [col]
                                               (p.common/get-format results_timezone col viz-settings))
                                             ordered-cols))))
        (csv/write-csv writer [(map (some-fn :display_name :name) ordered-cols)])
        (.flush writer))

      (write-row! [_ row _row-num cols {:keys [output-order]}]
        (let [[ordered-row
               ordered-cols] (if output-order
                               (let [row-v  (into [] row)
                                     cols-v (into [] cols)]
                                 [(for [i output-order] (row-v i))
                                  (for [i output-order] (cols-v i))])
                               [row cols])]
          (csv/write-csv writer [(map (fn [c r]
                                        (let [formatter (@formatters c)]
                                          (formatter (common/format-value r))))
                                      ordered-cols ordered-row)])
          (.flush writer)))

      (finish! [_ _]
        ;; TODO -- not sure we need to flush both
        (.flush writer)
        (.flush os)
        (.close writer)))))
```

The formatters for each column are build in the `begin!` function and then looked up in each `write-row!`.  The existing `format-value` is used to produce a string then passed into our looked up column formatter.

Note that the new unit tests simulate a pulse and grab the provided temp files as attachments and analyzes those for correctness. This should work in a CI environment so long as the test process has permission to both write attachments to the temp directory and read those attachments back out. Also note that these tests can be slow (but not terribly so).

Primary changes:
- `metabase.email.messages` - fix spelling
- `metabase.pulse.render.body` - move `get-format` out of this ns
- `metabase.pulse.render.common` - move `get-format` into this ns
- `metabase.query-processor.streaming.csv` - new logic to apply pulse renderer formatting to csv exports
- `metabase.pulse.pulse-integration-test` - adding unit tests

One TODO before a final commit of this PR is to move the `get-format` logic out of a pulse ns into something more general. Ultimately, it would be nice if this was a common capability used by both BE and FE.

* Refactoring formatting code for common utility

This PR refactors `metabase.pulse.render.common` to `metabase.formatter` as this is code we want applied to several locations, not just in pulses. It also updates references to these nses and the consistent alias.

A key observation of this formatting code, and reason for the refactor, is that it is "metabase-aware" in that it takes into account metadata columns and visualization settings when building formatters rather than just being a simple generic date or number formatter. This is a common code path that should be used any time we are rendering static assets and could potentially be used as common FE code with future development.

Moves:
- `metabase.pulse.render.common` to `metabase.formatter`
- `metabase.pulse.render.datetime` to `metabase.formatter.datetime`
- `metabase.pulse.render.common-test` to `metabase.formatter-test`
- `metabase.pulse.render.datetime-test` to `metabase.formatter.datetime-test`

* Ordering consistent aliases in kondo config

* Rebase fix to use formatter ns in streaming.csv

* Adding `metabase.formatter` require.

* Updating require alias on new test.

* Adding results_metadata to qp middleware

This adds two middlewares to `metabase.query-processor.middleware.results-metadata`:
- `inject-result-metadata`: Adds `result_metadata` from the context, if present, into the query map during preprocessing.
- `merge-existing-metadata`: A post-processing middleware that merges `results_metadata` (if present) from the query data into the metadata.

This has the effect of preserving existing metadata, which is particularly important for user-curated metadata, such as semantic type overrides.

This may also facilitate addressing the following TODO in the same ns:

```
;; 1. Is there some way we could avoid doing this every single time a Card is ran? Perhaps by passing the current Card
;;    metadata as part of the query context so we can compare for changes
```

* Preserving result_metadata in query processor

This PR uses `qp.util/combine-metadata` to simplify the combination of provided `:result_metadata` with computed metadata. It also updates pulse rendering code to use this new code path and adds and updates tests to vet this logic.

* fmt

* Modifying PR to document and use existing fns

The current qp pipeline will correctly propagate custom metadata if a question is derived from a model or if the model is processed like so:

```clojure
(qp/process-query
  (assoc-in dataset_query [:info :metadata/dataset-metadata] result_metadata))
```

This PR fixes rendering unit tests to clarify this and provides a much better working example in the `dev.render-png` ns.

* Updating test

* Adding Pulse w/Metadata Tests

This pr adds a unit test that constructs a dashboard and simulates a pulse email, then checks the resulting HTML for correctness.

A dependency on the [hickory](https://github.com/clj-commons/hickory) library was added to dev. This allows us to easily parse HTML into data and navigate the parse tree.

Note that it does NOT check the attachments for formatting correctness.

Spelling and import consistency changes were also made.