* `POST /api.ee/serialization/serialize/data-model` endpoint

* Sort namespaces

* Don't use `tru` for endpoint params validation

The YAML file names have an optional `:label` portion that becomes the
latter part of the filename. Reconstructed paths (eg. from
`serdes-dependencies`) don't have those labels.

This change makes the YAML ingestion code able to find a file with a
human-readable label even if the request didn't include it.
No ambiguity results because the file names are always based on the
unique serdes `:id`, usually an `entity_id`.

- Limit the scanning of directories and files to those named after
  models; don't try to ingest `.git`, `README.md`, etc.
- `table_id` and `collection_id` are optional on Cards
- Deserialization was not resolving some deeply nested `:field`s inside MBQL
  queries.

Pass `--collections 123,456,789` to dump only these collections and
their transitive `serdes-descendants`.

This is a simple expedient: convert our entities to a `(sorted-map)` before
passing them to the YAML writer!
This gives a consistent, platform-agnostic order between dumps.
(It might have already been fine, since it's based on Clojure's `hash`?)

* Add `serdes-descendants` for "containment" to serialize a subtree

This allows naming eg. a Collection and will recursively serialize: all
dashboards, cards and dashcards it contains directly, plus all child
collections and everything they contain.

Currently this words on Collection, Dashboard, DashboardCard, and Card.

* lint

* Switch to plural `extract-subtrees` with a list of `targets`

* [Toucan 2 prep] Don't invoke Toucan models as functions

* Some fixes

* Test fixes

* Test fix

* [Toucan 2 prep] Don't call `type` or `class` on Toucan models

* Test fixes

* More test fixes 

* [Toucan 2 prep] Don't invoke Toucan models as functions

* Some fixes

* Test fixes

FieldValues can be rebuilt by the sync process, but they are portable
and including them aids in the plans for content moderation via git.

* Remove the `Dependency` model code

* Appease the namespace linter again.

* Serdes v2: Add `--v2 true` flag to the `dump` and `load` commands

This is the last piece for end-to-end serialization and deserialization.
The results should be similar to v1, though probably not identical.

`PulseChannelRecipients` are a `:recipients` field on `PulseChannels`.

This requires some careful handling when inserting or updating a
`PulseChannel` to upsert the recipients.

Cam suggested most of these changes in another PR, and I'm tackling them
here.

* Actions multimethod refactor.

* Fix function called with wrong number of args

* Test fixes

* Bulk create action (second pass)

* Appease clj-kondo

* Fix bulk insert happy path

* Implement/fix everything

* Linter and test fixes 



* Fix bad metadata (thanks Eastwood!)

* Add some comments about why we're doing what we're doing

* Serdes v2 for Native Query Snippets (#23961)

Also standardizes date/time output to `ZonedDateTime`, rather than whatever the JDBC happens to return.

Co-authored-by: Braden Shepherdson <Braden.Shepherdson@gmail.com>

Also standardizes date/time output to `ZonedDateTime`, rather than whatever the JDBC happens to return.

Serialization of Databases, Tables, Fields

This brought a few core changes:
- Add `serdes-entity-id` to abstract the field used for the ID
- Pass the options to `extract-one` so it can eg. do encryption things.
- Handle dates in YAML storage and ingestion
- `:serdes/meta` now holds the entire hierarchy, not just the leaf model+ID pair.

There's an open problem here about the right way to handle secrets like
a database's password. Do we assume both sides have the same
`MB_ENCRYPTION_SECRET_KEY`? Provide a serdes-specific password the user
just made up, and every secret gets decrypted with the source key, encrypted with
the serdes key, stored, decrypted with the serdes key, and encrypted with
the destination key?

Write `storage.yaml` and `ingest.yaml` to serialize all the way to YAML files and back.

Lots of generative testing to check it's isomorphic.

This supports serialization of only Collections and Settings so far, but
it demonstrates the design of the new serialization system.

`metabase.models.serialization.base` defines the multimethods, which
are to be implemented by all the exported models eventually.
The actual serialization code that drives the larger process is in
`metabase_enterprise.serialization.v2.extract` and `.merge`, since
serialization is an enterprise feature.

The design calls for two matching phases on each side:
- Serialization is extract + store;
- Deserialization is ingest + load.

Extract and load deal with vanilla Clojure maps with a `serdes/meta` key
giving common details; they deliberately know nothing about files.

Store and ingest deal with the storage medium and the process of
listing and reading a stored export.

Laziness is retained: the `load` process ingests full details on demand,
so only the metadata of the importing database needs to fit in memory.

Background:

We had two lint configs: lint-config.edn which was enfored by Github Actions, and
.clj-kondo/config.edn, the default location used by editors.


Problem:

Patchy enforcement of the (larger) config in .clj-kondo/config.edn since not
everyone uses an editor.

Some new lint, like canonical namespace aliases (#21738, #19930) is not enforced
automatically and has begun to bit-rot.

Solution:

Combine both configs, fix any outstanding lint, and update our tooling to use
the unified config.

Caveats:

Anyone who has their local editor configured to use lint-config.edn will have
their linting broken by this change.

* Make namespace aliasing consistent everywhere; enforce with clj-kondo

See the table of aliases in .clj-kondo/config.edn

Notable patterns:
- `[metabase.api.foo :as api.foo]`
- `[metabase.models.foo :as foo]`
- `[metabase.query-processor.foo :as qp.foo]`
- `[metabase.server.middleware.foo :as mw.foo]`
- `[metabase.util.foo :as u.foo]`
- `[clj-http.client :as http]` and `[metabase.http-client :as client]`

Fixes #19930.

* Clear out existing DashboardCards when running load with --mode update

* Namespace cleanup

* Fix circular references

* Another test fix

* Settings cache needs to be per-app-DB

* (Experimental -- new application DB dynamic var)

* Create truly rebindable `metabase.db.connection/*application-db*`

* call-on-change-fn -> call-on-change

* Revert unneeded commit

* Namespace cleanup

* Add missing docstrings

* Appease linters again

* Fix handler stats logging middleware

* PR feedback

* PR feedback: slight optimization

* Remove NOCOMMIT

* PR feedback

* Clean namespace

* Create truly rebindable `metabase.db.connection/*application-db*`

* call-on-change-fn -> call-on-change

* Namespace cleanup

* Add missing docstrings

* Appease linters again

* Fix handler stats logging middleware

* PR feedback

* PR feedback: slight optimization

* Rename setting/get and setting/all; GeoJSON via env var tests

* Fix typo (thanks @noahmoss)

* Support Database-local Settings in metabase.models.setting itself

* Rework Settings code so it can handle possibly-already-deserialized values

* Database-local Settings

* Remove empty part of docstring

* Appease linters

* Update dox again

* Use text.type for the new column

* Test fixes 

* Test fix 

* Test fix 

* All negative integer values when setting :integer Setting with a string

* Rename setting/get and setting/all; GeoJSON via env var tests

* Fix typo (thanks @noahmoss)

* Fix serialization dump error when there are no collections

Update `select-collections` to correctly handle the case where there are no collections

Adding new test that removes all collections, then ensures that dump works with no errors

* Fix serialization load error into empty/blank target DB

Rethrowing exception in cmd when overall load fails

Add test to ensure that a dump containing a user can be loaded into a blank target app DB successfully

Adding a few missing bindings to the `with-temp-empty-app-db` code to set the connection vars under metabase.db.connection

In upsert, add hooks for a :pre-insert-fn and :post-insert-fn to be invoked for new entity instances created by the upsert process, since whether an entity will be an insert or update isn't necessarily known by the load process (only by upsert once identity-condition is checked for each)

In load, setting the pre and post insert functions for a User instance to initialize :password with a random value, and to generate and send a password reset email to the newly inserted user's email, respectively

NOTE: this post insert fn (to send a password reset email for newly inserted users) is NOT hooked up for User as of x.41 release, since it is considered a bugfix, but this can be enabled in a future release)

Adding new defs for the magic permission group names to make those easier to override from tests that might need to (such as the one added for this commit)

* Fix serialization objects being incorrectly updated on skip

Remove `maybe-fixup-card-template-ids!`, which was forcing mode :update, since the existing retry logic should cover what it was trying to do

Update Card and Metric models to delete any Dependency instances for which they are the `:model_id` (to make serialization tests after other tests created temporary Card/Dependency pairs)

Adding missing assertions for Dependency serialization

* Serialization: Fix reload entity logic

Now that the `:mode` is always respected instead of being ignored sometimes, we need to update our "second pass" reload functions to always make the mode `:update` on the second pass, or else that entity would just be skipped, which is bad

Also updated the test to use `:mode` `:skip` from the beginning to be more stringent

* Enable clj-kondo warnings

* (when (seq ...) ...) instead of (when-not (empty? ...) ...)

Update visualization_settings.cljc to handle the table.columns submap

Update dump and load to handle :visualization_settings on a Card (basically identically to how they are handled for a DashboardCard)

Updating test to have a card include :visualization_settings and ensure they fields are properly handled

Updating visualization_settings test case to incorporate table.columns as well as :show_mini_bar (under ::column_settings)

Serialization refactoring and fixes

*********************************************************
* Support loading linked cards in different collections *
*********************************************************

Architectural changes:
Lay foundation for more generic retry mechanism
Establish a keyword to represent a failed name->id lookup that can happen during load, and map that to information about which part of the card resolution failed (for logging purposes)
Similar to dashboards, add a separate fn for performing a load of cards
Invoking the load-cards fn from the load multimethod implementation when some cards failed name resolution

Test enhancements:
Add new multimethod to perform more specific assertions on the loaded entities
For cards - check that the native SQL of the loaded card matches that of the original (pre-dump) instance
For dashboards - check that the linked card for each dashboard card series matches

Bug fixes:
Apply the mbql-id->fully-qualified-name fn on serialize to filter, breakout, and aggregation clauses
Updating "My Test" card to include filter clause so that fix is exercised
Instead of upserting users with a "changeme" password, simply do not attempt to change the password
Adding :engine to the identity condition of Database (so unrelated test-data instances don't get trashed on load)

Cleanup:
Remove unused specs namespace

****************************************
* Support dashboard card click actions *
****************************************

Architectural changes:
Add visualization settings shared model namespace with utility functions

Adding tests for viz settings

Changes to serialization:
Update visualization_settings in dump to convert column_settings keys from field ID to fully qualified name, and update the click action target with the fully qualified name
Also converting targetId in top level click action to fully qualified name

Doing the reverse in load, using the same utility code

Changes to serialization tests:
Updating the existing dashcard in the "world" roundtrip test to include the click action

Adding assertion for the click action in the assert-loaded-entity implementation for Dashboard

******************************************************************
* Stop nesting cards and check for unresolved query dependencies *
******************************************************************

In names.clj, stop qualifying a card's name by its dataset query's source card ID (this is what caused the "nested" directory structure)

In load.clj, start checking that either the source query [:dataset_query :query] is resolved properly as a card ID or table ID, and if neither, then propagate an ::unresolved-names entry up into the card (causing it to be reloaded on second pass)

Fix logging of unresolved name info

Test changes:
Add two new cards to the "world", one that lives in the root collection but depends on a card within a collection for its query, and one opposite (i.e. lives in a collection but depends on a root card for its query)

Adding new check that collection names match after loading

************************
* Logging improvements *
************************

Include causes when logging errors to aid with diagnosis

Add BEGIN and END messages for both dump and load commands

Add dynamic var to suppress printing of name lookup error on first pass load

*************************************************
* Handle collection namespaces and retry pulses *
*************************************************

Architectural changes:
Considering a collection namespace to be part of its fully qualified name (putting in a subdirectory called :namespace when writing to disk)
Accounting for that change when dumping and loading, and upsert (i.e. namespace is part of unique columns)
Bumping serialization format version to 2 because of this

Changes to load:
Add load-pulses fn that works similarly to others, to facilitate reloading
Add similar logic as other entities to reload a pulse if the :card_id is not found

Model changes:
Add hack fn to compare namespace values for the "namespace does not change" assertion with appropriate comment

Test changes:
Added test for pulse with reload

***********************************************
* Handle dimension entry within template-tags *
***********************************************

Add :field to the recognized clauses under mbql-entity-reference? (since that is how field refs are now represented in MBQL)

Add temp-field helper fn in test_util.clj to reduce some of the "world" verbosity

Adding new test card to world that uses template-tags with a field reference under dimension

************************
* Handle textbox cards *
************************

Add support for passing textbox (virtual) cards through visualization_settings.cljc

Updating load test to add a textbox type card to the dashboard and ensuring it is persisted

***************************************
* Handle dimension parameter mappings *
***************************************

In visualization_settings.cljc:
Adding handling for parameterMapping
Adding new roundtrip test for parameter mappings
Some refactoring to split up larger fns

In serialize.clj, convert all parts of the map that contain the dimension key to use fully qualified names for the respective fields

In load.clj, do the opposite (convert back to new IDs)

Updating test to add parameter mapping to an existing click action, and adding new assertions for that to ensure it is preserved

Fixing serialize test now that the "ids" can themselves be maps (i.e within dimension vector)

In visualization_settings_test.cljc
Adding new roundtrip test for problem dashboard

***********************************
* Handle field literals correctly *
***********************************

On load, only attempt to translate field name to ID if it's actually a fully qualified field name

Adding new fn to names.clj to provide that logic

Adding new test card that uses such a field reference

**************************************
* Accept only-db-ids option for dump *
**************************************

Accept options map for the dump command.  Currently, the only supported option is `:only-db-ids`, which is expected to be a set of DB IDs to dump.  If set, then only these databases (and their respective tables, fields, and segments) will be dumped to the dump-dir.  If not set, then the previous behavior takes effect (all DBs).

Update the load_test.clj to set this to be only the new temporary DB to avoid tramping on existing `sample-dataset` and other instances

*****************************************************
* Don't include personal collections of other users *
*****************************************************

Use the `select-collections` fn to filter only collections that are public, or owned bythe given user (by way of email), plus all collections with those as an ancestor

Updating test to assert a non-owned personal collection is not persisted (including nested), but all nested personal collections are

Adding new wrapped macro to also bind collection/*allow-deleting-personal-collections* true around the with-temp* call so that personal collections can legitimately be deleted (since they're temporary); the name of this macro was chosen to preserve the same amount of spacing within `with-world`

*********************************
* Support native query snippets *
*********************************

Adding support for NativeQuerySnippet:

- in names
- in dump (putting into subdirectory of collection, as cards)
- in upsert (identity condition of :name :collection_id)
- in load (no retry fn necessary since they should not have any dependencies themselves)

Adding a snippet and a card referencing it to roundtrip test (including a deeply nested snippet)

Fixing up handling of snippet namespaced collections and adding relevant name tests

*********************************
* Handle source-table for joins *
*********************************

Adding `mbql-fully-qualified-names->ids*` for recursive calls, which does not normalize its args, keeping mbql-fully-qualified-names->ids` as a wrapper to that

Adding clause to `mbql-fully-qualified-names->ids*` to resolve `:source-table` when it's a fully qualified table name

Adding util fn to names.clj to support that scenario

Updating "My Card" test card to include a join

Skipping two more drivers from `dump-load-entities-test` - SQLite and Spark because they don't support FKs

***************************
* Cleanup and refactoring *
***************************

Responding to PR feedback

Updating upsert_test.clj to check that the identity-condition keys actually exist for the models

Removing legacy MBQL syntax in a couple places

Adding ordering by `:id` to serialization load command for predictability

Lots of fixes and refactorings in visualization_settings.cljc:
 - added lots of specs and fdefs
 - added test fixture that instruments all the speced fns and required it from both visualization_settings.cljc and load_test.clj
 - renamed a bunch of fns for more consistency
 - added and fixed some docstrings
 - fix implementation of `parse-json-string` for cljs

*************************************************
* Preserve temporal-unit in source-field clause *
*************************************************

Update serialize and match macros to keep the rest of the match, when modifying the source field ID

Adding a bunch of new tables in order to perform a "bucketing on implicit join" type query

Updating test_util.clj to initialize the query processor with all the new temp tables so that queries actually work

**************************
* Misc cleanup and fixes *
**************************

Fixing the implementation of `fully-qualified-name->context` to work properly when the entity name itself is also a model name

***************************
* Handle joining to cards *
***************************

Recognize :source-table pointing to a fully qualified card name in `mbql-fully-qualified-names->ids*`, and adding the unresolved name capture to that lookup

Adding new util fns to capture the unresolved named entries from anywhere within the MBQL query tree: `paths-to-key-in` and `gather-all-unresolved-names`

Updating `resolve-card-dataset-query` to call this new `gather-all-unresolved-names` to pull all unresolved names from the MBQL query tree into card itself

Renaming `fully-qualified-name->id-rec` to `resolve-source-query` for clarity, and removing now unneeded clause

To the test case, adding a card that joins to another card

Make the load multimethod return a function, which serves as a "reload function", to be called after the first entire pass is complete (via cmd)

Updating various places to combine these functions together and invoke at the top level from cmd.clj when load finishes

Updating dashboard load implementation to check for presence of cards referred to by its dashcards, and if missing, reload those dashboards(s) on a second pass

Splitting out a separate load-dashboards fn that contains the bulk of the body from the previous multimethod implementation (which now delegates to it) to support easier reload functionality

Background

The original version of MBQL (known as the "Structured Query" language at the time, which was a little confusing) just referred to all fields by integer ID. e.g.

{:filter ["STARTS_WITH" 1 "abc"]}
That made clauses like this ambiguous:

{:filter ["=" 1 2]} ; is 2 a Field, or the number 2?
To clear up ambiguity and to let you filter Fields against Fields, we added the :field-id clause to make it explicit that you were referring to a Field, not an Integer:

{:filter [:= [:field-id 1] [:field-id 2]]} ; Field 1 == Field 2
We soon added a couple of new types of Field clauses, :fk-> and :datetime-field, both of which wrap the original :field-id:

;; refer to Field 2 from a different Table, use Field 1 from the current Table to perform the join
[:fk-> [:field-id 1] [:field-id 2]]

;; bucket Field 3 by month
[:datetime-field [:field 3] :month]
So far things were still pretty reasonable, the worst you'd have to do is something like

[:datetime-field [:fk-> [:field-id 1] [:field-id 2]] :month]
Things started to get out-of-hand IMO when we continued to add more Field clause types that just wrapped everything else. We added :binning-strategy:

[:binning-strategy <field> :num-buckets 10]
then :field-literal, to refer to a Field from a nested native source query:

[:field-literal "my_field" :type/Text]
then we added :joined-field to specify the Table you're explicitly joining against that is the source of a Field:

[:joined-field "my_join" [:field-id 4]]
This ends up getting really hairy when you combine things. This is an actual real clause you can use right now:

[:binning-strategy
 [:datetime-field
  [:joined-field "my_join"
   [:field-literal "my_field" :type/DateTimeWithLocalTZ]]
  :month]
 :num-bins 10]
And even with all of those clauses, we still can't pass around arbitrary extra information in a way that would make it easy to implement performance optimizations. If we ever try to add another new clause, the whole house of cards is going to come crashing down.

The proposal
Combine all of the Field clauses into a single new :field clauses with the schema

[:field id-or-name options-map]
Here are some before & after examples:

[:field-id 1] 
=> 
[:field 1 nil]

[:field-literal "my_field" :type/Text] 
=> 
[:field "my_field" {:base-type :type/Text}]

[:datetime-field [:field 1] :month] 
=> 
[:field 1 {:temporal-unit :month}]

[:binning-strategy [:field 1] :num-bins 10] 
=>
[:field 1 {:binning {:strategy :num-bins, :num-bins 10}}]

[:joined-field "my_join" [:field 1]] 
=> 
[:field 1 {:join-alias "my_join}]

[:fk-> [:field 1] [:field 2]] 
=> 
[:field 2 {:source-field 1}]

[:binning-strategy
 [:datetime-field
  [:joined-field "my_join"
   [:field-literal "my_field" :type/DateTimeWithLocalTZ]]
  :month]
 :num-bins 10]
=>
[:field "my_field" {:base-type :type/DateTimeWithLocalTZ, :join-alias "my_join", :binning {:strategy :num-bins, :num-bins 10}}]

* Proposal: no more prefix namespaces

* Fix a few missing files