These fields hold JSON, which can contain field IDs in a few places.

Particularly nasty is the `:column_settings` subfield, which is a map
whose *keys* are JSON strings with field IDs.

This PR handles the other JSON-encoded MBQL snippets I was able to find.

These snippets contain `Table` and `Field` IDs, and so are not portable. These
fields are expanded during serialization and the IDs replaced with portable
references, then converted back in deserialization.

Note that the referenced field must already be loaded before it has a valid ID.
`serdes-dependencies` defines this order, therefore each entity depends on those
tables and fields referenced in its MBQL fragments.

The complete set of fields I found to convert:

- `Metric.definition`
- `Segment.definition`
- `DashboardCard.parameter_mappings`
- `Card.parameter_mappings`

These snippets contain raw `Database`, `Table`, and `Field` IDs, and so are not portable. This PR proves out the concept for `Card.dataset_query`, paving the way for the others in a follow-up PR.

* Remove the `Dependency` model code

* Appease the namespace linter again.

* smarter hashing for advanced fieldvalues

* saving some db calls

* updates some docs and remove an unecessary macro

* Add a check to PUT /user/:id to disallow name edits if an SSO user

* Clean up After SAML SSO tests

The `:sso_source` key is set for the Rasta user in some SAML tests, but is expeted to be nil in subsequent tests, so
we clean up in the SAML test ns.

* Add a test to ensure SSO user names can't be changed via API

* Missed a change I had made while adjusting tests

* valid-name-update? take 1 name, to allow better error messages

Let's the user know that first or last name is the cause of a problem, rather than just 'names'.

* Remove unneeded thread macro

* Use partial=

* slight change to local fn for reusability

* Serdes v2: Add `--v2 true` flag to the `dump` and `load` commands

This is the last piece for end-to-end serialization and deserialization.
The results should be similar to v1, though probably not identical.

`PulseChannelRecipients` are a `:recipients` field on `PulseChannels`.

This requires some careful handling when inserting or updating a
`PulseChannel` to upsert the recipients.

* fixing tests and make chain-filter dashboard endpoints returns the flag

* remove using anonymous function in update

* Update dashboard endpoints and `FieldValuesWidget` to handle `has_more_values` scenario (#24022)

* update dashboard action/reducer

* Update FieldValuesWidget

* Update isSearchable to accept map of args

* Add unit tests for various FVW utils

* Split isSearchable logic a little

Co-authored-by: Dalton <daltojohnso@users.noreply.github.com>

* Remove `u/key-by`

* Fix typo

* SORT THE NAMESPACES

Cam suggested most of these changes in another PR, and I'm tackling them
here.

* Lets use the main thread

* Strip out channel stuff and rename

* 202 -> 200 response

When returning a channel we return a 202. A map is just a 200. Since we
no longer need to have the main stuff async (as opposed to the metadata
stuff) we can just return the map with a 200 instead of this long
running channel stuff and a 202.

* Last test

* renames, logging, ensure query is the same before saving metadata

* Sandbox test 202 -> 200

* Another 202 -> 200

* Put timeout on async metadata saving

timeout of 15 minutes before we give up on the async metadata saving. It
is possible this cuts things off but hard to tell if work is still being
done at that point.

* outdated comment

* Return json error message, not text/plain

this is a subtle one that I'm not happy about. Our error handling will
return text/plain if you throw an `(ex-info "something" {:status-code
400})`.

```shell
❯ http post localhost:3000/api/timeline-event/ name=some-name description=Bob timestamp=2022 timezone=America/Central time_matters:=false timeline_id:=1629  Cookie:$COOKIE
HTTP/1.1 404 Not Found
Content-Type: text/plain

Timeline with id 1,629 not found
```

But if you add extra information to the map to the ex-info, you get
json!

```clojure
(defmethod api-exception-response Throwable
  [^Throwable e]
  (let [{:keys [status-code], :as info} (ex-data e)
        other-info                      (dissoc info :status-code :schema :type)
        body                            (cond
                                          (and status-code (empty? other-info))
                                          ;; If status code was specified but other data wasn't, it's something like a
                                          ;; 404. Return message as the (plain-text) body.
                                          (.getMessage e)

                                          ;; if the response includes `:errors`, (e.g., it's something like a generic
                                          ;; parameter validation exception), just return the `other-info` from the
                                          ;; ex-data.
                                          (and status-code (:errors other-info))
                                          other-info

                                          ;; Otherwise return the full `Throwable->map` representation with Stacktrace
                                          ;; and ex-data
                                          :else
                                          (merge
                                           (Throwable->map e)
                                           {:message (.getMessage e)}
                                           other-info))]
    {:status  (or status-code 500)
     :headers (mw.security/security-headers)
     :body    body}))
```

So this fix is a _very_ subtle way to get to what we want, although it
does add a bunch of extra junk to our response.

```javascript
{
 ...
 "message": "Invalid Field Filter: Field 26 \"PRODUCTS\".\"CATEGORY\"
belongs to Database 1 \"Sample Database\", but the query is against
Database 990 \"copy of sample dataset\"",
 "data": {
   "status-code": 400,
   "query-database": 990,
   "field-filter-database": 1}
 ...
}
```

Reminder of what we want: the frontend is saving a card. We added a
field filter on a database and then changed the source database of the
query. So the backend needs to reject the field filter as being on the
wrong db. The FE expects a json response with a message and will then
show that message in the save/edit modal.

Why did this come up now: these endpoints for saving and editing a card
used to always return 202 streaming responses. This means they would
have to tuck any errors inside of an already open 202 response. Which is
why you get a message as a json response. But now that they are sync,
the api can just return a proper 400 with a reason. But we still want
that to be a json response for the FE.

* error layout fix

* Several Cleanups

- make sure numbers have units at the end (-ms)
- use (u/minutes->ms 15) rather than a more opaque (* 15 60 1000)
- move the scheduled metadata saving into its own function

This last bit is actually a bit more than that. I was previously
throwing everything in a thread submitted to the pooled executor. I'm
now using a `future` but with a caveat: All of the waiting for the
timeout, checking if we got metadata is done in a simple `a/go` block
now, and the thread does just the last bit of IO. We have to select the
card as it is now to ensure the query is the same and then save.

Refresher on why we have to check if the query is the same. We have up
to 15 minutes to wait for the query metadata to come back. Completely
possible for them to have edited the query and then our metadata is
useless.

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

* first pass storing linked-filter fieldvalues

* limit linked-filter to not explode

* no check perms for fields when getting params values on dashboard

Also standardizes date/time output to `ZonedDateTime`, rather than whatever the JDBC happens to return.

* logout when session expires, login when session appears

* add setting UI

* Add last_activity column to session table

* Start implementing session middleware to check for expired sessions

* Change last_activity field to include timezone offset

* Update session middleware to check user activity timeout

* Update last_activity after checking the timeout, or not at all if the setting is nil

* Move session-timeout settings to server.middleware.session

* Handcode timeout for testing

* Fix migrations validation error

* Fix whitespace

* Change session timeout to use metabase.TIMEOUT cookie with expiry

* Remove migration for last_activity column on session table

* Revert changes to logout endpoint

* Revert change to Session model pre-update

* Remove tap>

* Fix tests to include cookie value

* Fix timeout when user is logged out. Timeout loop should only start when a user is logged in

* Update comment and date format

* Store the session-timeout setting as json and convert it to seconds on the fly

* Set zoned date time to use GMT instead of default time zone

* Refactor for testing

* refactor session listener (#23686)

* remove old session listener

* Clear the timeout cookie when user signs out

* Clear session cookie if the timeout cookie expires

* fe tweaks

* Update expires attribute for session and timeout cookies together

* Reapply minimum limit on session-timeout

* Rename functions and fix lint warnings

* Fix resetting session-timeout

* Fix sign out

* Fix tests

* Whitespace

* Get full-app-embeds working

* Add test for embedded session

* session timeout ui tweaks

* fix security issue

* Fix test

* Fix tests

* Do not redirect to "/" if there isn't any redirect URL

* Add test for session-cookies setting

* Fix bug when toggling off timeout and adjust tests

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>

Pursuant to #23451.

The end effect of whitespace existing in a SAML response is us choking on it as reported in #23451. Two possible interpretations of causes of this bug:

There was an upstream change in our fork of the clojure SAML lib as flamber noted,
The decoding of base64 in our SAML endpoint (which uses the SAML lib) chokes on whitespace.
The proximate cause is the second one and ultimate cause is the first. However, I tend to believe that fixing the second one would be the better fix. For comparison, onelogin's first party SAML thing for java decodes base64 (https://github.com/onelogin/java-saml/blob/master/core/src/main/java/com/onelogin/saml2/util/Util.java) via apache's lib, which seems to do the thing that a lot of base64 decoders do of skipping whitespace.

* Allow group managers to see members even if they're sandboxed

* Remove failing test

* Delete test file

* Fix (unrelated) failing E2E test

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

Serialization of Databases, Tables, Fields

This brought a few core changes:
- Add `serdes-entity-id` to abstract the field used for the ID
- Pass the options to `extract-one` so it can eg. do encryption things.
- Handle dates in YAML storage and ingestion
- `:serdes/meta` now holds the entire hierarchy, not just the leaf model+ID pair.

There's an open problem here about the right way to handle secrets like
a database's password. Do we assume both sides have the same
`MB_ENCRYPTION_SECRET_KEY`? Provide a serdes-specific password the user
just made up, and every secret gets decrypted with the source key, encrypted with
the serdes key, stored, decrypted with the serdes key, and encrypted with
the destination key?

* Replace interval-hours and anchor-time with cron

Removed the two settings and replaced with a single cron schedule setting

Renamed the /set-interval endpoint to /set-refresh-schedule

* Ignore "year" part in schedule endpoint

* Fix variable

* Use a temp scheduler and initialize the refresh job

Running into errors when updating the triggers for each database's
refresh job because the "job" itself didn't exist. Reminder of what's
going on here: There's a single refresh job. It has a trigger for each
database. So updating the trigger would fail since it doesn't exist
since there was no job to hold the triggers.

This error is quite clear in the tests run locally:

```
ERROR in metabase.api.persist-test/set-refresh-schedule-test (util.clj:421)
Uncaught exception, not in assertion.

        clojure.lang.ExceptionInfo: Error in with-temporary-setting-values: Couldn't store trigger 'DEFAULT.metabase.task.PersistenceRefresh.database.trigger.1' for 'DEFAULT.metabase.task.PersistenceRefresh.job' job:The job (DEFAULT.metabase.task.PersistenceRefresh.job) referenced by the trigger does not exist.
    location: metabase.public-settings/persisted-models-enabled
     setting: "persisted-models-enabled"
       value: true
```

But this logging is absent from the logging in Github annoyingly:

```
FAIL in metabase.api.persist-test/set-refresh-schedule-test (persist_test.clj:26)
Setting new cron schedule reschedules refresh tasks
Setting :persisted-models-enabled = true
expected: "0 0 0/12 * * ? *"
  actual: (nil)
```
Whic doesn't give us the error, just the test result.

* update to newer API `set-interval` -> `set-refresh-schedule`

```
;; old
{:hours 4}

;; new
{:cron "0 0 0/1 * * ? *"}
```

Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>

* LDAP now also properly allows `nil` in names.

There were still places where "Unknown" was substituted in to name keys, which we don't need to do anymore as 'nil' is
valid for first/last names of users.

* Remove unused namespaces/refers

* Adjust LDAP so that even nil name values update the Metabase user

Also changed a test in both the OSS and EE LDAP tests to make sure this is indeed true that 'nil' values are correctly
set in the app-db when LDAP does not send `givenName` and/or `sn`.

Write `storage.yaml` and `ingest.yaml` to serialize all the way to YAML files and back.

Lots of generative testing to check it's isomorphic.

* Add test for user list endpoint when user is a sandboxed group manager

* Improve test description

* Store Sandboxed FieldValues

* locked-filter -> linked-filter

* resolve Noah and Braden comments

* Ensure current code does not confuse with the new added FieldValues type (#23601)

* makes sure the old code returns the correct FieldValues after adding new FieldValues types

* use threading macro to make at test easier to read

* Job to clean expired advanced field values (#23437)

* add a clean job

* merge upstream

* use java-time for max-age var

* simplify a reduce function and update some tests to make it easier to understand

* Send alert emails on persisted model refresh fail

* Fixing formatting and making sure all the information is available for template

* Fix circular dependency

* Fixing tests and only check advanced perm users if they exist

* Fix merge cruft that was renamed

* Only send emails in persist-refresh

* Add temporary endpoint for testing persist failure emails

* Use `border-box` box sizing in emails

* Use Lato font in emails

* Style model caching error email

* Revert "Add temporary endpoint for testing persist failure emails"

This reverts commit 3d3b8f123060f278374aa5a9301c67d547eb42f7.

* Add is-not-first to error context to help styling

* Only send to admins if advanced-permissions is off

From @noahmoss:
> This is an edge case, but it's possible for an EE instance to downgrade back to OSS. When this happens, users with DB and/or monitoring perms lose these permissions in the app, since it's an EE-only feature. But we don't run any migrations on the permissions table when downgrading happens, so they'll still show up here.
> To fix this you'll need to call metabase.public-settings.premium-features/enable-advanced-permissions? to check whether the instance is EE and advanced perms are enabled before including these users in this query.

* Test private fn

* More robust admin email handling and tests

* Remove commented out code

* Avoid `nth-child` selector for padded sections

* Don't hardcode colors

* Fix line-height

* Use line-height fix for Outlook

* Avoid media-queries and `box-sizing`

* Fix E2E test

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* add exceeded_limit column and set it to true when values of a list field exceed our defined limits

* move exceeded_limit to has_more_values in metabase_fieldvalues

* make fieldvalues search not do in-memory search if has_more_values=true

* resolve Noah's comments and make the comments clearer

* application-colors have new keys that may be added via the getter

Some colors previously had multiple usage contexts. For example, `accent1` was used both in charts and other parts of
the summarize UI. Now, those notions are being separated, so `accent1` remains a valid key, and `summarize` is a new
valid key. To make sure behaviour remains the same for existing whitelabel users who may have set these keys, when
such a key exists, it is 'split' by the getter.

For example, if the existing application-colors json contains `accent1`, the getter will add a new key `summarize`
with the same value as `accent1`, but only if `accent1` already exists, otherwise it does nothing. This is also true
for keys `brand`, which adds `accent0`, and `accent7`, which adds `filter`

* Make application colors getter make change only once

* Premium feature flag for test

This supports serialization of only Collections and Settings so far, but
it demonstrates the design of the new serialization system.

`metabase.models.serialization.base` defines the multimethods, which
are to be implemented by all the exported models eventually.
The actual serialization code that drives the larger process is in
`metabase_enterprise.serialization.v2.extract` and `.merge`, since
serialization is an enterprise feature.

The design calls for two matching phases on each side:
- Serialization is extract + store;
- Deserialization is ingest + load.

Extract and load deal with vanilla Clojure maps with a `serdes/meta` key
giving common details; they deliberately know nothing about files.

Store and ingest deal with the storage medium and the process of
listing and reading a stored export.

Laziness is retained: the `load` process ingests full details on demand,
so only the metadata of the importing database needs to fit in memory.

* Allow for disabling settings

Disabled settings will return their default value (else nil if no
default is set). This allows us to have enterprise override settings and
use them from regular OSS code without classloaders, extra vars,
remembering to check if the feature is enabled, etc.

Motivating examples are the appearance settings. We allow
`application-font` setting to change the font of the application. This
is an enterprise feature, but anyone can post to
`api/setting/application-font` and set a new value or startup as
`MB_APPLICATION_FONT=comic-sans java -jar metabase.jar` and have the
functionality.

Same thing for application colors in static viz. The calling code just
calls `(settings/application-colors)` and uses them but doesn't check if
the enterprise settings are enabled. To do this correctly, you have to
remember to implement the following onerous procedure:

A whole namespace for a setting
```clojure
(ns metabase-enterprise.embedding.utils
  (:require [metabase.models.setting :as setting :refer [defsetting]]
            [metabase.public-settings :as public-settings]
            [metabase.public-settings.premium-features :as premium-features]
            [metabase.util.i18n :refer [deferred-tru]]))

(defsetting notification-link-base-url
  (deferred-tru "By default \"Site Url\" is used in notification links, but can be overridden.")
  :visibility :internal
  :getter (fn []
            (when (premium-features/hide-embed-branding?)
              (or (setting/get-value-of-type :string :notification-link-base-url)
                  (public-settings/site-url)))))
```

And then in the calling code you have to do the procedure to
conditionally require it and put it behind a var that can handle it
being nil:

```clojure
;; we want to load this at the top level so the Setting the namespace defines gets loaded
(def ^:private site-url*
  (or (u/ignore-exceptions
        (classloader/require 'metabase-enterprise.embedding.utils)
        (resolve 'metabase-enterprise.embedding.utils/notification-link-base-url))
      (constantly nil)))

;; and then the usage
(defn- site-url
  "Return the Notification Link Base URL if set by enterprise env var, or Site URL."
  []
  (or (site-url*) (public-settings/site-url)))
```

Far nicer to just place the following into the regular public-settings
namespace:

```clojure
(defsetting notification-link-base-url
  (deferred-tru "By default \"Site Url\" is used in notification links, but can be overridden.")
  :visibility :internal
  :enabled?    premium-features/hide-embed-branding?)
```

Then no need for a custom namespace to hold this setting, no need to
have an extra var to point to the setting else a fallback value.

Note that this feature is not required on every enterprise feature we
have. We a namespace `metabase-enterprise.sso.integrations.sso-settings`
that has 24 settings in it, all of which are enterprise features. But
these features are used in our enterprise sso offerings and are directly
referenced from the enterprise features. No need for the extra var to
point to them and the flag checks happen in other parts.

* Mark the UI/UX customization settings as requiring whitelabeling

Mark the following settings as requiring
premium-settings/enable-whitelabeling? (aka token check)

- application-name
- loading-message (override of "doing science")
- show-metabot (override of showing our friendly metabot)
- application-colors
- application-font
- application-logo-url
- application-favicon-url

Updates the helper functions for colors to use the setting rather than
feeling entitled to use a lower level `setting/get-value-of-type`. We
need the higher level api so it takes into account if its enabled or
not.

* Move notification-link-base-url into regular settings with enabled?

* Cleanup ns

* Add reply-to setting for emails

* Add reply-to to SMTPSettings

* Change setting UI to match spec

* Remove WIP comment

* Add email-from-name setting

* Support comma delimited list of emails for reply-to setting

* Remove extra require

* Remove console.logging

* Improve the setting descriptions

* update email settings

* For now, limit reply-to field to one email

* Change back SMTPSettings to closed map

* Change test to only one reply-to email

* Fix SMTPSettings reply-to type

* Fix email-api-test

* Fix e2e email setup

* Add more checks in e2e tests

* Fix smoketest

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

They're useful in v2 serialization as well.

* Audit Queries fall back to email when first/last names are nil

The audit page displays user information, and one column is "Name". Since we now allow nil first/last names, this
value can return as a single space character. Instead, I've changed `common/user-full-name` to fall back to returning
the user's email address if they have no first/last name values.

Additionally:
- since names can be nil, I've added sort by email to some of the queries for the audit page
- display names have been changed from "Name" to "Member" in several queries to better fit the fallback to email

* Add a test to check that audit pages have proper name fallback

* Try to get test passing in CI

* Make query work in MySQL by coalescing 'null' to empty string