* Join slack channels with slack-id

Fixes https://github.com/metabase/metabase/issues/23229

We upload images to a channel and then send messages to the desired
channel referencing those images from the channel we uploaded. But the
slack bot must be in the channel to upload images.

We handle this in `slack/upload-image!` where we watch the error message
and join the channel if we recognize that is our issue. This special
upload channel is set in the admin section when setting up, typed in by
a human.

Slack now requires us to use the internal id of the channel when
joining. IE we used to use "metabase_files" but now we need to use
something like "C87LQNL0Y23". But we do not have this information and we
don't want humans to have to look this up.

SOLUTION:
change our cache. We currently just get a list of channels and users
```
["#general" "@dan" ...]
```

Change this to
```
{:version 2,
 :channels [{:display-name "#random",
             :name "random",
             :id "CT2FNGZSRPL",
             :type "channel"}
            {:display-name "#general",
             :name "general",
             :id "C87LQNL0Y23",
             :type "channel"}
            {:display-name "@dan",
             :type "user",
             :name "dan",
             :id "UR65C4ZJVIW"}
            ...]}
```

Now we have slack internal ids present. When we attempt to join the
slack channel, look for this id and attempt to use that.

This has some knock-on effects. The UI still lists the channels in a
channel picker when sending pulses. The list sent over the wire still
mimics the previous shape (a flat list) and the choice is still the
human readable name.

In the future we should switch over to using the stable ids rather than
solely channel names. Channel names can be renamed.

I didn't go down this route because of the files channel. It is set at
setup before we have a channel list. We could do some kind of run time
migration but it is difficult because it would change the type of
`slack-files-channel` from string to :json to handle the more complex
type. Or perhaps we could make another setting to hold the json form and
set that when we can positively identify things.

In either case, these changes were not required yet to fix our slack
issue. We just upgrade the information we have about slack channels,
downgrade it when it hits the wire so the UI needs no changes, and use
the extra information in the one spot where we need it.

The cache is populated at startup and every four hours after that. So we
do not need to worry about the old cache shape. If the new code is
running, its the new cache.

* Send #channel and @user forms over wire

We store `{"channel": "#slack-pulses"}` in the pulse_channel.details
column so we should keep those types of values around.

We use the bare portion ("slack-pulses") rather than with the hash on it
so we seem to be mixing usernames and channels. But these sets are
distinct and you cannot create a channel with the same name as a
user. Also, channel names are lowercase while channel-ids are uppercase
so those are also non-overlapping sets.

* Put slack token so slack reports as configured

* Errant tap>

* Alignment and docstring fixes

* Remove slack-cache version information

remove the `:version 2` from the cache. We are always in charge of the
cache, and we compute it on startup so there's little risk of other data
shapes being present.

* Stop caching /api/geojson/:key requests

* Remove extra require

This supports serialization of only Collections and Settings so far, but
it demonstrates the design of the new serialization system.

`metabase.models.serialization.base` defines the multimethods, which
are to be implemented by all the exported models eventually.
The actual serialization code that drives the larger process is in
`metabase_enterprise.serialization.v2.extract` and `.merge`, since
serialization is an enterprise feature.

The design calls for two matching phases on each side:
- Serialization is extract + store;
- Deserialization is ingest + load.

Extract and load deal with vanilla Clojure maps with a `serdes/meta` key
giving common details; they deliberately know nothing about files.

Store and ingest deal with the storage medium and the process of
listing and reading a stored export.

Laziness is retained: the `load` process ingests full details on demand,
so only the metadata of the importing database needs to fit in memory.

* Allow for disabling settings

Disabled settings will return their default value (else nil if no
default is set). This allows us to have enterprise override settings and
use them from regular OSS code without classloaders, extra vars,
remembering to check if the feature is enabled, etc.

Motivating examples are the appearance settings. We allow
`application-font` setting to change the font of the application. This
is an enterprise feature, but anyone can post to
`api/setting/application-font` and set a new value or startup as
`MB_APPLICATION_FONT=comic-sans java -jar metabase.jar` and have the
functionality.

Same thing for application colors in static viz. The calling code just
calls `(settings/application-colors)` and uses them but doesn't check if
the enterprise settings are enabled. To do this correctly, you have to
remember to implement the following onerous procedure:

A whole namespace for a setting
```clojure
(ns metabase-enterprise.embedding.utils
  (:require [metabase.models.setting :as setting :refer [defsetting]]
            [metabase.public-settings :as public-settings]
            [metabase.public-settings.premium-features :as premium-features]
            [metabase.util.i18n :refer [deferred-tru]]))

(defsetting notification-link-base-url
  (deferred-tru "By default \"Site Url\" is used in notification links, but can be overridden.")
  :visibility :internal
  :getter (fn []
            (when (premium-features/hide-embed-branding?)
              (or (setting/get-value-of-type :string :notification-link-base-url)
                  (public-settings/site-url)))))
```

And then in the calling code you have to do the procedure to
conditionally require it and put it behind a var that can handle it
being nil:

```clojure
;; we want to load this at the top level so the Setting the namespace defines gets loaded
(def ^:private site-url*
  (or (u/ignore-exceptions
        (classloader/require 'metabase-enterprise.embedding.utils)
        (resolve 'metabase-enterprise.embedding.utils/notification-link-base-url))
      (constantly nil)))

;; and then the usage
(defn- site-url
  "Return the Notification Link Base URL if set by enterprise env var, or Site URL."
  []
  (or (site-url*) (public-settings/site-url)))
```

Far nicer to just place the following into the regular public-settings
namespace:

```clojure
(defsetting notification-link-base-url
  (deferred-tru "By default \"Site Url\" is used in notification links, but can be overridden.")
  :visibility :internal
  :enabled?    premium-features/hide-embed-branding?)
```

Then no need for a custom namespace to hold this setting, no need to
have an extra var to point to the setting else a fallback value.

Note that this feature is not required on every enterprise feature we
have. We a namespace `metabase-enterprise.sso.integrations.sso-settings`
that has 24 settings in it, all of which are enterprise features. But
these features are used in our enterprise sso offerings and are directly
referenced from the enterprise features. No need for the extra var to
point to them and the flag checks happen in other parts.

* Mark the UI/UX customization settings as requiring whitelabeling

Mark the following settings as requiring
premium-settings/enable-whitelabeling? (aka token check)

- application-name
- loading-message (override of "doing science")
- show-metabot (override of showing our friendly metabot)
- application-colors
- application-font
- application-logo-url
- application-favicon-url

Updates the helper functions for colors to use the setting rather than
feeling entitled to use a lower level `setting/get-value-of-type`. We
need the higher level api so it takes into account if its enabled or
not.

* Move notification-link-base-url into regular settings with enabled?

* Cleanup ns

* make schema for parameters and parameter_mappings more strict

* no duplicate id for parameterlist in mbql schema

* reverse a change in mbql schema

* allow blank name and slug for parameter

* add sectionId

* trying to use mbql.s/Parameter instead

* reverse change to use mbql.s/Parameter, there are too many failing tests

* Grant Snippets root collection for all users

* update perms path

* update cypress test

* Update repro for #17268

* remove trailing spaces

* don't run migration on existing EE instances

* fix migration failed in mysql and mariadb

* run migration for new instances only

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Add collection_preview field to report_card table
* Convert bits from MySQL/MariaDB union queries into booleans

* Add reply-to setting for emails

* Add reply-to to SMTPSettings

* Change setting UI to match spec

* Remove WIP comment

* Add email-from-name setting

* Support comma delimited list of emails for reply-to setting

* Remove extra require

* Remove console.logging

* Improve the setting descriptions

* update email settings

* For now, limit reply-to field to one email

* Change back SMTPSettings to closed map

* Change test to only one reply-to email

* Fix SMTPSettings reply-to type

* Fix email-api-test

* Fix e2e email setup

* Add more checks in e2e tests

* Fix smoketest

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

* first pass at updating parameter validation to include notebook cards and match template tags by either ID or name

* fix existing tests

* fix more tests

* new test

* more refactor and another test

* Update src/metabase/query_processor/card.clj

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* update names to ngoc's suggestions

* Update test/metabase/query_processor/card_test.clj

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Update src/metabase/query_processor/card.clj

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* specific error msg for missing id on request param

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

Fix compilation of temporal arithmetic for BigQuery and Mongo 5+

* Mongo 4 doesn't support $dateAdd so the generated filters result in an exception.
* Support adding field to interval too (time intervals were not allowed in the first place of an addition)
* Support temporal arithmetic with more than two operands for Mongo

MBQL goes through a different path. This is only for native
queries. Also the diff is huge. Ignoring whitespace shows a very modest
diff:

```diff
    "month"   {:unit-range month-range
               :to-period  t/months}
+   "quarter" {:unit-range relative-quarter-range
+              :to-period  (comp t/months (partial * 3))}
    "year"    {:unit-range year-range
               :to-period  t/years}})

...

   "past2months~"   {:end "2016-06-30", :start "2016-04-01"}
   "past13months"   {:end "2016-05-31", :start "2015-05-01"}
+  "past2quarters"  {:end "2016-03-31", :start "2015-10-01"}
+  "past2quarters~" {:end "2016-06-30", :start "2015-10-01"}
   "past1years"     {:end "2015-12-31", :start "2015-01-01"}
   "past1years~"    {:end "2016-12-31", :start "2015-01-01"}
```

Helpful Testing Strategies
--------------------------

Sample DB
=========

`select id, created_at from orders where {{created_at}}`
and set a field filter of type "Date Filter"

Custom Table
============

Create a table that has entries in the middle of each quarter.

```sql
esting=# create table quarters (id serial primary key not null, description text, t timestamptz);
CREATE TABLE
testing=# insert into quarters (description, t) values ('Q1 2022', '2022-02-01'), ('Q2 2022', '2022-05-01'), ('Q3 2022', '2022-08-01'), ('Q4 2022', '2022-11-01');
INSERT 0 4
testing=# select * from quarters;
 id | description |           t
----+-------------+------------------------
  1 | Q1 2022     | 2022-02-01 00:00:00-06
  2 | Q2 2022     | 2022-05-01 00:00:00-05
  3 | Q3 2022     | 2022-08-01 00:00:00-05
  4 | Q4 2022     | 2022-11-01 00:00:00-05
(4 rows)
```

Before this change
------------------

> Cannot invoke "clojure.lang.IFn.invoke(Object)" because "to_period" is null

(note, if you cannot reproduce this its because you haven't gotten
https://github.com/metabase/metabase/pull/23346 in your local version
which ensured errors always appear as errors on the frontend. java 11
has no message for NPE so the Frontend missed it was an error and
displayed no results as if the query had succeeded)

This was the case for the following scenarios:
- "last1quarters"
- "last1quarters~"
- "thisquarter"
- "next1quarters"
- "next1quarters~"

where the ~ means to include the current quarter.

After this change
-----------------

Running the queries against the custom table I made (current time is Jun
15, Q2)

- "last1quarters": "Q1 2022" "February 1, 2022, 6:00 AM"
- "last1quarters~": "Q1 2022" "February 1, 2022, 6:00 AM" | "Q2 2022" "May 1, 2022, 5:00 AM"
- "thisquarter": "Q2 2022" "May 1, 2022, 5:00 AM"
- "next1quarters" "Q3 2022" "August 1, 2022, 5:00 AM"
- "next1quarters~": "Q2 2022" "May 1, 2022, 5:00 AM" | "Q3 2022" "August 1, 2022, 5:00 AM"

And of course added tests into the matrix for the date parsing.

Pursuant to #22967.

Previously the notion of the identifier in json-query in postgres assumed a singular identifier at the top-level of the Identifier record. This is not always the case in complicated binning scenarios - the identifier could also just comprise an entire tree of Identifier record maps with the actual Identifier we're looking to turn into the JSON query in some leaf somewhere.

Therefore, the json-query's determination of what the identifier is now walks that identifier record tree and replaces the identifiers that have JSON field semantics with the reification of ToSql from json-query.

Previous attempt at solution hacked together something at the json-query level, changing the reification, but that only worked for one level down, instead of just walking the Identifier tree and therefore getting stuff at arbitrary tree depth. And we do get nontrivial tree depth in the bucketing in #22967.

* Ensure :error is present for frontend

The frontend checks for a string at "error" to determine if there's an
error or not. Any error that lacks a message (`(.getMessage e) = nil`)
will report as a successful query with no results. This was hard to
reproduce at first because java 14+ added a nice error message to NPEs
in [JEP-358](https://openjdk.org/jeps/358).

```java
;; java 11
jshell> HashSet x = null;
x ==> null

jshell> x.contains(3);
|  Exception java.lang.NullPointerException
|        at (#2:1)

jshell>

;; java 17
jshell> HashSet x = null;
x ==> null

jshell> x.contains(3);
|  Exception java.lang.NullPointerException: Cannot invoke "java.util.HashSet.contains(Object)" because "REPL.$JShell$11.x" is null
|        at (#4:1)
```

```clojure
;; java 17
catch-exceptions=> (let [x nil] (x 3))
Execution error (NullPointerException) at metabase.query-processor.middleware.catch-exceptions/eval130385 (REPL:41).
Cannot invoke "clojure.lang.IFn.invoke(Object)" because "x" is null

;; java 11

catch-exceptions=> (let [x nil] (x 3))
Execution error (NullPointerException) at metabase.driver.sql.parameters.substitution/eval118508 (REPL:17).
null
```

Here are the responses to the FE edited to the relevant bits:

```javascript
// objects are edited for clarity/brevity:
// from java-11
{
  "error_type": "qp",
  "status": "failed",
  "class": "class java.lang.NullPointerException",
  "stacktrace": [...],
  "error": null,   // <---- the FE ignores all of the other data and only sees this
}

// vs java-17
{
  "error_type": "qp",
  "status": "failed",
  "class": "class java.lang.NullPointerException",
  "stacktrace": [...],
  "error": "Cannot invoke \"clojure.lang.IFn.invoke(Object)\" because \"to_period\" is null",,
}
```

Also, note that we were still logging

> ERROR middleware.catch-exceptions :: Error processing query: null

because we were looking for `(:error format-exception)` instead of
`(:error formatted-exception)`. `format-exception` is a multimethod and
lookup on non-associative things will happily return nil.

* Tests

* Use (str ...) forms to translate multi-line messages
* Make H2 error messages refer to the correct field

* Migration to allow 'null' first and last names for users

This is part of improving how we handle user names, related to SSO, but we also have an opportunity to make login via
email simpler as well -> just loosen the requirement for names.

* When only last name is given, :common_name should not have a space

* WIP making optional first/last names safe in the backend.

Trying to fix up some assumptions about first/last name keys in backend implementations.

* Only add `:common_name` key when non-nil

* Adjust setup tests to pass if use first/last name are nil

* Fix up setup tests to work with null first/last names.

* User API endpoints altered to handle nil names

* Remove name validation test since they can be nil now

* Alter JWT/SAML tests for nil user names

* Remove "unknown" default in SAML

* Add tests to make sure users are updating appropriately

* use good naming convention in local function

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Simplify truthy check in test

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Simplify truthy check in test

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Indentation fix

* Fix up syntax issue from github merge.

* Fix missed function name changes in test

* Clean up how sets are built up in the PUT user/:id endpoint

* Better implementation for update-user-name

* Avoid relying on 'vals' since order of keys is not guaranteed

* Fixing little things, clarification in a comment

* I'm tired... missed an obvious thing here. oops

* indentation and clarify some testing

* Change post-select implementation slightly

* expected status is now an optional arg no dynamic var binding needed

* 'f' is not optional, so we take it with first, then args are rest

* simplify destructuring

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

Define identity-hash for fairly robust de-duplication when deserializing

This is a fallback for fully robust de-duplication based on `entity_id`
fields.

All serialized models should support identity-hash, and there is a test
to enforce that.

* add API to fetch and search card parameters

* remove code to support linked filters

* remove unecessary data in tests

* fix from Noah's comments

* some chain-filter -> param-values

* allow str form as the first argument to deferred-tru

* add tests and update doc strings

* migrate more setting descriptions

* Handle multiline translation sources

```clojure
enumerate=> x
(deferred-tru
 (str
  "Whether an introductory modal should be shown after the next database connection is added. "
  "Defaults to false if any non-default database has already finished syncing for this instance."))
enumerate=> (form->string-for-translation x)
"Whether an introductory modal should be shown after the next database connection is added. Defaults to false if any non-default database has already finished syncing for this instance."
```

We get the form from `(g/grasp <file> ::translate)` which returns the
`x`. And then simply pick through it for either a string literal or a
call to `(str <literal>+)`

Co-authored-by: dan sutton <dan@dpsutton.com>

Previously we defaulted to integer and float type in all cases for json unfolding type determination, even with bigints and bigdecimals. Now if we encounter a bignum in json unfolding we call it a bignum.

* clear Slack cache last-updated value when resetting cache

* use clear-channel-cache! function on invalid tokens as well

* Save query executions for cached queries

Previously had addressed
https://github.com/metabase/metabase/pull/16720/ because we were
including cache hit query executions in the running query execution
time. IE, queries take 60 seconds to run but the cache hit takes
100ms. Our average thus would be (60s + 100ms)/2 ~ 30 seconds and the
query duration falls below the cache threshold so we abandon the cache
until the average creeps up back up again. Original issue:
https://github.com/metabase/metabase/issues/15432

So the first fix was a poor one and it skipped recording the query
execution at all which was way too heavy handed.

```clojure
(when-not (:cached acc)
  (save-successful-query-execution! (:cached acc) execution-info @row-count))
```

This approach takes the more considered approach: skip updating average
execution time but record the query-execution.

```clojure
;; conditionally save average execution time
(when-not (:cache_hit query-execution)
    (query/save-query-and-update-average-execution-time! query query-hash running-time))

;; unconditionally (modulo having a context) save the query execution
(if-not context
  (log/warn (trs "Cannot save QueryExecution, missing :context"))
  (db/insert! QueryExecution (dissoc query-execution :json_query)))
```

One question
------------

I'm unsure about one change here. What are the implications of not
updating this execution time. There are two operations involved:
ensuring there is a `Query` (table query) entry for a query and updating
its average execution time. I'm assuming that since a query is cached it
has a query table entry. But this code creates the Query if it doesn't
exist and then updates its average execution time. It's not clear to me
if we do in fact want to ensure a Query entry exists. But it feels
harmless both in the case it is missing and also that it most likely
isn't missing since we have a cached query and when it was cached this
entry would have been created or updated. But adding a note just in case.

* Fix test

distinguish between the calls to save query execution and the calls to
update average execution duration. The former should be called
twice (for the uncached query and the cached query run) while the latter
should only be called on the first run (aka, uncached results affect
query duration stats, cached runs do not affect it)

* Checksums are no more

metadata became editable for models and the checksum really didn't add any value, just asserted that it had not been
changed.

We have already had support for server authentication based on custom
certificates. This change adds support for authenticating the client
based on custom client key and certificate.

Do trivial identity for passthrough of symbols for nested field column instead of default processing (#23136)

Should whack both #23026 and #23027.

Previous (default) behavior of the reducible query that gets the limited contents of the JSON to break out the nested field columns is to lowercase identifiers. This is root cause of #23026 and #23027.

But we want the proper case of those identifiers in order to be modifying the query correctly when we query the JSON. So we set the reducible query to just pass through the identifiers instead of default behavior.

Fixes #22626.

* dashboard api to get params value should return 404 instead of 500 when param not found

* on the 2nd thought, let's return 400

* update the tests too

* wording

* add parameters to report_card and card APIs

* make sure migration can work with existing cards and add migration tests

* remove debug code

* When parameters is empty, return template tag parameters

* update schema message

* update migraiton test name

* minor changes in test

* fix failing tests

* Always use card.parameters in PublicQuestion

* Add defaults and readd isEmpty check

* Default to undefined/use template tags in PublicQuestion...

* parameters should be in the writableProperties

* Native query card: construct parameters from template tags

* Separate the generation of parameter id

* Add parameter_mappings to report_card (#23003)

* add parameter_mappings to report_card

* fix from Noah's comments

* fix from Noah's comments

* Update a parameter from an updated template tag

* Correct the parameters construction

* Also add `parameter_mappings` to writableProperties

* CI test: bust the npm module cache

* Revert "CI test: bust the npm module cache"

This reverts commit 5a327b616f0220f43a90f7f871e0bd877ffa6f47.

Co-authored-by: Dalton Johnson <daltojohnso@users.noreply.github.com>
Co-authored-by: Ariya Hidayat <ariya@metabase.com>

Adds option to turn JSON column unfolding off. This does not constitute an actual fix of boat161 but flamber wanted it

That is: collection, dimension, metric, native_query_snippet, pulse,
report_card, report_dashboard, report_dashcard, segment, timeline

Notably that doesn't include database, table, or field, since those all
have external unique IDs that are used instead.

remove param_values hydration keys

Limit the length of the row for JSON parsing. This was the simplest way to make another attack at #22636.

* Add can-manage to databases to help front-end

* Only return persisted true when the state agrees

* Add event for persisted-info to catch new models and turn on peristence when appropriate

* Fix bad threading

* Add comment about the case being detected to add the persisted-info

* Fix pre-existing bug in expand-db-details where a function was used as the condition

* Indicate field responsible for database error

* Implement new interface

* Make error showing logic a little easier to understand

* Show individual field error in database form

* Add backend tests for connection errors when adding a database

* Add new form error component

* Make database form error message pretty

* Make main database form error message field bold

* Change create account database form according to feedback

* Fix failed E2E tests from UI changed.

* Make it easier to tree-shake "lodash"

* Change according to PR review

* Cleanup + remove FormError (to be included in a separate PR)

Co-authored-by: Mahatthana Nomsawadi <mahatthana.n@gmail.com>

* Add anchor time to persistence schedule

Introduces a new public-setting persisted-model-refresh-anchor-time,
representing the time to begin the refresh job. Defaults to midnight
"00:00".

Also peg the cron jobs to run in the first of: reporting timezone, system timezone, or
UTC. This means that if a user needs the refresh to be anchored consistently to a certain time in a certain place, they need to update reporting timezone to match.

* Add tests for anchor time

* Force anchor-time to midnight if refresh hours is less than 6

* Need to check that hours was passed in

* Fix ns lint

* Add tests to ensure timezone is being set on trigger