* Make Database details NOT NULL; add defaults for Database created_at and updated_at

* Add default value for `:details` to SerDes load-xform

* Fix MySQL UTF-8 test

* Reset changes and go thru everything again to make sure it makes sense

* Test fixes 

* indentation only

* Fix creation check for mysql

* Test that all persisting dbs can persist

* Erran tap> (whoops)

* `POST /api.ee/serialization/serialize/data-model` endpoint

* Sort namespaces

* Don't use `tru` for endpoint params validation

* Implement advanced date/time/zone manipulation, part 1

Incorporate new functions into MBQL and add tests:
 - get-year
 - get-quarter
 - get-month
 - get-day
 - get-day-of-week
 - get-hour
 - get-minute
 - get-second

* Fix BigQuery implementations to call extract

Mark as not supported in legacy driver

* Add date extraction fns for Postgres

* Disable in MongoDB (for now at least)

Disable in BigQuery (legacy driver)

Add implementations for presto-jdbc

* Misc cleanup from Jeff's PR

* Update Jeff's implementation of bigquery-cloud-sqk

* Reorganized tests

* Mongo

* Oracle

* Sqlserver

* Sqlite

* Add casting supports for presto

* Remove Jeff's implementation of presto-jdbc because  its parent is
sql-jdbc

* Update presto-jdbc tests to use the same catalog for all datasets

* Add date extraction functions to the expression editor (#25382)

* make sure the semantic type of aggregated columns are integer

* no recursive call in annotate for date-extract func

* get-unit -> temporal-extract(column, unit)

* desguar nested datetime extraction too

* Refactor static waterfall chart to use a single entrypoint

* Consolidate static watetrfall chart

* Update waterfall chart text size to match combo-chart

* Move `<Values />` from XYChart/ to be under components/

Since it'll be used by waterfall chart

* Fix `<Values />` key error (forgot to add key attribute)

* Show data point values on waterfall chart

* Fix failed BE tests

* Fix test -> wrong `waterfall-type` used. And test both types

The waterfall component was changed to take a type argument, instead of having 2 functions for nearly identical
output. The failing test just passed the wrong string to the key argument.

* Change waterfall keys from strings to Clojure keys

* Remove redundant test. Categorical also handled in `waterfall-test`

* Fix clj-kondo lint error

* Cleanup conditional statements

* Miscellaneous code cleanup

* Update waterfall test to render data point values

* Fix waterfall chart data point values not center in the bar

* Make some API for static waterfall chart with total saner

Co-authored-by: Adam James <adam.vermeer2@gmail.com>

* Switch to humane-are

* Fix some busted `are`s

* Remove unused namespaces

* Implement DB specific execution permissions

Separate execution permissions from data permissions as a new dimension
like application features. Create a new endpoint for getting and setting
these permissions.

* Substitute persisted queries in parameter card references

```sql
select o.total, o.quantity, model_p.category, model_p.title
from orders o
left join {{#14-pg-products}} model_p -- reference to products model
on o.product_id = model_p.id
limit 4
```

Testing:

I'm unhappy with how verbose the test is. At some point we're going to
need a better, standardized way to test persistence. But the test makes
a persisted table of the test-data.categories
table (test/metabase/test/data/dataset_definitions/test-data.edn). It
runs some sql to update the values in the table to turn `"Winery"` ->
`"Winery from cached table"`, joins the two tables together to have both
at once.

There's a lot of moving pieces and the setup is a bit verbose. There's
also no obvious place where these types of tests should go. We'll need
to consolidate them in the future (and extend them).

* persisted macro

* docstring for `persisted-info-native-query`

* alignments and small nits

* Reuse persistence macro

* clj-kondo cleanups

* remove clj-kondo change before rebase

* Introduce /execution/ permission

* the fix

* adding some tests

* alternative solution in which we don't merge metadata for aggregation fields at all

* wording for the docstring

* use partial= and remove smart quotes. Thanks Cam

* Handle bigint and boolean casts properly

- Add JSON -> SerializedJSON to `database-type->base-type`

* adds tests json bigint and boolean types for mysql

* Fix most of the tests

- still need to figure out :type/BigInteger vs :type/Integer

* fiddling with tests

* handle json booleans via default mechanism

* fix tests

* remove not(not(...)) from test

* bonk the test

- also fixes ddl.execute-with-timeout-test thrown? regex

* remove inline def

* try to get logs off ci

* don't test json on mariadb -- that never worked

* cleanup tests

* revert change to driver/database-supports? :mysql :nested-field-columns

* respond to code review comments

* 347 errors, 34 warnings

* 89 errors, 66 warnings

* 63 errors, 39 warnings

* 52 errors, 33 warnings

* 23 errors, 22 warnings

* 13 errors, 4 warnings

* Fix the last few errors.

* Sort Kondo config

* allow disabling custom geojson via env var

* add env var to docs and tweak setting description

* wip changes

* new endpoint

* fix test

* more test fixes

* add test for EE settings permissions

* try to fix another BE test

* address ngoc's comment

* fix cypress test

* fix another cypress test to hit new settings endpoint

* allow turning off mysql json unwrapping

- somehow this value was set to always be true, but it should be toggle-able.

- also for ignoring INIT=... on the conn string

* remove toc generation from api docs

* update api docs

* update test

* Don't use persisted model tables for segmented users

This actually isn't a bug, but due to very subtle and arbitrary reasons.

For background about why we need to ensure this never happens, we cannot
use persisted models when sandboxing is at play. A simple example is as
follows: make a model on a products table that does not select the
category. Have a sandbox on category such that someone can only see
products of category "Gizmo". the model lacks the category column but we
insert a where clause that still works. When the model is persisted,
there is no category column in the underlying table so sandboxing cannot
possibly work: the data necessary to filter is no longer associated with
the rest of the data in the model.

The fix for this is quite simple: in
`metabase.query-processor.middleware.fetch-source-query` we only splice
in the persisted query if the user is not a segmented user (product name
for sandboxing).

```clojure
(and persisted-info/*allow-persisted-substitution*
     (not (segmented-user?))  ;; <----- new check
     (:active card)
     (:definition card)
     (:query_hash card)
     (= (:query_hash card) (persisted-info/query-hash (:dataset_query card)))
     (= (:definition card) (persisted-info/metadata->definition (:result_metadata card)
                                                                (:table_name card)))
     (= (:state card) "persisted"))
```

Technical details about why this bug did not manifest

When swapping out a card__<id> to a source query, if its a model we will
see if it is persisted, and if so, we will use the native sql to select
from the persisted table. It does this by adding the native sql at a key
called `:persisted-info/native` and a middleware
`#'qp.persistence/substitute-persisted-query` walks the query replacing
the query with the native:

```clojure
;; metabase.query-processor.middleware.persistence

    (mbql.u/replace query
      (x :guard (every-pred map? :persisted-info/native))
      {:native (:persisted-info/native x)})
```

There is also a middleware that walks through the query looking for
tables with gtaps on them and replacing them. By change, the sandboxing
middleware runs immediately before the substitute-persisted middleware!

```clojure
   ;; literally the previous middleware
   (resolve 'ee.sandbox.rows/apply-sandboxing)
   #'qp.persistence/substitute-persisted-query
```

If you swap the order of these two sandboxing is broken. As is, it
"works" but not by design, just by happenstance. The sandboxing
middleware just did not know that the `:persisted-info/native` key meant
that a native query was to be substituted. In the reverse order, the
native query is already substituted and there is no change for the
sandboxing to occur.

The obvious fix is to ensure that we never even attempt to use the
persisted tables and that is what this PR does.

* Eastwood doesn't like shadowing like this

* Rearrange check order for tests

`segmented-user?` throws if there is no bound user. A test in
`fetch-source-query-test` was failing because there was no user bound,
but it wasn't attempting to swap out a persisted table, it just didn't
expect to need a user.

Moving it lower lets it short circuit on other bits that are bound to
fail (definition, query_hash, etc) requiring persistence before we check
for a bound user

* first pass at LDAP setup improvements

* fix lint errors

* fix LDAP api tests

* WIP test for new setting setter

* fix setting test

* set ldap-enabled to true in ldap server macro

* try to fix java11 tests

* Update src/metabase/integrations/ldap.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* add transaction

* remove ldap-ever-enabled? setting and revert some of the logic that is no longer necessary

* set ldap-enabled via the ldap api and add tests

* fix tests and lint

* fix error on settings save

* fix cypress test

* actually fix cypress

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* initial prometheus sketch

need env variable to start: MB_PROMETHEUS_SERVER_PORT=9191

sample of output:

```
'# HELP jvm_threads_current Current thread count of a JVM
'# TYPE jvm_threads_current gauge
jvm_threads_current 81.0
'# HELP jvm_threads_daemon Daemon thread count of a JVM
'# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 36.0
'# HELP jvm_threads_peak Peak thread count of a JVM
'# TYPE jvm_threads_peak gauge
jvm_threads_peak 81.0
'# HELP jvm_threads_started_total Started thread count of a JVM
'# TYPE jvm_threads_started_total counter
jvm_threads_started_total 104.0
'# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
'# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
```

request:

```
❯ http localhost:9191/metrics
HTTP/1.1 200 OK
Content-Length: 7329
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Wed, 31 Aug 2022 16:13:38 GMT
Server: Jetty(9.4.48.v20220622)

'# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
'# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 41.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.586
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
'# HELP jvm_threads_current Current thread count of a JVM
'# TYPE jvm_threads_current gauge
```

* Log on unparseable prometheus port

* Clean up prometheus and save test

* typehint

* Jetty collector

* Reset system to nil when shutting down

* c3p0 stats

* Clean up, document, and add tests

* Error message for failure to bind to port

Starting up with prometheus port set to the main webserver port to get
the error:

```
MB_JETTY_PORT=3006 MB_DB_CONNECTION_URI="postgres://..." \
MB_PROMETHEUS_SERVER_PORT=3006 java -jar locally-built.jar
```
yields the following error:

```shell
2022-09-09 10:08:52,000 INFO metabase.core :: Setting up prometheus metrics
2022-09-09 10:08:52,002 INFO metabase.prometheus :: Starting prometheus metrics collector
2022-09-09 10:08:52,016 INFO metabase.prometheus :: Starting prometheus metrics web-server on port 3,006
2022-09-09 10:08:52,036 ERROR metabase.core :: Metabase Initialization FAILED
clojure.lang.ExceptionInfo: Failed to initialized Prometheus on port 3,006 {:port 3006}
[stacktrace ...]
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:3006
[stacktrace ...]
Caused by: java.net.BindException: Address already in use
```

* Test for error message

* Move prometheus to analytics folder

* Str port so log does not add commas to number

eg:
> Starting prometheus metrics web-server on port 9,191

* make some test functions private

* docstring on defsetting

* align lets

* ns docstring changes

include the env variable for ease of understanding

* sort ns

* remove some reflection warnings

* reorder FE lines due to some new linter

* Cleanup: i18n descriptions, typo, List/of

* bit more concise c3p0 collection

* no longer need the helper `->array`

* clean up doseq for c3p0 measurements

* Make table column width reasonable

Truncate column name that is longer than 16 characters

* Add a test to check header truncation of tables

Also adjusted the shape of the header fn's return data a bit -> the `[:th ... ]` vectors aren't wrapped in a seq anymore.

* Fix wrong max column character length 



* Update static viz table style

* Address PR comment, simplifying the code

Co-authored-by: Adam James <adam.vermeer2@gmail.com>

* Allow slugs in question template tags, like `{{#123-a-question}}`

* Update CardTagEditor to add slug names to the query text, instead of IDs

* Fix dispatchApiErrorEvent prop for entity object loader

* Ignore errors loading names of referenced questions

* Fix e2e tests

* Implement autocomplete for model slugs

* Prioritize models above questions

* Make the ace_autocomplete popup slightly wider to account for longer question names

* Reorder requires

* Remove unnecessary ? from endpoint string

* Add e2e test for card tag autocomplete

* Fix NativeQueryEditor not swapping back to standard completers

* Refactor replaceCardSlug

* Add e2e test for updating tags when card name is changed

* Refactor updateReferencedQuestionNames

* Add slug method to Question class

* Use map for autocomplete API results

* Rename endpoint path

* Rename for consistency

* Add docstrings

* Update slug() to guard against undefined names

* Fix read permissions for the endpoint

* Add backend test

* Fix unit tests

* Fix permissions error

* Make regex simpler

* Undo change of value on QuestionPicker

* Fix tests

* Fix models e2e test

* Simplify updateReferencedQuestionNames

* Fix bugs in updateReferencedQuestionNames

* Rename variable

* Move NativeQuery methods to new native-query utils module

* Fix typescript errors

* Fix e2e tests

* Fix e2e test

* Fix BE tests

* Actually fix BE test

* Refactor updateQuestionTagNames

* Align comments

* Reorder autocomplete results

* Refactor: move all query text tag utils in one place

* Add tests for regex

* Add unit tests for cardIdFromTagName

* Add schema to autocomplete_suggestions endpoints

* Add perms checks to autocomplete tests

* Add unit test for updateCardTagNames

* Rename updateQuestionTagNames -> updateCardTagNames

* Tidy card-autocomplete-suggestions-test

* Rename questionReference-> cardTag for consistency

* Rename card-reference -> card-tag for consistency

* Use ternary operator for comprehension

* Fix broken swapCompleters

* Use prefix match on name for 123-foo case

* thinking

* killing it

* lab image bundle for lines proper

* linter shutting up

* get rid of viz-settings binding at all

* Remove accidentally re-added categorical-area

* Merged master, and temporarily comment out some tests

* Simplify conditional for detecting chart type

* Remove a few old fns/symbols

* Clean up the detect-chart a bit

* Remove sparkline. It's an old concept that's not used

We can acheive similar results with :line directly. If we care about a trend, we can use a :smartscalar.

* Fix up tests that now also use combo charts

* Missed a key change for area test

* Get rid of deprecated sparkline stuff

* Remove commented out sparkline tests

* Delete this function. I was using it in the REPL, it's not needed

Co-authored-by: Adam James <adam.vermeer2@gmail.com>

* [Apps] Add scaffolded pages to existing apps

Reuse app scaffolding in order to add to existing apps. This will not
check that the tables you choose have not been previously added to the app.

Also update scaffolding:
- Add back to list button
- Update list question to order by newest row on top.
- Use linkType: "page" instead of linkType: "dashboard"

* Fix H2 test, no need to add app-id to name

* Remove unused require

* Addressing review comments - suggested test uncovered an ordering bug

* Put priority back to ascending

* Implement navigation between data app pages (#25366)

* Add `DataAppPagePicker`

* Rename `QuestionDashboardPicker`

* Add page to `EntityCustomDestinationClickBehavior`

* Fix comment

* Use ellipsis character

* Fix custom destination type

* Add "page" custom destination to pickers

* Pass router props to dashboard click actions

* Remove not used function arguments

* Implement "page" custom destination click behavior

* Fix "page" click behavior isn't treated as valid

* Reorganize `LinkedEntityPicker`

* Fix drills from the homepage

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

When creating a new question, you can choose an existing saved question
as your starting point. However, we want to exclude is_write cards from
that list.

* Support starting Metabase without initializing the Quartz scheduler

* Fix

* Show a basic banner if token status is "unpaid" or "past_due"

* Add warning emoji to banner message

* Swap unpaid and past-due meanings

* Make links consistent between messages

* Replace useLicense with token-status setting

* Add token-status to SettingName

* Extract AppBanner logic to selector and reuse Banner component

* Create TokenStatus type and createMockTokenStatus

* Remove hard-coded status

* Make token-status have internal visibility

* Revert "Make token-status have internal visibility"

This reverts commit 8983617a0f7f1e1a9e36206d93d871df5c290965.

* Write test for token-status setting

* Bring back AppBanner to render message correctly

* Test banner isn't present for normal users

* Address Alex's comments

* use .p12 file extension for pkcs-12 keys

* test

* fix messed up indentation

* default to empty string ssl password

* address feedback

* fix tests

* Respect custom series names in static viz

* Remove change that isn't actually necessary

* Revert "Remove change that isn't actually necessary"

This reverts commit 617ff4a9719bd5386bf69651cad696948a1fea7b.

* Remove uneeded change

* Add helper fn

* Add back a change I actually did need... brain fart, oops

* Add a test to make sure the correct :name key is used for series

* Remove unused function

* Add goal line settings to the viz-settings map passed to combo chart

* Add a test to check that goal lines are added to line/area/bar/combo

* [Apps] Prototype backend app scaffolding

WIP

Given a set of table-ids, we try to build out an app. Ideally the
produced scaffold is done on the frontend and passed in to

1. avoid code duplication
2. allow the front end to maintain ownership of visualization_settings
3. avoid regressions on frontend code changes

However, this currently builds the scaffold on the backend due to FE dev
bandwidth. In theory, it is done in such a way that it both matches the
FE as closely as possible as well as becomes easy to change this code to
accept a scaffold rather than generating one itself.

It uses `scaffold-target` to map cards and dashboards in the scaffold
with the inserted ids.

Things that are still being worked out elsewhere:
1. The shape of nav-items
2. The shape of implicit action buttons

* Sort namespace

* Update and fix scaffold based on demo

* Address review comments

Deduplicate table-ids and make sure they are valid.

i18n page suffixes.

Check that tables have exactly one primary key column.

Check that card scaffold has a `scaffold-target`

Remove redudant check in scaffold-target replacement that the map lookup took care of.

* [Apps] Allow extra parameters to be passed to action execution

For actions, unlike queries, the user will be asked to fill in unmapped
parameters. This change allows the dashboard action execution to accept
incoming parameters without an "id" but with a "target". If that target
exists on the action, it will be accepted.

* Update doc string and validation message

* Review changes

Remove action-id from the execute endpoint, this will be looked up
through the dashcard.

Revert map-parameters and split parameters into an extra_parameters
value. extra_parameters are combined with the mapped parameters and
passed to execution.

* Fixing tests

Hydrate app_id for dashboard activities

Part of #25254.

Only the /recent_views and /popular_items endpoints have been extended.
The / endpoint currently doesn't deliver additional information like
the other endpoints do. (The other endpoints deliver at most 5 items,
but the / endpoint doesn't limit the number of results.)

* adds schema based api error messages for maps

- e.g. in defendpoint.

see: the error for `metabase.api.action/HTTPActionTemplate`:

value must be a map with schema: (
  body (optional) : value may be nil, or if non-nil, value must be a string.
  headers (optional) : value may be nil, or if non-nil, value must be a string.
  parameter_mappings (optional) : value may be nil, or if non-nil, value must be a map.
  parameters (optional) : value may be nil, or if non-nil, value must be an array. Each value must be a map.
  method : value must be one of: `DELETE`, `GET`, `PATCH`, `POST`, `PUT`.
  url : value must be a string.
)

* use deferred-tru for error message

- fix spacing in api-error-message
- spruce up test

* sort ns

* swap another test to using str/join "\n"