* remove toc generation from api docs

* update api docs

* update test

* Don't use persisted model tables for segmented users

This actually isn't a bug, but due to very subtle and arbitrary reasons.

For background about why we need to ensure this never happens, we cannot
use persisted models when sandboxing is at play. A simple example is as
follows: make a model on a products table that does not select the
category. Have a sandbox on category such that someone can only see
products of category "Gizmo". the model lacks the category column but we
insert a where clause that still works. When the model is persisted,
there is no category column in the underlying table so sandboxing cannot
possibly work: the data necessary to filter is no longer associated with
the rest of the data in the model.

The fix for this is quite simple: in
`metabase.query-processor.middleware.fetch-source-query` we only splice
in the persisted query if the user is not a segmented user (product name
for sandboxing).

```clojure
(and persisted-info/*allow-persisted-substitution*
     (not (segmented-user?))  ;; <----- new check
     (:active card)
     (:definition card)
     (:query_hash card)
     (= (:query_hash card) (persisted-info/query-hash (:dataset_query card)))
     (= (:definition card) (persisted-info/metadata->definition (:result_metadata card)
                                                                (:table_name card)))
     (= (:state card) "persisted"))
```

Technical details about why this bug did not manifest

When swapping out a card__<id> to a source query, if its a model we will
see if it is persisted, and if so, we will use the native sql to select
from the persisted table. It does this by adding the native sql at a key
called `:persisted-info/native` and a middleware
`#'qp.persistence/substitute-persisted-query` walks the query replacing
the query with the native:

```clojure
;; metabase.query-processor.middleware.persistence

    (mbql.u/replace query
      (x :guard (every-pred map? :persisted-info/native))
      {:native (:persisted-info/native x)})
```

There is also a middleware that walks through the query looking for
tables with gtaps on them and replacing them. By change, the sandboxing
middleware runs immediately before the substitute-persisted middleware!

```clojure
   ;; literally the previous middleware
   (resolve 'ee.sandbox.rows/apply-sandboxing)
   #'qp.persistence/substitute-persisted-query
```

If you swap the order of these two sandboxing is broken. As is, it
"works" but not by design, just by happenstance. The sandboxing
middleware just did not know that the `:persisted-info/native` key meant
that a native query was to be substituted. In the reverse order, the
native query is already substituted and there is no change for the
sandboxing to occur.

The obvious fix is to ensure that we never even attempt to use the
persisted tables and that is what this PR does.

* Eastwood doesn't like shadowing like this

* Rearrange check order for tests

`segmented-user?` throws if there is no bound user. A test in
`fetch-source-query-test` was failing because there was no user bound,
but it wasn't attempting to swap out a persisted table, it just didn't
expect to need a user.

Moving it lower lets it short circuit on other bits that are bound to
fail (definition, query_hash, etc) requiring persistence before we check
for a bound user

* first pass at LDAP setup improvements

* fix lint errors

* fix LDAP api tests

* WIP test for new setting setter

* fix setting test

* set ldap-enabled to true in ldap server macro

* try to fix java11 tests

* Update src/metabase/integrations/ldap.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* add transaction

* remove ldap-ever-enabled? setting and revert some of the logic that is no longer necessary

* set ldap-enabled via the ldap api and add tests

* fix tests and lint

* fix error on settings save

* fix cypress test

* actually fix cypress

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* initial prometheus sketch

need env variable to start: MB_PROMETHEUS_SERVER_PORT=9191

sample of output:

```
'# HELP jvm_threads_current Current thread count of a JVM
'# TYPE jvm_threads_current gauge
jvm_threads_current 81.0
'# HELP jvm_threads_daemon Daemon thread count of a JVM
'# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 36.0
'# HELP jvm_threads_peak Peak thread count of a JVM
'# TYPE jvm_threads_peak gauge
jvm_threads_peak 81.0
'# HELP jvm_threads_started_total Started thread count of a JVM
'# TYPE jvm_threads_started_total counter
jvm_threads_started_total 104.0
'# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
'# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
```

request:

```
❯ http localhost:9191/metrics
HTTP/1.1 200 OK
Content-Length: 7329
Content-Type: text/plain; version=0.0.4; charset=utf-8
Date: Wed, 31 Aug 2022 16:13:38 GMT
Server: Jetty(9.4.48.v20220622)

'# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
'# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 41.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.586
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
'# HELP jvm_threads_current Current thread count of a JVM
'# TYPE jvm_threads_current gauge
```

* Log on unparseable prometheus port

* Clean up prometheus and save test

* typehint

* Jetty collector

* Reset system to nil when shutting down

* c3p0 stats

* Clean up, document, and add tests

* Error message for failure to bind to port

Starting up with prometheus port set to the main webserver port to get
the error:

```
MB_JETTY_PORT=3006 MB_DB_CONNECTION_URI="postgres://..." \
MB_PROMETHEUS_SERVER_PORT=3006 java -jar locally-built.jar
```
yields the following error:

```shell
2022-09-09 10:08:52,000 INFO metabase.core :: Setting up prometheus metrics
2022-09-09 10:08:52,002 INFO metabase.prometheus :: Starting prometheus metrics collector
2022-09-09 10:08:52,016 INFO metabase.prometheus :: Starting prometheus metrics web-server on port 3,006
2022-09-09 10:08:52,036 ERROR metabase.core :: Metabase Initialization FAILED
clojure.lang.ExceptionInfo: Failed to initialized Prometheus on port 3,006 {:port 3006}
[stacktrace ...]
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:3006
[stacktrace ...]
Caused by: java.net.BindException: Address already in use
```

* Test for error message

* Move prometheus to analytics folder

* Str port so log does not add commas to number

eg:
> Starting prometheus metrics web-server on port 9,191

* make some test functions private

* docstring on defsetting

* align lets

* ns docstring changes

include the env variable for ease of understanding

* sort ns

* remove some reflection warnings

* reorder FE lines due to some new linter

* Cleanup: i18n descriptions, typo, List/of

* bit more concise c3p0 collection

* no longer need the helper `->array`

* clean up doseq for c3p0 measurements

* Make table column width reasonable

Truncate column name that is longer than 16 characters

* Add a test to check header truncation of tables

Also adjusted the shape of the header fn's return data a bit -> the `[:th ... ]` vectors aren't wrapped in a seq anymore.

* Fix wrong max column character length 



* Update static viz table style

* Address PR comment, simplifying the code

Co-authored-by: Adam James <adam.vermeer2@gmail.com>

* Allow slugs in question template tags, like `{{#123-a-question}}`

* Update CardTagEditor to add slug names to the query text, instead of IDs

* Fix dispatchApiErrorEvent prop for entity object loader

* Ignore errors loading names of referenced questions

* Fix e2e tests

* Implement autocomplete for model slugs

* Prioritize models above questions

* Make the ace_autocomplete popup slightly wider to account for longer question names

* Reorder requires

* Remove unnecessary ? from endpoint string

* Add e2e test for card tag autocomplete

* Fix NativeQueryEditor not swapping back to standard completers

* Refactor replaceCardSlug

* Add e2e test for updating tags when card name is changed

* Refactor updateReferencedQuestionNames

* Add slug method to Question class

* Use map for autocomplete API results

* Rename endpoint path

* Rename for consistency

* Add docstrings

* Update slug() to guard against undefined names

* Fix read permissions for the endpoint

* Add backend test

* Fix unit tests

* Fix permissions error

* Make regex simpler

* Undo change of value on QuestionPicker

* Fix tests

* Fix models e2e test

* Simplify updateReferencedQuestionNames

* Fix bugs in updateReferencedQuestionNames

* Rename variable

* Move NativeQuery methods to new native-query utils module

* Fix typescript errors

* Fix e2e tests

* Fix e2e test

* Fix BE tests

* Actually fix BE test

* Refactor updateQuestionTagNames

* Align comments

* Reorder autocomplete results

* Refactor: move all query text tag utils in one place

* Add tests for regex

* Add unit tests for cardIdFromTagName

* Add schema to autocomplete_suggestions endpoints

* Add perms checks to autocomplete tests

* Add unit test for updateCardTagNames

* Rename updateQuestionTagNames -> updateCardTagNames

* Tidy card-autocomplete-suggestions-test

* Rename questionReference-> cardTag for consistency

* Rename card-reference -> card-tag for consistency

* Use ternary operator for comprehension

* Fix broken swapCompleters

* Use prefix match on name for 123-foo case

* thinking

* killing it

* lab image bundle for lines proper

* linter shutting up

* get rid of viz-settings binding at all

* Remove accidentally re-added categorical-area

* Merged master, and temporarily comment out some tests

* Simplify conditional for detecting chart type

* Remove a few old fns/symbols

* Clean up the detect-chart a bit

* Remove sparkline. It's an old concept that's not used

We can acheive similar results with :line directly. If we care about a trend, we can use a :smartscalar.

* Fix up tests that now also use combo charts

* Missed a key change for area test

* Get rid of deprecated sparkline stuff

* Remove commented out sparkline tests

* Delete this function. I was using it in the REPL, it's not needed

Co-authored-by: Adam James <adam.vermeer2@gmail.com>

* [Apps] Add scaffolded pages to existing apps

Reuse app scaffolding in order to add to existing apps. This will not
check that the tables you choose have not been previously added to the app.

Also update scaffolding:
- Add back to list button
- Update list question to order by newest row on top.
- Use linkType: "page" instead of linkType: "dashboard"

* Fix H2 test, no need to add app-id to name

* Remove unused require

* Addressing review comments - suggested test uncovered an ordering bug

* Put priority back to ascending

* Implement navigation between data app pages (#25366)

* Add `DataAppPagePicker`

* Rename `QuestionDashboardPicker`

* Add page to `EntityCustomDestinationClickBehavior`

* Fix comment

* Use ellipsis character

* Fix custom destination type

* Add "page" custom destination to pickers

* Pass router props to dashboard click actions

* Remove not used function arguments

* Implement "page" custom destination click behavior

* Fix "page" click behavior isn't treated as valid

* Reorganize `LinkedEntityPicker`

* Fix drills from the homepage

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

When creating a new question, you can choose an existing saved question
as your starting point. However, we want to exclude is_write cards from
that list.

* Support starting Metabase without initializing the Quartz scheduler

* Fix

* Show a basic banner if token status is "unpaid" or "past_due"

* Add warning emoji to banner message

* Swap unpaid and past-due meanings

* Make links consistent between messages

* Replace useLicense with token-status setting

* Add token-status to SettingName

* Extract AppBanner logic to selector and reuse Banner component

* Create TokenStatus type and createMockTokenStatus

* Remove hard-coded status

* Make token-status have internal visibility

* Revert "Make token-status have internal visibility"

This reverts commit 8983617a0f7f1e1a9e36206d93d871df5c290965.

* Write test for token-status setting

* Bring back AppBanner to render message correctly

* Test banner isn't present for normal users

* Address Alex's comments

* use .p12 file extension for pkcs-12 keys

* test

* fix messed up indentation

* default to empty string ssl password

* address feedback

* fix tests

* Respect custom series names in static viz

* Remove change that isn't actually necessary

* Revert "Remove change that isn't actually necessary"

This reverts commit 617ff4a9719bd5386bf69651cad696948a1fea7b.

* Remove uneeded change

* Add helper fn

* Add back a change I actually did need... brain fart, oops

* Add a test to make sure the correct :name key is used for series

* Remove unused function

* Add goal line settings to the viz-settings map passed to combo chart

* Add a test to check that goal lines are added to line/area/bar/combo

* [Apps] Prototype backend app scaffolding

WIP

Given a set of table-ids, we try to build out an app. Ideally the
produced scaffold is done on the frontend and passed in to

1. avoid code duplication
2. allow the front end to maintain ownership of visualization_settings
3. avoid regressions on frontend code changes

However, this currently builds the scaffold on the backend due to FE dev
bandwidth. In theory, it is done in such a way that it both matches the
FE as closely as possible as well as becomes easy to change this code to
accept a scaffold rather than generating one itself.

It uses `scaffold-target` to map cards and dashboards in the scaffold
with the inserted ids.

Things that are still being worked out elsewhere:
1. The shape of nav-items
2. The shape of implicit action buttons

* Sort namespace

* Update and fix scaffold based on demo

* Address review comments

Deduplicate table-ids and make sure they are valid.

i18n page suffixes.

Check that tables have exactly one primary key column.

Check that card scaffold has a `scaffold-target`

Remove redudant check in scaffold-target replacement that the map lookup took care of.

* [Apps] Allow extra parameters to be passed to action execution

For actions, unlike queries, the user will be asked to fill in unmapped
parameters. This change allows the dashboard action execution to accept
incoming parameters without an "id" but with a "target". If that target
exists on the action, it will be accepted.

* Update doc string and validation message

* Review changes

Remove action-id from the execute endpoint, this will be looked up
through the dashcard.

Revert map-parameters and split parameters into an extra_parameters
value. extra_parameters are combined with the mapped parameters and
passed to execution.

* Fixing tests

Hydrate app_id for dashboard activities

Part of #25254.

Only the /recent_views and /popular_items endpoints have been extended.
The / endpoint currently doesn't deliver additional information like
the other endpoints do. (The other endpoints deliver at most 5 items,
but the / endpoint doesn't limit the number of results.)

* adds schema based api error messages for maps

- e.g. in defendpoint.

see: the error for `metabase.api.action/HTTPActionTemplate`:

value must be a map with schema: (
  body (optional) : value may be nil, or if non-nil, value must be a string.
  headers (optional) : value may be nil, or if non-nil, value must be a string.
  parameter_mappings (optional) : value may be nil, or if non-nil, value must be a map.
  parameters (optional) : value may be nil, or if non-nil, value must be an array. Each value must be a map.
  method : value must be one of: `DELETE`, `GET`, `PATCH`, `POST`, `PUT`.
  url : value must be a string.
)

* use deferred-tru for error message

- fix spacing in api-error-message
- spruce up test

* sort ns

* swap another test to using str/join "\n"

* Support filtering by "app" and "page" models
* Fetch pages instead of dashboards for app sidebar
* Fix pages display

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* Rename DashboardCard `sizeX` and `sizeY` to `size_x` and `size_y` (#16344)

* Fix migration

* Fix MySQL/MariaDB

Fixes #25213.

* Return app collections as apps, page dashboards as pages
* Fix bug: descriptions of datasets should also be searchable

* Send show_values key to static-viz js

The static viz components expect certain keys on the settings map passed into the js. At this time, the contract
between the backend (Clojure) and the static-viz (js) feels a little under-specified. However, for the change to XY
Charts here, we follow the existing method in code of checking the backend viz-settings map for some key, and passing
it into js in a 'simplified' form. For instance, in this change, `:graph.show-values` is taken and placed into the
settings map as `:show_values` for the static-viz js to use.

* De-dupe key

* Keep this PR super simple, don't eliminate sparkline stuff yet

* Add :line earlier in cond, try to fix tests that fail

* Get the :line conditional in the right place, allowing :scalar

* Format Pie Chart Legend Labels as Month, YYYY

On the backend, the pie chart legend is not rendered in the javascript, it is instead re-created server side as a set
of html elements. This means "YYYY-MM-DD" formatted dates do not get passed through the frontend code responsible for
formatting date times (I believe this is done with a library called 'moment.js'). And, since the SVG cannot contain
html foreignObjects (the backend SVG renederer 'batik' ignores foreign objects), the legend is left as a
responsiblility for Clojure land.

This PR adds a `format-month` function to handle this case.

Current caveats with this approach:

- doesn't consider localization (yet)
- adjusted the html to use a table instead of a div full of spans and the formatting could use some tweaks
- assumes any string that is parse-able into a Jave DateTime object will be displayed in this Month, Year format,
- which might not be true in all cases

* Improve label formatting by using viz-settings transform fns

* Rework Static Pie Chart legend

Try to make it look a bit nicer:
- splits legends into 2 tables if there are more than 8 entries
- larger colored circle
- use tables to help lay things out more evenly (CSS flexbox doesn't seem to work)
- handle some legend label formatting cases beyond just timestamps (eg weeks of year, yearly quarters)

Also fixed the failing test

* Adding a month formatter.

Sometimes, the label values are not passed in as timestamps, but just numbers (as strings). In these cases, the
Datetime formatters won't be useful, so there are some cases built to handle these.

* Remove print statement

* Change temporal format fns to more closely follow viz-settings

Various changes are made to improve static-viz's use of viz-settings from the frontend.

- abbreviation is considered (eg. so "January" is rendered as "Jan" when setting is true)
- abbreviation also works for Days of week
- x-of-y cases are handled a bit better
- separator settings are properly reflected (eg. 2022/05 becomes 2022-05 if separator changed to "-")

* Remove unused key from the let

* Fix some failing tests

* Adding more tests

* Prevent non-temporal legend labels from being rendered via datetime

If a legend's label is some string, we want to pass it unmodified. If it's a number or a timestamp, we can pass it
through to the date time render function to format the string to match the viz-settings/column units

* Add missed docstring

* Don't lowercase 'D' format, pass correct lowercase 'd' before

'D' format = day of year, which in most human-readable formats is not what we want, but it IS what we want in a
:day-of-year case.

Instead of always lowercasing, just make sure the lowercase 'd' is in the defaults/overrides

* Better way to handle day of week and month of year

Also got rid of bad use of read-string by using parse-long

* Use a java lib. to format ordinal numbers, respecting site locale

Fixes #25143 and #25157.

For recent_views and popular_items hydrate the is_app_page flag for dashboards.

* Add `serdes-descendants` for "containment" to serialize a subtree

This allows naming eg. a Collection and will recursively serialize: all
dashboards, cards and dashcards it contains directly, plus all child
collections and everything they contain.

Currently this words on Collection, Dashboard, DashboardCard, and Card.

* lint

* Switch to plural `extract-subtrees` with a list of `targets`

Hydrate app_id on collections in the search endpoint

See #25097 for context.

* Remove sort-by first from combo chart rendering

The card query (seems to) return the rows in a vector in the same order presented on the frontend UI. Having the row
selection function sort the selected rows by first will always just sort things alphabetically. This may be fine in
some generic case, but can break in situations where a user has created a query with a sort on some quantity.

* Fix test

* [Apps] Hydrate action on dashcards

It is useful for the frontend if we hydrate :action in the same way that
we hydrate :card on :ordered_cards.

* Add docstring

* [Apps] Hydrate action_id on is_write Card

It'll be helpful for the frontend if we send back the created action_id
when saving an `is_write` Card.

Also update the `api/card` tests to ensure that a QueryAction (and by
fk constraints the Action) is created/deleted as expected when
changing `is_write`.

* Add extra arg to more result-fns

Like `parameters`, it is desirable to use an `is_write` card's
visualization_settings to store information for form customizations.
When fetching an action, pull up the card's visualization_settings to
the root of action, later http actions can add a visualization_settings
field and they can be unified with query actions in the same way that
wwe do parameters.

* Add disable-max-results? to query middleware options when refreshing cached card results

* Add docstrings

* Add test for `add-default-limit`

* [App] Add action_id to dashcard and endpoint for execution

With app forms, we will be inlining action execution. So this PR is in
preparation for that inlining and removing the need for createing
Emitters in order to execute actions.

Mirrors card querying from dashcards. This expects parameter_mappings
of the dashboardcard to hold the mappings for execution, in the same
shape as other dashcards. Instead of referencing a card_id, we reference
an action_id.

Emitters may be removed entirely in a future PR.

* Fix extra require lint

* Add checks for superuser and feature enabled to execution route. Fix code review comments

* Add missing comment to migration