* Adding results_metadata to qp middleware

This adds two middlewares to `metabase.query-processor.middleware.results-metadata`:
- `inject-result-metadata`: Adds `result_metadata` from the context, if present, into the query map during preprocessing.
- `merge-existing-metadata`: A post-processing middleware that merges `results_metadata` (if present) from the query data into the metadata.

This has the effect of preserving existing metadata, which is particularly important for user-curated metadata, such as semantic type overrides.

This may also facilitate addressing the following TODO in the same ns:

```
;; 1. Is there some way we could avoid doing this every single time a Card is ran? Perhaps by passing the current Card
;;    metadata as part of the query context so we can compare for changes
```

* Preserving result_metadata in query processor

This PR uses `qp.util/combine-metadata` to simplify the combination of provided `:result_metadata` with computed metadata. It also updates pulse rendering code to use this new code path and adds and updates tests to vet this logic.

* fmt

* Modifying PR to document and use existing fns

The current qp pipeline will correctly propagate custom metadata if a question is derived from a model or if the model is processed like so:

```clojure
(qp/process-query
  (assoc-in dataset_query [:info :metadata/dataset-metadata] result_metadata))
```

This PR fixes rendering unit tests to clarify this and provides a much better working example in the `dev.render-png` ns.

* Updating test

* Adding Pulse w/Metadata Tests

This pr adds a unit test that constructs a dashboard and simulates a pulse email, then checks the resulting HTML for correctness.

A dependency on the [hickory](https://github.com/clj-commons/hickory) library was added to dev. This allows us to easily parse HTML into data and navigate the parse tree.

Note that it does NOT check the attachments for formatting correctness.

Spelling and import consistency changes were also made.

* Adding Percent Metadata Type

This adds the :type/Percentage metadata type to Metabase. It does
not currently do any fingerprinting. It just enables setting of the
type for the FE.

* Making :type/Percentage a semantic type as well as a decimal.

* support Percentage setting in the semantic type editor, infer percentage column formatting default

* Support for static-viz rendering when user metadata is `:type/Percentage`

This PR adds in user-defined metadata in `prep-for-html-rendering` so that `number-formatter` can properly render the column as a percentage. This may actually fix a family of issues as user-defined metadata doesn't appear to be used anywhere in this ns. We still need to consider cases where the formatting is specified by the viz settings, but this one step in the right direction.

* Percent semantic types render as percents in tables

This PR adds logic to properly capture a percent semantic type and renders it as such in a table static viz. Note that the `number-formatter` function in `metabase.pulse.render.common` has logic that renders the percent as a truncated int. IDK if this the desired long term behavior. I'll have to ask product that, but if we just make the change it could have potential impacts on places that expect this behavior. It seems like it would make more sense to be like Excel and allow each column to set its significant digits.

* Revert "support Percentage setting in the semantic type editor, infer percentage column formatting default"

This reverts commit e001e6312e6f0bd5700c1d9eb28c88829d87a70a.

---------

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

Use `(dev.fe-helpers/current-query)` from a CLJS REPL to get the query currently open in the UI.

Thanks to @dpsutton for the help here! I investigated an alternative approach of reusing the machinery that we use in
tests to determine whether the database already exists, but that turned out to rely on a private atom that stored a
list of databases created by the tests. I didn't want to muck around with those internals, and it turned out to be
simpler than expected to make `can-connect?` work (although I did need to catch the exception thrown when we couldn't
connect, which seems a little odd - I would have expected `can-connect?` to catch that and return `false`).

Fixes https://github.com/metabase/metabase/issues/9962

* Fix #33083 (part 2)

* Equality nonsense

* Test fix

* Remove leftover test

* Cljs fix

* Appease Kondo

* Appease Kondo more

* PR feedback

* `index-of-matching-metadata`

* Update some other things to use `index-of-closest-matching-metadata`

* PR feedback

* Don't use metadata provider for ref <=> metadata comparisons

* mark-selected-columns doesn't need metadata provider

* Update some join stuff to match with metadata

* Update more stuff to use `closest-matching-metadata`

* Test fix 

* Test fix

* Take advantage of the QP now using MLv2 metadata providers to simplify test

* Well ok maybe we do need the 3-arity after all =(

* Test fix

* Remove stray NOCOMMIT

* Some cleanup and extra tests

* Hopefully the last few test fixes

* Ok NOW I think I fixed stuff.

* Add a test for binning display name (degrees)

* Fix indentation

* Give up on the Card nonsense for now

* Ok I give up

* Revert changes to testing

* QP: use Malli

* Fix schema names

* Fix missing namespaces

* Fix busted stuff

* Simplify the new schema

* Test fix 

* Test fixes 

* MongoDB fixes 

* FAST mu/defn

* Malli registry

* Misc fixes

* Some cleanup

* Misc cleanup and test fixes 

* mu/disable-enforcement

* Test fix

* Fix Cljs version of `disable-enforcement`

* Misc improvements

* Update tests

* Fix Kondo warnings

* Fix Kondo warnings

* Fix busted Malli schema

* PR feedback

* Fix busted schema

* Kondo fixx

* Test fixes 

* Use Malli version of `describe` now that PR was merged

* Work around https://github.com/metosin/malli/issues/924

* Test fixes 

* Improvements

* Test fix

* Squash

* Replace connection-with-timezone usages

* Add deprecation warning.

* Fix docstring

* Database to toucan2

* Table to toucan2

* Field to toucan 2

Before:
```
❯ clj -M:"$ALIASES"
WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.algo.generic.math-functions, being replaced by: #'clojure.algo.generic.math-functions/abs
operator.clj:172 recur arg for primitive local: sum is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: sum
Clojure 1.12.0-alpha2
user=> (do (dev) (start!))
Warning: environ value /Users/dan/.sdkman/candidates/java/current for key :java-home has been overwritten with /Users/dan/.sdkman/candidates/java/17.0.1-zulu/zulu-17.jdk/Contents/Home
2023-05-12 14:26:37,091 INFO metabase.util :: Maximum memory available to JVM: 8.0 GB
2023-05-12 14:26:47,248 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
..instrumented #'metabase.util.malli/with-api-error-message
..instrumented #'metabase.util.honey-sql-2/identifier
..instrumented #'metabase.util.honey-sql-2/normalize-type-info
..instrumented #'metabase.util.honey-sql-2/with-database-type-info
..instrumented #'metabase.util.honey-sql-2/cast
..instrumented #'metabase.util.honey-sql-2/quoted-cast
```

After:

```
❯ clj -M:"$ALIASES"
WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.algo.generic.math-functions, being replaced by: #'clojure.algo.generic.math-functions/abs
operator.clj:172 recur arg for primitive local: sum is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: sum
Clojure 1.12.0-alpha2
user=> (do (dev) (start!))
Warning: environ value /Users/dan/.sdkman/candidates/java/current for key :java-home has been overwritten with /Users/dan/.sdkman/candidates/java/17.0.1-zulu/zulu-17.jdk/Contents/Home
2023-05-12 15:11:21,678 INFO metabase.util :: Maximum memory available to JVM: 8.0 GB
2023-05-12 15:11:30,806 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
util.cljc:346 recur arg for primitive local: cumulative_byte_count is not matching primitive, had: Object, needed: long
Auto-boxing loop arg: cumulative-byte-count
WARNING: infinite? already refers to: #'kixi.stats.math/infinite? in namespace: kixi.stats.core, being replaced by: #'clojure.core/infinite?
2023-05-12 15:11:55,615 INFO driver.impl :: Registered abstract driver :sql  
```

* Tidy database-supports? defmethods

* Fix oracle build

* Fix driver

* Fix typo

* prefer driver/database-supports? instead of driver/supports?

* driver/supports? -> driver/database-supports? in tests

* Fix foreign-key support for sqlite in tests

* supports? -> database-supports?

* `supports?` -> `database-supports?`

* supports? -> database-supports?

* Update supports? docstring

* Fix tests

* Fix test

* Remove unused require

* Add mt/with-everything-store in tests

* fix test

* Fix tests

* Add note in driver changelog

* Don't load metabase.query-processor-test.test-mlv2 until we load QP

* Add schema and tests for `:value`

* Don't skip tests for #29895

* Conversion logic for `:value`

* pMBQL normalization should handle `:effective-type`

* Options should validate `:base-type`, `:effective-type`, etc. if present

* Use `:effective-type` for type checking if present

in query_processor.clj we were requiring

```clojure
(when-not config/is-prod?
  (classloader/require 'metabase.query-processor-test.test-mlv2))
```

And this was blowing up as it was not present in the aliases used for
cypress and the FE crew.

Now user.clj requires it in the same way that it requires
`[metabase.test-runner.assert-exprs]` so we can run tests with
`partial=` and their ilk. And it is required by the test entry namespace
so CI and test runners find it as well.

The `resolve` command in query_processor.clj now works in dev and CI

```clojure
query-processor=> (resolve 'metabase.query-processor-test.test-mlv2/post-processing-middleware)
,#'metabase.query-processor-test.test-mlv2/post-processing-middleware
```
(comma to prevent commit message from eliding as a comment)

so presumably tests that require the middleware tricks from this test
namespace will continue to work.

* Add QP test middleware to validate converted pMBQL queries against MLv2 schema

* Revert 'fix' for #29944

* Fix ups

* Hack

* Fix final test failure

* Remove unused namespace

* Fix Kondo warning

* Skip #29958

* convert db\/insert! to t2

* db/select-one-field -> t2/select-one-fn

* `db/select-field` -> `t2/select-fn-set`

* db/select-ids -> t2/select-pk-set

* db/select-field->field -> t2/select-fn->fn

* db/select-field->id -> t2/select-fn->pk

* `db/select-id->field` -> `t2/select-pk->field`

* add discourage var

* db/select-one -> t2/select-one

* db/select-one-field -> t2/select-one-fn

* add discourage var

* Mind = blown 

* Remove printls

* Simplify things

* Better exampls

* Update dev.clj

Adding a dev-level docker-compose file. Located in /dev and is self-documented.

* Log a warning when extraneous parameters are passed into a request

* Massive Metabase logging performance improvement: use configured log levels in logs API endpoint

---------

Co-authored-by: Cam Saul <github@camsaul.com>

insert next-gen permission paths alongside currently active permission paths + classification (#27911)

* implement move, which returns v2 paths

- TODO: insert these into the db

(move v1-path) => [v2 paths]

* cleanup + add some schemas

* generative tests 4 permission path classification

* whitespace lint

* detect data, query, and paths for v2

* calling move on v2 paths is a no-op

* differentiate between v1 and v2 permissions

quickchecking for move, classify-path, and classify-data-path

* fix tests + add idempotency test

* add tests for classification of permission paths

- rename move to ->v2-path
- move some fxns around
- ascii art in test

* making the legos line up

- need to insert both v1 and v2 versions of paths (of course)
- valid-path? has to allow v2 paths to be inserted

* replace mu/with-api-error-message

* linter + code quality fixes

* privatize rx->kind

* remove some changes that should be 4 anotherbranch

* revert ns

* delete v2 permissions in

- they aren't handled by the perm graph parser, so they don't get propagated into "old graph", so the diff between old and new indicates that they need to be rewritten.

* only delete v2 paths for the current group_id

- reorder declarations in models.permissions

* remove extra line break

* Update src/metabase/models/permissions.clj

fix typo

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Switch to the new Hawk test runner

* Fix some stuff

* Fix everything

* Commentz

* Bump init timeout?

* Fix Kondo config

* Remove unused

* Bump test runner version to get `:exclude-tags` support

Replaced, as needed, instances of str/lower|upper-case with the -en version. Also updated rules in our kondo config.

* Implicit action pre-fetch values (#25940)

* [Apps] Implicit action pre-fetch values

WIP To return a parameter-id: value map in order to prefill ui.

* Use jsonstring encoded parameters

* Change parameters exposed by implicit actions

Previously, we would look at a model's source-table to determine which
parameters were valid for implicit actions. So if an implicit action is
on a User model that hides the user.password column, you could still
update or insert the password.

With this commit, if the model doesn't expose a source table column, it
is not a valid parameter. In the example user.password could not be
inserted/updated or fetched for prefilling.

* Fixing tests

* Show data app page ID params in editing mode (#25943)

* Data apps layout updates (#25930)

* play with grid sizes to make apps feel appier [ci skip]

* move app nav to bar inside app

* only hide other header buttons outside of edit mode

* tweak overflow issue

* fix header width on data apps

* add control to see app contents

* set data apps layout constants contextually [ci skip]

* remove hardcoded value [ci skip]

* extract conextual header styles [ci skip]

* set max-width on paramaters for data apps [ci skip]

* move data apps nav deeper, only show if not editing [ci skip]

* add spacing to contents trigger, rm old code [ci skip]

* rm table thing for now [ci skip]

* Fixes for data app layout updates (#25962)

* Reorder import

* Fix type

* Fix missing translation, move out URL

* Remove not used import

* Rework `getSelectedItems` utility for data apps

* Fix selected data app page highlighting

* Don't reload app navbar on page change

* Change nav item links look

* Remove `DataAppPageSidebarLink`

* Turn nav into HTML list

* Use styled components

* Fix edit button covered by visualizations

* Fix opening homepage

* Remove redundant prop

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* First take at parameterized data app page titles (#25938)

* Bring `DataAppContext` back

* Pass `onFocus` and `onBlur` to `EditableText`

* Export `EditableTextProps`

* Add basic `DataAppPageTitle` component

* Use `DataAppPageTitle`

* Fix component name

* Add primitive autocompletion to page title input

* Add `title_template` to `DataAppNavItem` type

* Tweak value management in `DataAppPageTitle`

* Handle `null` values in titles more gracefully

* Don't show suggestions without detail cards

* Don't add whitespace when pasting a token

* Don't update app's collection all the time

* Add data app related selectors to dashboard selectors

* Add redux state to keep title template changes

* Update page title template on page save

* Wire up `DataAppPageTitle` with dashboard code

* Fix unit tests

* Reset state on exiting editing mode

* Add back button to template data picker

* Fix test that didn't include pk on insert

* Use the actual pk value

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>

* Revert "Remove data app permission code (#26016)"

This reverts commit b8c92cf1.

* Remove unused imports

* Don't require the table_id on result_metadata elements (#26073)

In some cases (which ones still has to be clarified) the result metadata doesn't
contain the table_id and so no parameters are provided. In fact the table ID is
probably not needed anyway because fields are already table specific.

* Support filtering cards by used model (#26088)

This change adds a new filter option for the GET /card endpoint.

* Support filtering apps using a given model (#26124)

* Handle type errors more cleanly (#26120)

* [Apps] Handle type errors more cleanly

Adds a number of tests for different db datatypes, these can be expanded
on as we solve.

Catches most exceptions during execution and logs then throws a generic
message that leaks less of the details.

* Make sure we order-by expected results

* Fix tests, allow purposeful exception through

* Accept model-ids for scaffolding apps (#26139)

* [Apps] Accept model-ids for scaffolding apps

WIP - accept model-ids but there's still some issues, might be able to
start building UI from this point though.

* Maybe working model

* Work with card__n table-ids passed in. Fix problems with detail page

* Dynamically add actions that exist on models

* Update tests for existing custom actions

* Can't def tru

* Address review comments

* Add required to action parameters (#26296)

Looks at `database_required` field values for implicit actions, and
`required` template-tags for custom query actions. Bubble those values
up onto `action.parameters`. Finally, look at `requires_pk` to see if pk
fields should be marked as required regardless.

* Cleanup scaffolded back button (#26317)

* Forward action execution error messages to the frontend (#26469)

This is a crude temporary solution necessary to be able to debug problems related to executing actions.
Although error parsing has been implemented for Postgres (only) with the goal to provide information for
the FE like `{"errors":{"id":"violates foreign key constraint note_subject_id_fkey"}}`. This code doesn't
care about the error message itself, it just forwards the error message from the DB to the FE.

* Actions for public pages (#26554)

* [Apps] Testing out public dashboards

* Add WIP endpoints for public execution

* Add throttle check to public execution route

* Bump attempts-threshold to 1000

* Bump threshold to 5000

* Add test

* Fix linter

* Review changes

* Fix lint

* Run publicly shared actions with full permissions (#26610)

* Support actions with MySQL and MariaDB (#26573)

Support actions with MySQL/MariaDB

* Check permissions for implicit actions (#26652)

* Check permissions when executing implicit actions

* Fix some action tests and reduce permissions boilerplate

* Revert "[Apps] Disable data-apps prior to release (#25982)"

This reverts commit 51147027.

* WIP: Unify action representation (#26717) (#27349)

* [Apps] WIP initial db migrations for giving implicit actions an action_id and removing card from query_action

* Make crud work for http actions

* Make implicit crud work

* Rename implicit_action.namespace to kind. Rewrite merged-model-actions to actions-with-implicit-parameters

* Fix query action creation

* Make actions.test-util/with-action create a holding model

* Fix unified-action-create-test

* Fix get-action-test

* Add tests for creating implicit and http dashcard actions

* Enable query action in dashcard-action-create-update-test

* Fix metabase.api.dashboard-test/dashcard-query-action-execution-test

* Fix metabase.api.dashboard-test/dashcard-http-action-execution-test

* Fix metabase.api.dashboard-test/dashcard-action-execution-auth-test

* Fix metabase.api.dashboard-test/dashcard-action-execution-granular-auth-test

* Make implicit action execution work

* Scaffolding implicit actions from table

* Fix metabase.api.action-test

* Make scaffolding work with existing models. Update with-action macro to allow multiple actions

* Fix metabase.api.public-test

* Remove unused names from metabase.models.action

* Remove some unused imports

* Remove an unused import

* Remove model_action and fix a bunch of tests

* Fix dashcard-implicit-action-only-expose-and-allow-model-fields

* Fix action type inheritance

* Fill in remarks and add description to action

* Add forward migration for existing actions and rollbacks that remove existing actions

* Prevent overwriting parameters of custom actions

* Support setting description on actions

* Update migration to work with mysql - rollback is still not quite working in mysql due to database_id fk

* Rename with-action macro to with-actions and document it

* Remove report_card.is_write

* Fix some more tests

* Fix rollback across all appdb drivers, add not null constraints, use action.name to hold implicit slugs

* Pull name and description from is_write card

* Add name to scaffolded implicit actions

* Remove ModelAction

* Remove is_write from comments too

* Different databases reference capture groups in regexp_replace differently. Also drop model_action table

* Update migrations to pass linter.

Introduce ${capturegroup.ref} to handle the different ways to reference
capture groups within a regexp_replace. This allows collapsing large
data migrations into a single `sql` element that the linter requires.

* Add kind to action-mapper so we can sort as we wish

* Add name to with-temp Action

* Adjust FE to query actions migration (#26764)

* Remove `is_write` card notion

* Remove action `slug` from action types

* Fix typo in `estimateCardSize` name

* Remove hanging card's `is_write` prop

* Remove not used writeback actions

* Update endpoints

* Remove not used prop

* Remove action `slug` usage

* Update actions entity

* Tweak action creator to handle new actions shape

* Fix saving action dashcards

* Fix actions API usage

How did that work before? 



* Pass `database_id` when creating query actions

* Remove custom list API implementation for actions

* Clean up types

Co-authored-by: Tamás Benkő <tamas@metabase.com>

* Change regexp_replace to substring to support mysql - this is ok because we have consistent json serialization.

* Check permissions on the model in the action API

* Remove unused required namespace

* Throw an exception when demoting a model with actions to question

* Allow saved parameters on implicit actions to add to implicit parameters

Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Case Nelson <case@metabase.com>

* empty commit to kick CI

* bump CI

* Fix down migrations

 change v45.00-002 sets up a fk fk_query_action_ref_action_id. h2 v2 for
 some strange reason requires us to remove the constraint before we can
 remove the primary key. But then we have to restore the state that
 v45.00-002 creates manualy

* Allow for version for `(migrate! :down 45)`

* Remove data apps parts not planned for 0.46

Removed app container, app scaffolding and pages (dashboards with is_app_page)

* Address linter issues

* Make some cosmetic changes to reduce the diff

* Make new permission API endpoints use defendpoint-schema

* Fix bad automerge

* Skip Mongo 4.2 instead of 4.0

Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>
Co-authored-by: dan sutton <dan@dpsutton.com>

- add the rest of malli reserved aliases
- malli.util :as [mu -> mut] (had collision with metabase.util.malli)

* Make Dimension unique on `field_id` and deduplicate

* Fix rollback for new migrations

* Don't require migration comments to contain 'added' anymore since it's part of the version ID now

* Don't require 'Added <version>' in migration comments anymore

* Fix SerDes test that created duplicate dimensions for one Field

* Silly fix to fix MySQL 5.7

* add the library, and dev/start-malli!

* Update deps.edn

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* enable the use of regexes as malli schemas

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>