* Metabase lib PoC RFC [ci skip]

* Cleanup [ci skip]

* `:lib/uuid` overhaul [ci skip]

* More tests [ci skip]

* Basic join support [ci skip]

* Simplify a few things, more tests, etc. [ci skip]

* Pipeline queries FTW [ci skip]

* Metabase® Lib™ v2

* Fix cljs test assertions

* No bees please

* Update docstring

* Mega simplification

* Fix Kondo warnings

* Address PR feedback

* Fix Malli util for Cljc

* Oops, `instrument!` should be public

* Cloverage do not instrument JS util namespace

* Fix Cloverage again

* Don't count MySQL or Postgres namespaces for Coverage

* Address PR feedback

* Move dispatch value keyword function to util

* Revert acccidental require

* Test fix

* Fix Cljs =?

* Fix totally unrelated broken links

* expose number of active users as a setting

* Fix static-viz render with black background when values > 1000

* fix to work with any separator settings

* Adds tests for chain filters security issue

* add another test

* Users with collection permissions can view chained filters for dashboard

- this obeys sandboxing

* Users with collection access should be allowed to view chain filters

* add test + fix case for block filters

* check with premium features

* check premium features + revert with-premium-features in tests

* remove block check

* respond to review comments

* ensure data-perms are there + check blocked

* rewrite e2e test so that it actually tests chain filters

* properly toggle linked filters on

- don't use random classes, etc

* rename *param-values-query* dynamic var

* update chain-filter docstring

* token-features should still be private

* initial implementation of query_execution retention policy

* fix logic and add tests for setting

* refactor and more tests

* set default on cloud to 0

* try to fix test by loading ee ns if available

* address feedback and small refactors

* interface file for truncate-audit-log

* extract common definitions into an interface to avoid cyclic dependency

* fix indentation

* Ensure query action's db matches model id

This is only a half-way measure. Needs some FE loving as well.

The +New -> Action workflow lets you create an action. This starts with
the user choosing the database, writing the action. On save it prompts
you to choose a model it should live under. But this model's db and the
db chosen originally are not checked in the FE.

Ideally we only show models on that existing db, but it kinda seems
weird to have an action do things to a random table and it live under a
model that just needs to be on the db.

* reword error message

Co-authored-by: Maz Ameli <maz@metabase.com>

* Check both databases for enabled actions when they disagree

a custom action can have its own query which brings its own
database. When creating a new custom action from the models detail page,
these go hand in hand. When using the workflow from the "+New" button,
the model is chosen at the last step, well after the action's db and
query is set. And there's nothing holding these two together.

Asked if this was a bug and was told no. But we want to make sure we
check both for actions enabled.

* fix lint

---------

Co-authored-by: Maz Ameli <maz@metabase.com>

* Delay substituting persisted native query until after analysis

We were substituting persisted queries into the query at too early a
stage.

Card 729 is a persisted model of `select * from categories`

```clojure
{:type :query,
 :query {:source-table "card__729",
         :expressions {:adjective ["case"
                                   [[[">" ["field" 100 nil] 30]
                                     "expensive"]
                                    [[">" ["field" 100 nil] 20]
                                     "not too bad"]]
                                   {:default "its ok"}]},
         :aggregation [["count"]],
         :breakout [["expression" "adjective" nil]
                    ["field" 101 nil]]},
 :database 3,
 :parameters []}
```

This would immediately get turned into an equivalent

```clojure
{:type :query,
 :query {:source-table 12
         :persisted-info/native "select \"id\", \"ean\", \"title\", \"category\", \"vendor\", \"price\", \"rating\", \"created_at\" from \"metabase_cache_424a9_3\".\"model_729_products_r\""
         :expressions {:adjective ["case"
                                   [[[">" ["field" 100 nil] 30]
                                     "expensive"]
                                    [[">" ["field" 100 nil] 20]
                                     "not too bad"]]
                                   {:default "its ok"}]},
         :aggregation [["count"]],
         :breakout [["expression" "adjective" nil]
                    ["field" 101 nil]]},
 :database 3,
 :parameters []}
```

And the obvious failure here is the use of `[:field 100 nil]` (price)
against an inner query of `select ... price ... from cache_table`. This
could break many queries and caused a flood of

```
WARN middleware.fix-bad-references :: Bad :field clause [:field 100 #:metabase.query-processor.util.add-alias-info{:source-table :metabase.query-processor.util.add-alias-info/source, :source-alias "price"}] for field "products.price" at [:expressions "adjective" :case :>]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
WARN middleware.fix-bad-references :: Bad :field clause [:field 100 #:metabase.query-processor.util.add-alias-info{:source-table :metabase.query-processor.util.add-alias-info/source, :source-alias "price"}] for field "products.price" at [:expressions "adjective" :case :>]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
```

* forgot to commit new test file

* Handle persisted before native

* Add test for native models

- have to populate the metadata. You can do this by running the query,
but there are some requirements around the `:info` object to actually
save it. So just take the more mechanical route
- `wait-for-result` returns a time out value, but it's namespaced
autoresolved to a namespace inside of the metabase.test forest and not
mt itself. So easy to mess up and watch for the wrong keyword. Want an
arity so i can check my own timeout value and throw a helpful error
message. I checked around and noone seems to care. So not great, but at
least people aren't checking `::mt/timed-out` when it should be
`::tu.async/timed-out` or
`:metabase.test.util.async/timed-out`. Possibly best scenario is we
remove the autoresolved bit and have it return `:async/timed-out` or
something stable.

* properly compile the native query

* bump timeout to set redshift metadata

Cleans/fixes serdes v2 export :collections handling in cmd and API

Dev experience:
- CLJS now has source maps
- Devtools extension renders CLJS data in Chrome (et al) Devtools
- Use a debug build of CLJS in the tests
- Fixed line numbers and breakpoints being off-by-one in Devtools

CLJC infra:
- More robust direct approach to JS<->CLJS conversions, powered by
  Malli transformers.
- Powerful `define-getters-and-setters` macro to generate the accessors.

Added post-collection retrieval function `remove-other-users-personal-collections` that will filter out any personal collection that doesn't belong to the current user when the `exclude-other-user-collections=true` path param is used. Also added unit tests for both the `api/collection/` and `api/collection/tree` endpoint.

* Add comment for database-enable-actions

* Enforce actions not supported implies actions are disabled

* Remove test and update check

* Fix typos

* Revert "Remove test and update check"

This reverts commit 8b1f14f44bd97514cd8ce0c9b0a5df048d713b16.

* Fix test

* Remove json fields from action parameters

```sql
json-test=# select * from json_table;
     json_val     |  description
------------------+---------------
 {"a": 1, "b": 2} | a description
(1 row)
```

This ends up in our db as:

```clojure
({:database_type "json",
  :name "json_val",
  :effective_type :type/Structured,
  :nfc_path nil,
  :id 73,
  :display_name "Json Val",
  :database_required true}
 {:database_type "bigint",
  :name "json_val → a",
  :effective_type :type/Integer,
  :nfc_path ("json_val" "a"),
  :id 72,
  :display_name "Json Val → A",
  :database_required false}
 {:database_type "bigint",
  :name "json_val → b",
  :effective_type :type/Integer,
  :nfc_path ("json_val" "b"),
  :id 74,
  :display_name "Json Val → B",
  :database_required false}
 {:database_type "text",
  :name "description",
  :effective_type :type/Text,
  :nfc_path nil,
  :id 13670,
  :display_name "Description",
  :database_required false})
```

AKA, exploded. But we cannot enter values for these fields for implicit
actions. Before this change, the action thought its parameters were:

```clojure
 :parameters ({:id "json_val",
               :target [:variable [:template-tag "json_val"]],
               :type :type/Structured,
               :required true}
              {:id "json_val_%E2%86%92_a",
               :target [:variable
                        [:template-tag "json_val_%E2%86%92_a"]],
               :type :type/Integer,
               :required false}
              {:id "json_val_%E2%86%92_b",
               :target [:variable
                        [:template-tag "json_val_%E2%86%92_b"]],
               :type :type/Integer,
               :required false}
              {:id "description",
               :target [:variable [:template-tag "description"]],
               :type :type/Text,
               :required false})
```

And we just don't know how to handle the exploded json values in the
insert statement to create a nested json structure. So lets omit
them. After this change:

```clojure
action=> (:parameters (select-action :id 14))
({:id "json_val",
  :target [:variable [:template-tag "json_val"]],
  :type :type/Structured,
  :required true}
 {:id "description",
  :target [:variable [:template-tag "description"]],
  :type :type/Text,
  :required false})
```

* Translate `:type/Structured` to JSON for pg

Fields have effective_type, and base_type `:type/Structured`. But
several other types get marked as this as well:

```clojure
actions=> (descendants :type/Structured) -> #{:type/SerializedJSON :type/XML :type/JSON}
```

So for the purposes of this, we _assume_ that structured is synonymous
with json. Certainly will error if its xml. This makes the implicit
action cast the text input for a json field to json. If the underlying
field in pg is jsonb the driver cleans that up for us with no error.

* Remove json fields, exploded json fields, unrecognized type fields

In order to do this, had to change how we load json fields in test
datasets for postgres. It just stuffed them in as text. Mysql would
correctly make them json fields but postgres just bailed and left them
as text strings.

This changes `load-data/load-data!` to look for json columns, and if so,
cast them to json when loading.

* Remove json and xml checks from dashboard execution tests

these are not doing what you think they are doing. they inspire
confidence that json works but the underlying fields are actually always
text. So it's unfounded confidence; misleading confidence.

If you fix this, you quickly run into issues with expressivity in
parameters and differences in dbs. A few:
- no parameter on a dashboard can express that its type is json
- mysql doesn't have an xml type
- postgres does have an xml type, and you get back a native PGXML type
- since you can't have an implicit type in the parameter, each sql has
to cast in its query. And then you start getting differences in return
types, queries, and it stops making sense for this all to be in the same
query.

And at the end of the day, of course you can write a query that casts a
string to some native type. That's not really action related.

* Let's not do this yet

i think this is the correct call but want more tests on it when we flip
the switch

* Test pg enums in implicit actions

* alignment and unused binding

* Exclude in maria

maria doesn't see the json_bit field as a json column. It's just a text
column. So it is not excluded and all of the exploded fields like
"json_bit → 1234" are not extracted. So this test is not applicable
because it's just a generic implicit action on three fields.

* accept suggestions from tamas

* fixes nil schema being passed into ->malli-io-link

- happens when function args aren't annotated.

* (u/pprint-to-str nil) should be "nil\n", not nil

* Mark `hx/` and `h1x/` namespaces and their vars as deprecated

* Fix snowflake failures

* Remove unneeded honey-sql-version bindings

* Decouple checking ddl from classifying h2 stmts

- should enable followup for easily blocking more kinds of queries
- check all statements to make sure they aren't "ddl".

* fix classify-query

* linter fixes + get-field refactor

* return the CommandInterface values as ints

Reach into the CommandList when needed

* docstring wording

* catch invalid queries -- they can't be classified

* Remove action subtypes from inlined-models

* Add ddl check for `execute-write-query!`

* Remove ACCESS_MODE_DATA

* Check queries are single select statements

* Add test for sample database privileges

* Fix single-select check

* Add single-select test

* Rename and add more tests for checking read only commands

* commands -> statements

* Fix check-disallow-ddl-commands

* new line

* Add more read-only statements to the tests

* Update error text

* Use are

* Add integration test for executing actions with disallowed commands

* Add test before inserting row

* Run GRANT ALL ON SCHEMA "PUBLIC" TO GUEST

* Restore classify-query

* whitespace

* Whitelist command types for actions

* Add comment

* Rename

* Remove dupes

* Add truncate test

* Add DDL commands

* Rename to query-classification

* Update error message

* Fix test

---------

Co-authored-by: Bryan Maass <bryan.maass@gmail.com>

Fixes #22872

In #21839, all order-bys were removed from pivot queries because columns
in `ORDER BY` had to appear in `GROUP BY` clauses. Unfortunately that
removed legal order by columns (e.g. count) from being applied.

This now keeps aggregation columns in field defs to restore the sorting
behaviour.

* Snowflake Honey SQL 2

* Test fix 

* Use Honey SQL 2 for GTAP compilation in sandboxing tests

* Test fix 

* new clojure migration for v1->v2 permissions

* updates

* fix rebase issue

* fix reflection warning

* fix final test and address comment about macro name

* fix indentation

* revert changes to defmigration macro to try to fix final test

* Decouple checking ddl from classifying h2 stmts

- should enable followup for easily blocking more kinds of queries
- check all statements to make sure they aren't "ddl".

* fix classify-query

* linter fixes + get-field refactor

* return the CommandInterface values as ints

Reach into the CommandList when needed

* docstring wording

* catch invalid queries -- they can't be classified

* Remove action subtypes from inlined-models

* Add ddl check for `execute-write-query!`

* Remove ACCESS_MODE_DATA

* Check queries are single select statements

* Add test for sample database privileges

* Fix single-select check

* Add single-select test

* Rename and add more tests for checking read only commands

* commands -> statements

* Fix check-disallow-ddl-commands

* new line

* Add more read-only statements to the tests

* Update error text

* Use are

* Add integration test for executing actions with disallowed commands

* Add test before inserting row

* Run GRANT ALL ON SCHEMA "PUBLIC" TO GUEST

* Restore classify-query

* whitespace

---------

Co-authored-by: Bryan Maass <bryan.maass@gmail.com>

* add logics to skip generate subscription contents for link cards and actions

* Add `model_id` to action search results so that URLs can be made

[Fixes #28565]

* Fix typo in Cypress filename

* set action.model_id to cascade on delete

* add rollback

* Strip semicolons from card refs in native queries

Fixes #28218

While source-query substitution handled semicolons in native questions
within outer mbql queries, we used a different code path when splicing
into native queries. This brings the two paths inline.

* Reference issue in testing

* Add docstring

* Resolve nested questions in native queries

Fixes #25988.

* Decouple checking ddl from classifying h2 stmts

- should enable followup for easily blocking more kinds of queries
- check all statements to make sure they aren't "ddl".

* fix classify-query

* linter fixes + get-field refactor

* return the CommandInterface values as ints

Reach into the CommandList when needed

* docstring wording

* catch invalid queries -- they can't be classified

* respond to review comments

- fix see: comment
- remove CommandList typehint
- rename command-list -> command
- skip checking disallowed ddl commands when query is `nil`

then also:
- fix a linter error
- add a test for `nil` query ddl command checking

* rename command-list-type -> first-command-type

* fix schema + handle h2-parser build failure

* Fix action loading and extraction of action type

* Remove comment

* Fix handling required parameters for public basic actions (#28415)

* Fix presenting required parameters

* Add tests

* Add explanation text to the action editor (#28406)

* Add explanation text to the action editor

* Replace link

* Add missing `nock.cleanAll` calls (#28441)

* [CI] Improve `codenotify` workflow (#28432)

* Add path filtering to codenotify workflow

* Upgrade action to its latest version (v0.6.4)

* [E2E] Prevent app permission flakes (#28445)

* Ensure dashcards are deleted after actions are deleted (#28446)

(c.f. discussion at https://metaboat.slack.com/archives/C04FG88HL95/p1676649795131089

)

* Show not found view when opening archived model (#28443)

* Add `isArchived` method to `Question` class

* Show nice not found view when opening archived model

* [CI] Unify E2E tests workflows (#28431)

* [Ci] Fix wildcard glob pattern in path filters (#28447)

[ci skip]

* Remove dupe tests

* Add database_id to implicit actions

* Tidy

* Update select-actions docstring

* Add test for database_id in action dashcards

* Remove database_id before insert and update

* whitespace

* Refactor select-actions

* Add database_id to action-create-test

* Add test for create+update+select for implicit actions

* Fix test

* Remove conditional

* style

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Tim Macdonald <tim@metabase.com>

* If a model is converted to a card, archive the associated actions
* If an action is unarchived and its model _is_ a card, throw an error

* MISC improvements

* Update test/metabase/models/secret_test.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update .clj-kondo/hooks/clojure/test.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update .clj-kondo/hooks/clojure/test.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Improves consistency of substring in MBQL

Standard SQL SUBSTRING() uses a 1-based start index, with various drivers
handling a 0 start index in different ways. To make the behavior of substring
consistent across drivers, we will always convert a 0 start index in MBQL
to a 1 in SQL. The ->honeysql implementation of substring has been generalized
and drivers can now override just the name of the substring function, or the
entire expression if necessary. SQL Server now calls the parent implementation.

Resolves #12445

* Fixes substring from 0 in Mongo

* Adds arg to substring parent call

* Fix docs and helper text for substring()

* Adds schema to require positive integer index for substring

* WIP, adds frontend validation for substring index arg

* Adds frontend test for substring arg validation

* Updates substring test to expect exception on index 0

* Update src/metabase/driver/sql/query_processor.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Use s/if instead of s/conditional

* Fix substring call in mysql substring spec

* Reverts driver and sql.qp changes for substring index

* Reverts changes to dispatch-by-clause-name-or-class

* Fixes typeinferencer test substring index

* Adds test for canonicalizing substring

* Fixes substring canonicalization with extra args

* Updates string extracts test to account for substring canonicalization

* Rewrites resolver validation application

* Adds test case for frontend expression validation errors

* Fixes frontend compiler spec for new substring validation

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Misc QP test improvements

* Add test

This'll be revisited as part of v47

[Fixes #28425]

* Fix linter

* Make sure we're inlining num in substitution and sum aggregations

* Set hsql version

* Fixing tests

* Fixing tests after the h2 conversion

* Use h2x/extract in h2, fix test differences

* Fixing tests for honeysql2

* Only numbers get inlined

* More test fixes

* Fixing tests

* Undo bad commit

* Inline Ratio must produce doubles or h2 will do integer division where 1/2 = 0

* Fix bad case statement

* We can't run native queries with AS for oracle

---------

Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Make joins work

* Run fuzzy-field-info-test only against H2

* Set driver/*driver* when performing actions

* Restrict expressions+joins-test to DBs supporting date arithmetics

* Fix ordering of (top level) fields in mongo tables

* Explicitly exclude mongo from tests requiring foreign key support

* Optimize away $sort stage for implicit order-by.

* add dashcard testing mocks

* improve virtual card overlays

* update types

* add search entity types

* add metabase entity support to link cards

* simplify link displays

* update types and support collections and tables

* use edit button to replace entity

* test entity links

* fix rebase

* update types

* address review comments

* handle restricted link entities

* Hydrate link cards info (#28297)

* hydrate link cards info

* we need to hydrate collections too

* Address Cal's comments:
- model use keywrod -> string
- use model hydration
- comments here and there

* Link cards serdes (#28321)

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

(c.f. discussion at https://metaboat.slack.com/archives/C04FG88HL95/p1676649795131089)

* Removes :presto driver and tests
* Merges :presto-common into :presto-jdbc
* Adds migration to update presto databases to presto-jdbc

---------

Co-authored-by: Cam Saul <github@camsaul.com>