* `:metadata/field` => `:metadata/column`

* Merge master

* initial prototype w/out statement count parameter

* new approach

* default-database-role driver method

* migration for connection_impersonations table

* conn impersonation model, API and tests

* impersonation fetch and deletion endpoints

* switch test to t2 with-temp

* read conn impersonation settings from DB

* fix merge issue

* add impersonated key to data perms graph and treat it the same as full self-service access

* include impersonated key in returned permissions graph

* make sure impersonated graph passes StrictDataPerms validation

* fix boolean logic

* make sure impersonated keyword doesnt cause error when deleting gtaps

* clear impersonations as necessary when perms graphs changes

* add impersonation support for postgres

* fix typo

* make sure impersonation updates are a non-lazy seq

* add impersonated-user? fn

* fix impersonation api tests

* fix snippet tests

* fix build & exclude connection impersonations from serialization

* switch a test to use t2.with-temp

* add with-impersonations helper and util tests

* move macro and add a connection impersonation driver-level test for postgres

* fix rebase issue

* more tests and code reorganization

* add snowflake test

* clarify comment

* fix lint errors

* fix final kondo error

* reorganization

* fix one test

* fix lint errors

* revert change to sql_jdbc.execute from bad merge

* make sure perms for all users gets reset after conn impersonation tests

* ignore exceptions when restoring perms

* fix postgres test

* refactor to address bryan's comment

* add note about new methods to database changelog

* driver method refactor

* Squash

* Replace connection-with-timezone usages

* Add deprecation warning.

* Fix docstring

Resolves #28792 

* Always create a transient dataset for BigQuery tests

* Simplifies BigQuery test data code

* More BigQuery test data code cleanup

* More BigQuery test data code cleanup, test fix

* Extracts test-db-name in bigquery tests, restores normalize-name

* Fixes BigQuery QP tests for new test DB name scheme

* Removes extraneous clojure.core

* Updates bigquery QP test helper code style

* Cleans up comments and code in bigquery test data

* Update modules/drivers/bigquery-cloud-sdk/test/metabase/driver/bigquery_cloud_sdk/query_processor_test.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Updates Snowflake driver to decode uploaded private key file data if necessary

Use MLv2 for metadata calculation in QP (part 1); make column name calculation consistent with QP (#29583)

* Simplified impl

* Add #31266 test

* Yay

* Very nice

* Test fixes

* Fix Kondo warnings

* Ban `tt/with-temp` and convert usages to Toucan 2

* Ban `tt/with-temp*` and convert usages to Toucan 2

* Revert changes not meant for this PR

* Ban `mt/with-temp` and replace usages with `t2.with-temp/with-temp`

* Fixes 



* Fix indentation

* fix syntax in models.secret_test

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Ban `tt/with-temp` and convert usages to Toucan 2

* Ban `tt/with-temp*` and convert usages to Toucan 2

* Revert changes not meant for this PR

This should resolve vulnerability found by Trivy scan.
https://github.com/metabase/metabase/security/code-scanning/95

* Fix native source query handling for mongo

* Add test for mongo nested native query

* Fix mongo `aggregation-at-index` usage

* Add test for source query with aggregation and sort

* Adjust test for presto and oracle

* Add PR suggestions

betwen -> between

[ci skip]

Fixes #29678.

Formatting of BigQuery will use the MySQL dialect for now.

* feat: allow authorization token for druid database

* fix:  adjust in basic auth  header and new field username

* fix: add scenario test

* fix: remove debug code

* fix: use metabase-plugin for new fields and improve tests

* fix: change type of auth-token to password

* fix: adjustment tested

* fix: adjustment tested

---------

Co-authored-by: Nícolas Botelho <nicolas.botelho@piposaude.com.br>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Fix subsitution of exclude- params

* Rename fn to `not-single-date-type?`

* Add docstring

* Remove comment

* Fix substitution

* Fix errors when param values are not strings

* Exclude bigquery from the test

* Fix testing description

* Use users table instead

* Fix docstring

Each test run can create schemas where we put persisted models. Normally
these are thrown away when the test runner disconnects the db. But cloud
is a persistent resource and they don't just fall away automatically.

We create a cache_info table in each persisted schema:

```sql
-- postgres/redshift
test-data=# select * from metabase_cache_424a9_379.cache_info ;
       key        |                value
------------------+--------------------------------------
 settings-version | 1
 created-at       | 2023-03-29T14:16:24.849866Z
 instance-uuid    | 407e4ba8-2bab-470f-aeb5-9fc63fd18c4e
 instance-name    | Metabase Test
(4 rows)
```

And we delete these schemas in redshift when creating our test databases
when their `created-at` is over 6 hours old.

At the moment, there are (`(count (:all-schemas info))`) 420 schemas in
redshift that we use on CI. 122 of those are cache schemas. Under a 6
hour threshold, 121 are recent. Under a 1 hour threshold, 2 are recent.

We've run out of tables in CI so I'm lowering the threshold for what
"recent" is.

* Database to toucan2

* Table to toucan2

* Field to toucan 2

* Add schemas driver feature flag

* Add sql_jdbc schemas test

* Add more general test

* Make schemas feature default to true

* Implement supports? for other drivers

* Add feature to the driver changelog

* Remove redundant schemas from h2

* Spelling fix

Pre-rebase SHA: 8a2c7e0 or 5686b2f

* MySQL does *not* support schemas

* Bad rebase

* Typo?

Co-Authored-By: Tim Macdonald <tim@metabase.com>

* Tidy database-supports? defmethods

* Fix oracle build

* Fix driver

* Fix typo

* prefer driver/database-supports? instead of driver/supports?

* driver/supports? -> driver/database-supports? in tests

* Fix foreign-key support for sqlite in tests

* supports? -> database-supports?

* `supports?` -> `database-supports?`

* supports? -> database-supports?

* Update supports? docstring

* Fix tests

* Fix test

* Remove unused require

* Add mt/with-everything-store in tests

* fix test

* Fix tests

* Add note in driver changelog

Fixes #26054.

The Snowflake JDBC driver is buggy: schema and table name are interpreted as patterns
in DatabaseMetaData.getPrimaryKeys and DatabaseMetaData.getImportedKeys calls.

This PR replaces the default JDBC functions with ones that escape the names.

* MLv2: use kebab-case keys in Clojure & camelCase in JS

* Revert changes to tests

Fixes #30262.

* Use type/JSON instead of type/SerializedJSON

* Tidy migration

* Update migration

* Fix rollback migration for h2

* whitespace

* Add test for migration

* Fix test

* Add rollback

* whitespace

* Test JSONB type as well as JSON

* Don't fingerprint JSON columns

* Remove comment, that might be wrong in the future

* Update test

* Add tests for fingerprinting

* Use base-type JSON for fingerprinting base query

* Add test for visibility-type=details-only

* undo .

* Change migration id

* Fix migration test

* Merge master

* Exclude mariadb from tests

* Make is-mariadb? public

* Migration for adding nfc_enabled

* Add nfc_enabled to field settings

* Update describe-nested-field-columns to only unfold fields that have not been disabled

* Remove spy

* Tidy

* Fix

* Clear nested fields immediately if folding is disabled

* Clear nested fields on nfc_enabled change

* Trim trailing whitespace

* Tidy

* Fix

* Add enable-json-unfolding-test

* Move to field api test

* nfc_enabled -> json_unfolding

* Tidy test

* Make json-unfolding in database details just the default for new settings

* Restore original visibility-type logic

* Fix

* Sync field json_unfolding according to db json_folding

* Rename to json-unfolding-default

* Add test for the case when the json-unfolding is false for the database

* Implement default json unfolding for first sync

* Update comment

* Update comment

* Restore PUT field

* Migration for populating json_unfolding for mysql and postgres

* Remove migration and use default on the frontend instead

* Update json-unfolding database setting copy

* Move Unfold JSON setting under semantic type setting

* Separate sentences with spaces

* Capitalize

* Restore handleChangeSemanticType

* Use base_type TYPE.JSON instead

* Change order of migrations

* Add h2 migration

* Add test for migration

* Update test description

* Remove validCheckSum

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Use ?? instead of ||

* Fix tests

* Remove outdated serdes stuff

* Remove unnecessary and

* Fix mysql migration

* Remove unused require

* Fix test

* Add false default value for json_unfolding

* Fix json-unfolding nil case

* Other suggestions

* whitespace

* Tidy describe-nested-field-columns

* Update comment

* Remove unused clear-nested-fields!

* Remove unused return value

* Fix H2 migration to use base_type not database_type

* Always set json-unfolding during sync

* Fix test

* Fix test

* Fix test

* Add comment explaining nested-field-column support for MySQL

* Fix tests

* Fix tests

* Fix test

* Fix test

* Default json_unfolding to false for new fields

* Fix merge

* Add json_unfolding to mock tables

* Don't capitalize prepositions

* Update setting description

* whitespace

* whitespace

* Fix fetch_metadata

* Fix fetch_metadata

* Fix tests

* Fix test

* Fix clj-kondo

* Remove postgres database-supports test

* Fix postgres test

* Fix postgres test

* Fix mysql migration

* Fix clj-kondo

* Don't test mariadb

* Fix mysql test

* Fix mysql json-unfolding nil case

* Add comments to test

* Add upterm step to mariadb test

* Move upterm step before tests

* Fix postgresql migration

* move upterm step to test-driver action

* Remove upterm step from drivers.yml

* Comment out everything else in test-driver

* Fix mariadb migration

* Remove upterm action

* Whitespace

* Only handle JSONObjects with JSON_VALUE, not JSONArray

* Fix fields.sync_metadata/update-field-metadata-if-needed!

* Add comment explaining `json-unfolding-default`

* Add comment for JSON_VALUE test

* Update json-unfolding-default-true-test to use fresh db

* Fix test descriptions

* Don't capitalize setting display name

* Fix update json_unfolding

* Fix unfolding json for only one JSON column, not all

* Add e2e test for JSON unfolding setting

* Fix test

* whitespace

* Fix test

* Fix json-unfolding for nested field columns

* Fix mysql and postgres tests for nested field columns

* Coerce nil json-unfolding to false for sync_metadata

* Add test for nil json-unfolding for sync_metadata

* Fix test

* Don't update json_unfolding from intial sync

* Add json_unfolding to update-field-test

* Fix tests

* Use cypress-testing-library commands

* Fix e2e test

* Fix e2e test

* Add json-simple to deps

* Update uberjar.yml

* import org.json.simple

* Replace SQL migrations and H2 UDF with a clojure migration

* Remove json-simple from deps

* Restore uberjar.yml

* Rename to set-json-unfolding!

* Make the DB JSON unfolding setting override the field setting, instead of being a default.

* Remove unused var

* Fix error

* Remove custom clojure migration

* Fix migration

* Fix migration again

* Fix migration and kondo

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Fixes inlining of numeric params in native queries
* Updates namespaces to use sql.qp/with-driver-honey-sql-version

* Delete old persistence schemas in redshift

Reminder that recently a change occurred that populates a cache_info
table in each persisted schema:

```sql
-- postgres/redshift
test-data=# select * from metabase_cache_424a9_379.cache_info ;
       key        |                value
------------------+--------------------------------------
 settings-version | 1
 created-at       | 2023-03-29T14:16:24.849866Z
 instance-uuid    | 407e4ba8-2bab-470f-aeb5-9fc63fd18c4e
 instance-name    | Metabase Test
(4 rows)
```

So for each cache schema, we can classify it as
- old style (a commit before this change. more and more infrequent)
- new style recent
- new style expired

And we can delete them accordingly in startup:
```
2023-04-11 20:09:03,402 INFO data.redshift :: Dropping expired cache schema: metabase_cache_0149c_359
2023-04-11 20:09:04,733 INFO data.redshift :: Dropping expired cache schema: metabase_cache_0149c_70
2023-04-11 20:09:05,557 INFO data.redshift :: Dropping expired cache schema: metabase_cache_0149c_71
2023-04-11 20:09:06,284 INFO data.redshift :: Dropping expired cache schema: metabase_cache_0149c_90
...
2023-04-11 20:20:33,271 INFO data.redshift :: Dropping expired cache schema: metabase_cache_fe4a7_90
2023-04-11 20:20:34,284 INFO data.redshift :: Dropping old cache schema without `cache_info` table: metabase_cache_8f4b8_358
2023-04-11 20:20:35,076 INFO data.redshift :: Dropping old cache schema without `cache_info` table: metabase_cache_8f4b8_69
...
```

It's possible this will at first cause a few flakes if we are unlucky
enough to drop a cache schema without `cache_info` for an instance that
is running tests at that point. But the `cache_info` table has been
backported so the chances of that become smaller each day.

I've let a week elapse from that change before committing this so
hopefully it is not an issue in practice.

Number of queries:
Makes a single query to get all schemas, then for each schema makes a
query to classify it. This can be unified into a single query with some
shenanigans like

```clojure
(sql/format {:select [:schema :created-at]
             :from   {:union-all
                      (for [schema schemas]
                        {:select [[[:inline schema] :schema]
                                  [{:select [:value]
                                    :from   [(keyword schema "cache_info")]
                                    :where  [:= :key [:inline "created-at"]]}
                                   :created-at]]})}}
            {:dialect :ansi})
```

But i found that this query is extremely slow and does not offer any
benefit over the simpler, repeated queries. And as we run this on each
commit now, the number of schemas will be far lower and it will be on
the order of 5-10 schemas (and therefore queries) and therefore not an
issue.

* Ngoc's suggestions

- docstring for `delete-old-schemas!`
- combine nested `doseq`
- use java-time over interop with java.time

* Rename stage field to fields

* Convert join condition to conditions

* Do legacy to mbql conversion in lib.util/pipeline

* Fixing tests

* Do not convert questions twice

* Fix linter

* Fix tests

* Fix mongo tests that were building incorrect legacy queries

* Fix mongo properly

* Adapt unique schema logic in CI from Snowflake to Redshift

* Ok don't stomp on all the old schemas just yet.

* Fix missing vars

* Test fixes 

* Test fixes 

* Revert `^:parallel` on tests that shouldn't be allowed to use it

* Don't log in bigquery results hotpath

Right now we log in the parser of bigquery results for unrecognized
types. But the problem is that we log for each _value_ and not each
column. This is causing an _enormous_ amount of logs and performance
penalty.

See
- https://github.com/metabase/metabase/issues/29118 (performance)
- https://github.com/metabase/metabase/issues/28868 (filling disk space)

This log was added between 45.1 and 45.3

```diff
❯ git diff v0.45.1..v0.45.3 modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/**
diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
index a0d8081c30..f367199b55 100644
--- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
+++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
@@ -22,7 +22,7 @@
             [metabase.util :as u]
             [metabase.util.date-2 :as u.date]
             [metabase.util.honeysql-extensions :as hx]
-            [metabase.util.i18n :refer [tru]]
+            [metabase.util.i18n :refer [trs tru]]
             [pretty.core :refer [PrettyPrintable]]
             [schema.core :as s])
   (:import [com.google.cloud.bigquery Field$Mode FieldValue]
@@ -88,7 +88,8 @@
     (parse-fn v)))

 (defmethod parse-result-of-type :default
-  [_ column-mode _ v]
+  [column-type column-mode _ v]
+  (log/warn (trs "Warning: missing type mapping for parsing BigQuery results of type {0}." column-type))
   (parse-value column-mode v identity))
```

The result is that selecting 50,000 rows for download in excel:

| version                      | time       |
|------------------------------|------------|
| 0.45.1                       | 28 seconds |
| 0.45.3                       | 52 seconds |
| 0.45.3 with logging disabled | 30 seconds |

(disable logging by adding `<Logger
name="metabase.driver.bigquery-cloud-sdk.query-processor"
level="ERROR"/>` and `-Dlog4j2.configurationFile=log4j2.xml` to jar
startup)

For the query (3 string columns, 5 rows):

```sql
SELECT game_id, first_name, last_name
FROM `bigquery-public-data.ncaa_basketball.mbb_players_games_sr`
LIMIT 5
```

BEFORE:

```
```
2023-03-31 17:17:52,146 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,155 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 795.2 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (2 idle, 0 queued) (192 total active threads) Queries in flight: 0 (0 queued)
```

Note this is 15 logs (3 columns x 5 rows)

AFTER:

```
2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column game_id of type STRING.
2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column first_name of type STRING.
2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column last_name of type STRING.
2023-03-31 17:19:15,757 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 973.5 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (3 idle, 0 queued) (193 total active threads) Queries in flight: 0 (0 queued)
```

* unused require to appease our overlords

This reverts commit 25a1b259.

* Use type/JSON instead of type/SerializedJSON

* Tidy migration

* Update migration

* Fix rollback migration for h2

* whitespace

* Add test for migration

* Fix test

* Add rollback

* whitespace

* Test JSONB type as well as JSON

* Don't fingerprint JSON columns

* Remove comment, that might be wrong in the future

* Update test

* Add tests for fingerprinting

* Use base-type JSON for fingerprinting base query

* Add test for visibility-type=details-only

* undo .

* Change migration id

* Fix migration test

* Merge master

* Exclude mariadb from tests

* Make is-mariadb? public

* Migration for adding nfc_enabled

* Add nfc_enabled to field settings

* Update describe-nested-field-columns to only unfold fields that have not been disabled

* Remove spy

* Tidy

* Fix

* Clear nested fields immediately if folding is disabled

* Clear nested fields on nfc_enabled change

* Trim trailing whitespace

* Tidy

* Fix

* Add enable-json-unfolding-test

* Move to field api test

* nfc_enabled -> json_unfolding

* Tidy test

* Make json-unfolding in database details just the default for new settings

* Restore original visibility-type logic

* Fix

* Sync field json_unfolding according to db json_folding

* Rename to json-unfolding-default

* Add test for the case when the json-unfolding is false for the database

* Implement default json unfolding for first sync

* Update comment

* Update comment

* Restore PUT field

* Migration for populating json_unfolding for mysql and postgres

* Remove migration and use default on the frontend instead

* Update json-unfolding database setting copy

* Move Unfold JSON setting under semantic type setting

* Separate sentences with spaces

* Capitalize

* Restore handleChangeSemanticType

* Use base_type TYPE.JSON instead

* Change order of migrations

* Add h2 migration

* Add test for migration

* Update test description

* Remove validCheckSum

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Use ?? instead of ||

* Fix tests

* Remove outdated serdes stuff

* Remove unnecessary and

* Fix mysql migration

* Remove unused require

* Fix test

* Add false default value for json_unfolding

* Fix json-unfolding nil case

* Other suggestions

* whitespace

* Tidy describe-nested-field-columns

* Update comment

* Remove unused clear-nested-fields!

* Remove unused return value

* Fix H2 migration to use base_type not database_type

* Always set json-unfolding during sync

* Fix test

* Fix test

* Fix test

* Add comment explaining nested-field-column support for MySQL

* Fix tests

* Fix tests

* Fix test

* Fix test

* Default json_unfolding to false for new fields

* Fix merge

* Add json_unfolding to mock tables

* Don't capitalize prepositions

* Update setting description

* whitespace

* whitespace

* Fix fetch_metadata

* Fix fetch_metadata

* Fix tests

* Fix test

* Fix clj-kondo

* Remove postgres database-supports test

* Fix postgres test

* Fix postgres test

* Fix mysql migration

* Fix clj-kondo

* Don't test mariadb

* Fix mysql test

* Fix mysql json-unfolding nil case

* Add comments to test

* Add upterm step to mariadb test

* Move upterm step before tests

* Fix postgresql migration

* move upterm step to test-driver action

* Remove upterm step from drivers.yml

* Comment out everything else in test-driver

* Fix mariadb migration

* Remove upterm action

* Whitespace

* Only handle JSONObjects with JSON_VALUE, not JSONArray

* Fix fields.sync_metadata/update-field-metadata-if-needed!

* Add comment explaining `json-unfolding-default`

* Add comment for JSON_VALUE test

* Update json-unfolding-default-true-test to use fresh db

* Fix test descriptions

* Don't capitalize setting display name

* Fix update json_unfolding

* Fix unfolding json for only one JSON column, not all

* Add e2e test for JSON unfolding setting

* Fix test

* whitespace

* Fix test

* Fix json-unfolding for nested field columns

* Fix mysql and postgres tests for nested field columns

* Coerce nil json-unfolding to false for sync_metadata

* Add test for nil json-unfolding for sync_metadata

* Fix test

* Don't update json_unfolding from intial sync

* Add json_unfolding to update-field-test

* Fix tests

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* db/insert-many! -> t2/insert! (mostly)

* Configure foreign key support for Mongo to allow implicit joins

* (db/transaction body) -> (t2/with-transaction [_conn] body)

* (db/with-call-counting) -> (t2/with-call-count)

* db/execute -> t2/query-one