* Support overriding ROWCOUNT for SQL Server

Add new "ROWCOUNT Override" connection property for `:sqlserver`, which will provide a DB-level mechanism to override the `ROWCOUNT` session level setting as needed for specific DBs

Change `max-results-bare-rows` from a hardcoded constant to a setting definition instead, which permits a DB level override, and move the former constant default to a new def instead (`default-max-results-bare-rows`)

For `:sqlserver`, set the DB-level setting override (if the connection property is set), via the `driver/normalize-db-details` impl

Add test to confirm the original scenario from #9940 works using this new override (set to `0`)

Move common computation function of overall row limit to the `metabase.query-processor.middleware.limit` namespace, and invoke it from execute now, called `determine-query-max-rows`

Add new clause to the `determine-query-max-rows` function that preferentially takes the value from `row-limit-override` (if defined)

* Apply schema inclusion/exclusion filtering to sql-jdbc drivers

Update `sql-jdbc` namespaces to handle schema inclusion/exclusion patterns when filtering schemas

Add new generic schema inclusion/exclusion test for sql-jdbc drivers that define the property

Update Snowflake and Redshift driver manifests to include schema filtering property

Create `db-details->schema-filter-patterns` util fn to turn DB details into the inclusion/exclusion patterns

Move schema inclusion/exclusion filtering code to new namespace (since it's not strictly used by `:sql-jdbc` derived drivers)

Move existing tests accordingly

Add schema inclusion/exclusion check to the new `filtered-syncable-schemas` multimethod (and updating docstring)

Change `:redshift` impl of `filtered-syncable-schemas` to call the `:sql-jdbc` version instead

Use new multimethod instead for `filtered-syncable-schemas`, and have default impl of `syncable-schemas` call that

Mark `syncable-schemas` as deprecated and include notes on the new method (and update driver markdown file accordingly)

* Add drive scope to BigQuery client

Adding new drive auth scope in order to be able to query Google Drive files created in BigQuery as external tables

Adding new test to confirm such a table can be queried successfully

* Add test for #7487

* Fix error message

* Update test for new error message

* Add test for #15538

* Add extra validation for native source queries

* Add test for sql-source-query validation

* Remove hardcoded project-id from `:bigquery-cloud-sdk` tests

Pull out common helper fn to extract project-id from credentials

Update `bigquery_cloud_sdk_test.clj` to have a var that loads the service-account-json from the env var, then uses aforementioned function to determine project-id

Change sensitive-data-redacted-test to use an arbitrary/fake project-id to remove any sort of confusion

Nested Queries Overhaul 2022: Split logic for determining appropriate table & column aliases out of SQL QP (#19384)

Move bulk of dataset-id normalization logic into new private helper fn

Perform app DB update for db-details after dataset-id has been turned into inclusion filter

Add test to confirm that normalization only happens once

* Don't update tables in a schema one by one

Code had wanted to log each table which is admirable. But on bigquery
there can be 15,000 tables and this will lock up calls to
/api/database. On stats they are taking about 18s for me. Need to do
once and let the db do the heavy lifting

* Also where schema is nil

* slightly better query

Add functions to describe-database namespace for performing inclusive/exclusive schema filtering

Add tests for these functions in isolation

Update `:bigquery-cloud-sdk` driver and QP code to remove the single `dataset-id` conn-param

Changing :bigquery-cloud-sdk` driver to use dataset inclusion/exclusion filters instead

Updating `add.cy.spec.js` Cypress test to account for new structure

Add data upgrade (via `driver/normalize-db-details` implementation) to change `dataset-id` property to inclusion filter

Add test to confirm data upgrade

Adding server side expansion of `:schema-filters` type connection property

* Allow override of WEEK_START in Snowflake driver

Incorporate the following sources of WEEK_START, in decreasing order of precedence:
1) the connection string, a.k.a. additional-options (ex: WEEK_START=N)
2) the Metabase "start-of-week" setting (i.e. day of a week)
3) the default value (7: Sunday)

Adding new fn to the driver.common namespace to parse a connection string param out of additional-settings (plus a test for that)

Adding Snowflake driver test to ensure that all cases above are covered as expected, by invoking the DAYOFWEEK function on a set of fixed dates

Update documentation for start-of-week setting to clarify that it now influences this behavior

Swap out parse-additional-options-value for additional-options->map instead (more generic)

Update test accordingly

* Rename setting/get and setting/all; GeoJSON via env var tests

* Fix typo (thanks @noahmoss)

* Support Database-local Settings in metabase.models.setting itself

* Rework Settings code so it can handle possibly-already-deserialized values

* Database-local Settings

* Remove empty part of docstring

* Appease linters

* Update dox again

* Use text.type for the new column

* Test fixes 

* Test fix 

* Test fix 

* All negative integer values when setting :integer Setting with a string

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

* Fix file upload of secret values

Fix logic in `db-details-client->server` to properly consider the `-options` suffixed connection property from the client version, to determine the treatment to be applied (i.e. base64 decode)

Change logic in `db-details-client->server` to `assoc` nil rather than `dissoc` unused keywords (so they override the saved db details on merge)

Update `driver.u/database->driver*` so that the engine loaded from the database instance is *always* a keyword

Update `metabase.test.data.oracle/connection-details` to pass the property vals through the client->server translation layer

Change `creator_id` in `secret` model to be nullable (since, when creating test DBs, the API current user is deliberately set to nil)

Switch the logic in Oracle SSL test to iterate on the file-path and uploaded variants to the test itself to get around weird CircleCI/environment issues

Use `rds_root_ca_truststore.jks` for the test instead since we don't need the full cacerts for this

* Remove references to flag in FE code

* Remove flag from BE code

* Remove/update tests related to the feature flag

* Show widget-type selection of 'String' for category & location template tags

Remove any parent fields from the field list when building projections, which should have the same effect as before (only child fields selected) while being compatible with MongoDB 4.4+

Add test to confirm the parent fields are removed

* Refactor: move Card and Dashboard QP code into their own qp.* namespaces

* Disable extra validation for now so a million tests don't fail

* WIP

* Validate template tag :parameters in query in context of a Card

* Fixes

* Disable strict validation for now

* Test fixes [WIP]

* Make the parameter type schema a little more forgiving for now

* Tests & test fixes 

* More test fixes 

* 1. Need more tests
2. Need to actually validate stuff

* More test fixes. 

* Test fixes (again)

* Test fix 

* Some test fixes / PR feedback

* Disallow native queries with a tag widget-type of "none"

Template tags with a widget-type that is undefined, null, or "none" now
cause the query's isRunnable method to return false. Existing questions
that have this defect won't be runnable until they are resaved with a
set widget-type.

* Fix prettier error

* add snippet and card types to validation pass

* Make sure template tag map keys + `:names` agree + test fixes

* Have MBQL normalization reconcile template tags map key and :name

* Test fix 

* Fix tests for Cljs

* Fix Mongo tests.

* Allow passing :category parameters for :text/:number/:date for now.

* Dashboard subscriptions should use qp.dashboard code for executing

* Make sure Dashboard QP parameter resolution code merges in default values

* Add a test for sending a test Dashboard subscription with default params

* Prettier

* If both Dashboard and Card have default param value, prefer Card's default

* Test fix 

* More tests and more fixes 



Co-authored-by: Dalton Johnson <daltojohnso@users.noreply.github.com>

Add new deps for finding close misspellings of map keys (spell-spec) and printing more readable spec errors (expound)

Adding spec definitions for driver YAML format

Add call to driver YAML validation from verify.clj

Fixing a couple issues in existing drivers found by the validation

* Enable Postgres driver to use secrets for configuring SSL parameters

Adding secret related properties for SSL options to Postgres driver

Move `conn-props->secret-props-by-name` to secret.clj since it's needed directly there, too

Add `us-east-2-bundle.pem` to `test-resources/certificates` for testing Postgres with SSL connectivity to our RDS instance (and a README.md explaining how it differs from the existing `ssl` directory)

Updating `value->file!` to have better logic for building the error message when the existing file is not found (for better UX from the Database admin page)

Updating frontend to support the `visible-if` value being an array, in which case any value may match the current details value, in order for that field to be visible

Adding secret related properties for SSL options to Postgres driver

Updating CircleCI config.yml to refer to the RDS instance when running Postgres SSL test

Implement server side expansion of transitive visible-if dependency chain

Update shouldShowEngineProvidedField on client to consider multiple key/value pairs in the visible-if map, and only show the field if ALL are true

Adding new test to confirm transitive visible-if resolution, and cycle detection

Add Cypress test for SSL field visibility

* Fix google-api-client version from google driver

Update google-api-client version `1.30.7` -> `1.32.1`, so that `google-http-client` ends up as `1.39.2` to match the other drivers

Add explicit `google-http-client-jackson2` dependency for google driver since it's no longer included transitively for some strange reason, but the `:google` driver itself directly uses it

Add new CircleCI test run for multiple Google related drivers (to ensure there aren't cross-driver classpath issues)

Generate correct connection string when srv is defined, including `authDb`

Default `authDb` to `admin` if not set

Update bigquery and googleanalytics drivers dependencies to newer versions, such that their transitive dependencies on google-api-client/google-http-client match the versions used in the new BigQuery driver

* Add SSL keystore and truststore secret properties to Oracle driver YAML files


Update Oracle driver to set keystore and truststore connection options from secret values

Adding new `select-keys-sequentially` helper function for dealing with assertions on transformed conn props

Add new function to the secret namespace to return a lighter weight "secret map", and remove that functionality from the `handle-db-details-secret-prop!`, so that it can be reused from connection testing

Modifying CircleCI to set the corresponding truststore secret conn prop vars instead of JVM-level truststore settings, as the test had been doing previously

Expand Oracle test connection details to incorporate all SSL related properties, and also make it a function instead of delayed def (so changes can be picked up later)

Misc. fixes in Oracle driver YAML files

Update `exception-classes-not-to-retry` to include SSLHandshakeException

Fix misc issues in secret and database model code

Update CircleCI config to use correct secret based env var keys and values

Add *database-name-override* dynamic var in `metabase.test.data.interface` to allow for DB name overriding

Fix up ssl connectivity test so all parts pass successfully

Get rid of crazy with-redefs to swap in the existing test-data's schema, and instead, just dropping the DB if it was created under a different schema (similar to what H2 does)

Co-authored-by: Gustavo Saiani <gustavo@poe.ma>

Update new BigQuery driver to only use project-id in qualified names if it differs from credentials (#18843)

* Update new BigQuery driver to only use project-id in qualified names if it differs from credentials

Add new key to db-details for :bigquery-cloud-sdk when a database is updated, called :project-id-from-credentials, and extract the project-id from the service account creds when the database is saved

Updating logic in bigquery-cloud-sdk QP to only include the project-id when qualifying an Identifier, if the project-id (override) value is set, and its value differs from the :project-id-from-credentials value mentioned above

Adding a test to confirm that a card query continues to work after changing the associated DB's driver from :bigquery to :bigquery-cloud-sdk

Change all syncs in the test to {:scan :schema} to speed them up (deeper level not needed)

Add a new dynamic var - *source-query* - to sqp.qp to bind the current source-query, and bind that in the same place that *table-alias* is currently bound

Update bigquery-cloud-sqp qp implementation to check the sqp.qp/*source-query*, and only qualify the source query alias if the type of source query is a table (which is apparently required; see #15074)

Add test for #18742

* Make the checkins_interval transient datasets more uniquely named

Append a timestamp suffix (initialized when the namespace was loaded) to transient dataset names

Add verbose comments explaining what we're doing

Add transient-dataset? functions for both BQ drivers to put logic in one place (which currently just see if the dataset is named `checkins_interval_*`

Ditch the "legacydriver_" prefix from the legacy BigQuery driver when creating new datasets; it's no longer needed now that transient datasets will have a much more unique suffix

Update the logic for the transient dataset suffix so that it can be more easily detected/parsed later (for cleanup)

Remove old (more than 2 hours) transient datasets when test extensions namespaces load

Add various docstrings

This reverts commit c6688f5caf226c6ec5be424859fc00b93665620f.

Redshift only supports VACUUM command at a time on the whole
cluster, so running VACUUM can fail with parallel executions.

v2.0.0.3 -> v2.1.0.1

Removing duplicate property declaration from presto-jdbc driver YAML

Add test that executes against all drivers to confirm that no duplicate names come out of connection-properties

Change the way the test runs to avoid needing to initialize test data namespace (to make googleanalytics happy)

Unskip repro Cypress test