* Replace AOT Spark SQL deps with a `proxy` and a `DataSource`

* Support `connection-details->spec` returning a `DataSource`

* Remove target/classes

* Don't need to deps prep drivers anymore

* Fix duplicate `this` params in `proxy` methods; add `:test` alias for Eastwood to make it be a little quieter

* Make sure to call `pool/map->properties`

* Upgrade Hive JDBC driver version from 1.2.2 -> 3.1.2; Bump Spark SQL from 2.1.1 to 3.2.1

* Clean the namespaces

* Don't need to register the a proxy JDBC driver since we're not even using it anymore

* Fix Spark SQL :schema sync for newer versions

* Remove unneeded override

* Fix day-of-week extract for new :sparksql

* Hive/Spark SQL needs to escape question marks inside QUOTED identifiers now 

* Some minor SQL generation improvements to avoid duplicate casts

* Revert change to debug test

* Add failing test for #15978

* Improved test

* Add new metabase.driver.query-processor.escape-join-aliases QP middleware

* Test fix 

* Add reference to #20307

* Add some extra dox

* Test fixes for BigQuery drivers

* revert unneeded change

* Fix :bigquery and :bigquery-cloud-sdk mixup

* Test fixes 

* Test fix 

* Remove comment I meant to remove

* Fix BigQuery not qualifying source columns

* Don't fix on 42 for the deprecated BigQuery driver

* You know what I'll fix it after all. Why not.

* Oops I updated the wrong part of the test

* Remove :bigquery driver

Add "migration" to convert existing Database instances from :bigquery to :bigquery-cloud-sdk (with error log if using outdated OAuth mechanisms), by way of `normalize-db-details`

Removing references to :bigquery from various places in the code

* Remove from modules/drivers/deps.edn

* Remove stuff from CircleCI that is running the old driver

Remove `driver-switch-test` since there is no practical way to run this anymore (since we can't initialize the old driver)

Remove buggy redef of `isa?` from `semantic-type-migration-tests`

Co-authored-by: Cam Saul <github@camsaul.com>

* Bump backend dependencies (Jan 2022)

* Revert java-time version upgrade for now until https://github.com/dm3/clojure.java-time/issues/77 is fixed

* Add license overrides

* Bump a few more deps (again). Revert Google/BigQuery and Vertica version bumps

* Revert MariaDB and Redshift version changes

Break out separate util fn to return the given `authdb` or the canonical default ("admin")

Add invocation to new util fn from the `mcred/create` call

* Support overriding ROWCOUNT for SQL Server

Add new "ROWCOUNT Override" connection property for `:sqlserver`, which will provide a DB-level mechanism to override the `ROWCOUNT` session level setting as needed for specific DBs

Change `max-results-bare-rows` from a hardcoded constant to a setting definition instead, which permits a DB level override, and move the former constant default to a new def instead (`default-max-results-bare-rows`)

For `:sqlserver`, set the DB-level setting override (if the connection property is set), via the `driver/normalize-db-details` impl

Add test to confirm the original scenario from #9940 works using this new override (set to `0`)

Move common computation function of overall row limit to the `metabase.query-processor.middleware.limit` namespace, and invoke it from execute now, called `determine-query-max-rows`

Add new clause to the `determine-query-max-rows` function that preferentially takes the value from `row-limit-override` (if defined)

* Apply schema inclusion/exclusion filtering to sql-jdbc drivers

Update `sql-jdbc` namespaces to handle schema inclusion/exclusion patterns when filtering schemas

Add new generic schema inclusion/exclusion test for sql-jdbc drivers that define the property

Update Snowflake and Redshift driver manifests to include schema filtering property

Create `db-details->schema-filter-patterns` util fn to turn DB details into the inclusion/exclusion patterns

Move schema inclusion/exclusion filtering code to new namespace (since it's not strictly used by `:sql-jdbc` derived drivers)

Move existing tests accordingly

Add schema inclusion/exclusion check to the new `filtered-syncable-schemas` multimethod (and updating docstring)

Change `:redshift` impl of `filtered-syncable-schemas` to call the `:sql-jdbc` version instead

Use new multimethod instead for `filtered-syncable-schemas`, and have default impl of `syncable-schemas` call that

Mark `syncable-schemas` as deprecated and include notes on the new method (and update driver markdown file accordingly)

* Add drive scope to BigQuery client

Adding new drive auth scope in order to be able to query Google Drive files created in BigQuery as external tables

Adding new test to confirm such a table can be queried successfully

* Add test for #7487

* Fix error message

* Update test for new error message

* Add test for #15538

* Add extra validation for native source queries

* Add test for sql-source-query validation

* Remove hardcoded project-id from `:bigquery-cloud-sdk` tests

Pull out common helper fn to extract project-id from credentials

Update `bigquery_cloud_sdk_test.clj` to have a var that loads the service-account-json from the env var, then uses aforementioned function to determine project-id

Change sensitive-data-redacted-test to use an arbitrary/fake project-id to remove any sort of confusion

Nested Queries Overhaul 2022: Split logic for determining appropriate table & column aliases out of SQL QP (#19384)

Move bulk of dataset-id normalization logic into new private helper fn

Perform app DB update for db-details after dataset-id has been turned into inclusion filter

Add test to confirm that normalization only happens once

* Don't update tables in a schema one by one

Code had wanted to log each table which is admirable. But on bigquery
there can be 15,000 tables and this will lock up calls to
/api/database. On stats they are taking about 18s for me. Need to do
once and let the db do the heavy lifting

* Also where schema is nil

* slightly better query

Add functions to describe-database namespace for performing inclusive/exclusive schema filtering

Add tests for these functions in isolation

Update `:bigquery-cloud-sdk` driver and QP code to remove the single `dataset-id` conn-param

Changing :bigquery-cloud-sdk` driver to use dataset inclusion/exclusion filters instead

Updating `add.cy.spec.js` Cypress test to account for new structure

Add data upgrade (via `driver/normalize-db-details` implementation) to change `dataset-id` property to inclusion filter

Add test to confirm data upgrade

Adding server side expansion of `:schema-filters` type connection property

* Allow override of WEEK_START in Snowflake driver

Incorporate the following sources of WEEK_START, in decreasing order of precedence:
1) the connection string, a.k.a. additional-options (ex: WEEK_START=N)
2) the Metabase "start-of-week" setting (i.e. day of a week)
3) the default value (7: Sunday)

Adding new fn to the driver.common namespace to parse a connection string param out of additional-settings (plus a test for that)

Adding Snowflake driver test to ensure that all cases above are covered as expected, by invoking the DAYOFWEEK function on a set of fixed dates

Update documentation for start-of-week setting to clarify that it now influences this behavior

Swap out parse-additional-options-value for additional-options->map instead (more generic)

Update test accordingly

* Rename setting/get and setting/all; GeoJSON via env var tests

* Fix typo (thanks @noahmoss)

* Support Database-local Settings in metabase.models.setting itself

* Rework Settings code so it can handle possibly-already-deserialized values

* Database-local Settings

* Remove empty part of docstring

* Appease linters

* Update dox again

* Use text.type for the new column

* Test fixes 

* Test fix 

* Test fix 

* All negative integer values when setting :integer Setting with a string

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

* Fix file upload of secret values

Fix logic in `db-details-client->server` to properly consider the `-options` suffixed connection property from the client version, to determine the treatment to be applied (i.e. base64 decode)

Change logic in `db-details-client->server` to `assoc` nil rather than `dissoc` unused keywords (so they override the saved db details on merge)

Update `driver.u/database->driver*` so that the engine loaded from the database instance is *always* a keyword

Update `metabase.test.data.oracle/connection-details` to pass the property vals through the client->server translation layer

Change `creator_id` in `secret` model to be nullable (since, when creating test DBs, the API current user is deliberately set to nil)

Switch the logic in Oracle SSL test to iterate on the file-path and uploaded variants to the test itself to get around weird CircleCI/environment issues

Use `rds_root_ca_truststore.jks` for the test instead since we don't need the full cacerts for this

* Remove references to flag in FE code

* Remove flag from BE code

* Remove/update tests related to the feature flag

* Show widget-type selection of 'String' for category & location template tags

Remove any parent fields from the field list when building projections, which should have the same effect as before (only child fields selected) while being compatible with MongoDB 4.4+

Add test to confirm the parent fields are removed

* Refactor: move Card and Dashboard QP code into their own qp.* namespaces

* Disable extra validation for now so a million tests don't fail

* WIP

* Validate template tag :parameters in query in context of a Card

* Fixes

* Disable strict validation for now

* Test fixes [WIP]

* Make the parameter type schema a little more forgiving for now

* Tests & test fixes 

* More test fixes 

* 1. Need more tests
2. Need to actually validate stuff

* More test fixes. 

* Test fixes (again)

* Test fix 

* Some test fixes / PR feedback

* Disallow native queries with a tag widget-type of "none"

Template tags with a widget-type that is undefined, null, or "none" now
cause the query's isRunnable method to return false. Existing questions
that have this defect won't be runnable until they are resaved with a
set widget-type.

* Fix prettier error

* add snippet and card types to validation pass

* Make sure template tag map keys + `:names` agree + test fixes

* Have MBQL normalization reconcile template tags map key and :name

* Test fix 

* Fix tests for Cljs

* Fix Mongo tests.

* Allow passing :category parameters for :text/:number/:date for now.

* Dashboard subscriptions should use qp.dashboard code for executing

* Make sure Dashboard QP parameter resolution code merges in default values

* Add a test for sending a test Dashboard subscription with default params

* Prettier

* If both Dashboard and Card have default param value, prefer Card's default

* Test fix 

* More tests and more fixes 



Co-authored-by: Dalton Johnson <daltojohnso@users.noreply.github.com>

Add new deps for finding close misspellings of map keys (spell-spec) and printing more readable spec errors (expound)

Adding spec definitions for driver YAML format

Add call to driver YAML validation from verify.clj

Fixing a couple issues in existing drivers found by the validation

* Enable Postgres driver to use secrets for configuring SSL parameters

Adding secret related properties for SSL options to Postgres driver

Move `conn-props->secret-props-by-name` to secret.clj since it's needed directly there, too

Add `us-east-2-bundle.pem` to `test-resources/certificates` for testing Postgres with SSL connectivity to our RDS instance (and a README.md explaining how it differs from the existing `ssl` directory)

Updating `value->file!` to have better logic for building the error message when the existing file is not found (for better UX from the Database admin page)

Updating frontend to support the `visible-if` value being an array, in which case any value may match the current details value, in order for that field to be visible

Adding secret related properties for SSL options to Postgres driver

Updating CircleCI config.yml to refer to the RDS instance when running Postgres SSL test

Implement server side expansion of transitive visible-if dependency chain

Update shouldShowEngineProvidedField on client to consider multiple key/value pairs in the visible-if map, and only show the field if ALL are true

Adding new test to confirm transitive visible-if resolution, and cycle detection

Add Cypress test for SSL field visibility

* Fix google-api-client version from google driver

Update google-api-client version `1.30.7` -> `1.32.1`, so that `google-http-client` ends up as `1.39.2` to match the other drivers

Add explicit `google-http-client-jackson2` dependency for google driver since it's no longer included transitively for some strange reason, but the `:google` driver itself directly uses it

Add new CircleCI test run for multiple Google related drivers (to ensure there aren't cross-driver classpath issues)

Generate correct connection string when srv is defined, including `authDb`

Default `authDb` to `admin` if not set

Update bigquery and googleanalytics drivers dependencies to newer versions, such that their transitive dependencies on google-api-client/google-http-client match the versions used in the new BigQuery driver