Fixes #47426

Date operations return values based on the timestamp offset rather than
the session/db timezone. Converting to `timestamp_ltz` first ensures
that we get predictable results.

In the test, it is important that database timezone matches
report_timezone so that post-processing gives proper results.

* fix: bigquery add null checks to result processing

Fixes #47339

On the related issue there are different stacktraces indicating
likely sources of null pointer exceptions.

1. `.getNextPage` is likely returning a nil value. I was unable to reproduce this but one thing I did notice is that `hasNextPage` is recommended over checking `.getNextPageToken`. Added nil handling around `page` possibly being nil.

2. `cancel-chan` may be triggered before processing begins and such `execute-bigquery` would pass nil as a TableResult to the initial reducer. Testing if cancel-chan happens at just the right moment would be too flaky for CI testing but I was able to reproduce this locally and is fixed by the nil handling added.

3. `cancel-chan` may be triggered during query processing. This is covered by a test now.

* Check hasNextPage in test:

* Add test for null getNextPage

* Fix cljfmt

* fix: sqlserver handle uniqueidentifier uuids

Fixes #46148

Include sqlserver in `uuid-type` handling as its `uniqueidentifier` type
stores uuids.

* Don't be so precise with varchar size

* Add seam for drivers to cast to text type

* Fix arg order

Fixes #30602

We were overriding localdate in bigquery driver to produce
offsetdatetimes for no apparent reason. This causes a number of problems
when dealing with date type columns.

* Bigquery verify bignumeric fingerprint and bin

Fixes #28573

Field values must be scanned.

In 45.3, these fields were not being fingerprinted properly. (tag `v1.45.3`)
In a 45 point release that appears to have been fixed. (branch `release-x.45.x`)
Confirmed fixed in master, adding test

* cljfmt

Previously, truncating a `:type/DateTime` column by `:month` or `:day`
would return a `:type/Date`, which subtly broke the query.

In particular, if you try to order-by the breakout column `Created At (month)`
then it would not get de-duplicated, causing a SQL error about conflicting
ORDER BY clauses.

Fixes #46992.

Lotsa fixes for this

* Add database type adjustments to server side generated temporal values

* Speculative base type change

* Infer timezone only from _actual_ user, not global user level

* Set timezone for actual user in test extensions

* Add test case

* Update type info generation

* Adjust server side relative datetime

* Update src/metabase/driver/sql/query_processor.clj

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* Fix base type logic

* Fix typo

* Fix formatting

* Update user name handling

* cljfmt

* Address review comments

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* feat: BigQuery nested fields

Updates dataset definitions to allow `nested-fields`

Adds the `::sql.qp/nfc-path` to include nfc-path in the field
identifier.

* Add bigquery to nested-fields feature. Rework geographical-tips to allow adding to bigquery

* Parse RECORD type into maps

* Fix tests

* Fix tests

* Fix formatting

* Handle arrays of records

* Fix formatting

* Conditionally add nested-fields

* Fix test

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Properly detect when Snowflake is using a local key file when expanding connection properties (#46348)

* Cljfmt config part 2

* Backport updated config and linter fork from part 3

* Update formatting

* Cljfmt

* Fix new GH action

* Kondo Config cleanup

* Restore warnings for with-log-messages-for-level until #28827 is merged

* Updated clojure.test hooks

* Fix kondo warning

* Test defn/defmacro exclamation point linter should also use :parallel/disallowed (part 1)

* WIP

* Time for me to learn to spell

* Finish the cleanup

* Remove code I was typing as an example

* Fix renamed var

* Fix the SAML tests

* Fix Kondo warning

* Add Kondo linter to disallow hardcoded driver names in tests

* Fix all tests using hardcoded driver names

* Test fixes 

* Appease Kondo

* Remove normal-drivers-except

* Test fix 

* Test fixes 

* MongoDB test fixes

dep is brought in by google cloud

From `clj -X:deps tree :aliases '[:ee :drivers]'`

```
  . metabase/bigquery-cloud-sdk metabase/modules/drivers/bigquery-cloud-sdk
    . com.google.cloud/google-cloud-bigquery 2.38.1
      . com.google.cloud/google-cloud-core 2.35.0
      [...]
      . javax.annotation/javax.annotation-api 1.3.2
```

Before this change:

```
❯ clj -X:deps tree :aliases '[:ee :drivers]' | grep 'javax.annotation/javax.annotation-api'
      . javax.annotation/javax.annotation-api 1.3.2
```

After:

```
❯ clj -X:deps tree :aliases '[:ee :drivers]' > deps-master

❯ echo $?
0
```

* fix: mongo ObjectId should be orderable

Fixes #46259

Mongo ObjectId derive :type/TextLike but that can't be an orderable
type, so we mark the specific Mongo type as orderable for malli schemas.

* Fix test

* Avoid adding temporal-unit to lhs cols

* Use :default temporal-unit

* Simplify logic, update comments

* Update substitute-field-filter-test

* Update align-temporal-unit-with-param-type-test

* Update field-filter-date-test

* Use end-excludding gte lt filter for DateTime fields

* Update substitute-field-filter-test

* Update align-temporal-unit-with-param-type-test

* Update field-filter-date-test

* Update bigquery test + comment

* Update date-str->qp-aware-offset-dt

* Add guard for unexpected date string format

This is just for the completeness, I haven't encountered it.

* Address review remarks

* Update comment

* add mu/defn-

* use new built-in mu/defn-

* mu/defn- preserves metadata

---------

Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* Bigquery Pagination Use next page to get proper results

Fixes 45953

WIP

* use new-page in values-iterator

* disable just the part of the test that is still flaky

Want to keep the other part of the test that does prove we're not getting duplicates due to incorrect pagination.

* Potential Redshift sync flake fix

* Make Redshift sync much more robust

* Add basic auth to dbms-version

* Add test

* Post sync hook stub for implicit joins testing stub

* Add join alias to field lvalues to enable sorting on joined fields

* Disable :foreign-keys on Mongo

* Require :left-join support for implicit joins instead of :foreign-keys

* Update implicit joins tests

* Adjust sync-fields-test

* Update implicit joins feature check test

* Transform post-sync-hook to normal function

* Add foreign key relationships only for dbmses without :foreign-keys feature

* Update test to handle Oracle correctly

* Split convoluted fn

* Avoid unnecessary computations for datasets with no fks

* Update docstring

* Fix driver usage

* Transform :foreign-keys to :metadata/key-constraints in test data loading code

* Update sync_test.clj

* Update driver_test.clj

* Update moviedb.clj

* Update dataset_definition_test.clj

* Update fetch_metadata.clj

* Update fields_test.clj

* Update driver.clj

* Update driver/sql.clj

* Set sql driver join support to true for all joins

Deriving drivers are expected to set to false where applicable.

* Update sqllite.clj

* Remove foreign-keys from spark

* Remove :foreign-keys from presto

* Remove :foreign-keys from Athena

* Remove foreign-keys from big query

Reading docs it seems fk inference should be ok. Let's see the test results. Act based on that.

* Update test_metadata.cljc

* Set key-constraints to false for bigquery

* Add foreign keys to sqlite manually until sync is fixed

* Return driver require to Athena

* Correct typo

* Add naive primary key heuristic

* Update pk fk logic to handle name components correctly

* Add alias escaping to presto

* Add ordering to test

* Add order by to test

* Update test

* Remove use of rewrite-fields-to-force-using-column-aliases in order by fields

* Add exception to alias forcing

* Different approach to exception from alias forcing

* Alternative approach for prefixing idents in bigq

* All seelcted fields by desired alias

* Rewrite only fields not from this source table

* Update test

* Enable breakout-on-fk-field-test for :left-join drivers

* Add feature comment

* Explicit joins tests foreign-keys removal

* Update nested_queries_test.clj

* Update remapping tests

* Update tests

* Update tests to handle sqlite results format

* Disable metadata/key-constraints on sqlite during tests until

* Address remarks

* Remove mt/with-mock-fks-for-drivers-without-fk-constraints

* Update bigquery test

* Update tests

* Adjust row level restrictions

* Add parameterized-sql feature

* Update comment

* Update leftovers

* Order keys

* Remove foreign keys from frontend

* Fix FE unit

* Update driver changelog

* Address review remark

* Update test/metabase/query_processor/test_util.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update test

* Update comment and reduce expression

* Update comment

* Address remarks

* Fix merge

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* tests for drivers with spaces in their identifiers

* Add driver feature

* Fix tests

* Remove ambiguous alias for sqlite

* Try to fix bigquery and sqlite tests not seeing left-join support

* Add explicit support for left-join because of foreign-keys during test hacks

* Handle left-join support weirdness

* Fix test

* Reuse connection for test data loading & JDBC data loading overhaul

* Revert silly change

* Remove unneeded logic for recording which datasets are loaded or not-yet-loaded

* Ok so I guess we do need the code to detect duplicate dataset loading after all

* B I G  improvements

* Appease Kondo

* Test fixes 

* More test fixes 

* Fix dataset-already-loaded?-test

* Fix test

* Fix Presto JDBC and greatly improve test data load speed for it

* Postgres test fix 

* Fix compilation errors

* MySQL fixes 

* Fix syntax error

* MySQL fixes 

* Implement `dataset-already-loaded?` for Athena and cache the set of existing databases

* Fix Redshift

* Fix Presto JDBC

* Vertica test fix 

* Fix Oracle dataset-already-loaded?

* Test fixes and parallelize some API tests.

* Test fixes 

* Oracle test fix? 

* Another test fix 

* Stop putting random spaces in code bodies this is not JavaScript code

* with-temp-vals-in-db should merge in original value of Database settings

* The most test fixes 

* 3 or 4 test fixes 

* Add schemas around stuff to catch errors

* Appease Kondo

* Implement dataset-already-loaded? for vertica

* Implement dataset-already-loaded? for SQLite and Snowflake

* [WIP] Testing driver filters

* Only test returned values in CI

* Add :uuid-type for dbs that support that type (test oracle's support)

* Try again for oracle test, redshift does not support uuid-type

* Add tests, support where available

* Add empty/null tests

* Adjust test query for athena

* Reset logging

* Fix comments

* Add Athena uuid conversion

* Address PR review

* Add comments

* Address PR comments

* Use varchar for portability. Add inline uuid casting for athena

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Fix deserialization of mongo native query string

* Add test

* Sync Snowflake `VARIANT` type as :type/SnowflakeVariant`

* Test fix 

* get-sub-props -> ->sub-props because it doesn't 'get' anything

* add defendpoint dox

* make style consistent

* respond to review comment

* fix formatting

* fix: add timezone to bigquery param substitution

Fixes #43597

* Fix indentation and remove report-timezone lookup from loop

This comes in two parts.

**Rebuilt query logic**

I rewrote the query exec logic to use one `future` and one `a/go` block,
reporting results to a `promise`. This is easier to follow and doesn't leak
threads like the original implementation with two `future`s.

The single future is guaranteed to exit. It blocks on the BigQuery call.
That will either complete or throw an exception. Either the exception or
the BigQuery result is sent to the promise, and then the future is
complete. The only reference to that future is overwritten after the `promise`
resolves, so the `future`, its closure, and its thread don't persist longer than needed.

**Return a reducible result**

Previously the results were returned as a lazy sequence, but that brings
a lot of `seq` overhead, and is a risk of retaining the entire result
set in memory until the query is all done. Above all, it doesn't support
`reduced?` to explicitly drop any further results.

The `thunk` function style of `qp.reducible/reducible-rows` doesn't fit
neatly here, so I implemented the reducible logic directly on top of the
`Iterable` BigQuery results.

**Verifying**

This is a bit tricky. I've been testing it like this:

- Add a BigQuery database to my instance
- Create a saved question and note its ID (say, 123)
- Create an API key in (EE) Metabase and save it in an env var
- Install the `siege` load testing tool
- Install VisualVM and monitor my Clojure REPL process's thread count
- `siege --concurrent=15 --time=30s --header="x-api-key: $API_KEY" "http://localhost:3000/api/card/123/query POST"`

That will create a bunch of new threads while it's getting called, but
within a minute of the last queries they'll gradually drop out of the
thread pools and it will return to the baseline number of threads.

Fixes #39789.

* Split unrelated changes into separate PR

* Fix test since user perms set reset fix split into other PR

* Fix syntax error

* Test fixes 

* Test fix??? 

* Test fix 

* Support compiling SQL with inlined parameters directly

* Test fix

* Make `format-honeysql` a multimethod

* Add test for #32543

* Test also fixes #44915

* Add explicit test for #44915

* Better test for #44915

* More test fixes 

* Fix Kondo error