Co-authored-by: Demetrio <53023423+ddowbnac@users.noreply.github.com>

Previously, cumulative aggregators were being done in clojure after the mongo query was finished.  However, this
didn't work on multi-step queries where the cumulative aggregator wasn't the last step (for example, do a cumulative
sum and then filter the result).  Implementing cumulative operators directly in mongo fixes this issue.

Co-authored-by: William <william@metabase.com>

* test: see which drivers pass these tests

* Fix build

* Change test to use standard dataset, add initial implementation for bigquery

* Use fully qualified names for views

* Don't use transactions for databricks

* Make drop table only if exists, fix describe-index in mongo for views

* don't sync system.views

* Fix oracle drop-views, disable non-syncing views

* Use metabase-instance to find the table, this is important because redshift and oracle munge table name

* Fix oracle tests

* Fix oracle views

* Fix formatting

* Fix snowflake qualified components

* Change tests to opt out

* Fix sqlsserver

* Fix postgres test

* Disable h2 describe views test

* Address review

Co-authored-by: Case Nelson <case@metabase.com>

Co-authored-by: Alexander Solovyov <alexander@solovyov.net>

Fixes: #50422

Although we passed additional-options through subname in url parameters
to DriverManager, we were also passing `:schema nil` as connection property.

Co-authored-by: Case Nelson <case@metabase.com>

* Add catalog validation in can-connect? implementation

* Remove redundant let

* Adjust description

* Update modules/drivers/databricks/src/metabase/driver/databricks.clj



---------

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Fixes: #49913

In #49786 we handled parameterized STRING types like `STRING(255)`

Here we add handling for the other other types https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#parameterized_data_types

`BIGDECIMAL` and `DECIMAL` are aliases for `BIGNUMERIC` and `NUMERIC`
https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#parameterized_decimal_type

* Map Oracle DATE to type/DateTime

* Modify test

* Add test

Fixes: #49786

* Bump Oracle driver

Move to ojdbc11

* Update modules/drivers/oracle/deps.edn

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

---------

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* bump snowflake for java 17+

* Update modules/drivers/snowflake/deps.edn

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Update deps.edn

---------

Co-authored-by: Luis Paolini <paoliniluis@gmail.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* fix: add UserAgentEntry to databricks connection

Fixes: #49350

* Add test

* Fix formatting

* Use system schema

* Use data only from selected catalog

* Adjust tests

* Update modules/drivers/databricks/test/metabase/driver/databricks_test.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Comments

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* perf: Implement faster sync methods for postgres

Fixes #48575

Pulls work from redshift into the common postgres driver.

* Fix tests and formatting

* Move nested-field-column sync to sync functions so describe-fields will also get them

* Fix test

* Fix test

* Remove fixed safety test

* Add test specific database-supports feature for pk metadata

* Fix test

* Adrress PR feedback

* Fix test

* Don't use subselect for field-comment

* Fix quoting weird identifiers

* Make format string inline

* Mongo objects should download as JSON, not EDN

Fixes #48198

Prior to this change, object columns (base or effective type of :type/Dictionary) were just formatted with `(str
value)` which results in a csv or json download containing EDN formatted objects.

This is a bug because we present object column values as json in the app, so the expected formatting of the download
should match this.

The formatter function now takes this type into account. As well, since this is a type of formatting that should be
always applied (even when format_rows is false), the function is modified to unconditionally apply the json/encode
formatting to dictionary types when encountered.

* add a test

* add proper condition to test

* card-download should be public

* uncomment json encoding formatter

* set-cell! should keep encoded json string for Objects

I think this is the correct change; I don't really understand the reason for wrapping, encoding, decoding, and then
string-ing that value. Maybe I'm missing something.

* Adjusted xlsx Object set-cell! implementation

* forgot the not... inverted

* Use type date for case expression when there are Date and DateTime args

* Use logic for getting case expr type from annotate middleware

* Add e2e test

* Add bigquery test

* Update test

* cljfmt and comment

* Adjust e2e test

* fix: Always cast json number types as decimal

Fixes #48507

* Fix tests

* Fix tests

* Fix test

* Use information_schema for describe-database

* Unrelated fix: set dynamic-dataset-loading defaults to true

* cljfmt

* Fix test

This PR reimplements driver/describe-table for MongoDB. Before we would query a sample of documents from a collection and analyse them in Clojure. Instead, we now now execute a query that does a similar aggregation, but most of the calculation is done in the Mongo database.

Based on a few tests the performance is slightly slower when the collection contains small or deeply nested documents but much faster for large ones. But the main difference is in memory usage. This uses very little memory in the Metabase instance because all of the aggregation is done in the database.


Nested fields are a naturally recursive problem but here we unroll potential recursions to a `max-depth` number of queries that look for nesting at each depth level.

* ~ use DB to describe the table

* ~ optimize root query

* ~ nested-level-query works and gets objects too

* + root query gets objects too

* + driver/describe-table :mongo works

* ~ remove old implementation

* Various fixes for faster sync

Upgraded driver to 5.2.0
Updated data load to insert many rather than 1 row at a time.
Dropped max-depth to 7, see comment.

---------

Co-authored-by: Case Nelson <case@metabase.com>

* Address initial remarks

* Extract hive-like to separate module and set it as dependency

* Remove hive-like also from spark

* Parallel driver tests PoC

* Set fail-fast to false for now

* Try splitting up non-driver tests to see how broken tests are

* Whoops fix plain BE tests

* Ok nvm I'll test this in another branch

* Fix fail-fast

* Experiment with the improved Hawk split logic

* Fix some broken/flaky tests

* Experiment: try splitting MySQL 8 tests into FOUR jobs

* Divide other Postgres and MySQL tests up and use num-partitions = 2

* Another test fix 

* Flaky test fix 

* Try making more stuff fast

* Make athena fast??

* Fix a few more things

* Test fixes? 

* Fix configs

* Fix Mongo job syntax

* Fix busted test from #46942

* Fix Mongo config again

* wait-for-port needs to specify shell I guess

* More cleanup

* await-port can't have a timeout-minutes I guess

* Let's only parallelize MySQL for now.

* Cleanup action

* Cleanup wait-for-port action

* Fix another flaky test

* NOW driver tests will be FAST

* Need to mark driver tests too

* Fix wrong tag

* Use Hawk 1.0.5

* Fix busted metabase.public-settings-test/landing-page-setting-test

* Fix busted `metabase.api.database-test/get-database-test` etc. (hopefully)

* Fix busted `metabase.sync.sync-metadata.fields-test/sync-fks-and-fields-test` for Oracle

* Maybe this fixed `metabase.query-processor.middleware.permissions-test/e2e-ignore-user-supplied-perms-test` maybe not

* Fix busted metabase.api.dashboard-test/dependent-metadata-test because endpoint had differemt sort order than test

* Ok my test fix did not work

* Fix metabase.sync.sync-metadata.fields-test/sync-fks-and-fields-test for Redshift

* Better test name

* More test fixes 

* Schema fix

* PR feedback

* Split off test partitioning into separate PR

* Fix failing Oracle tests

* Another round of test fixes, hopefully

* Fix failing Redshift tests

* Maybe the last round of test fixes

* Fix Oracle

* Fix stray line

* fix: bigquery more resilient querying

Inline some function calls to make it easier to track what's happening.

Make sure that cancellation during the initial query and subsequent page
fetches are handled properly. Explicitly throw when cancelled.

Only retry queries if bigquery says they are retryable.

Try to cancel the BigQuery job if an exception or cancellation occurs.

* Add comment explaining execution flow

* Bump bigquery deps

* Bump biquery dependencies

* Fix tests

* Fix formatting

This was old logic to support certain drivers (eg. pre-JDBC Druid) and
isn't required for most. It's perfectly sound to filter or even break
out on a datetime column without bucketing.

Adds a new `:temporal/requires-default-unit` driver feature, and enables it only for the legacy Druid driver.

Fixes #47341

* Databricks JDBC driver base

* Add databricks CI job

* WIP data loading -- it works, further cleanup needed

* Cleanup

* Implement ->honeysql to enable data loading

* Hardcode catalog job var

* Implement driver methods and update tests

* Derive hive instead of sql-jdbc

* Cleanup leftovers after deriving hive

* Run databricks tests on push

* Cleanp and enable set-timezone

* Disable database creation by tests

* Add Databricks to broken drivers for timezone tests

* Exclude Databricks from test

* Enable have-select-privilege?-test

* Restore sql-jdbc-drivers-using-default-describe-table-or-fields-impl post rebase

* Restore joined-date-filter-test

* Adjust to work with dataset definition tests

* Adjust alternative date tests

* Remove leftover reflecttion warning set

* Update test exts

* cljfmt vscode

* Add databricks to kondo drivers

* Update metabase-plugin.yaml

* Update databricks_jdbc.clj

* Rework test extensions

* Update general data loading code to work with Databricks

* Reset tests to orig

* Use DateTimeWithLocalTZ for TIMESTAMP database type

* Convert to LocalDateTime for set-parameter

* Update test extensions field-base-type->sql-tyoe

* Update database-type->base-type

* Enable creation of time columns in test data even though not supported

* Fix typo

* Update tests

* Udpate tests

* Update drivers.yml

* Disable dynamic dataset loading tests

* Adjust the iso-8601-text-fields-should-be-queryable-date-test

* Update load-data/row-xform

* Add time type exception to test

* Update test data loading and enable test

* Whitespace

* Enable all driver jobs

* Update comment

* Make catalog mandatory

* Remove comment

* Remove log level from spec generation

* Update sql.qp/datetime-diff

* Update read-column-thunk

* Remove comment

* Simplify date-time->results-local-date-time

* Update comment

* Move definitions

* Update test extension types mapping

* Remove now obsolete ddl/insert-rows-honeysql-form implementation

* Update sql-jdbc.conn/connection-details->spec for perturb-db-details

* Update load-data/do-insert!

* Remove ssh tunnel from driver as tests do not work with it

* Update test

* Promote ::dynamic-dataset-loading to :test/dynamic-dataset-loading and modify corresponding tests

* Adjust to broken TIMESTAMP_NTZ sync

* Update read-column-thunk to return timestamps always in Z

* Comment

* Disable tests for dynamic datasets

* Return spark jobs into drivers.yml

* Update Databricks CI catalog

* Remove vscode cljfmt tweak

* Update iso-8601-text-fields-expected-rows

* Update datetime-diff

* Formatting

* cljfmt

* Add placeholder test

* Remove comment

* cljfmt

* Use EnableArrow=0 connection param

* Remove comment

* Comment

* Update tests

* cljfmt

* Update driver's deps.edn

* Update tests

* Implement alternative `describe-table`

* WIP Workaround for timestamp_ntz sync, will be thrown away probably

* Update metabase-plugin.yaml with schema filters

* Update driver to use schema filters and remove now redundant sync implemnetations

* Update tests

* Update tests extensions

* Update test

* Add feature flags for fast sync

* Implement describe-fields

* Implement describe-fks-sql

* Enable fast sync features

* Use full_data_type

* Comment

* Add exception for timestamp_ntz columns to new sync code

* Implement db-default-timezone

* Add timestamp_ntz ignored test

* Add db-default-timezone-test

* Fix typo

* Update setReadOnly

* Add comment on setAutoCommit

* Update chunk-size

* Add timezone-in-set-and-read-functions-test

* Drop Athena from driver exceptions

* Use set/intersection instead of a filter

* Add explicit fast-sync tests

* Update describe-fields-sql and add comment

* Add preprocess-additional-options

* Add leading semicolon test

* Disable dataset creation and update comment

* Rename driver to `databricks`

* Use old secret names

* Fix wrongly copied hsql list

* Temporarily allow database creation

* Add *allow-database-deletion*

* Temporarily allow database creation

* Disable database creation

* cljfmt

* cljfmt

* feat: BigQuery Faster Sync

Use describe-fields for much faster big-query sync.

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Fixes #47426

Date operations return values based on the timestamp offset rather than
the session/db timezone. Converting to `timestamp_ltz` first ensures
that we get predictable results.

In the test, it is important that database timezone matches
report_timezone so that post-processing gives proper results.

* fix: bigquery add null checks to result processing

Fixes #47339

On the related issue there are different stacktraces indicating
likely sources of null pointer exceptions.

1. `.getNextPage` is likely returning a nil value. I was unable to reproduce this but one thing I did notice is that `hasNextPage` is recommended over checking `.getNextPageToken`. Added nil handling around `page` possibly being nil.

2. `cancel-chan` may be triggered before processing begins and such `execute-bigquery` would pass nil as a TableResult to the initial reducer. Testing if cancel-chan happens at just the right moment would be too flaky for CI testing but I was able to reproduce this locally and is fixed by the nil handling added.

3. `cancel-chan` may be triggered during query processing. This is covered by a test now.

* Check hasNextPage in test:

* Add test for null getNextPage

* Fix cljfmt

* fix: sqlserver handle uniqueidentifier uuids

Fixes #46148

Include sqlserver in `uuid-type` handling as its `uniqueidentifier` type
stores uuids.

* Don't be so precise with varchar size

* Add seam for drivers to cast to text type

* Fix arg order

Fixes #30602

We were overriding localdate in bigquery driver to produce
offsetdatetimes for no apparent reason. This causes a number of problems
when dealing with date type columns.

* Bigquery verify bignumeric fingerprint and bin

Fixes #28573

Field values must be scanned.

In 45.3, these fields were not being fingerprinted properly. (tag `v1.45.3`)
In a 45 point release that appears to have been fixed. (branch `release-x.45.x`)
Confirmed fixed in master, adding test

* cljfmt

Previously, truncating a `:type/DateTime` column by `:month` or `:day`
would return a `:type/Date`, which subtly broke the query.

In particular, if you try to order-by the breakout column `Created At (month)`
then it would not get de-duplicated, causing a SQL error about conflicting
ORDER BY clauses.

Fixes #46992.

Lotsa fixes for this

* Add database type adjustments to server side generated temporal values

* Speculative base type change

* Infer timezone only from _actual_ user, not global user level

* Set timezone for actual user in test extensions

* Add test case

* Update type info generation

* Adjust server side relative datetime

* Update src/metabase/driver/sql/query_processor.clj

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* Fix base type logic

* Fix typo

* Fix formatting

* Update user name handling

* cljfmt

* Address review comments

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* feat: BigQuery nested fields

Updates dataset definitions to allow `nested-fields`

Adds the `::sql.qp/nfc-path` to include nfc-path in the field
identifier.

* Add bigquery to nested-fields feature. Rework geographical-tips to allow adding to bigquery

* Parse RECORD type into maps

* Fix tests

* Fix tests

* Fix formatting

* Handle arrays of records

* Fix formatting

* Conditionally add nested-fields

* Fix test

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>