Co-authored-by: Oleksandr Yakushev <alex@bytopia.org>

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* XLSX Pivot Exports Now Use The Sort Settings from Viz Settings

Prior to this PR, the xlsx pivot exports worked correctly except for the fact that no sorting (ascending or
descending) was applied to the pivot rows or columns. Now, those settings are used when they exist.

* only try to set sort when a correct setting exists

* add ooxml-full dep for correct classes

* Update src/metabase/query_processor/streaming/xlsx.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* Try snowflake 3.19.0

* Deafault to java 21 for drivers

* Trigger CI

* Try running things with --add-opens to see if it solves our problems

* Fix `update-view-dashboard-timestamp-test`

* Try snowflake 3.19.0

* Deafault to java 21 for drivers

* Try running things with --add-opens to see if it solves our problems

* Fix `update-view-dashboard-timestamp-test`

* Update deps.edn

* Switch to Java 21

* Docker image needs to use --add-opens option

* Add note about change to Java 21 to driver changelog

---------

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>

### Description

This PR updates Macaw to the latest version, which exposes a new analyzer which handles compound queries.

It also finishes the migration to using tagged-union result maps consistently across all the interfaces.

This should also remove the atrociously verbose analysis error messages in CI.

### Description

This is a low level step towards applying table-level permissions to native queries.

It replaces the more generic and powerful `:ast-walker-1` analyzer with the more conservative `:basic-select` version we recently landed. The advantage of this version is that it protects against dynamic table references and false negatives due to masking.

it's been long ago that Ring has been updated to depend on org.apache.commons/commons-fileupload2-core

There was a new version that supports jetty 11.0.24 https://github.com/ring-clojure/ring/blob/master/CHANGELOG.md

https://github.com/jetty/jetty.project/security/advisories/GHSA-g8m5-722r-8whq

* [Notification] Migrate user invited email (#48215)

* [Notification] Migrate alert create email (#48292)

* [Notification] Migrate slack token error email (#48333)

* Collapse `metabase.shared.*` namespaces

* Fix Kondo warnings

* Does updating the stories-data keys fix the failing tests?

* Appease msgcat

* Appease msgcat

* Fix typo

* Make the build happy

* Appease fslint

* Address initial remarks

* Extract hive-like to separate module and set it as dependency

* Remove hive-like also from spark

* Parallel driver tests PoC

* Set fail-fast to false for now

* Try splitting up non-driver tests to see how broken tests are

* Whoops fix plain BE tests

* Ok nvm I'll test this in another branch

* Fix fail-fast

* Experiment with the improved Hawk split logic

* Fix some broken/flaky tests

* Experiment: try splitting MySQL 8 tests into FOUR jobs

* Divide other Postgres and MySQL tests up and use num-partitions = 2

* Another test fix 

* Flaky test fix 

* Try making more stuff fast

* Make athena fast??

* Fix a few more things

* Test fixes? 

* Fix configs

* Fix Mongo job syntax

* Fix busted test from #46942

* Fix Mongo config again

* wait-for-port needs to specify shell I guess

* More cleanup

* await-port can't have a timeout-minutes I guess

* Let's only parallelize MySQL for now.

* Cleanup action

* Cleanup wait-for-port action

* Fix another flaky test

* NOW driver tests will be FAST

* Need to mark driver tests too

* Fix wrong tag

* Use Hawk 1.0.5

* Fix busted metabase.public-settings-test/landing-page-setting-test

* Fix busted `metabase.api.database-test/get-database-test` etc. (hopefully)

* Fix busted `metabase.sync.sync-metadata.fields-test/sync-fks-and-fields-test` for Oracle

* Maybe this fixed `metabase.query-processor.middleware.permissions-test/e2e-ignore-user-supplied-perms-test` maybe not

* Fix busted metabase.api.dashboard-test/dependent-metadata-test because endpoint had differemt sort order than test

* Ok my test fix did not work

* Fix metabase.sync.sync-metadata.fields-test/sync-fks-and-fields-test for Redshift

* Better test name

* More test fixes 

* Schema fix

* PR feedback

* Split off test partitioning into separate PR

* Fix failing Oracle tests

* Another round of test fixes, hopefully

* Fix failing Redshift tests

* Maybe the last round of test fixes

* Fix Oracle

* Fix stray line

* Do not cache all token check failures

We want to cache token checks to avoid an issue where we repeatedly ask
the store "hey, is this token valid?? is this token valid?? is this
token valid??" for the same token.

However, transient errors can also occur. For example, maybe a network
issue causes the HTTP request to fail entirely. In this case, if we
cache the result, the user needs to restart metabase (or wait 5 minutes
until the cache is cleared) before they can attempt to validate their
token again.

This PR moves the cache logic deeper into the stack. We want to cache
"successful" responses from the store API - cases where the store has
told us categorically that the token is or is not valid. We don't want
or need to cache other things that might happen. Maybe your token isn't
the right length - we can recalculate that, it's ok. Maybe you get a 503
error from the Store - we should let you retry. Maybe your network is
having issues and you can't contact the Store at all - again, we should
let you retry.

The one potential issue I see here is that if the store goes down, we'll
massively increase the number of requests we send to the store,
potentially making it harder to recover. If this is a concern, I can add
a circuit breaker: if we repeatedly get errors back from the store, back
off and stop making requests for a while.

* Add a circuit breaker to store API requests

In the pathological case where the store goes down for >5 minutes, the
cache will expire and all instances everywhere will start repeatedly
making requests for token validation at once. This might make recovering
from an outage more difficult.

This adds a circuit breaker around the API request. If the call
repeatedly throws (5XX errors, socket timeouts, etc.) then we'll pause
for 1 minute, during which time all calls to token validation will
immediately fail without making any request to the API.

After one minute, we'll allow one request through to the API. If it
succeeds, we'll go back to normal operation. Otherwise, we'll wait
another minute.

* Databricks JDBC driver base

* Add databricks CI job

* WIP data loading -- it works, further cleanup needed

* Cleanup

* Implement ->honeysql to enable data loading

* Hardcode catalog job var

* Implement driver methods and update tests

* Derive hive instead of sql-jdbc

* Cleanup leftovers after deriving hive

* Run databricks tests on push

* Cleanp and enable set-timezone

* Disable database creation by tests

* Add Databricks to broken drivers for timezone tests

* Exclude Databricks from test

* Enable have-select-privilege?-test

* Restore sql-jdbc-drivers-using-default-describe-table-or-fields-impl post rebase

* Restore joined-date-filter-test

* Adjust to work with dataset definition tests

* Adjust alternative date tests

* Remove leftover reflecttion warning set

* Update test exts

* cljfmt vscode

* Add databricks to kondo drivers

* Update metabase-plugin.yaml

* Update databricks_jdbc.clj

* Rework test extensions

* Update general data loading code to work with Databricks

* Reset tests to orig

* Use DateTimeWithLocalTZ for TIMESTAMP database type

* Convert to LocalDateTime for set-parameter

* Update test extensions field-base-type->sql-tyoe

* Update database-type->base-type

* Enable creation of time columns in test data even though not supported

* Fix typo

* Update tests

* Udpate tests

* Update drivers.yml

* Disable dynamic dataset loading tests

* Adjust the iso-8601-text-fields-should-be-queryable-date-test

* Update load-data/row-xform

* Add time type exception to test

* Update test data loading and enable test

* Whitespace

* Enable all driver jobs

* Update comment

* Make catalog mandatory

* Remove comment

* Remove log level from spec generation

* Update sql.qp/datetime-diff

* Update read-column-thunk

* Remove comment

* Simplify date-time->results-local-date-time

* Update comment

* Move definitions

* Update test extension types mapping

* Remove now obsolete ddl/insert-rows-honeysql-form implementation

* Update sql-jdbc.conn/connection-details->spec for perturb-db-details

* Update load-data/do-insert!

* Remove ssh tunnel from driver as tests do not work with it

* Update test

* Promote ::dynamic-dataset-loading to :test/dynamic-dataset-loading and modify corresponding tests

* Adjust to broken TIMESTAMP_NTZ sync

* Update read-column-thunk to return timestamps always in Z

* Comment

* Disable tests for dynamic datasets

* Return spark jobs into drivers.yml

* Update Databricks CI catalog

* Remove vscode cljfmt tweak

* Update iso-8601-text-fields-expected-rows

* Update datetime-diff

* Formatting

* cljfmt

* Add placeholder test

* Remove comment

* cljfmt

* Use EnableArrow=0 connection param

* Remove comment

* Comment

* Update tests

* cljfmt

* Update driver's deps.edn

* Update tests

* Implement alternative `describe-table`

* WIP Workaround for timestamp_ntz sync, will be thrown away probably

* Update metabase-plugin.yaml with schema filters

* Update driver to use schema filters and remove now redundant sync implemnetations

* Update tests

* Update tests extensions

* Update test

* Add feature flags for fast sync

* Implement describe-fields

* Implement describe-fks-sql

* Enable fast sync features

* Use full_data_type

* Comment

* Add exception for timestamp_ntz columns to new sync code

* Implement db-default-timezone

* Add timestamp_ntz ignored test

* Add db-default-timezone-test

* Fix typo

* Update setReadOnly

* Add comment on setAutoCommit

* Update chunk-size

* Add timezone-in-set-and-read-functions-test

* Drop Athena from driver exceptions

* Use set/intersection instead of a filter

* Add explicit fast-sync tests

* Update describe-fields-sql and add comment

* Add preprocess-additional-options

* Add leading semicolon test

* Disable dataset creation and update comment

* Rename driver to `databricks`

* Use old secret names

* Fix wrongly copied hsql list

* Temporarily allow database creation

* Add *allow-database-deletion*

* Temporarily allow database creation

* Disable database creation

* cljfmt

* cljfmt

* Bump Kondo version to 2024.08.29

* Fix Kondo warnings from new version of Kondo

* Fix failing tests because schema was broken

* Implement a simple greedy approximation for dedupe-joins

Includes https://github.com/metabase/macaw/pull/97, which will prevent
some false positives

* perf: Use optimized stats functions for computing insights

* perf: Rewrite fingerprinters/with-error-handling to not generate closures

* fix: Don't reuse global-fingerprinter object

* Add `clojure -M:kondo` and `clojure -M:kondo:kondo/all` and bump version

* Fix Kondo errors

* Fix Kondo+LSP issues with `defendpoint`, `defenterprise`, etc.

* Use replace-deps instead of deps for speed

* Ok apparently maybe we do need to copy configs when we run Kondo on CI

* Oops `./bin/kondo.sh` should not try to use `clj-kondo`

* Remove references to GA driver folders

* Cljfmt config part 2

* Backport updated config and linter fork from part 3

* Update formatting