* Apply Column Sort To Pivot Sections

Fixes #49437

This PR fixes the dataset API so that column sorts (ascending/descending settings on pivot-rows and pivot-cols) are
properly parsed and passed through the QP.

After that, I've also modified the post processor to use the sort orders properly in pivoted exports.

* println debugging :)

* fixing a few tests

* fix a few more tests

* Try snowflake 3.19.0

* Deafault to java 21 for drivers

* Trigger CI

* Try running things with --add-opens to see if it solves our problems

* Fix `update-view-dashboard-timestamp-test`

* Try snowflake 3.19.0

* Deafault to java 21 for drivers

* Try running things with --add-opens to see if it solves our problems

* Fix `update-view-dashboard-timestamp-test`

* Update deps.edn

* Switch to Java 21

* Docker image needs to use --add-opens option

* Add note about change to Java 21 to driver changelog

---------

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>

* Fixes https://github.com/metabase/metabase/issues/33037

* Align query and fix the linter

* Add blurb from C3P0 docs to comment for unreturnedConnectionTimeout

* Enable debugUnreturnedConnectionStackTraces in data-warehouse-connection-pool-properties

Closes #47981

* Add defsetting for jdbc-data-warehouse-debug-unreturned-connection-stack-traces

And only enable debugUnreturnedConnectionStackTraces in the C3P0 config if the setting is enabled. There is some
memory cost to enabling the setting, and since maxPoolSize is user-configurable, it seems safer to disable
debugUnreturnedConnectionStackTraces by default.

* Add metabase.logger/level-enabled?

* Log a warning if C3P0 logs are not configured correctly to log stack traces

Log a warning if the user enables the jdbc-data-warehouse-debug-unreturned-connection-stack-traces but does not raise
the log level for the com.mchange package to INFO or higher.

* Add docs for MB_JDBC_DATA_WAREHOUSE_DEBUG_UNRETURNED_CONNECTION_STACK_TRACES

* Improve markdown docs for MB_JDBC_DATA_WAREHOUSE_DEBUG_UNRETURNED_CONNECTION_STACK_TRACES

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Keep Pivoted Option works in Subscriptions

Fixes: #49525

Even though the user could toggle 'Keep data pivoted` in the Sharing Sidebar UI, the setting was not being propagated
to the backend, so attachments would not be pivoted.

* add a cypress test

* add viz settings to get the pivot working

* fix for linter

* remove eslint ignore

---------

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

* Revert "[streaming-response] Use HttpEndpoint.isOpen() instead of reading a byte from TCP socket (#49460)"

This reverts commit 060baf0c.

* Disable connection reuse for requests that serve streaming responses

* Revert "Remove old `MB_API_KEY` setting (#48592)"

This reverts commit 187e9508.

* preserve metadata of the wrapped handler

* Correct embedding hash options type semantically

* Pass `X-Metabase-Locale` to embed downloads

* wip locale header middleware

* Allow smart scalar text to be translated

* Stop using global variable

* Add more details to embedding types

* Extend the E2E util type and correct some properties types

* Add e2e test

* Fix dashboard name

* Add a test for static embed question

* Make a type name follows a convention

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Fix failed E2E tests due to missing default values

* Fix public embed not working with `?locale` anymore

* Revert "wip locale header middleware"

This reverts commit 83d8d135.

* Preserve locale when in the `as-admin` macro is used

Motivating issue:

```shell
❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124' x-metabase-locale:fr -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "Count"
]
```

I'm running a query that has aggregated columns and it is returning the
untranslated. Note that these are computed and dynamically given a
display name.

```clojure
(case tag
  :count     (i18n/tru "Count")
  :cum-count (i18n/tru "Cumulative count"))
```

The problem is that we run embedding things as admin. And that gets the
current user, which in embed and public spaces is nil, and so binds a
nil to user-locale, which defaults to english when translating.

After:

```
❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124' x-metabase-locale:fr -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "Nombre de lignes"
]

❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124' x-metabase-locale:zh -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "行数"
]

❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124

' x-metabase-locale:es -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "Contar"
]
```

This is fetching a dashboard's card which is a query of orders grouped
by created at month, sum of total.

---------

Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>

* docs: cloud environment variables

* Apply suggestions from code review

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Pivot Measures Order Used in Pivot Exports

Fixes #48442

A pivot table can have any number of measures, and the user can order these by dragging in the UI. Before this PR,
that order was ignored and measures would alway be in index order, which is confusing for any user who needs the
measures to be displayed in a particular order, especially if they've re-ordered them in the pivot viz settings UI.

A test has been added to check that measure order is used.

A few minor changes to the pivot qp and post-processor
 - measure indices are looked up in the pivot qp and added in viz-settings order
 - the pivot measures are only added if the qp has not already added them.
 - pivot-opts Malli spec has been made in the namespace, adjusted to allow `nil` as a valid pivot-opts output, and
 used in relevant functions

* address review points.

* add a rows order test

* Filter out Empty Rows from Pivot Exports

Fixes #49353

The linked issue is not actually related to pivot export size but is instead related to the 'Min of Created At: Month'
aggregation; the default aggregation function was `+`, so it broke when the date string was encountered.

That was fixed.

As I was trynig to keep the export small if possible, I noticed that in some cases empty rows are appended, so I added
the filter so that if a pivot row's values are completely empty, it doesn't add it.

Finally, this PR also adds the 'sub section totals' which I noticed were missing from the exports. This comes up when
you have 3+ pivot-rows configured, so you can see the subtotals for the first pivot row and the subtotals nested
within those sections for the second pivot row, and so on.

* add test for non-numeric values

* Make sure the new name refs match on aggregations not just breakouts

* cljfmt

* Fix :pivot-measures with column names

* Fix the test

* Update src/metabase/query_processor/pivot.clj

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* Add a test

* Add a test

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>

Adds filters to generative testing for queries.

There's a lot of new code here, but don't panic, it's quite straightforward:
generating filter values, in the right shapes for various filter expressions.

The query can contain all kinds of things, in particular the whole original query in the case of pivot questions, so
we should check only the part that actually gets resolved by the middleware.

* perf: Implement faster sync methods for postgres

Fixes #48575

Pulls work from redshift into the common postgres driver.

* Fix tests and formatting

* Move nested-field-column sync to sync functions so describe-fields will also get them

* Fix test

* Fix test

* Remove fixed safety test

* Add test specific database-supports feature for pk metadata

* Fix test

* perf: faster mysql sync with describe-fields

Fixes: #49010

* Adrress PR feedback

* Fix tests

* Fix test

* Add nil table-schema

* Don't use subselect for field-comment

* Fix quoting weird identifiers

* Make format string inline

* Update src/metabase/driver/mysql.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update src/metabase/driver/mysql.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Fix tests

* Fix database-type

* Fix tests

* Fix test

* Fix tests

* Exclude mysql table_schema

* Handle tinyInt1IsBit

* Fix test

* Only get one db at a time

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

These are helpful for creating literal dates and times in a
cross-platform way. Useful for generating queries, in particular for
expression and filter values.

The ref (usually) contains enough information to unambiguously find the
matching implictly-joined column, even if there are several FKs that
could join the target table.

The remaining ambiguous case is if the *same* FK column exists
multiple times - current MBQL uses raw field IDs for the FK field, so it
can't distinguish between eg. an FK in the main table and the one from a
self-join.

* [streaming-response] Use HttpEndpoint.isOpen() instead of reading a byte from TCP socket

* Comment out cancellation test