Make sure the query to fetch result_metadata for native sandboxes doesn't use the streaming results handler (#50049)

* XLSX Pivot Exports Now Use The Sort Settings from Viz Settings

Prior to this PR, the xlsx pivot exports worked correctly except for the fact that no sorting (ascending or
descending) was applied to the pivot rows or columns. Now, those settings are used when they exist.

* only try to set sort when a correct setting exists

* add ooxml-full dep for correct classes

* Update src/metabase/query_processor/streaming/xlsx.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

Fixes: #50072

When we `describe-fields` we are passing db details, but sometimes we
have the database name in `dbname` rather than in `db` depending on the
connection specs and possibly environment. So we check for both now.

== Goal ==

Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them
  - This is what we're adding for H2 snapshots (dumps) with this PR.

The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.

== Implementation ==

In 592360c9 I wrongly understood that
database details would be omitted when dumping to H2, but this is only
true for dumping H2 databases.

Fix this by omitting database details also when dumping databases with
`is_attached_dwh` set.

== How to test ==

To prepare, download a H2 JAR matching the H2 version used by Metabase:
```
wget https://repo1.maven.org/maven2/com/h2database/h2/2.1.214/h2-2.1.214.jar
```

If the H2 version does not match, you will get an error when trying to
open the H2 shell:
```
Exception in thread "main" org.h2.jdbc.JdbcSQLNonTransientConnectionException: Unsupported database file version or invalid file header in file "[REDACTED]" [90048-232]
```

=== New behaviour ===

Setting the `is_attached_dwh` field hides the database details from H2 dumps:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - In addition to the fields you would normally set, also set
     `is_attached_dwh: true`.
   - This also works when adding this flag to a database that previously
     did not have this flag set.
2. Start your Metabase instance.
3. Run `${metabase} dump-to-h2 ./dump-file-h2` to create the H2 snapshot file.
4. Run `java -cp h2-2.1.214.jar org.h2.tools.Shell -url "jdbc:h2:file:./dump-file-h2;ifexists=true"` to open a H2 shell.
5. Verify that `SELECT name,details FROM metabase_database;` shows `{}` for the database you added in step 1

=== Original behaviour ===

Behaviour without setting the `is_attached_dwh` field is unchanged:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - Only set the fields you would normally set.  Do not set
     `is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Run `${metabase} dump-to-h2 ./dump-file-h2` to create the H2 snapshot file.
4. Run `java -cp h2-2.1.214.jar org.h2.tools.Shell -url "jdbc:h2:file:./dump-file-h2;ifexists=true"` to open a H2 shell.
5. Verify that `SELECT name,details FROM metabase_database;` shows a non-empty object (i.e. not `{}`) for the database you added in step 1

Fixes: 592360c9
Closes: https://github.com/metabase/harbormaster/issues/5526

* sync the attached datawarehouse

requires either "sync_db" true or a table name

it will error
```shell
❯ http post 'localhost:3000/api/notify/db/attached_datawarehouse' x-metabase-apikey:apikey -pb
Must provide `sync_db` or a `table_name` to sync
```

It will sync the db when sync_db is true [NOTE: this is outdated. now just omitting the table name will sync the db]
```shell
❯ echo '{"synchronous?": true, "sync_db": true}' | http post 'localhost:3000/api/notify/db/attached_datawarehouse' x-metabase-apikey:apikey -pb
{
    "success": true
}
```

It will sync existing tables
```shell
❯ http post 'localhost:3000/api/notify/db/attached_datawarehouse' table_name=existing schema_name=public x-metabase-apikey:apikey -pb
{
    "success": true
}
```

it will error if it cannot find a table
```shell
❯ http post 'localhost:3000/api/notify/db/attached_datawarehouse' table_name=new schema_name=public x-metabase-apikey:apikey -pb
{
    "cause": "Unable to identify table 'public.new'",
    "data": {
        "schema_name": "public",
        "status-code": 404,
        "table_name": "new"
    },
    "message": "Unable to identify table 'public.new'",
    "schema_name": "public",
    "table_name": "new",
    "trace": [],
    "via": [
        {
            "data": {
                "schema_name": "public",
                "status-code": 404,
                "table_name": "new"
            },
            "message": "Unable to identify table 'public.new'",
            "type": "clojure.lang.ExceptionInfo"
        }
    ]
}
```

if i create that table
```sql
attached=# create table new (id int);
CREATE TABLE
```

it will then find and sync it

```shell
❯ http post 'localhost:3000/api/notify/db/attached_datawarehouse' table_name=new schema_name=public x-metabase-apikey:apikey -pb
{
    "success": true
}
```

* formatting

* Tests for attached datawarehouse

* PR nits

typo in `thorw` -> `throw`
comment explaining why `find-and-sync-new-table` is always sync
gensym the db name in test
remove a binding for a fn `sync!` that's only called once
test `{:sync_db true}` syncs the whole db

* stupid formatting 

* Use absence of table_name as indicator to sync database

* Apply Column Sort To Pivot Sections

Fixes #49437

This PR fixes the dataset API so that column sorts (ascending/descending settings on pivot-rows and pivot-cols) are
properly parsed and passed through the QP.

After that, I've also modified the post processor to use the sort orders properly in pivoted exports.

* println debugging :)

* fixing a few tests

* fix a few more tests

* Try snowflake 3.19.0

* Deafault to java 21 for drivers

* Trigger CI

* Try running things with --add-opens to see if it solves our problems

* Fix `update-view-dashboard-timestamp-test`

* Try snowflake 3.19.0

* Deafault to java 21 for drivers

* Try running things with --add-opens to see if it solves our problems

* Fix `update-view-dashboard-timestamp-test`

* Update deps.edn

* Switch to Java 21

* Docker image needs to use --add-opens option

* Add note about change to Java 21 to driver changelog

---------

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Luis Paolini <paoliniluis@gmail.com>

* Fixes https://github.com/metabase/metabase/issues/33037

* Align query and fix the linter

* Add blurb from C3P0 docs to comment for unreturnedConnectionTimeout

* Enable debugUnreturnedConnectionStackTraces in data-warehouse-connection-pool-properties

Closes #47981

* Add defsetting for jdbc-data-warehouse-debug-unreturned-connection-stack-traces

And only enable debugUnreturnedConnectionStackTraces in the C3P0 config if the setting is enabled. There is some
memory cost to enabling the setting, and since maxPoolSize is user-configurable, it seems safer to disable
debugUnreturnedConnectionStackTraces by default.

* Add metabase.logger/level-enabled?

* Log a warning if C3P0 logs are not configured correctly to log stack traces

Log a warning if the user enables the jdbc-data-warehouse-debug-unreturned-connection-stack-traces but does not raise
the log level for the com.mchange package to INFO or higher.

* Add docs for MB_JDBC_DATA_WAREHOUSE_DEBUG_UNRETURNED_CONNECTION_STACK_TRACES

* Improve markdown docs for MB_JDBC_DATA_WAREHOUSE_DEBUG_UNRETURNED_CONNECTION_STACK_TRACES

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Keep Pivoted Option works in Subscriptions

Fixes: #49525

Even though the user could toggle 'Keep data pivoted` in the Sharing Sidebar UI, the setting was not being propagated
to the backend, so attachments would not be pivoted.

* add a cypress test

* add viz settings to get the pivot working

* fix for linter

* remove eslint ignore

---------

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

* Revert "[streaming-response] Use HttpEndpoint.isOpen() instead of reading a byte from TCP socket (#49460)"

This reverts commit 060baf0c.

* Disable connection reuse for requests that serve streaming responses

* Revert "Remove old `MB_API_KEY` setting (#48592)"

This reverts commit 187e9508.

* preserve metadata of the wrapped handler

* Correct embedding hash options type semantically

* Pass `X-Metabase-Locale` to embed downloads

* wip locale header middleware

* Allow smart scalar text to be translated

* Stop using global variable

* Add more details to embedding types

* Extend the E2E util type and correct some properties types

* Add e2e test

* Fix dashboard name

* Add a test for static embed question

* Make a type name follows a convention

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Fix failed E2E tests due to missing default values

* Fix public embed not working with `?locale` anymore

* Revert "wip locale header middleware"

This reverts commit 83d8d135.

* Preserve locale when in the `as-admin` macro is used

Motivating issue:

```shell
❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124' x-metabase-locale:fr -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "Count"
]
```

I'm running a query that has aggregated columns and it is returning the
untranslated. Note that these are computed and dynamically given a
display name.

```clojure
(case tag
  :count     (i18n/tru "Count")
  :cum-count (i18n/tru "Cumulative count"))
```

The problem is that we run embedding things as admin. And that gets the
current user, which in embed and public spaces is nil, and so binds a
nil to user-locale, which defaults to english when translating.

After:

```
❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124' x-metabase-locale:fr -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "Nombre de lignes"
]

❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124' x-metabase-locale:zh -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "行数"
]

❯ http get 'http://localhost:3000/api/embed/dashboard/eyJhbGciOiJIUzI1NiJ9.eyJyZXNvdXJjZSI6eyJkYXNoYm9hcmQiOjEyfSwicGFyYW1zIjp7fX0.QZ-tEWpgJeSMUuaPfuH6_VQwM2yXNMcuureWCtHsszU/dashcard/162/card/124

' x-metabase-locale:es -pb | jq '.data|.cols|map(.display_name)'
[
  "State",
  "Contar"
]
```

This is fetching a dashboard's card which is a query of orders grouped
by created at month, sum of total.

---------

Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>

* docs: cloud environment variables

* Apply suggestions from code review

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Pivot Measures Order Used in Pivot Exports

Fixes #48442

A pivot table can have any number of measures, and the user can order these by dragging in the UI. Before this PR,
that order was ignored and measures would alway be in index order, which is confusing for any user who needs the
measures to be displayed in a particular order, especially if they've re-ordered them in the pivot viz settings UI.

A test has been added to check that measure order is used.

A few minor changes to the pivot qp and post-processor
 - measure indices are looked up in the pivot qp and added in viz-settings order
 - the pivot measures are only added if the qp has not already added them.
 - pivot-opts Malli spec has been made in the namespace, adjusted to allow `nil` as a valid pivot-opts output, and
 used in relevant functions

* address review points.

* add a rows order test

* Filter out Empty Rows from Pivot Exports

Fixes #49353

The linked issue is not actually related to pivot export size but is instead related to the 'Min of Created At: Month'
aggregation; the default aggregation function was `+`, so it broke when the date string was encountered.

That was fixed.

As I was trynig to keep the export small if possible, I noticed that in some cases empty rows are appended, so I added
the filter so that if a pivot row's values are completely empty, it doesn't add it.

Finally, this PR also adds the 'sub section totals' which I noticed were missing from the exports. This comes up when
you have 3+ pivot-rows configured, so you can see the subtotals for the first pivot row and the subtotals nested
within those sections for the second pivot row, and so on.

* add test for non-numeric values

* Make sure the new name refs match on aggregations not just breakouts

* cljfmt

* Fix :pivot-measures with column names

* Fix the test

* Update src/metabase/query_processor/pivot.clj

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* Add a test

* Add a test

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>

Adds filters to generative testing for queries.

There's a lot of new code here, but don't panic, it's quite straightforward:
generating filter values, in the right shapes for various filter expressions.