* Make Formatting on Exports Optional

Formatting on exports can be turned off via a middleware key:

`:format-export?` which will control if the 'app style' formatting will be applied to the exported results.
`:format-export? false` will return the file as in previous Metabase versions.

* format_export's default value provided in the QP middleware

Should still default to formatted for v. 49, as the formatting is intended as a feature. This default value should be
provided in a central place, so is added to the middleware and is therefore not necessary for the API to provide a default.

* weird indentation

* Add format_export param to dashboard dashcard query endpoint

* Add format_export option to embedded dashboard downloads endpoint

* Move default true value to the streaming-results-writer impls

* Default true but for real this time

* Add format_export to the public Question endpoint

* Add test to card api

* Dashboard exports endpoint test

* Add test to dashboard embeds

Test still fails and I don't know why yet

* the dangers of println debugging....

Well, I was passing a string false the whole time. Turns out the problem was me

* Public Card downloads endpoint now has format_export and test

* export_format also works for adhoc queries using api/dataset

* Let defendpoint do the boolean parsing

* Embed endpoint should now work too.

* Unify format-rows and format-export middlewares

* Sneaky endpoint!

* In xlsx, always parse temporal strings so that viz-settings formatting is applied

* Snuck in an incomplete change by accident. removed it.

* Can now specify format/no format on subscriptions

* Fix and move migration

* default format rows to true in attachments

* Ok, I think that should actually work for subscriptions.

* Change endpoints to use format_rows instead of confusing format_export

* format_rows everywhere

* Fix more test failures

* redef'd function had wrong signature

* Make format export optional (FE part) (#40643)

* Refactor download URL builder code

Need to handle query `params` and request `body` separately

* Export `useHover` from `metabase/ui`

* Add unformatted export for dashboard cards

* Add unformatted export for saved questions

* Fix POST body encoding

* Add unformatted export for ad-hoc questions

* Ensure `format_export` is always false for Excel

* Add unformatted export to public questions

* Add unformatted export to embedded questions and dashboards

* Fix behavior for PNG format

* Add tests for `QueryDownloadPopover`

* Fix e2e downloads helper (allow URL query params)

* Rework alt key handling

* Add e2e test

* Switch to `format_rows` instead of `format_export`

* Add unformatted export option to subscriptions

* Fix `isHoldingAltKey` initial state

* Rework e2e test to use Total column

* Fix initial checkbox state for subscriptions

* Use "Alt" in tooltip label for Windows/Linux

* Show the subscriptions toggle only for CSV attachments

---------

Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>

* Address review feedback

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* Parse wildcards in native queries

Add query_field.direct_reference

* Do away with old Model refs in Card model test

* Do not show stale cards with a select *

Fixes #40328

[Our documentation](https://www.metabase.com/docs/latest/people-and-groups/managing#group-managers) states that:

> Group managers can:
>
> - View all people in the Admin settings > People tab.

This fixes enforcement to be aligned with the documentation. This behavior makes sense, because as the docs also state, Group Managers should be allowed to *add* people to the group they manage, which they can only do if they can see those people!

Initially, I also removed a faulty test, which was described as:
```
;; Non-segmented users are allowed to ask for a list of all of the users in the Metabase instance. Pulse email lists
;; are an example usage of this. Segmented users should not have that ability. Instead they should only see
;; themselves. This test checks that GET /api/user for a segmented user only returns themselves
```

but actually failed to test this in a relevant way (because it was testing the `GET /api/user` endpoint rather than the `GET /api/user/recipients` endpoint). The test had continued to pass only because the user was the only member of the group they managed.

I initially thought this behavior wasn't desired, but as it turns out, it *is* in fact documented behavior to disallow sandboxed users from seeing any email suggestions. Investigating, I found that a bug was allowing sandboxed users to see all users on the `/api/user/recipients` endpoint if the user-visibility setting was set to `:all`.

Thus, the second commit fixes the bug and re-adds the (fixed) test. A sandboxed user now only sees their own user when hitting `/api/user/recipients`.

* Use expression fix on both pk-ref and value-ref

The gist: inside the query expressions are referred to as [:expression
"foo"]. But as a nested query, it's just another field, and the fact
that it was an expression is irrelevant and should not leak from that
stage. So "downstream" should just be [:field "foo" base-type].

Oddly, if you just fix one field or the other, the query would be
"valid", would log an error, but no valid values would be returned.

It seems like the invalid expression reference was just dropped so we
only selected a single column and those were removed (we run `(filter
valid-tuples?)` as we only want valid key/value pairs).

So let's add a `(> (count values) 0)` which was failing previously.

* Fix indexing error

When updating the model index, it diffs the existing values vs the new
values and deletes appropriate and adds the new values. But it was using
the wrong column when updating the model index values. It used `pk_ref`
but that table uses `model_pk`. This led to the following errors on
stats:

```sql
select pk_ref, value_ref, state, error from model_index where state = 'error'

pk_ref,value_ref,state,error
"[""field"",563028,null]","[""field"",563018,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",534030,null]","[""field"",534037,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",545945,null]","[""field"",545948,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
```

Now those errors will be fixed and the model indexes will correctly
track the values

* Bump indexing threshold to 25,000

Had been limited to 5,000 rows of model indexing. This was a made up
number as we went into this endeavor. But it's proving to be far too
small. Bumping to 25,000

* Parititon deletions and inserts

With a higher limit need to ensure that we don't send too large of a
query.

```clojure
model-index=> (t2/insert! ModelIndexValue
                          (map (fn [id]
                                 {:name           (str (gensym "value"))
                                  :model_pk       id
                                  :model_index_id 1})
                               (range 25000)))
Execution error (PSQLException) at org.postgresql.jdbc.PgPreparedStatement/<init> (PgPreparedStatement.java:105).
PreparedStatement can have at most 65,535 parameters. Please consider using arrays, or splitting the query in several ones, or using COPY. Given query has 75,000 parameters
```

* move comment and fix test failure explanation

comment should be next to the number? check. and also need to use
`mc/explain` which returns info rather than `mc/validate` which returns
a bool:

```clojure
;; bad
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
                     (-> (mc/validate [:sequential [:tuple number? string?]] values)
                         (me/humanize)))
nil
;; good
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
                     (-> (mc/explain [:sequential [:tuple number? string?]] values)
                         (me/humanize)))
[nil nil [["should be a number"]]]
```

* Split from #40146: only the filter changes

* Don't include format-rows changes here.

* Test fixes 

* Test fix 

* Misc test fixes 

* Test fix 

* Tweak SQL Server fix

They were showing "break out by" and quick filters, when they should be
showing "break out by" and "See these Orders".

These are the `pivot` and `underlying-records` drills, internally.

Fixes #40174.

* Sort official collections first in API endpoints

For both `/api/collection/tree` and `/api/collection/items`, sort
collections with official collections coming first.


Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

Fixes #40383

If a native query parameter is set up with a string widget type that
supports a `case-sensitive` option, and that parameter is tied to a
dashboard filter that did not support that option, the option was still
leaking into the filter as an option where options were not expected.

For instance the native query parameter used `string/contains` and the
dashboard used `string/=`, the QP would try to build a filter like `[:=
field value {:case-sensitive false}]` which is an illegal filter.

Include postgresql tables that are visible through column grants instead of full table grants for simple questions (#40034)

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Funnel Chart Static-Viz should handle row data more robustly

The row data coming into the funnel chart render method can evidently have a few slightly different shapes, and may
need to be 'coordinated' with the :funnel.rows key from viz-settings (names and order are specified there).

This is a WIP to make the method more robust to different row data shapes. For example:

[[1 100] [2 200]] should work as well as [["A" 100] ["B" 200]] and correctly handle the viz-settings

WIP because I need to add a test or two, try to get a handle on the exact schema that is allowed for 'row shape' and
the funnel.row viz key, and see if there are failing cases that I haven't considered yet.

* Funnel rows should now work more effectively with possible raw-rows

* Add a test that checks the success of the render and correct order

* Update src/metabase/pulse/render/body.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Update src/metabase/pulse/render/body.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Addressing feedback from the review

 - renamed build-funnel-rows to funnel-rows
 - used 'funnel-viz' as the binding for the {:key "asdf" :name "asdf" :enabled true} maps from the viz settings
 - added docstring to the function to try clarify its 2 branches (keys vs indices on the rows)

---------

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Trend Chart Viz Should Use Proper Card execute in Rendering

WIP

Fixes: #39854

The trend chart follows a code path shared with multi-series static-viz. This means there was an implicit expectation
that a dashcard is associated with the card at all times. This isn't true for alerts, so the code is fixed to remove
the failure.

* Add a test

* add missed :breakout that actually made the test work

oops

* Adding comments to try clarify what's going on

* /api/collection/:id/items gets `here` and `below`

During postprocessing of `collection` children retrieved by
`/api/collection/:id/items`, annotate each of the children with `below`
and `here` keys representing the presence of questions, models, or
collections at this level of the hierarchy or below it.

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

Fixes: #40306

Our datetime formatter relies on the `java-time.api`, for which there are many different, sometimes confusing, formatter patterns: https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatterBuilder.html#appendPattern-java.lang.String-

In this case, 'YYYY' is a week-of-year style year, which calculates which week a date falls into before returning the
year. Sometimes days near the start/end of a year will fall into a week in the wrong year.

For example, apparently 2023-12-31 falls into the 1st week of 2024, which probably not the year you'd expect to
see. What we probably do want is 'yyyy' which calculates what day of the year the date is and then returns the year
based off of that instead of the week number.

For an explanation, you can check out this SO answer: https://stackoverflow.com/a/46395342 provides an explanation.

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* E2E support for CRUD + running pMBQL queries; new MLv2 normalization

* More work on improved normalization

* Update schema references

* Test fixes 

* Move the MBQL parameters schema

* Performance improvements

* Improved pivot code etc.

* Clean namespaces

* Fixes 

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Split pivot stuff off

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Test fix

* Another fix

* Test fix 

* decrypt and read values from airgap token

- Adds notion of "AirgapToken"
- Adds max-users and company optional fields to `TokenStatus`
- Adds branch to `fetch-token-status*` for handling airgapped tokens

* Add support for handling airgap tokens

- validate airgap token valid-thru date

* enforce user creation limits using airgap token

* fix airgap token reads + fill in the token data

* in oss mode don't try to read the airgap_ token

* use airgap-token? helper

* tighten the user-creation logic

- make the check correct
- account for archived users

Co-authored-by: John Swanson <john.swanson@metabase.com>

* check airgap user count

setting premium-embedding-token
app startup

* move decryption code into ee namespace

* add some tests for token decryption

* fix tests and add tests

* move ee features into ee tests

* add typehint + warn on reflection

* add a check for missing public key resource

* respond to review

- add docstring to AirgapToken
- remove outdated comment
- fix off-by-1 error

* valid-now? takes a TokenStatus instead of a :map

* revert usage of mt/with-temporary-setting-values

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>

* Rename :model/Metric to :model/LegacyMetric
* Rename ::lib.schema.metadata/metric tp ::lib.schema.metadata/legacy-metric
* Rename models.metric/Metric to models.metric/LegacyMetric
* Move /api/metric endpoints to /api/legacy-metric