Goal is to properly document the interface between MLv2 and the FE. In
particular I want to call out the health and fate of each function. Some
of them only exist to support legacy compatibility, for example, and
that should be documented to discourage new usage and make clear the
conditions for removing the functions.

* Enable not emptyable arguments for is-empty and not-empty clauses

* Add review remarks

* Shorten dataset name so it is usable with some drivers

* Disable test for Athena

* Handle possibly missing base type

* Update comment

* Missing base type field is treated as not emptyable

* Update desugar middleware

Now middleware adds base type to fields with integer id that are missing that first. This ensures correct function of is-empty expansion.

* Update desugar middleware

Cleanup base types necessary for desugaring is-empty

* Update desugar middleware cleanup

* Update test

* Adjust tests

* Remove now redundant e2e test

* Update comment

The first time you zoom in on a bin in a binned query, it works fine.

For example, Orders COUNT by SUBTOTAL auto-binned has a bin for 40-60.
If you zoom in on that, it correctly shows the range 40 to 60 broken
down into smaller bins which are 2.5 units wide. If you then try to
zoom in on one of these smaller bins, say 50-52.5, there are two
problems:

1. The filters from the first zoom (`>= 40` and `< 60`) are still
   there, and so are the new ones (`>= 50` and `< 52.5`).
2. The new filter is actually wrong; it uses the big width (20)
   from the original bins, not the width of the nested bins (2.5),
   so the new filters are `>= 50` and `< 70`.

This PR fixes both issues, so nested bins (1) remove the old filters,
and (2) use the correct width for the nested bins.

This logic was trying to match `[name join-alias]` pairs previously, but
that won't work since the later stage field doesn't have a `:join-alias`
on it. This falls back to matching on only the ID or name, but only if
it's unambiguous.

Fixes #40252.

This middleware shouldn't throw exceptions when `limit` or `offset`
query parametesr aren't integers.

* Remove old config + fix typo

* Add new analytics for 4 new illustration settings

* Add login page illustration (#39299)

* Move Metabot setting to a new place

* Add setting for login page illustration

* fix UI styling

* Fix not able to upload the same file twice

* Update the admin > appearance > conceal metabase copy

* Prepare the component for tests

* Add some unit tests

* Prevent settings not saved sometimes

After setting the value to/from `custom` one of the setting could be
failed to save, I suspect this happens because the race condition from
trying to save the setting values relatively at the same time.

* Make unit tests easier to read

* Add all unit test test cases

* Add basic E2E test

* Show fallback icon for broken images

* Make use of the proper component

* Update the comment to affect the reality

* Add an error message when file size is over 2MB

* Show an error message when the file size > 2MB

* Add the error message tests

* Preserve the preview image aspect-ratio

* Detect corrupted images

* Fix failing unit tests

* Address feedback: Type

* Feedback: Create meaningful variables

* Feedback: Make selector API more sensible

* Prevent race condition

Makes sure we save the settings completely before making other actions
that also change settings otherwise, there could be race conditions.

* Feedback: tighten up a type

* Feedback: Fix inconsistency callback calls

* 2. Add the rest 3 illustration settings (#40487)

* Add the rest 3 illustration settings

* Don't make the X icon feel lonely

* Fix illustration widget saving

* Fix type error

* 3. Update the select UX to match new design (#40491)

* Update the select UX to match new design

* Move info dot next to file upload button

* Simplify IllustrationWidget API

* Update setting title copy

* Clean up test comments

* Feedback: Clarify `any` type

* Feedback: Use shorthand object creation

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Feedback: Use global CSS glass instead of inlining values

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Feedback: Remove unnecessary curly brackets in JSX

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

---------

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* 4. Connect illustrations to new setting values (#40139)

* Wire landing page illustration to the setting value

* Wire the unsubscribe page to the setting value

* Prepare no rows error to be wired to the setting value

* Wire no question results illustration to the setting value

* Wire no search results illustration to setting value

* Fix error from fixing conflict

* Fix required unnecsseary prop on Image

* Use Image component

* Link dashboard card no results image to the setting value

* Move the no data results to more a generic directory

* Link collection picker no results image to the setting value

* Make illustrations more accessible

* Fix failed tests

* Test unscribe page illustration + test no illustration

* Add landing page tests

* Simplify Cypress selectors

* Refactor: Rename settings to match the new requirements

No question data -> No Data
No search results -> No object

These changes are made so, these 2 settings are generic enough to
include other illustrations later on.

* Add no data illustration test

* Add no object illustration tests

* Fix unsubscribe page background style

* Fix not able to build static viz

* 5. Follow-up on milestone 1 (#40710)

* Update custom background illustration scale behavior

* Remove the need to put an SVG file in static folder

* Fix test from rebasing off master with @testing-library/user-event upgrade

---------

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Send creator emails every week

mod the email by 52 instead of by 12 and use the current week as an
anchor

```clojure
user=> (require '[java-time.api :as t])
nil
user=> (let [every-day (take 1000 (t/iterate t/plus (t/local-date 2024 1 1) (t/days 1)))
             f         (fn [d]
                         (let [wf (java.time.temporal.WeekFields/of (java.util.Locale/getDefault))]
                           (.get d (.weekOfWeekBasedYear wf))))]
         (sort (keys (frequencies (map f every-day)))))
(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
 40 41 42 43 44 45 46 47 48 49 50 51 52)
```

```clojure
user=> (metabase.util.cron/cron-string->schedule-map "0 0 2 ? * 7")
{:schedule_minute 0,
 :schedule_day "sat",
 :schedule_frame nil,
 :schedule_hour 2,
 :schedule_type "weekly"}
```

* fix typo in creator sentiment email, clean up a bit

typo: had `:verison` instead of `version`.

And while I'm in there, make the payload all in one file and have the
email namespace json->bytes->b64-encode so the data is all in one place
and the setting that allows it to be present or not is co-located.

... among other things.

The order of arguments was wrong here, causing the default
implementation of `post-process-collection-childre` to be called. The
implementation for `:card` ends up removing the `:dataset_query`
entirely, which is what was intended.

For collections retrieved from the Search API, attach `location` and
`effective_location` so the frontend knows... where they're located.

* Cards with Results Should Show up in Subscriptions

Fixes: #40726

This PR fixes a bug where setting 'Hide thie card if there are no results' in a Dashcard's viz-settings would hide the
card even when there were results.

* adjust test to show that the setting is respected

Before, the setting would always hide the card. Now, the card is only hidden when there are no results. This test sets
up 4 cards: 2 are normal, 2 have no results, but 1 has the setting to hide empty. THerefore, we should expect to see 3 results.

* Make Formatting on Exports Optional

Formatting on exports can be turned off via a middleware key:

`:format-export?` which will control if the 'app style' formatting will be applied to the exported results.
`:format-export? false` will return the file as in previous Metabase versions.

* format_export's default value provided in the QP middleware

Should still default to formatted for v. 49, as the formatting is intended as a feature. This default value should be
provided in a central place, so is added to the middleware and is therefore not necessary for the API to provide a default.

* weird indentation

* Add format_export param to dashboard dashcard query endpoint

* Add format_export option to embedded dashboard downloads endpoint

* Move default true value to the streaming-results-writer impls

* Default true but for real this time

* Add format_export to the public Question endpoint

* Add test to card api

* Dashboard exports endpoint test

* Add test to dashboard embeds

Test still fails and I don't know why yet

* the dangers of println debugging....

Well, I was passing a string false the whole time. Turns out the problem was me

* Public Card downloads endpoint now has format_export and test

* export_format also works for adhoc queries using api/dataset

* Let defendpoint do the boolean parsing

* Embed endpoint should now work too.

* Unify format-rows and format-export middlewares

* Sneaky endpoint!

* In xlsx, always parse temporal strings so that viz-settings formatting is applied

* Snuck in an incomplete change by accident. removed it.

* Can now specify format/no format on subscriptions

* Fix and move migration

* default format rows to true in attachments

* Ok, I think that should actually work for subscriptions.

* Change endpoints to use format_rows instead of confusing format_export

* format_rows everywhere

* Fix more test failures

* redef'd function had wrong signature

* Make format export optional (FE part) (#40643)

* Refactor download URL builder code

Need to handle query `params` and request `body` separately

* Export `useHover` from `metabase/ui`

* Add unformatted export for dashboard cards

* Add unformatted export for saved questions

* Fix POST body encoding

* Add unformatted export for ad-hoc questions

* Ensure `format_export` is always false for Excel

* Add unformatted export to public questions

* Add unformatted export to embedded questions and dashboards

* Fix behavior for PNG format

* Add tests for `QueryDownloadPopover`

* Fix e2e downloads helper (allow URL query params)

* Rework alt key handling

* Add e2e test

* Switch to `format_rows` instead of `format_export`

* Add unformatted export option to subscriptions

* Fix `isHoldingAltKey` initial state

* Rework e2e test to use Total column

* Fix initial checkbox state for subscriptions

* Use "Alt" in tooltip label for Windows/Linux

* Show the subscriptions toggle only for CSV attachments

---------

Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>

* Address review feedback

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

if a previous run had an error, it will remain present but the state
would be "indexed". We should just clear that error.

* Parse wildcards in native queries

Add query_field.direct_reference

* Do away with old Model refs in Card model test

* Do not show stale cards with a select *

Fixes #40328

[Our documentation](https://www.metabase.com/docs/latest/people-and-groups/managing#group-managers) states that:

> Group managers can:
>
> - View all people in the Admin settings > People tab.

This fixes enforcement to be aligned with the documentation. This behavior makes sense, because as the docs also state, Group Managers should be allowed to *add* people to the group they manage, which they can only do if they can see those people!

Initially, I also removed a faulty test, which was described as:
```
;; Non-segmented users are allowed to ask for a list of all of the users in the Metabase instance. Pulse email lists
;; are an example usage of this. Segmented users should not have that ability. Instead they should only see
;; themselves. This test checks that GET /api/user for a segmented user only returns themselves
```

but actually failed to test this in a relevant way (because it was testing the `GET /api/user` endpoint rather than the `GET /api/user/recipients` endpoint). The test had continued to pass only because the user was the only member of the group they managed.

I initially thought this behavior wasn't desired, but as it turns out, it *is* in fact documented behavior to disallow sandboxed users from seeing any email suggestions. Investigating, I found that a bug was allowing sandboxed users to see all users on the `/api/user/recipients` endpoint if the user-visibility setting was set to `:all`.

Thus, the second commit fixes the bug and re-adds the (fixed) test. A sandboxed user now only sees their own user when hitting `/api/user/recipients`.

* Use expression fix on both pk-ref and value-ref

The gist: inside the query expressions are referred to as [:expression
"foo"]. But as a nested query, it's just another field, and the fact
that it was an expression is irrelevant and should not leak from that
stage. So "downstream" should just be [:field "foo" base-type].

Oddly, if you just fix one field or the other, the query would be
"valid", would log an error, but no valid values would be returned.

It seems like the invalid expression reference was just dropped so we
only selected a single column and those were removed (we run `(filter
valid-tuples?)` as we only want valid key/value pairs).

So let's add a `(> (count values) 0)` which was failing previously.

* Fix indexing error

When updating the model index, it diffs the existing values vs the new
values and deletes appropriate and adds the new values. But it was using
the wrong column when updating the model index values. It used `pk_ref`
but that table uses `model_pk`. This led to the following errors on
stats:

```sql
select pk_ref, value_ref, state, error from model_index where state = 'error'

pk_ref,value_ref,state,error
"[""field"",563028,null]","[""field"",563018,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",534030,null]","[""field"",534037,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",545945,null]","[""field"",545948,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
```

Now those errors will be fixed and the model indexes will correctly
track the values

* Bump indexing threshold to 25,000

Had been limited to 5,000 rows of model indexing. This was a made up
number as we went into this endeavor. But it's proving to be far too
small. Bumping to 25,000

* Parititon deletions and inserts

With a higher limit need to ensure that we don't send too large of a
query.

```clojure
model-index=> (t2/insert! ModelIndexValue
                          (map (fn [id]
                                 {:name           (str (gensym "value"))
                                  :model_pk       id
                                  :model_index_id 1})
                               (range 25000)))
Execution error (PSQLException) at org.postgresql.jdbc.PgPreparedStatement/<init> (PgPreparedStatement.java:105).
PreparedStatement can have at most 65,535 parameters. Please consider using arrays, or splitting the query in several ones, or using COPY. Given query has 75,000 parameters
```

* move comment and fix test failure explanation

comment should be next to the number? check. and also need to use
`mc/explain` which returns info rather than `mc/validate` which returns
a bool:

```clojure
;; bad
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
                     (-> (mc/validate [:sequential [:tuple number? string?]] values)
                         (me/humanize)))
nil
;; good
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
                     (-> (mc/explain [:sequential [:tuple number? string?]] values)
                         (me/humanize)))
[nil nil [["should be a number"]]]
```

* Split from #40146: only the filter changes

* Don't include format-rows changes here.

* Test fixes 

* Test fix 

* Misc test fixes 

* Test fix 

* Tweak SQL Server fix

They were showing "break out by" and quick filters, when they should be
showing "break out by" and "See these Orders".

These are the `pivot` and `underlying-records` drills, internally.

Fixes #40174.

* Sort official collections first in API endpoints

For both `/api/collection/tree` and `/api/collection/items`, sort
collections with official collections coming first.


Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

Fixes #40383

If a native query parameter is set up with a string widget type that
supports a `case-sensitive` option, and that parameter is tied to a
dashboard filter that did not support that option, the option was still
leaking into the filter as an option where options were not expected.

For instance the native query parameter used `string/contains` and the
dashboard used `string/=`, the QP would try to build a filter like `[:=
field value {:case-sensitive false}]` which is an illegal filter.

* In #37160 we added a migration to convert all datetime columns to
timestamp with time zone by using a fixed list of what to convert. but
it turns out some customer instances has a different list. So this PR
removed the fixed list and use a query to get a list of all datetime
columns.