* Enable not emptyable arguments for is-empty and not-empty clauses

* Add review remarks

* Shorten dataset name so it is usable with some drivers

* Disable test for Athena

* Handle possibly missing base type

* Update comment

* Missing base type field is treated as not emptyable

* Update desugar middleware

Now middleware adds base type to fields with integer id that are missing that first. This ensures correct function of is-empty expansion.

* Update desugar middleware

Cleanup base types necessary for desugaring is-empty

* Update desugar middleware cleanup

* Update test

* Adjust tests

* Remove now redundant e2e test

* Update comment

The first time you zoom in on a bin in a binned query, it works fine.

For example, Orders COUNT by SUBTOTAL auto-binned has a bin for 40-60.
If you zoom in on that, it correctly shows the range 40 to 60 broken
down into smaller bins which are 2.5 units wide. If you then try to
zoom in on one of these smaller bins, say 50-52.5, there are two
problems:

1. The filters from the first zoom (`>= 40` and `< 60`) are still
   there, and so are the new ones (`>= 50` and `< 52.5`).
2. The new filter is actually wrong; it uses the big width (20)
   from the original bins, not the width of the nested bins (2.5),
   so the new filters are `>= 50` and `< 70`.

This PR fixes both issues, so nested bins (1) remove the old filters,
and (2) use the correct width for the nested bins.

This logic was trying to match `[name join-alias]` pairs previously, but
that won't work since the later stage field doesn't have a `:join-alias`
on it. This falls back to matching on only the ID or name, but only if
it's unambiguous.

Fixes #40252.

This middleware shouldn't throw exceptions when `limit` or `offset`
query parametesr aren't integers.

* Remove old config + fix typo

* Add new analytics for 4 new illustration settings

* Send creator emails every week

mod the email by 52 instead of by 12 and use the current week as an
anchor

```clojure
user=> (require '[java-time.api :as t])
nil
user=> (let [every-day (take 1000 (t/iterate t/plus (t/local-date 2024 1 1) (t/days 1)))
             f         (fn [d]
                         (let [wf (java.time.temporal.WeekFields/of (java.util.Locale/getDefault))]
                           (.get d (.weekOfWeekBasedYear wf))))]
         (sort (keys (frequencies (map f every-day)))))
(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
 40 41 42 43 44 45 46 47 48 49 50 51 52)
```

```clojure
user=> (metabase.util.cron/cron-string->schedule-map "0 0 2 ? * 7")
{:schedule_minute 0,
 :schedule_day "sat",
 :schedule_frame nil,
 :schedule_hour 2,
 :schedule_type "weekly"}
```

* fix typo in creator sentiment email, clean up a bit

typo: had `:verison` instead of `version`.

And while I'm in there, make the payload all in one file and have the
email namespace json->bytes->b64-encode so the data is all in one place
and the setting that allows it to be present or not is co-located.

For collections retrieved from the Search API, attach `location` and
`effective_location` so the frontend knows... where they're located.

* Cards with Results Should Show up in Subscriptions

Fixes: #40726

This PR fixes a bug where setting 'Hide thie card if there are no results' in a Dashcard's viz-settings would hide the
card even when there were results.

* adjust test to show that the setting is respected

Before, the setting would always hide the card. Now, the card is only hidden when there are no results. This test sets
up 4 cards: 2 are normal, 2 have no results, but 1 has the setting to hide empty. THerefore, we should expect to see 3 results.

* Make Formatting on Exports Optional

Formatting on exports can be turned off via a middleware key:

`:format-export?` which will control if the 'app style' formatting will be applied to the exported results.
`:format-export? false` will return the file as in previous Metabase versions.

* format_export's default value provided in the QP middleware

Should still default to formatted for v. 49, as the formatting is intended as a feature. This default value should be
provided in a central place, so is added to the middleware and is therefore not necessary for the API to provide a default.

* weird indentation

* Add format_export param to dashboard dashcard query endpoint

* Add format_export option to embedded dashboard downloads endpoint

* Move default true value to the streaming-results-writer impls

* Default true but for real this time

* Add format_export to the public Question endpoint

* Add test to card api

* Dashboard exports endpoint test

* Add test to dashboard embeds

Test still fails and I don't know why yet

* the dangers of println debugging....

Well, I was passing a string false the whole time. Turns out the problem was me

* Public Card downloads endpoint now has format_export and test

* export_format also works for adhoc queries using api/dataset

* Let defendpoint do the boolean parsing

* Embed endpoint should now work too.

* Unify format-rows and format-export middlewares

* Sneaky endpoint!

* In xlsx, always parse temporal strings so that viz-settings formatting is applied

* Snuck in an incomplete change by accident. removed it.

* Can now specify format/no format on subscriptions

* Fix and move migration

* default format rows to true in attachments

* Ok, I think that should actually work for subscriptions.

* Change endpoints to use format_rows instead of confusing format_export

* format_rows everywhere

* Fix more test failures

* redef'd function had wrong signature

* Make format export optional (FE part) (#40643)

* Refactor download URL builder code

Need to handle query `params` and request `body` separately

* Export `useHover` from `metabase/ui`

* Add unformatted export for dashboard cards

* Add unformatted export for saved questions

* Fix POST body encoding

* Add unformatted export for ad-hoc questions

* Ensure `format_export` is always false for Excel

* Add unformatted export to public questions

* Add unformatted export to embedded questions and dashboards

* Fix behavior for PNG format

* Add tests for `QueryDownloadPopover`

* Fix e2e downloads helper (allow URL query params)

* Rework alt key handling

* Add e2e test

* Switch to `format_rows` instead of `format_export`

* Add unformatted export option to subscriptions

* Fix `isHoldingAltKey` initial state

* Rework e2e test to use Total column

* Fix initial checkbox state for subscriptions

* Use "Alt" in tooltip label for Windows/Linux

* Show the subscriptions toggle only for CSV attachments

---------

Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>

* Address review feedback

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* Parse wildcards in native queries

Add query_field.direct_reference

* Do away with old Model refs in Card model test

* Do not show stale cards with a select *

Fixes #40328

[Our documentation](https://www.metabase.com/docs/latest/people-and-groups/managing#group-managers) states that:

> Group managers can:
>
> - View all people in the Admin settings > People tab.

This fixes enforcement to be aligned with the documentation. This behavior makes sense, because as the docs also state, Group Managers should be allowed to *add* people to the group they manage, which they can only do if they can see those people!

Initially, I also removed a faulty test, which was described as:
```
;; Non-segmented users are allowed to ask for a list of all of the users in the Metabase instance. Pulse email lists
;; are an example usage of this. Segmented users should not have that ability. Instead they should only see
;; themselves. This test checks that GET /api/user for a segmented user only returns themselves
```

but actually failed to test this in a relevant way (because it was testing the `GET /api/user` endpoint rather than the `GET /api/user/recipients` endpoint). The test had continued to pass only because the user was the only member of the group they managed.

I initially thought this behavior wasn't desired, but as it turns out, it *is* in fact documented behavior to disallow sandboxed users from seeing any email suggestions. Investigating, I found that a bug was allowing sandboxed users to see all users on the `/api/user/recipients` endpoint if the user-visibility setting was set to `:all`.

Thus, the second commit fixes the bug and re-adds the (fixed) test. A sandboxed user now only sees their own user when hitting `/api/user/recipients`.

* Use expression fix on both pk-ref and value-ref

The gist: inside the query expressions are referred to as [:expression
"foo"]. But as a nested query, it's just another field, and the fact
that it was an expression is irrelevant and should not leak from that
stage. So "downstream" should just be [:field "foo" base-type].

Oddly, if you just fix one field or the other, the query would be
"valid", would log an error, but no valid values would be returned.

It seems like the invalid expression reference was just dropped so we
only selected a single column and those were removed (we run `(filter
valid-tuples?)` as we only want valid key/value pairs).

So let's add a `(> (count values) 0)` which was failing previously.

* Fix indexing error

When updating the model index, it diffs the existing values vs the new
values and deletes appropriate and adds the new values. But it was using
the wrong column when updating the model index values. It used `pk_ref`
but that table uses `model_pk`. This led to the following errors on
stats:

```sql
select pk_ref, value_ref, state, error from model_index where state = 'error'

pk_ref,value_ref,state,error
"[""field"",563028,null]","[""field"",563018,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",534030,null]","[""field"",534037,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
"[""field"",545945,null]","[""field"",545948,null]",error,"ERROR: column ""pk_ref"" does not exist Position: 68"
```

Now those errors will be fixed and the model indexes will correctly
track the values

* Bump indexing threshold to 25,000

Had been limited to 5,000 rows of model indexing. This was a made up
number as we went into this endeavor. But it's proving to be far too
small. Bumping to 25,000

* Parititon deletions and inserts

With a higher limit need to ensure that we don't send too large of a
query.

```clojure
model-index=> (t2/insert! ModelIndexValue
                          (map (fn [id]
                                 {:name           (str (gensym "value"))
                                  :model_pk       id
                                  :model_index_id 1})
                               (range 25000)))
Execution error (PSQLException) at org.postgresql.jdbc.PgPreparedStatement/<init> (PgPreparedStatement.java:105).
PreparedStatement can have at most 65,535 parameters. Please consider using arrays, or splitting the query in several ones, or using COPY. Given query has 75,000 parameters
```

* move comment and fix test failure explanation

comment should be next to the number? check. and also need to use
`mc/explain` which returns info rather than `mc/validate` which returns
a bool:

```clojure
;; bad
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
                     (-> (mc/validate [:sequential [:tuple number? string?]] values)
                         (me/humanize)))
nil
;; good
model-index-test=> (let [values [[1 "foo"] [2 "foo"] ["foo" "foo"]]]
                     (-> (mc/explain [:sequential [:tuple number? string?]] values)
                         (me/humanize)))
[nil nil [["should be a number"]]]
```

* Split from #40146: only the filter changes

* Don't include format-rows changes here.

* Test fixes 

* Test fix 

* Misc test fixes 

* Test fix 

* Tweak SQL Server fix

They were showing "break out by" and quick filters, when they should be
showing "break out by" and "See these Orders".

These are the `pivot` and `underlying-records` drills, internally.

Fixes #40174.

* Sort official collections first in API endpoints

For both `/api/collection/tree` and `/api/collection/items`, sort
collections with official collections coming first.


Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

Fixes #40383

If a native query parameter is set up with a string widget type that
supports a `case-sensitive` option, and that parameter is tied to a
dashboard filter that did not support that option, the option was still
leaking into the filter as an option where options were not expected.

For instance the native query parameter used `string/contains` and the
dashboard used `string/=`, the QP would try to build a filter like `[:=
field value {:case-sensitive false}]` which is an illegal filter.

Include postgresql tables that are visible through column grants instead of full table grants for simple questions (#40034)

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Funnel Chart Static-Viz should handle row data more robustly

The row data coming into the funnel chart render method can evidently have a few slightly different shapes, and may
need to be 'coordinated' with the :funnel.rows key from viz-settings (names and order are specified there).

This is a WIP to make the method more robust to different row data shapes. For example:

[[1 100] [2 200]] should work as well as [["A" 100] ["B" 200]] and correctly handle the viz-settings

WIP because I need to add a test or two, try to get a handle on the exact schema that is allowed for 'row shape' and
the funnel.row viz key, and see if there are failing cases that I haven't considered yet.

* Funnel rows should now work more effectively with possible raw-rows

* Add a test that checks the success of the render and correct order

* Update src/metabase/pulse/render/body.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Update src/metabase/pulse/render/body.clj

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Addressing feedback from the review

 - renamed build-funnel-rows to funnel-rows
 - used 'funnel-viz' as the binding for the {:key "asdf" :name "asdf" :enabled true} maps from the viz settings
 - added docstring to the function to try clarify its 2 branches (keys vs indices on the rows)

---------

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* Trend Chart Viz Should Use Proper Card execute in Rendering

WIP

Fixes: #39854

The trend chart follows a code path shared with multi-series static-viz. This means there was an implicit expectation
that a dashcard is associated with the card at all times. This isn't true for alerts, so the code is fixed to remove
the failure.

* Add a test

* add missed :breakout that actually made the test work

oops

* Adding comments to try clarify what's going on

* /api/collection/:id/items gets `here` and `below`

During postprocessing of `collection` children retrieved by
`/api/collection/:id/items`, annotate each of the children with `below`
and `here` keys representing the presence of questions, models, or
collections at this level of the hierarchy or below it.

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>