* Support parsing {{variable}}s and such in queries

[Fixes #39954]

[Fixes #39953]

* H2 compilation improvements: avoid unneeded casting

* Legacy MBQL schema improvements

* Add additional test for wrap-value-literals

* Simpler version of the fixes from #36068

* Test and lint fixes 

* Pivot QP use MLv2

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* PR feedback

* Resurrect remove-all-breakouts

* Test fixes 

These type checks are useful when constructing clauses with full MLv2
metadata available, but when converting from legacy the `:field` refs
have only `:base-type`.

If a column has its metadata overridden to eg. coerce a string or
integer into a date, its `:base-type` will remain `:type/Integer` or
`:type/Text` and the `:effective-type` will be `:type/Date` etc.

This change skips the type check while converting from legacy, and
while checking `can-run`. Eventually this should be replaced with
tracking `:effective-type` on refs or using real metdata, instead of
relying on the `:base-type` only.

Fixes #41122.

We need to serialize it for any card, since when a normal question depends on a native question - their result_metadata won't be updated when opening a dashboard, which leads to front-end not understanding how to connect the parameters from a dashboard to a card, and errors during queries.

* Group axes with the correct map shape.

Prior to this, any time axes were all explicitly set :left or :right by the user, the `group-axes-at-once` function
would incorrectly return the map in the following form:

`{:left ["a" "b" "c"]}` which is incorrect. It must instead look like this:

`{"a" :left "b" :left "c" :left}`

This fixes the map shape and permits the user to explicitly set all axes to the left or to the right.

* Add a test to confirm the axis settings are respected and do render

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* More Metric => LegacyMetric renaming

* metabase.models.metric-important-field => metabase.models.legacy-metric-important-field

* Even more renaming

* More renaming

* Even more fixes.

* More renaming

* Wow, even more renaming!!

* Use window functions for SQL drivers for cumulative count and sum

* H2/Snowflake/Redshift implementation

* Test fixes 

* Use SELECT expression positions as the default window function implementation

* Some test fixes.

* Fix bugs in add-alias-info

* MongoDB fixes

* Another pass

* BigQuery should use positions?

* BigQuery almost works 100% now.

* MLv2 fix conversion behavior for expressions with type info

* Add transformations for nesting cumulative aggregations in expressions

* Minor tweaks

* MLv2 should escape aliases using driver/escape-alias (dynamically bind this)

* Kondo fix

* Lint fixes 

* More test fixes and revert test changes

* Update some tests to use correct case for nominal Field literal refs

* Wow! BigQuery is fixed! Remove upgrade-field-literals middleware

* Disable tests for the failing drivers, we can fix in follow-on PRs.

* Enable BigQuery :window-functions

* Extra info

* Add GH issue numbers

* Update more GH issue numbers

* Fix SQL Server!!!! wooo

* I THOUGHT WE WERE DISABLING SPARK SQL!!!!!!!!!!1

* Cleanup to get things ready for review.

* Snowflake + Vertica shouldn't use output column numbers for ORDER BY in OVER

* Code cleanup

* Snowflake + Vertica shouldn't use output column numbers for ORDER BY in OVER

* Code cleanup

* Test fixes 

* Much simpler BigQuery implementation

* Add GH issue context

* Remove commented-out code

* Remove unused add-stage-at-index

* Documentation

* Test fix? 

* Test fix 

* Legacy MBQL documentation

* Do not commit dox

* Subject the legacy MBQL stuff to the same stricter linting rules as MLv2

* PR feedback

This optional fourth argument will disregard all joins at `position` and
later when suggesting conditions. This is useful when editing a join's
RHS, which wipes out the condition. We don't want to generate a
suggested condition that's based on keys from the RHS of the join we
just edited, or from later joins.

Fixes #40916. Part of #40890.

Fixes #41023.

New endpoint accepts entity and entity ids in form of `database=1&dashboard=2&question=3`. If you don't supply
`&include=overrides`, then it tries to find configs directly referencing supplied entities and updates their
`invalidated_at`. If you supply `&include=overrides`, all the referenced cards are
updated (`report_card.cache_invalidated_at`).

Cache strategies then select the maximum `invalidated_at` in their logic. EE-only.

resolves #40548

[Fixes #13922]

* MLv2 normalization: add :lib/uuid automatically to options maps if it's missing

* Remove unneeded normalization code in QP

* Fix 

* Ignore invalid metadata providers when passed in from the REST API

* Use window functions for SQL drivers for cumulative count and sum

* H2/Snowflake/Redshift implementation

* Test fixes 

* Use SELECT expression positions as the default window function implementation

* Some test fixes.

* Fix bugs in add-alias-info

* MongoDB fixes

* Another pass

* BigQuery should use positions?

* Disable tests for the failing drivers, we can fix in follow-on PRs.

* Extra info

* Add GH issue numbers

* Update more GH issue numbers

* Fix SQL Server!!!! wooo

* I THOUGHT WE WERE DISABLING SPARK SQL!!!!!!!!!!1

* Cleanup to get things ready for review.

* Snowflake + Vertica shouldn't use output column numbers for ORDER BY in OVER

* Code cleanup

* Test fix? 

* Enable not emptyable arguments for is-empty and not-empty clauses

* Add review remarks

* Shorten dataset name so it is usable with some drivers

* Disable test for Athena

* Handle possibly missing base type

* Update comment

* Missing base type field is treated as not emptyable

* Update desugar middleware

Now middleware adds base type to fields with integer id that are missing that first. This ensures correct function of is-empty expansion.

* Update desugar middleware

Cleanup base types necessary for desugaring is-empty

* Update desugar middleware cleanup

* Update test

* Adjust tests

* Remove now redundant e2e test

* Update comment

The first time you zoom in on a bin in a binned query, it works fine.

For example, Orders COUNT by SUBTOTAL auto-binned has a bin for 40-60.
If you zoom in on that, it correctly shows the range 40 to 60 broken
down into smaller bins which are 2.5 units wide. If you then try to
zoom in on one of these smaller bins, say 50-52.5, there are two
problems:

1. The filters from the first zoom (`>= 40` and `< 60`) are still
   there, and so are the new ones (`>= 50` and `< 52.5`).
2. The new filter is actually wrong; it uses the big width (20)
   from the original bins, not the width of the nested bins (2.5),
   so the new filters are `>= 50` and `< 70`.

This PR fixes both issues, so nested bins (1) remove the old filters,
and (2) use the correct width for the nested bins.

This logic was trying to match `[name join-alias]` pairs previously, but
that won't work since the later stage field doesn't have a `:join-alias`
on it. This falls back to matching on only the ID or name, but only if
it's unambiguous.

Fixes #40252.

This middleware shouldn't throw exceptions when `limit` or `offset`
query parametesr aren't integers.

* Remove old config + fix typo

* Add new analytics for 4 new illustration settings

* Send creator emails every week

mod the email by 52 instead of by 12 and use the current week as an
anchor

```clojure
user=> (require '[java-time.api :as t])
nil
user=> (let [every-day (take 1000 (t/iterate t/plus (t/local-date 2024 1 1) (t/days 1)))
             f         (fn [d]
                         (let [wf (java.time.temporal.WeekFields/of (java.util.Locale/getDefault))]
                           (.get d (.weekOfWeekBasedYear wf))))]
         (sort (keys (frequencies (map f every-day)))))
(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
 40 41 42 43 44 45 46 47 48 49 50 51 52)
```

```clojure
user=> (metabase.util.cron/cron-string->schedule-map "0 0 2 ? * 7")
{:schedule_minute 0,
 :schedule_day "sat",
 :schedule_frame nil,
 :schedule_hour 2,
 :schedule_type "weekly"}
```

* fix typo in creator sentiment email, clean up a bit

typo: had `:verison` instead of `version`.

And while I'm in there, make the payload all in one file and have the
email namespace json->bytes->b64-encode so the data is all in one place
and the setting that allows it to be present or not is co-located.

For collections retrieved from the Search API, attach `location` and
`effective_location` so the frontend knows... where they're located.

* Cards with Results Should Show up in Subscriptions

Fixes: #40726

This PR fixes a bug where setting 'Hide thie card if there are no results' in a Dashcard's viz-settings would hide the
card even when there were results.

* adjust test to show that the setting is respected

Before, the setting would always hide the card. Now, the card is only hidden when there are no results. This test sets
up 4 cards: 2 are normal, 2 have no results, but 1 has the setting to hide empty. THerefore, we should expect to see 3 results.

* Make Formatting on Exports Optional

Formatting on exports can be turned off via a middleware key:

`:format-export?` which will control if the 'app style' formatting will be applied to the exported results.
`:format-export? false` will return the file as in previous Metabase versions.

* format_export's default value provided in the QP middleware

Should still default to formatted for v. 49, as the formatting is intended as a feature. This default value should be
provided in a central place, so is added to the middleware and is therefore not necessary for the API to provide a default.

* weird indentation

* Add format_export param to dashboard dashcard query endpoint

* Add format_export option to embedded dashboard downloads endpoint

* Move default true value to the streaming-results-writer impls

* Default true but for real this time

* Add format_export to the public Question endpoint

* Add test to card api

* Dashboard exports endpoint test

* Add test to dashboard embeds

Test still fails and I don't know why yet

* the dangers of println debugging....

Well, I was passing a string false the whole time. Turns out the problem was me

* Public Card downloads endpoint now has format_export and test

* export_format also works for adhoc queries using api/dataset

* Let defendpoint do the boolean parsing

* Embed endpoint should now work too.

* Unify format-rows and format-export middlewares

* Sneaky endpoint!

* In xlsx, always parse temporal strings so that viz-settings formatting is applied

* Snuck in an incomplete change by accident. removed it.

* Can now specify format/no format on subscriptions

* Fix and move migration

* default format rows to true in attachments

* Ok, I think that should actually work for subscriptions.

* Change endpoints to use format_rows instead of confusing format_export

* format_rows everywhere

* Fix more test failures

* redef'd function had wrong signature

* Make format export optional (FE part) (#40643)

* Refactor download URL builder code

Need to handle query `params` and request `body` separately

* Export `useHover` from `metabase/ui`

* Add unformatted export for dashboard cards

* Add unformatted export for saved questions

* Fix POST body encoding

* Add unformatted export for ad-hoc questions

* Ensure `format_export` is always false for Excel

* Add unformatted export to public questions

* Add unformatted export to embedded questions and dashboards

* Fix behavior for PNG format

* Add tests for `QueryDownloadPopover`

* Fix e2e downloads helper (allow URL query params)

* Rework alt key handling

* Add e2e test

* Switch to `format_rows` instead of `format_export`

* Add unformatted export option to subscriptions

* Fix `isHoldingAltKey` initial state

* Rework e2e test to use Total column

* Fix initial checkbox state for subscriptions

* Use "Alt" in tooltip label for Windows/Linux

* Show the subscriptions toggle only for CSV attachments

---------

Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>

* Address review feedback

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>