* Fix embed download endpoint

We have added format_rows as a query parameter. In most endpoints this just works fine.

Here, however, the problem is that we use query parameters to pass parameter values. We run some validation over these
provided parameter values (eg. don't allow a user to over ride a 'locked' parameter). Since we have these 2 different
use cases for query parameters, the validation was failing on :format_rows, which isn't expected to exist at all.

Here, I dissoc the :format_rows key prior to this validation step, so normal validation can continue.

* Add test that uses this card embedding endpoint

* Nested queries have limits which can defeat their purpose

Consider a query

```sql
select count(*) from {{#199}}
```

This query should return the number of distinct rows in the query
defined by 199. But it's actually limited by the excel limit of
1048575. And that's because when the value of `{{#199}}` is expanded it
has that limit applied as normal.

```clojure
qp=> (let [card-id 199] ;; use a valid card id for you
       (-> {:database 1,
            :type :native,
            :native {:query "select count(*) from {{ref}}"
                     :template-tags {:ref {:card-id card-id
                                           :type :card
                                           :name "ref"
                                           :display-name "ref"}}}
            :middleware {:disable-max-results? true}}
           qp.compile/compile
           :query
           (metabase.db.query/format-sql )
           println))
select
  count(*)
from
  (
    SELECT
      "PUBLIC"."ORDERS"."ID" AS "ID",
      "PUBLIC"."ORDERS"."TOTAL" AS "TOTAL"
    FROM
      "PUBLIC"."ORDERS"
    LIMIT
      1048575
  )
```

But we can suppress this limit when substituting a query inside yielding

```sql
select
  count(*)
from
  (
    SELECT
      "PUBLIC"."ORDERS"."ID" AS "ID",
      "PUBLIC"."ORDERS"."TOTAL" AS "TOTAL"
    FROM
      "PUBLIC"."ORDERS"
  )
```

And this is proper because we want to limit the _outer_ query, not
internal queries.

* Remove limit from test expectation

* stupid trailing space

* another subquery test

* Use helper function to disable limit middleware

* Fix #39138 again

Fixes #40934

Got unfixed somehow in #40578. seems like some file renames caught it off guard

But the long and short is that its possible for a persisted model to end
up in the state "refreshing" (possible an instance restart during
refreshing). The refreshing job doesn't look for these so they become
effectively invisible.

Since the job to refresh them will only run one at a time cluster wide,
any jobs that are in the "refreshing" state when the refresher begins to
refresh are necessarilly stuck (no one else could currently be
refreshing them). So we can just add them to the queue of models to
refresh.

```clojure
(jobs/defjob ^{org.quartz.DisallowConcurrentExecution true ;; <----
               :doc "Refresh persisted tables job"}
  PersistenceRefresh [job-context]
  (refresh-job-fn! job-context))
```

* Ensure ee/oss pathways are taken in tests

Annoying little footgun here. CI does not run with an ee token, so in
order to ensure that test pathway goes through ee version of a
defenterprise we _must_ use the `mt/with-premium-features
{:cache-granular-controls}`,  but we also want to ensure that it goes
through the oss version.

So two options, a `doseq` on both features (empty set and the feature
that triggers this). But want a test in the enterprise folder as well to
ensure.

The real trickiness comes from running tests at a repl and CLI. My REPL
always has an ee token in it. My command line always lacks that as
well. So want to be explicit about the token features in effect at test
time.

That's why I'm essentially duplicating tests (ee in ee folder, oss in
regular pathway)

* Support parsing {{variable}}s and such in queries

[Fixes #39954]

[Fixes #39953]

* H2 compilation improvements: avoid unneeded casting

* Legacy MBQL schema improvements

* Add additional test for wrap-value-literals

* Simpler version of the fixes from #36068

* Test and lint fixes 

* Pivot QP use MLv2

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* PR feedback

* Resurrect remove-all-breakouts

* Test fixes 

These type checks are useful when constructing clauses with full MLv2
metadata available, but when converting from legacy the `:field` refs
have only `:base-type`.

If a column has its metadata overridden to eg. coerce a string or
integer into a date, its `:base-type` will remain `:type/Integer` or
`:type/Text` and the `:effective-type` will be `:type/Date` etc.

This change skips the type check while converting from legacy, and
while checking `can-run`. Eventually this should be replaced with
tracking `:effective-type` on refs or using real metdata, instead of
relying on the `:base-type` only.

Fixes #41122.

We need to serialize it for any card, since when a normal question depends on a native question - their result_metadata won't be updated when opening a dashboard, which leads to front-end not understanding how to connect the parameters from a dashboard to a card, and errors during queries.

* Group axes with the correct map shape.

Prior to this, any time axes were all explicitly set :left or :right by the user, the `group-axes-at-once` function
would incorrectly return the map in the following form:

`{:left ["a" "b" "c"]}` which is incorrect. It must instead look like this:

`{"a" :left "b" :left "c" :left}`

This fixes the map shape and permits the user to explicitly set all axes to the left or to the right.

* Add a test to confirm the axis settings are respected and do render

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* More Metric => LegacyMetric renaming

* metabase.models.metric-important-field => metabase.models.legacy-metric-important-field

* Even more renaming

* More renaming

* Even more fixes.

* More renaming

* Wow, even more renaming!!

* Use window functions for SQL drivers for cumulative count and sum

* H2/Snowflake/Redshift implementation

* Test fixes 

* Use SELECT expression positions as the default window function implementation

* Some test fixes.

* Fix bugs in add-alias-info

* MongoDB fixes

* Another pass

* BigQuery should use positions?

* BigQuery almost works 100% now.

* MLv2 fix conversion behavior for expressions with type info

* Add transformations for nesting cumulative aggregations in expressions

* Minor tweaks

* MLv2 should escape aliases using driver/escape-alias (dynamically bind this)

* Kondo fix

* Lint fixes 

* More test fixes and revert test changes

* Update some tests to use correct case for nominal Field literal refs

* Wow! BigQuery is fixed! Remove upgrade-field-literals middleware

* Disable tests for the failing drivers, we can fix in follow-on PRs.

* Enable BigQuery :window-functions

* Extra info

* Add GH issue numbers

* Update more GH issue numbers

* Fix SQL Server!!!! wooo

* I THOUGHT WE WERE DISABLING SPARK SQL!!!!!!!!!!1

* Cleanup to get things ready for review.

* Snowflake + Vertica shouldn't use output column numbers for ORDER BY in OVER

* Code cleanup

* Snowflake + Vertica shouldn't use output column numbers for ORDER BY in OVER

* Code cleanup

* Test fixes 

* Much simpler BigQuery implementation

* Add GH issue context

* Remove commented-out code

* Remove unused add-stage-at-index

* Documentation

* Test fix? 

* Test fix 

* Legacy MBQL documentation

* Do not commit dox

* Subject the legacy MBQL stuff to the same stricter linting rules as MLv2

* PR feedback

* Add new user-local setting `notebook-native-preview-preferences`

* Update user settings on FE

* Use two distinct setting entries

* Update FE types

* More explicit `notebook-native-preview-sidebar-width` definition

* Add selectors

* Initialize QB uiControls based on the BE settings

* Rename QB store entry

`isNativePreviewSidebarOpen` -> `isShowingNotebookNativePreview`

* Rename QB store entry

`nativePreviewSidebarWidth` -> `notebookNativePreviewSidebarWidth`

* Add Redux actions

* Dispatch user settings on top of Redux store

* Fix test title typo

* Add E2E test to make sure sidebar is resizable

* Expand the E2E test to cover persisting settings across sessions

* Be more explicit that we're resizing along x axis

* Store window breakpoint as a variable

* Calculate handle's left position more explicitly

This optional fourth argument will disregard all joins at `position` and
later when suggesting conditions. This is useful when editing a join's
RHS, which wipes out the condition. We don't want to generate a
suggested condition that's based on keys from the RHS of the join we
just edited, or from later joins.

Fixes #40916. Part of #40890.

Fixes #41023.

New endpoint accepts entity and entity ids in form of `database=1&dashboard=2&question=3`. If you don't supply
`&include=overrides`, then it tries to find configs directly referencing supplied entities and updates their
`invalidated_at`. If you supply `&include=overrides`, all the referenced cards are
updated (`report_card.cache_invalidated_at`).

Cache strategies then select the maximum `invalidated_at` in their logic. EE-only.

resolves #40548

[Fixes #13922]

* MLv2 normalization: add :lib/uuid automatically to options maps if it's missing

* Remove unneeded normalization code in QP

* Fix 

* Ignore invalid metadata providers when passed in from the REST API

* create ui for the full embedding homepage, without any logic (#40251)

* wip ui for the full embedding homepage

* adds missing link + fix keys

* remove done TODO

* Update frontend/src/metabase/home/components/EmbedHomepage/InteractiveTabContent.tsx

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

* destructure props in the function arguments

* rename customerFacingAnalyticsDocsUrl to analyticsDocsUrl

---------

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

* add new settings for embedding homepage (#40455)

* remove old minimal version

* adds new embedding homepage settings and sets them at the end of the setup

* comment e2e test

* add tests for embedding homepage flags

* more tests

* remove unit tests related to old minimal version

* fixes flags descriptions

* :keyword for embedding-homepage

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

* replace reloadSettings and refreshSettingsList with initializeSettings

* `value => value === true` -> `Boolean`

* toMatchObject -> toEqual for more strict tests

* remove useless undefined check for settings, the component is conditionally rendered based on a setting so it means they're loaded anyway

* fix: use :hidden as we changed the type to :keyword

* specifically check for the `embedding` feature in the tokenFeatures object instead of checking if any feature is active

* remove double \n

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

---------

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* connect settings to the embed homepage and show it (#40528)

* fix nested a warnings in browser console

* connect settings to the embed homepage and show it

* fix stories

* remove localStorage.clear() as we're not using it in the logic anymore

* dismiss without feedback for the embedding homepage (#40587)

* feat: add dismiss functionality to the embed homepage

* update setup e2e to check for the dismissal of embedding homepage

* use List from metabase/ui (#40638)

* set embedding-secret-key when embedding is enabled (#40813)

* set embedding-secret-key when embedding is enabled

* use "named" setters and getters as suggested by Noah

* Fix tests for ms1 after updating on master (#41069)

---------

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* Use window functions for SQL drivers for cumulative count and sum

* H2/Snowflake/Redshift implementation

* Test fixes 

* Use SELECT expression positions as the default window function implementation

* Some test fixes.

* Fix bugs in add-alias-info

* MongoDB fixes

* Another pass

* BigQuery should use positions?

* Disable tests for the failing drivers, we can fix in follow-on PRs.

* Extra info

* Add GH issue numbers

* Update more GH issue numbers

* Fix SQL Server!!!! wooo

* I THOUGHT WE WERE DISABLING SPARK SQL!!!!!!!!!!1

* Cleanup to get things ready for review.

* Snowflake + Vertica shouldn't use output column numbers for ORDER BY in OVER

* Code cleanup

* Test fix? 

Goal is to properly document the interface between MLv2 and the FE. In
particular I want to call out the health and fate of each function. Some
of them only exist to support legacy compatibility, for example, and
that should be documented to discourage new usage and make clear the
conditions for removing the functions.

* Enable not emptyable arguments for is-empty and not-empty clauses

* Add review remarks

* Shorten dataset name so it is usable with some drivers

* Disable test for Athena

* Handle possibly missing base type

* Update comment

* Missing base type field is treated as not emptyable

* Update desugar middleware

Now middleware adds base type to fields with integer id that are missing that first. This ensures correct function of is-empty expansion.

* Update desugar middleware

Cleanup base types necessary for desugaring is-empty

* Update desugar middleware cleanup

* Update test

* Adjust tests

* Remove now redundant e2e test

* Update comment

The first time you zoom in on a bin in a binned query, it works fine.

For example, Orders COUNT by SUBTOTAL auto-binned has a bin for 40-60.
If you zoom in on that, it correctly shows the range 40 to 60 broken
down into smaller bins which are 2.5 units wide. If you then try to
zoom in on one of these smaller bins, say 50-52.5, there are two
problems:

1. The filters from the first zoom (`>= 40` and `< 60`) are still
   there, and so are the new ones (`>= 50` and `< 52.5`).
2. The new filter is actually wrong; it uses the big width (20)
   from the original bins, not the width of the nested bins (2.5),
   so the new filters are `>= 50` and `< 70`.

This PR fixes both issues, so nested bins (1) remove the old filters,
and (2) use the correct width for the nested bins.

This logic was trying to match `[name join-alias]` pairs previously, but
that won't work since the later stage field doesn't have a `:join-alias`
on it. This falls back to matching on only the ID or name, but only if
it's unambiguous.

Fixes #40252.

This middleware shouldn't throw exceptions when `limit` or `offset`
query parametesr aren't integers.

* Remove old config + fix typo

* Add new analytics for 4 new illustration settings

* Add login page illustration (#39299)

* Move Metabot setting to a new place

* Add setting for login page illustration

* fix UI styling

* Fix not able to upload the same file twice

* Update the admin > appearance > conceal metabase copy

* Prepare the component for tests

* Add some unit tests

* Prevent settings not saved sometimes

After setting the value to/from `custom` one of the setting could be
failed to save, I suspect this happens because the race condition from
trying to save the setting values relatively at the same time.

* Make unit tests easier to read

* Add all unit test test cases

* Add basic E2E test

* Show fallback icon for broken images

* Make use of the proper component

* Update the comment to affect the reality

* Add an error message when file size is over 2MB

* Show an error message when the file size > 2MB

* Add the error message tests

* Preserve the preview image aspect-ratio

* Detect corrupted images

* Fix failing unit tests

* Address feedback: Type

* Feedback: Create meaningful variables

* Feedback: Make selector API more sensible

* Prevent race condition

Makes sure we save the settings completely before making other actions
that also change settings otherwise, there could be race conditions.

* Feedback: tighten up a type

* Feedback: Fix inconsistency callback calls

* 2. Add the rest 3 illustration settings (#40487)

* Add the rest 3 illustration settings

* Don't make the X icon feel lonely

* Fix illustration widget saving

* Fix type error

* 3. Update the select UX to match new design (#40491)

* Update the select UX to match new design

* Move info dot next to file upload button

* Simplify IllustrationWidget API

* Update setting title copy

* Clean up test comments

* Feedback: Clarify `any` type

* Feedback: Use shorthand object creation

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Feedback: Use global CSS glass instead of inlining values

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* Feedback: Remove unnecessary curly brackets in JSX

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

---------

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* 4. Connect illustrations to new setting values (#40139)

* Wire landing page illustration to the setting value

* Wire the unsubscribe page to the setting value

* Prepare no rows error to be wired to the setting value

* Wire no question results illustration to the setting value

* Wire no search results illustration to setting value

* Fix error from fixing conflict

* Fix required unnecsseary prop on Image

* Use Image component

* Link dashboard card no results image to the setting value

* Move the no data results to more a generic directory

* Link collection picker no results image to the setting value

* Make illustrations more accessible

* Fix failed tests

* Test unscribe page illustration + test no illustration

* Add landing page tests

* Simplify Cypress selectors

* Refactor: Rename settings to match the new requirements

No question data -> No Data
No search results -> No object

These changes are made so, these 2 settings are generic enough to
include other illustrations later on.

* Add no data illustration test

* Add no object illustration tests

* Fix unsubscribe page background style

* Fix not able to build static viz

* 5. Follow-up on milestone 1 (#40710)

* Update custom background illustration scale behavior

* Remove the need to put an SVG file in static folder

* Fix test from rebasing off master with @testing-library/user-event upgrade

---------

Co-authored-by: Denis Berezin <denis.berezin@metabase.com>