* add collection/list endpoint

* add children

* address changes

* address change

* fix test

* fix like

* address comments

* Update src/metabase/api/collection.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* address changes

* exclude other user collections in the sql query

* fix logic

* fix acrhive

* make remove-other-users-personal-collections cleaner

* fix tests

* fix errors

* fix other test

* add test

* fix merge

* fix formatting

* address comments

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* When creating a new model, don't call /api/model-index because that returns error code 400
* Add two unit tests of DatasetEditor
* Fixes #31671 

* Don't coerce add-implicit-clauses

Fixes #33861

Renames `::nest-query/outer-select` to `qp/ignore-coercion` since it was
already applied outside nest-query and currently only controls that aspect.

* Fix whitespace

* Only add ignore-coercion if temporal-unit or binning are present

* Fix tests

* Only add ignore-coercion flag when coercion strategy is present

* add reproducing e2e spec

* show failed button state and error message

* add failed button and error message to e2e spec

* Use findByLabelText and findByRole rather than cy.get

* Add role=alert and aria-label to FormErrorMessage

---------

Co-authored-by: Raphael Krut-Landau <raphael.kl@gmail.com>

Previously date arithmetic was only supported with a single date or
datetime +/- 0 or more intervals.

This adds a new flavour of temporal arithmetic - subtracting two
`:type/Date` or two `:type/DateTime` values yields a `:type/Interval`.

Fixes #37263.

* Remove unused empty dependency

* Sync yarn.lock

#37233 made it possible to get rid of the temporary placeholder method name `_getMLv2Query` and replace with with the `query`.

* Rename `_getMLv2Query` method to `query`

* Update all references of `_getMLv2Query` to `query`

* Rename query to legacyQuery

* Use step.query

* Port ExpressionStep to MLv2

* Use named export for ExpressionStep

* Move setup to the top of the file

* Add expressions parameter to createQueryWithClauses

* Update ExpressionStep unit tests

* Add a test for adding expressions

* Format code

* Add typing for openSteps and remove default parameter to prevent accidental omission

* Rename arguments

* Remove NotebookStep['update'] and update NotebookStepUiComponentProps['updateQuery']

* Remove NotebookStep['update'] from test util

* Add metadata argument to NotebookStepFn

* Remove unused NotebookStep['clean']

* Change previewQuery type to Query

* Update NotebookStep['revert'] to work with MLv2 only

* Use revert: null properly

* Clear breakouts before aggregations

* Update NotebookStepDef['active'] to work with MLv2 only

* Update NotebookStepDef['valid'], remove NotebookStepFn

* Remove TODO

* Update STEP.valid calls

* Port #36507 to master

* Migrate STEP.valid

* Format code

* Rename topLevelQuery to query and stagedQuery to legacyQuery

* Use INotebookStep["id"], simplify handleQueryChange

* Use database from metadata instead of question

* Format code

* Apply expressions before aggregations

* Improve typing in tests

* Fix expression name uniqueness

* Fix Lib.fields typing

* Use dropStageIfEmpty when removing expressions

* Fix unique name computation

* Port hasData

* Add stageIndex argument to NotebookStepDef['valid']

* Port query.isSourceQuery()

* Add junit.xml to .gitignore
- see https://github.com/metabase/metabase/pull/35929

* Make unit tests run

* Improve revert typing

* Update unit tests

* Update unit tests

* Update unit tests

* Rename filterStep1 to filterStep and filterStep2 to postAggregationFilterStep

* Update unit tests

* Update unit tests

* Update unit tests

* Update unit tests

* Convert unit tests to TS

* Simplify getQuestionStepsForMBQLQuery

* Refactor tests

* Remove test for new queries - a Lib.Query object cannot exist without the "data step"

* Lift database to reuse it

* Port getQuestionSteps

* Skip e2e test for metabase#19745 due to metabase#36574

- See https://github.com/metabase/metabase/issues/36574
- See https://metaboat.slack.com/archives/C04CYTEL9N2/p1702063378269379
- See https://metaboat.slack.com/archives/C0643FZ5E83/p1702365832126429?thread_ts=1702362649.548849&cid=C0643FZ5E83

* Update comment

* Revert type change

* Revert "Use dropStageIfEmpty when removing expressions"

This reverts commit d529503e793c93ecdc006f0ecf20c874f8cd091b.

* Format code

* Skip 19745 tests

* Format code

* Use dropStageIfEmpty instead of cleanNesting

* Try to drop only the updated stage

* Use stageIndex instead of -1 in preview3

* Format code

* Ignore question.isStructured() check

* Finish migrating getQuestionSteps

* Migrate query.clean in Notebook to MLv2

* Use reduceRight instead of reverse + reduce

* Fix typo

* Move hasBreakouts computation until after the redundant stage is removed

* Update e2e sandboxing test to assert showing permissions error instead of magically dropping an unsupported join
- @see https://metaboat.slack.com/archives/C01LQQ2UW03/p1704294434403599



---------

Co-authored-by: Tamás Benkő <tamas@metabase.com>
Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

* Add number-viz test to pulse integration test.

This is a currently failing test. The first card succeeds because the 'narrowing' to 1 field is done in the query
directly, so the expected single value is shown.

The second (currently failing) card has the viz for a 'number' applied but the query isn't 'narrowed' at all, so we
see the failure.

* Fix the :scalar render to look at the column specified in :scalar.field

* Addressed feedback: improved get-col-by-name impl. and added test comment

* Use simple `Secret`s for ApiKeys

I decided to POC the idea I had here:
https://metaboat.slack.com/archives/CKZEMT1MJ/p1703790833895029

This wraps the generated ApiKeys in a special Secret type. If the object
is accidentally printed, logged, etc. it will appear as "<< REDACTED
SECRET >>" rather than the actual value.

When we're actually *using* the Secret, we call `expose` on it to turn
it into a String.

When trying to render a card with viz settings containing a nonexistent dimension, an error would result in `mult-x-axis-rowfn` causing a failure to render.

This fix simply replaces `map` with `keep` in `metabase.util.ui-logic/mult-x-axis-rowfn` and `metabase.util.ui-logic/mult-y-axis-rowfn` so that `nil` values are dropped if a bad dimension or metric name is provided.

The following easily shows the broken case (now fixed). Previously, an error was rendered. Now you get the chart.

```clojure
;; REPRO
(comment
  (ns tickets.37266-slack-error
    (:require
     [clojure.test :refer :all]
     [dev.render-png :as render-png]
     [metabase.models :refer [Card]]
     [metabase.pulse.render.body :as body]
     [metabase.query-processor :as qp]
     [metabase.test :as mt]
     [toucan2.core :as t2]))

  (let [busted? true]
    (mt/dataset test-data
      ;; Note that this uses orders since we have that as test data.
      (mt/with-temp [Card {base-card-id :id}
                     {:dataset_query          {:database (mt/id)
                                               :type     :query
                                               :query    {:source-table (mt/id :orders)
                                                          :aggregation  [[:count]]
                                                          :filter       [:between [:field (mt/id :orders :created_at) nil] "2019-05-16" "2019-08-16"]
                                                          :breakout     [:field (mt/id :orders :created_at) {:temporal-unit :week}]}}
                      :display                :line
                      :visualization_settings {:table.pivot_column      "state"
                                               ;; This is also intentionally broken
                                               :graph.metrics           (if busted?
                                                                          ["frooby"]
                                                                          ["count"])
                                               :graph.show_trendline    false
                                               :graph.x_axis.title_text "Week"
                                               :graph.y_axis.title_text "PR Review Activity"
                                               ;; Using a missing dimension name breaks the results
                                               :graph.dimensions        (if busted?
                                                                          ["_sdc_extracted_at"]
                                                                          ["created_at"])
                                               :graph.show_values       false
                                               :table.cell_column       "count"}}]
        (render-png/render-card-to-png base-card-id)))))
```

* Error if DB downgrades are required

Before running migrations, check the major versions of:

- the latest database migration that was actually applied (e.g. if v49
ran migrations, this would be 49)

- the latest database migration that liquibase is aware of (e.g. if
we're currently running v48, this would be 48)

If the database has executed migrations that we don't know about, throw
an exception and exit with a helpful error message telling the user they
need to run `migrate down` from the newer version before they can
downgrade.

Fixes #27906

* Rename query to legacyQuery

* Fix legacy query ref

* [MLv2] Support exclude date filter for filter args display name

* Update spec

* Add excludes to filter args display

* [MLv2] Fix field-values-search-info returning string ids

* Skip join test because the text prompt still doesn't match

* Fix the test

---------

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

When we generate a `:common_name` on Users, we check for the presence of
both `first_name` and `last_name`. If either is `nil`, we don't generate
a `common_name`.

ApiKey users only have one name (the name of the ApiKey itself) but we
should set the `last_name` to an empty string. This way, the common name
will still be generated.

Because the `metabase.models.user/insert-new-user!` function requires a
NonBlankString for `first_name` and `last_name` if they're present, I
went back to calling `t2/insert-returning-instances!` directly.

For auto-requesting the reviews on backend PRs.

* add collection/list endpoint

* add children

* address changes

* address change

* fix test

* fix like

* address comments

* Update src/metabase/api/collection.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* address changes

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

This still throws an exception on trying to directly remove the final
JOIN condition. But if the condition is being removed because a field it
depends on is being removed (eg. a custom column, a breakout, a
previous stage aggregation) then the entire JOIN is also removed, in the
usual cascading fashion.

Fixes #36690.

* Fix stress-test workflow

Use `replay-chromium' browser for these tests as well.
This gives us a bit more confidence because we're using
the same browser in both "regular" and "stress-test" runs.

[ci skip]

* Enable Replay via envvar

[ci skip]

* Removing Time portion of Timestamp when formatted as Date in Excel

This is a starting changeset for fixing the issue "Timestamp is always present in dates when exporting to Excel in the embedded application. #14393", which demonstrates a failure mode in applying column-formatting to Excel downloads. This commit should fix the underlying issue as well as update formatting for remaining CSV file download discrepancies.

The goal of this commit is to get us to a baseline state for some small refactorings to combine our formatting paths for CSV, JSON, and Excel exports.

* Updating common logic and tests to reflect how global settings are passed in and applied to format-temporal-str

* Updating `number-format-strings` to not need semantic type.

* Adding tests for new Excel-export logic.

* Fixing cases for currency where we must specify that `::mb.viz/currency-in-header` is false.

* Fixing formatting tests

* Updating unit test to reflect a global abbreviated dates preference.

* Adding column type defaults for certain test cases.

* Updating logic and tests in xlsx exports

Logic in `metabase.query-processor.streaming.common` further unifies the idea of merging all settings first, then making decisions based on that. It also makes the downstream code more functional, as that code doesn't repeatedly reach for global settings.

`metabase.query-processor.streaming.xlsx-test` has been updated to reflect this change. In particular, the helper function `format-string` now builds the right input settings rather than passing settings through to be accumulated later.

* Fixing two more tests.

* Fixing remaining unit tests

- Update `metabase.api.card-test` with multiple test cases for different formats.
- Update `export-format-test` to lock temporary settings for consistency.
- `default-number-format?` to `unformatted-number?` and include "currency" as a potential unformatted type.

* Consistent application of global viz settings

This PR adds a function that will inject the global viz settings into `viz-settings-for-col` if not provided by the query processor. This also simplifies the `xslx-test` ns.

* Treating :default unit as no unit.

* Make brush filter tests more stable

* Make repro for 34382 less flaky

* fixup! Make repro for 34382 less flaky

* More asserts

* Implement truncate, add, and unit-diff date util functions for Cljs

* Implement filter update util fns (#37103)

* Fix circular references

* Test fixes 

* PR feedback

* Clarify things

The additional call to apiUpdateQuestion is not needed, since you can't edit an alert without first saving the question

Fixes #36395 

* Implement truncate, add, and unit-diff date util functions for Cljs

* Fix circular references

* Test fixes