* fix circular deps in query_builder/components

* js -> ts

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

This PR reimplements driver/describe-table for MongoDB. Before we would query a sample of documents from a collection and analyse them in Clojure. Instead, we now now execute a query that does a similar aggregation, but most of the calculation is done in the Mongo database.

Based on a few tests the performance is slightly slower when the collection contains small or deeply nested documents but much faster for large ones. But the main difference is in memory usage. This uses very little memory in the Metabase instance because all of the aggregation is done in the database.


Nested fields are a naturally recursive problem but here we unroll potential recursions to a `max-depth` number of queries that look for nesting at each depth level.

* ~ use DB to describe the table

* ~ optimize root query

* ~ nested-level-query works and gets objects too

* + root query gets objects too

* + driver/describe-table :mongo works

* ~ remove old implementation

* Various fixes for faster sync

Upgraded driver to 5.2.0
Updated data load to insert many rather than 1 row at a time.
Dropped max-depth to 7, see comment.

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Case Nelson <case@metabase.com>

* Incremental Pivot Processing for Exports

WIP

Fixes pivot exports for CSV and xlsx.

The CSV export should use less memory by incrementally building up the data structure and aggregating necessary row
data right away, so the memory overhead becomes only as large as the total pivot result.

In cases where the pivot rows/cols do combine into many many columns and rows, this can still be a large set of data,
but it should behave much better now in most cases.

The Excel export is a little more straightforward: create the export rows in the same fashion, streaming one row at a
time, and just post-process the sheet to add the pivot table in one shot at the end.

* WIP adding row totals.

* aggregate totals as rows are added

Row, column, section, and grand totals are all aggregated as each row is added.
This means the final step of building pivot output becomes just an exercise of lookups/arrangement, no further
aggregation is needed.

* CSV pivot works per-row, export respects formatting

This is a big step forward; we don't need to hold the entire dataset in memory, we instead aggregate a row's data into
the pivot datastructure, which only holds onto:

- unique values for each pivot-row in a sorted set
- unique values for each pivot-col in a sorted set
- grand total for each measure N values, where N is number of measures, ususally 1 or 2
- row totals for each combination of each pivot-row * N measures
- col totals for each combination of each pivot-col * N measures
- totals for each 'section', determined by unique values of first pivot-row * N measures
- values for each measure in every 'cell'; Row Combos * Col Combos * N Measures

So, there can still be a decent amount of data to store; but it will never hold onto all of the 'raw rows' from the
dataset.

We can never completely guarantee that Row Combos * Col Combos * N Measures remains small, but two things let us move
forward anyway:

- there's now visible feedback in the app that the download is running (or if it's failed)
- Pivot table utility diminishes rapidly with huge output anyway; users still need to curate/set up their data
- effectively to improve the table's utility, so we can assume that a slow-to-download pivot table is also slow to
- use/less effective, and will likely be something the user doesn't want (as often).

* some test fixes

* now, if we export 'raw pivot rows', they don't show pivot-grouping

and they also don't include the 'extra' rows for totals/subtotals/grand totals (any row with pivot-grouping > 0).

This means that now the non-pivot version of a pivot table export will match what a user sees if they change the viz
to a regular table.

* remove old test

* re-incorporate some changes from master

* fix csv for non-pivots due to oversight in my changes

This is just a temporary change, I think I should clean up this bit of the code a little, I can probably make it a
little more readable and use some cleaner logic regarding if the rows are 'raw pivot rows' or not.

* start moving format_rows to POST bod, add pivot_results too

There's still wiring work to do, but this starts to add format_rows and pivot_results to POST body for the various API
endpoints. Also modify tests to improve coverage/consistency across downloads and alerts/subscriptions.

The tests will not pass on this commit, but fixes will be incoming

* native pivot tables in xlsx

* add precondition to pass migration linter

* try to get migrations fixed

* pasing pivot-results through api and attachments

* fix tests for format_rows in BODY vs query param

* tests!

* might have the tests all fixed now

* the pivoted export now respects col/row totals settings

* add test coverage for public questions and dashboards

* col and row totals work as expected

* build-pivot refactor for clarity

* docstring change + tiny refactor in helper fn

* see if dashcard download works with format_rows

* csv pivot handles nil values

* pass format_rows and pivot_results in :params not :body

* fix some other tests

* pivot-grouping col filtered out of xlsx

* pivot-grouping-col removed for all rows

* configurable pivot exports and attachments (#47880)

* exports fe

* specs

* ui

* specs

* format/unformatted now works for xlsx

* format test changes for xlsx formatting

* embedding endpoints accept pivot_results

* cljfmt and eslint fix

* empty

* embedding test should have formatting defaulted to true

* embed test fixes

* Use `Chip` for export settings widget

* downloads e2e test fix

* fix public download limit test

* public card download defaults

* fix public download defaults in some tests

* Fix visual test

---------

Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* Implement inactive field removal

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Do not save metric viz settings to metric (#48609)

* Add repro for #44171

* Do not save viz settings for metrics

* Lowercase description

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

* Rename variable to be more descriptive

* Use visitDashboard

* Fix typo in excludeVisualisationSettings

* Check the request does not contain visualization_settings too

* Allow visitQuestion to work for metrics

---------

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

* Format code

---------

Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

* Refactor createDashboardWithQuestions, createQuestionAndAddToDashboard and editDashboardCard to TS function helpers

* Fix createNativeQuestionAndDashboard that got out of sync

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

* Update copy on metrics page

* Update copy in empty metric builder

* Fix e2e test

Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>

* increase time tolerance to make tests less flaky on ci when CPU is under heavy load

* Update e2e/test/scenarios/sharing/public-sharing-embed-button-behavior.cy.spec.js



---------

Co-authored-by: Nicolò Pretto <info@npretto.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

* Rename Common Metrics to Metrics

* Rename Basic Metrics to Summaries

* Update chill mode button copy to 'Add a summary or metric'

* Rename 'Pick the metric you want to see' button to include summary

* Fix type in test

* Rename Summaries to Basic functions

* Fix test

* Undo cross-version test changes

Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

Co-authored-by: bryan <bryan.maass@gmail.com>

* Collapse `metabase.shared.*` namespaces

* Fix Kondo warnings

* Does updating the stories-data keys fix the failing tests?

* Appease msgcat

* Appease msgcat

* Fix typo

* Make the build happy

* Appease fslint

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Revert "fix flakiness during resize (#48556)"

This reverts commit 03b6584b.

* Move .should call up to fix flaky test

* Use consistent naming

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

 Repro: App crashes when editing custom expression referencing metric when dependent metadata is not loaded (#48560) (#48683)

* Add repro for #47058

* Delay the response and test for the actual metric button

* Move setup to before and find summarize block using helper

* Use beforeEach

Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

* version and channel query params for version info

https://github.com/metabase/metabase/issues/48615



* omit blanks from query params for version info

Co-authored-by: John Swanson <john.swanson@metabase.com>

* Update GitHub link that has been moved

* Get rid of a failing link

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

Two reasons: it's very noisy, and the feature is not very user-visible, so does not make a lot of sense yet

Co-authored-by: Alexander Solovyov <alexander@solovyov.net>

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

fix(sdk): allow CLI to check React version without installing and allow continuing setup if React is missing (#48491) (#48528)

* check sdk version via package.json and allow continuing setup

* make package version checks more flexible

Co-authored-by: Phoomparin Mano <poom@metabase.com>

* adding snowplow event for archiving stale items

* adding total_items_archived to cleanup events

Co-authored-by: Nick Fitzpatrick <nick@metabase.com>

* Relax the arg types to ExpressionArg for concat expressions in the legacy schema

Relax the arg types to ExpressionArg for concat since many DBs allow to concatenate non-string types. This also aligns
with the corresponding MLv2 schema and with the reference docs we publish.

Fixes #39439

* Add nested concat schema tests

* Add nested-concat query-processor tests

Co-authored-by: appleby <86076+appleby@users.noreply.github.com>

Co-authored-by: Oisin Coveney <oisin@metabase.com>

* Moving diagnostic state to redux, adding CP action

* changing key binding

* adding e2e test

* adjusting e2e test

* type adjustment

Co-authored-by: Nick Fitzpatrick <nick@metabase.com>

Co-authored-by: Oisin Coveney <oisin@metabase.com>

* Add Next.js to SDK readme

* Fix typo

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

Co-authored-by: Metabase bot <metabase-bot@metabase.com>

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Write Vitess limitation for row limits in MySQL databases

* Update docs/databases/connections/mysql.md



---------

Co-authored-by: Abed Habli <154609371+abdhabli@users.noreply.github.com>
Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

Co-authored-by: Oisin Coveney <oisin@metabase.com>

Co-authored-by: Oisin Coveney <oisin@metabase.com>

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

* Always startup prometheus metrics

previously only started up when a port was provided

```
MB_PROMETHEUS_SERVER_PORT=9191 java -jar metabase.jar
```

But these counters are useful to be included in anonymous stats. So
let's start up the collectors and then we can get their values like:

```clojure
prometheus=> (dotimes [_ 500] (inc :metabase-email/messages))
nil
;; prometheus/value is iapetos.core/value here
prometheus=> (-> system :registry :metabase-email/messages prometheus/value)
501.0
```

* rename `metabase.analytics.prometheus/inc` to `inc!`

it side effects a value and now no longer shadows `clojure.core/inc` so
we're all happy

Co-authored-by: dpsutton <dan@dpsutton.com>

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

Co-authored-by: Oisin Coveney <oisin@metabase.com>

* don't set enable-embedding and setup-embedding-autoenabled in setup actions + cleanup

* update jest test

* shouldn't expect the secret key to be there either 



Co-authored-by: Nicolò Pretto <info@npretto.com>

* Move mbql.u/{mbql-clause?,is-clause?,check-clause} to schema.helpers

And remove duplicate definitions.

Closes #39333

* Fix docstring of schema.helpers.mbql-clause?

This func is apparently no longer used by code in `normalize`, which has it's own local def.

Co-authored-by: appleby <86076+appleby@users.noreply.github.com>