* Add new route

* Add the navigation link

* Guard the route

* Add first pass of the page structure

* Add the bottom section

* Add placeholder dummy YT videos

* Add images

* Track checklist being opened

* Track checklist item expanded

* Track checklist item CTA clicked

* PoC: Stop the video on accordion change

* Use actual onboarding videos

* Fix missing unique `key` prop

* Final content tweaks

* Override the accordion styles defined in the `theme`

* Use predefined t-shirt sizes where possible

* Update copy

* Conditionally display the example dashboard CTA

* Add very basic unit test coverage

* Show onboarding link to new instances only

* Remember the last position in the onboarding

* Fix the mock settings types

* Scroll last opened item into view

* Properly display docs links

* Another accordion style override

* Add tests for the Onboarding page contents

* Add E2E Snowplow tests

* Conditionally display xray item contents based on settings

* Split tests

* Make sure items scroll into view when expanded

* Do not show example question CTA if example dashboard doesn't exist

* Allow YT videos to play in fullscreen

* Add comments about the example question

* Update rules for when we show the onboarding nav link

* Test the conditional documentation links display

* Reduce E2E test flakiness

Co-authored-by: bryan <bryan.maass@gmail.com>

* ui implementation of new emb homepage, tests will come later

* change padding of the last card to be consistent with the top one

* update beta badge spacing

* update tests

* fix stories type error

* use quick-start for url

* update HomeContent tests

* update events + related e2e

* Update frontend/src/metabase/home/components/EmbedHomepage/StaticEmbedContent.tsx

* add athena and databricks logos + update paths

* Add fallback for `druid-jdbc`

---------

Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>

* Incremental Pivot Processing for Exports

WIP

Fixes pivot exports for CSV and xlsx.

The CSV export should use less memory by incrementally building up the data structure and aggregating necessary row
data right away, so the memory overhead becomes only as large as the total pivot result.

In cases where the pivot rows/cols do combine into many many columns and rows, this can still be a large set of data,
but it should behave much better now in most cases.

The Excel export is a little more straightforward: create the export rows in the same fashion, streaming one row at a
time, and just post-process the sheet to add the pivot table in one shot at the end.

* WIP adding row totals.

* aggregate totals as rows are added

Row, column, section, and grand totals are all aggregated as each row is added.
This means the final step of building pivot output becomes just an exercise of lookups/arrangement, no further
aggregation is needed.

* CSV pivot works per-row, export respects formatting

This is a big step forward; we don't need to hold the entire dataset in memory, we instead aggregate a row's data into
the pivot datastructure, which only holds onto:

- unique values for each pivot-row in a sorted set
- unique values for each pivot-col in a sorted set
- grand total for each measure N values, where N is number of measures, ususally 1 or 2
- row totals for each combination of each pivot-row * N measures
- col totals for each combination of each pivot-col * N measures
- totals for each 'section', determined by unique values of first pivot-row * N measures
- values for each measure in every 'cell'; Row Combos * Col Combos * N Measures

So, there can still be a decent amount of data to store; but it will never hold onto all of the 'raw rows' from the
dataset.

We can never completely guarantee that Row Combos * Col Combos * N Measures remains small, but two things let us move
forward anyway:

- there's now visible feedback in the app that the download is running (or if it's failed)
- Pivot table utility diminishes rapidly with huge output anyway; users still need to curate/set up their data
- effectively to improve the table's utility, so we can assume that a slow-to-download pivot table is also slow to
- use/less effective, and will likely be something the user doesn't want (as often).

* some test fixes

* now, if we export 'raw pivot rows', they don't show pivot-grouping

and they also don't include the 'extra' rows for totals/subtotals/grand totals (any row with pivot-grouping > 0).

This means that now the non-pivot version of a pivot table export will match what a user sees if they change the viz
to a regular table.

* remove old test

* re-incorporate some changes from master

* fix csv for non-pivots due to oversight in my changes

This is just a temporary change, I think I should clean up this bit of the code a little, I can probably make it a
little more readable and use some cleaner logic regarding if the rows are 'raw pivot rows' or not.

* start moving format_rows to POST bod, add pivot_results too

There's still wiring work to do, but this starts to add format_rows and pivot_results to POST body for the various API
endpoints. Also modify tests to improve coverage/consistency across downloads and alerts/subscriptions.

The tests will not pass on this commit, but fixes will be incoming

* native pivot tables in xlsx

* add precondition to pass migration linter

* try to get migrations fixed

* pasing pivot-results through api and attachments

* fix tests for format_rows in BODY vs query param

* tests!

* might have the tests all fixed now

* the pivoted export now respects col/row totals settings

* add test coverage for public questions and dashboards

* col and row totals work as expected

* build-pivot refactor for clarity

* docstring change + tiny refactor in helper fn

* see if dashcard download works with format_rows

* csv pivot handles nil values

* pass format_rows and pivot_results in :params not :body

* fix some other tests

* pivot-grouping col filtered out of xlsx

* pivot-grouping-col removed for all rows

* configurable pivot exports and attachments (#47880)

* exports fe

* specs

* ui

* specs

* format/unformatted now works for xlsx

* format test changes for xlsx formatting

* embedding endpoints accept pivot_results

* cljfmt and eslint fix

* empty

* embedding test should have formatting defaulted to true

* embed test fixes

* Use `Chip` for export settings widget

* downloads e2e test fix

* fix public download limit test

* public card download defaults

* fix public download defaults in some tests

* Fix visual test

---------

Co-authored-by: Aleksandr Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* [Notification] Migrate user invited email (#48215)

* [Notification] Migrate alert create email (#48292)

* [Notification] Migrate slack token error email (#48333)

* [Notification] Notification and subscription (#47707)

* [Notification] Notification and subscription (#47707)

* [Notification] Handlers + recipients (#47759)

* [Notification] Channel template table and model (#47782)

* [Notification] Render system event emails (#47859)

* [Notification] Strict type for channel template and notification recipient (#47910)

* [Notification] Event hydration (#47953)

* [Notification] Send asynchronously (#48200)

Co-authored-by: Oisin Coveney <oisin@metabase.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Nicolò Pretto <info@npretto.com>

* rename usage analytics and few fixes

* update e2e tests

* update some other strings too

---------

Co-authored-by: Ryan Laurie <iethree@gmail.com>

* Fix some minor collections issues

- explicitly provide access to the trash to all users, in the same way
we provide access to personal collections. Due to migration order, we
don't necessarily have a permissions row with the correct `perm_type`,
`perm_value`, and `collection_id` for the Trash collection. That's ok -
we don't actually move things to the Trash, so there isn't any
item where `collection_id=$trash.id` - but this may affect things like
effective ancestors for children of the trash. Let's be explicit about
the permissions that users have.

- replace a case where we manually calculated effective location and
then got the parent_id with just hydrating `:effective_parent`. This is
more efficient.

- replace the simple hydration method for `effective-location-path` with
a batched hydration method that fetches `visible-collection-ids` *once*
and then uses it to figure out the effective location path for each
collection passed.

* Add custom image for empty metrics

* Add useFetchMetrics helper

* Copy browse models page to metrics page

* Add route for browse metrics page

* Add useHasMetrics hook

* Add link to browse metrics page to the sidebar

* Simplify MetricsTable component

* Render description markdown in table too

* Add unit tests for browse metrics page

* Remove custom padding

* Fix line height of markdown cell

* Add e2e tests for browse page

* Fix metrics sorting

* Fix types

* Fix column widths and cell padding

* Typo

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

* Remove name check

* Simplify alphabet

* Remove resetSnowplow

* Use helper to get metric in table

* Use MouseEvent from React directly

* Use --mb-color-icon-primary

* Test ellipsification of description

* Simplify useHasMetrics

* Move useHasMetrics MainNavBar

* Merge sortMetrics and sortModels into sortCards

* Fix metric default description

* Extract SidebarSkeleton to a separate component

* Test sorting via Collection too

* Use ModelTableRow to fix cursor

* Share more components between ModelsTable and MetricsTable

* Use the same simplified structure in ModelsTable as for MetricsTable

* Render block-level elements as inline so ellipsis detection works

* Pick better testid for metric name

* Remove delay

* Avoid type specificty error in sortCards

* Add test for not opening in new tab

* Remove useFetchMetrics defaults

* Fix unit test import

* Test meta-click with cypress

* Add comment on inline markdown elements

* Use model-name test id

* Use correct colors for icon

* Remove delay from browse models page too

* Make meta key in test platform-dependent

* Fix hover color on metrics table

* Add focus styling for celss in the models and metrics table

* Use MarkdownPreview for table cells

* Define --mb-color-icon-primary

* Make the collection link take up the whole height of the cell

* Remove tabIndex from table rows

* Rename ModelsTable.module.css to BrowseTable.module.css

* Rename CardsTable to BrowseTable

* Remove single module.css class in favor of styled components

* Allow customising allowed elements

* Allow forcing all markdown on one line

* Render bold and italics in markdown descriptions and move content onto one line

* Fix legacy tooltip in test

* Make sure getValueForSorting returns a string

* Remove empty file

* Remove icon for metrics in the table

* Remove else statement.

Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

* Remove async findBy where possible

* Add github issue link for TODO item

* Disambiguate between getMetricDescription type

* Be explicit with sortModelOrMetric

---------

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>
Co-authored-by: Uladzimir Havenchyk <125459446+uladzimirdev@users.noreply.github.com>

This migration is supposed to be backported to v50 and thus needs a v50
migration ID.  Since v51.2024-08-27T00:00:00 was already applied to dev
instances, we set `preConditions[].onFail = "MARK_RAN"`, so that if the
table is already present, the backdated migration does not fail.  See
the discussion on PR #47419 for more details.

This will not need an (automatic) backport, since I will include it in
PR #47419.

Fixes: 592360c9
References: https://github.com/metabase/metabase/pull/47419#discussion_r1738929516

* [notification] New method: `channel/can-connect?` (#44955)

* [notification] Channel APIs (#45207)

* [notification] namespaced channel type (#45527)

* [Notification] Render alert for http channel (#45545)

* [notification] Add channel description (#45840)

* [notification] update API to enable http channels for alert (#45839)

* [Notification] Remove channel details for users without write perms (#46034)

* [Notification] Serdes channel (#46031)

* [Notification] Update http details schema (#45960)

* [Notification] Deactivate channels will delete PulseChannel (#46115)

* [Notification] audit log for channel create and update (#46113)

* [Notification] Disallow undefined key for http channel details (#46712)

* [Notification] Handle channel name conflicts (#46818)

* Webhooks Admin Section (#46194)

* [notification] Fix test pulse endpoint does not work properly for http channels (#46474) (#47050)

* [Notification] Fix unable to update multiple channels per type (#47111)

* [Notification] Record Task History when pulse sends channel message (#46218)

* Enabling Webhook Alerts (#47022)

* [Notification] fix cyclic deps (#47379)

* [notification] channel serdes spec (#47386)

Co-authored-by: Nick Fitzpatrick <nick@metabase.com>

* move migration in release to match master

* add column if not exists

== Goal ==

Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them

The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.

== Implementation ==

The Metabase backend already contains provisions in the implementation
of `metabase.models.interface/to-json` for `:model/Database` to hide the
`details` of the database in HTTP responses, if the user lacks write
permission on the database.  We utilize this by adding an
`is_attached_dwh` column to the `database` table and rejecting
`metabase.models.interface/can-write?` when this flag is enabled.  In
the "admin" UI, we show a replacement text instead of the edit form when
the flag is set.  (It might be correct to show this whenever `details`
is absent.  See below for possible follow-up work.)

However, several sections of the frontend code expected the `details`
field to always be present.  In order to make `details` optional, as the
backend seems to handle it, we fix the respective code to treat this
case in the way that appears appropriate in the context.

Database details are already generally excluded from H2 dump snapshots
(see `metabase.cmd.copy/*copy-h2-database-details*`), thus nothing
changes there.

== How to test ==

=== New behaviour ===

Setting the `is_attached_dwh` field hides the database details:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - In addition to the fields you would normally set, also set
     `is_attached_dwh: true`.
   - This also works when adding this flag to a database that previously
     did not have this flag set.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
   (`/admin/databases`).
4. Verify that clicking the database to see its details only reveals
   "This database cannot be modified."
5. Verify that responses from the backend do not include a `details`
   field for this database.

=== Original behaviour ===

Behaviour without setting the `is_attached_dwh` field is unchanged:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - Only set the fields you would normally set.  Do not set
     `is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
   (`/admin/databases`).
4. Verify that clicking the database to see its details only reveal the
   regular edit form, showing connection fields like `host`, `user`,
   etc. with the values you configured.

== How this will be rolled out ==

1. Upgrade existing Metabase Cloud instances with data warehouse to a
   Metabase version that supports `is_attached_dwh`.
2. Set `is_attached_dwh` in the database section of the config file for
   Metabase Cloud instances with a data warehouse.

== Possible follow-up work ==

In https://github.com/metabase/metabase/issues/25715, absent
`database.details` was identified as a bug.  Since then, `details` was
made `NOT NULL` in the application database, so this bug can no longer
occur.  However, today backend responses can be missing the `details`
field, if the current user lacks write permission to the database
setting (see above).  Fully re-evaluating the fix to #25715 in this
context is outside the scope of this PR.

Closes: https://github.com/metabase/harbormaster/issues/5051

This updates the instance analytics YAMLs:
- v50.22
- fixes several fk-target-field-id references

These migrations will be backported to v49.

Three changes:

- update IDs/locations of the migrations to v49 vs v51

- add one preCondition (when adding the foreign key constraint)

- add `onFail: MARK_RAN` to the index preconditions. I forgot this
before, so it blows up when the precondition doesn't hold.

* Increase log level to INFO from error

Closes #47396

logs before:
```
2024-08-29 08:37:03,836 INFO db.liquibase :: Checking if Database has unrun migrations...
2024-08-29 08:37:04,045 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken...
2024-08-29 08:37:04,047 INFO db.liquibase :: No migration lock found.
2024-08-29 08:37:04,047 INFO db.liquibase :: Migration lock acquired.
2024-08-29 08:37:04,167 INFO db.liquibase :: Running 83 migrations ...
```

logs after:

```
2024-08-29 08:39:45,019 INFO db.liquibase :: Running 83 migrations ...
...
2024-08-29 08:39:45,166 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:00::calherries ran successfully in 0ms
2024-08-29 08:39:45,182 INFO liquibase.changelog :: Columns is_defective_duplicate(boolean) added to metabase_field
2024-08-29 08:39:45,182 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:01::calherries ran successfully in 14ms
2024-08-29 08:39:45,199 INFO liquibase.changelog :: Custom SQL executed
2024-08-29 08:39:45,200 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:02::calherries ran successfully in 16ms
2024-08-29 08:39:45,202 INFO liquibase.changelog :: Foreign key fk_field_parent_ref_field_id dropped
2024-08-29 08:39:45,202 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:03::calherries ran successfully in 1ms
2024-08-29 08:39:45,204 INFO liquibase.changelog :: Foreign key constraint added to metabase_field (parent_id)
2024-08-29 08:39:45,205 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:04::calherries ran successfully in 2ms
...
```

* empty commit to bump CI

* handle cache config overflowed

* fix root too

* do not comment out test

* make cljfmt happy

---------

Co-authored-by: Alexander Solovyov <alexander@solovyov.net>

There are a few separate changes here:

- Migrations: add and populate indexed columns perm_value, perm_type, and collection_id to permissions

These fields allows us to efficiently run queries based on collection permissions in the DB without string manipulation. Keeping the table as permissions allows us to do this migration in-place. Note that in some cases collections may have been deleted from the database without deleting the associated permissions row (since there was no foreign key before). We need to be defensive here: if we have a permissions row without a corresponding collection, delete the row before running the rest of the migration.

- Write perm_value, perm_type, and collection_id for new collection permissions

A very simple before-insert method sets these fields before a collection permission is written to the DB.

- Replace collection/permissions-set->visible-collection-ids and collection/visible-collection-ids->honeysql-filter-clause with collection/honeysql-filter-clause

Previously, just about everywhere we used permissions-set->visible-collection-ids, what we were essentially doing was an in-app join: select all the collection IDs you have permission on, then convert it to a SQL clause like WHERE collection_id IN ( all of those collection IDs).

Replace both of these with honeysql-filter-clause, which uses the new fields we added to permissions above to construct a honeysql filter representing "all the collections I have permissions on", without needing to round-trip them to the application and back to the DB.

Of course, we can then write a function visible-collection-ids, which uses honeysql-filter-clause, for those cases where we do actually need the whole bunch in the application (we use this, for example, when constructing the effective-location for a collection).

I also added one more toggle to the VisibilityConfig that's passed into the honeysql-filter-clause (and used to be passed to permissions-set->visible-collection-ids), allowing you to select only the effective children of some collection.

* tests ensure blocked tables downgrade > blocked db

* implement the blocked table -> blocked db rollback

- move/add tests for it

Make sure we insert special segmented permission paths for sandboxed tables on downgrade to 49 (#47085)

I didn't consider that we might not have permissions necessary to
disable triggers.

* new version of data_access migration + run ANALYZE beforehand

* relax constraints on migration timestamps

* more migration rewrites

* fix bug in migration

* fix another bug and remove extraneous logging

* reorder migrations to fix ID validation

* start adapting the changes to mysql as well

* wip

* add more ANALYZE calls

* revert logging changes

* finish duplicating work for mysql + a bit of clean up

* create H2 versions of migrations without the ANALYZE calls

* remove ANALYZE call from view_data.sql (probably not necessary)

* fix mysql syntax

* fix another mysql bug

* another mysql fix

* Don't exclude inactive tables for data_access

* Don't write rows for Administrators group

* Don't write rows for Admins (H2/MySQL)

* disable triggers before perms migration

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>

* Add column_settings migration

* Remove old getColumnKey

* Remove old getColumnKey

* Fix export

* Change comment

* Change comment

* Change comment

* Change comment

* Add RemoveFieldRefsFromDashboardCardColumnSettings

* fix

* chore

* chore

* chore

* chore

* chore

* chore

* chore

* chore

* chore

* chore

* chore

* Fix

* Fix

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Fix tests

* Update loki

* Update loki

* Fix the homepage dashboard being loaded everywhere

* Fix the homepage dashboard being loaded everywhere

* Fix tests

* Fix tests

* Fix tests

* Fixes

* Fix tests

* Fix tests

* Fix tests

* Fix

* Fix

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fixes

* Fix typo

* Fix types

* Improve migration performance

* Fixes

* Revert changes

* Revert changes

* Revert changes

* Revert changes

* Remove migration

### Description

This change should fix some explosions in stats due to phantom fields, while also making things a bit more rigorous.

- Record the table schema in query_field - especially useful for unknown tables
- Allow null table names in query_field - they might be phantoms, or they might not
  - We still have the option to filter them out in tools we build on top
- Tighten up matching logic to consider schema