This migration is supposed to be backported to v50 and thus needs a v50
migration ID.  Since v51.2024-08-27T00:00:00 was already applied to dev
instances, we set `preConditions[].onFail = "MARK_RAN"`, so that if the
table is already present, the backdated migration does not fail.  See
the discussion on PR #47419 for more details.

This will not need an (automatic) backport, since I will include it in
PR #47419.

Fixes: 592360c9
References: https://github.com/metabase/metabase/pull/47419#discussion_r1738929516

* [notification] New method: `channel/can-connect?` (#44955)

* [notification] Channel APIs (#45207)

* [notification] namespaced channel type (#45527)

* [Notification] Render alert for http channel (#45545)

* [notification] Add channel description (#45840)

* [notification] update API to enable http channels for alert (#45839)

* [Notification] Remove channel details for users without write perms (#46034)

* [Notification] Serdes channel (#46031)

* [Notification] Update http details schema (#45960)

* [Notification] Deactivate channels will delete PulseChannel (#46115)

* [Notification] audit log for channel create and update (#46113)

* [Notification] Disallow undefined key for http channel details (#46712)

* [Notification] Handle channel name conflicts (#46818)

* Webhooks Admin Section (#46194)

* [notification] Fix test pulse endpoint does not work properly for http channels (#46474) (#47050)

* [Notification] Fix unable to update multiple channels per type (#47111)

* [Notification] Record Task History when pulse sends channel message (#46218)

* Enabling Webhook Alerts (#47022)

* [Notification] fix cyclic deps (#47379)

* [notification] channel serdes spec (#47386)

Co-authored-by: Nick Fitzpatrick <nick@metabase.com>

* move migration in release to match master

* add column if not exists

== Goal ==

Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them

The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.

== Implementation ==

The Metabase backend already contains provisions in the implementation
of `metabase.models.interface/to-json` for `:model/Database` to hide the
`details` of the database in HTTP responses, if the user lacks write
permission on the database.  We utilize this by adding an
`is_attached_dwh` column to the `database` table and rejecting
`metabase.models.interface/can-write?` when this flag is enabled.  In
the "admin" UI, we show a replacement text instead of the edit form when
the flag is set.  (It might be correct to show this whenever `details`
is absent.  See below for possible follow-up work.)

However, several sections of the frontend code expected the `details`
field to always be present.  In order to make `details` optional, as the
backend seems to handle it, we fix the respective code to treat this
case in the way that appears appropriate in the context.

Database details are already generally excluded from H2 dump snapshots
(see `metabase.cmd.copy/*copy-h2-database-details*`), thus nothing
changes there.

== How to test ==

=== New behaviour ===

Setting the `is_attached_dwh` field hides the database details:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - In addition to the fields you would normally set, also set
     `is_attached_dwh: true`.
   - This also works when adding this flag to a database that previously
     did not have this flag set.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
   (`/admin/databases`).
4. Verify that clicking the database to see its details only reveals
   "This database cannot be modified."
5. Verify that responses from the backend do not include a `details`
   field for this database.

=== Original behaviour ===

Behaviour without setting the `is_attached_dwh` field is unchanged:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - Only set the fields you would normally set.  Do not set
     `is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
   (`/admin/databases`).
4. Verify that clicking the database to see its details only reveal the
   regular edit form, showing connection fields like `host`, `user`,
   etc. with the values you configured.

== How this will be rolled out ==

1. Upgrade existing Metabase Cloud instances with data warehouse to a
   Metabase version that supports `is_attached_dwh`.
2. Set `is_attached_dwh` in the database section of the config file for
   Metabase Cloud instances with a data warehouse.

== Possible follow-up work ==

In https://github.com/metabase/metabase/issues/25715, absent
`database.details` was identified as a bug.  Since then, `details` was
made `NOT NULL` in the application database, so this bug can no longer
occur.  However, today backend responses can be missing the `details`
field, if the current user lacks write permission to the database
setting (see above).  Fully re-evaluating the fix to #25715 in this
context is outside the scope of this PR.

Closes: https://github.com/metabase/harbormaster/issues/5051

These migrations will be backported to v49.

Three changes:

- update IDs/locations of the migrations to v49 vs v51

- add one preCondition (when adding the foreign key constraint)

- add `onFail: MARK_RAN` to the index preconditions. I forgot this
before, so it blows up when the precondition doesn't hold.

* handle cache config overflowed

* fix root too

* do not comment out test

* make cljfmt happy

---------

Co-authored-by: Alexander Solovyov <alexander@solovyov.net>

There are a few separate changes here:

- Migrations: add and populate indexed columns perm_value, perm_type, and collection_id to permissions

These fields allows us to efficiently run queries based on collection permissions in the DB without string manipulation. Keeping the table as permissions allows us to do this migration in-place. Note that in some cases collections may have been deleted from the database without deleting the associated permissions row (since there was no foreign key before). We need to be defensive here: if we have a permissions row without a corresponding collection, delete the row before running the rest of the migration.

- Write perm_value, perm_type, and collection_id for new collection permissions

A very simple before-insert method sets these fields before a collection permission is written to the DB.

- Replace collection/permissions-set->visible-collection-ids and collection/visible-collection-ids->honeysql-filter-clause with collection/honeysql-filter-clause

Previously, just about everywhere we used permissions-set->visible-collection-ids, what we were essentially doing was an in-app join: select all the collection IDs you have permission on, then convert it to a SQL clause like WHERE collection_id IN ( all of those collection IDs).

Replace both of these with honeysql-filter-clause, which uses the new fields we added to permissions above to construct a honeysql filter representing "all the collections I have permissions on", without needing to round-trip them to the application and back to the DB.

Of course, we can then write a function visible-collection-ids, which uses honeysql-filter-clause, for those cases where we do actually need the whole bunch in the application (we use this, for example, when constructing the effective-location for a collection).

I also added one more toggle to the VisibilityConfig that's passed into the honeysql-filter-clause (and used to be passed to permissions-set->visible-collection-ids), allowing you to select only the effective children of some collection.

* tests ensure blocked tables downgrade > blocked db

* implement the blocked table -> blocked db rollback

- move/add tests for it

Make sure we insert special segmented permission paths for sandboxed tables on downgrade to 49 (#47085)

I didn't consider that we might not have permissions necessary to
disable triggers.

* new version of data_access migration + run ANALYZE beforehand

* relax constraints on migration timestamps

* more migration rewrites

* fix bug in migration

* fix another bug and remove extraneous logging

* reorder migrations to fix ID validation

* start adapting the changes to mysql as well

* wip

* add more ANALYZE calls

* revert logging changes

* finish duplicating work for mysql + a bit of clean up

* create H2 versions of migrations without the ANALYZE calls

* remove ANALYZE call from view_data.sql (probably not necessary)

* fix mysql syntax

* fix another mysql bug

* another mysql fix

* Don't exclude inactive tables for data_access

* Don't write rows for Administrators group

* Don't write rows for Admins (H2/MySQL)

* disable triggers before perms migration

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>

### Description

This change should fix some explosions in stats due to phantom fields, while also making things a bit more rigorous.

- Record the table schema in query_field - especially useful for unknown tables
- Allow null table names in query_field - they might be phantoms, or they might not
  - We still have the option to filter them out in tools we build on top
- Tighten up matching logic to consider schema

* Squash

* Clean up query modifier for validating card references (#46283)

* Fixup bad git merge on test expectation

* Tweak docstrings and de-densify logic

* Adds a middleware that sets 2 dynamic vars

- These vars will be used to insert context for embedding analytics

* migration adding embedding_client and embedding_version to qe and vl tables

* clarify function api

* headers come in -> dynamic vars -> written to db

When the vars are set we insert their values into query execution and
query_log under embedding_client and embedding_version.

* update update migrations

* add predconditions to migrations

* fix tablenames + dataset_test

* Code Review responses

- remove commented out requires
- clarify test for merge-count-maps

* Add `report_dashboard.last_viewed_at`

- defaults to `now()` (it's intended to be used as a timestamp after which
we're *sure* the item hasn't been used, not a definitive timestamp we're
sure the dashboard was actually used at)

- updated whenever the QP processes a query for a dashcard

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Nicer error messages for linting migrations

In general, just throw exceptions instead of using clojure spec here.

You'll only get one error per run, but that seems fine.

I did keep some specs around, but run them slightly differently. Rather
than validating the whole collection of changeSets at once with `s/+`,
just `doseq` through the changeSets and validate each one separately.

That way, the value that's presented as erroneous is much smaller (a
single changeSet) and it's easier to see what went wrong.

* Update bin/lint-migrations-file/test/lint_migrations_file_test.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* Update bin/lint-migrations-file/src/lint_migrations_file.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* Require preConditions for certain liquibase change types to encourage idempotence (#44578)

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* remove `!`s from non-side-effecty fns

* add preConditions to a few v51 migrations

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

Per discussion in Slack, I made a mistake with the migration IDs as I wanted to backport to 50 but used 51 IDs.
This PR uses 50 IDs and places the migrations in the correct place.

Not sure if it's mergable in this state because Stats has run with the PR containing the incorrect migration IDs.

* User Parameter Values Are Unique Per Dashboard

Fixes: #43001

Might be related: #44858

User Parameter Values previously stored just the `parameter-id` and `user-id` as data to look up the last used
parameter value. This isn't sufficient as the parameter id is not guaranteed to be unique, in particular this is true
when a dashboard is duplicated: the parameters are copied to the new dashboard without changing the parameter id at
all.

This means that we need to also store the dashboard-id in the user_parameter_value table and use that to update/get
the last used value.

The migration removes existing entries to the user_parameter_value table because I want a non-nullable constraint on
the dashboard_id column, and existing entries will not have a value. The only way to backfill those values would be to
look through every dashboard containing parameters, and for every parameter check for a matching ID. Even if you can
do this, there's no way to disambiguate if the same paramter exists on 2 dashboards anyway, so one of them will be
wrong. I think it's not worth the trouble considering that removing the entries in this table doesn't break anything;
it just becomes a mild inconvenience that a user has to select a filter value again (since the dashboard will use the
default value).

* alter test to check for uniqueness across dashboard

This test makes 2 dashboards with parameters of the same ID and asserts that changing the value on dashboard-a does
not change the value on dashboard-b.

* adjust migration to pass linter rules

* remove the extra rollback on migration

* Update src/metabase/models/user_parameter_value.clj

Co-authored-by: bryan <bryan.maass@gmail.com>

* Update src/metabase/models/user_parameter_value.clj

Co-authored-by: bryan <bryan.maass@gmail.com>

* adjust parameters in test so we don't get logged warnings

* put update!/insert! in a transaction to avoid any race conditions

---------

Co-authored-by: bryan <bryan.maass@gmail.com>

* Writing migration notes