== Goal ==

Hide attached DWH database details from anyone incl. admins:
* Do not show them in the UI
* Do not permit to change them
* Do not serialize them

The aim is that customers cannot gain access to (parts of) credentials,
and they cannot break a feature they are paying for by changing
connection details.

== Implementation ==

The Metabase backend already contains provisions in the implementation
of `metabase.models.interface/to-json` for `:model/Database` to hide the
`details` of the database in HTTP responses, if the user lacks write
permission on the database.  We utilize this by adding an
`is_attached_dwh` column to the `database` table and rejecting
`metabase.models.interface/can-write?` when this flag is enabled.  In
the "admin" UI, we show a replacement text instead of the edit form when
the flag is set.  (It might be correct to show this whenever `details`
is absent.  See below for possible follow-up work.)

However, several sections of the frontend code expected the `details`
field to always be present.  In order to make `details` optional, as the
backend seems to handle it, we fix the respective code to treat this
case in the way that appears appropriate in the context.

Database details are already generally excluded from H2 dump snapshots
(see `metabase.cmd.copy/*copy-h2-database-details*`), thus nothing
changes there.

== How to test ==

=== New behaviour ===

Setting the `is_attached_dwh` field hides the database details:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - In addition to the fields you would normally set, also set
     `is_attached_dwh: true`.
   - This also works when adding this flag to a database that previously
     did not have this flag set.
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
   (`/admin/databases`).
4. Verify that clicking the database to see its details only reveals
   "This database cannot be modified."
5. Verify that responses from the backend do not include a `details`
   field for this database.

=== Original behaviour ===

Behaviour without setting the `is_attached_dwh` field is unchanged:

1. Configure a database as described in https://www.metabase.com/docs/latest/configuring-metabase/config-file#databases.
   - Only set the fields you would normally set.  Do not set
     `is_attached_dwh` (or set it to `false`).
2. Start your Metabase instance.
3. Verify the database shows up in the "admin" section
   (`/admin/databases`).
4. Verify that clicking the database to see its details only reveal the
   regular edit form, showing connection fields like `host`, `user`,
   etc. with the values you configured.

== How this will be rolled out ==

1. Upgrade existing Metabase Cloud instances with data warehouse to a
   Metabase version that supports `is_attached_dwh`.
2. Set `is_attached_dwh` in the database section of the config file for
   Metabase Cloud instances with a data warehouse.

== Possible follow-up work ==

In https://github.com/metabase/metabase/issues/25715, absent
`database.details` was identified as a bug.  Since then, `details` was
made `NOT NULL` in the application database, so this bug can no longer
occur.  However, today backend responses can be missing the `details`
field, if the current user lacks write permission to the database
setting (see above).  Fully re-evaluating the fix to #25715 in this
context is outside the scope of this PR.

Closes: https://github.com/metabase/harbormaster/issues/5051

* Add `clojure -M:kondo` and `clojure -M:kondo:kondo/all` and bump version

* Fix Kondo errors

* Fix Kondo+LSP issues with `defendpoint`, `defenterprise`, etc.

* Use replace-deps instead of deps for speed

* Ok apparently maybe we do need to copy configs when we run Kondo on CI

* Oops `./bin/kondo.sh` should not try to use `clj-kondo`

* Remove references to GA driver folders

This updates the instance analytics YAMLs:
- v50.22
- fixes several fk-target-field-id references

These migrations will be backported to v49.

Three changes:

- update IDs/locations of the migrations to v49 vs v51

- add one preCondition (when adding the foreign key constraint)

- add `onFail: MARK_RAN` to the index preconditions. I forgot this
before, so it blows up when the precondition doesn't hold.

* allow re-ordering and re-naming pie chart slices

* aggregate rows with same dimension value

* show warning when dimension has unaggregated values

* change colors from sidebar

* implement renaming

* styles for slice name widget

* handle numeric slice keys for binned values relative dates etc

* implement slice disabling

* remove old colors setting

* implement merging new and removing existing rows from query results

* fix formatting bugs

* port settings to static viz

* fix bug with unaggregated data

* address todos

* add loki test

* fix measure column settings widget erroring

* fix settings sidebar crashing

* add e2e tests

* update snapshots

* fix pie_chart.cy.spec.js CI-only flakes

* fix #21504 repro failure

* fix click action value

* use nestedSettings instead of seriesSettings

* fix settings sidebar crashing

* fix row with 0 key not being draggable

* fix flaky test

* filter other slices from sidebar

* combine filter header buttons

* refined styling

* unit tests

* another import conflict

* omg another one

* remove double border for filter button

* use vars and update imports

* Increase log level to INFO from error

Closes #47396

logs before:
```
2024-08-29 08:37:03,836 INFO db.liquibase :: Checking if Database has unrun migrations...
2024-08-29 08:37:04,045 INFO db.liquibase :: Database has unrun migrations. Checking if migration lock is taken...
2024-08-29 08:37:04,047 INFO db.liquibase :: No migration lock found.
2024-08-29 08:37:04,047 INFO db.liquibase :: Migration lock acquired.
2024-08-29 08:37:04,167 INFO db.liquibase :: Running 83 migrations ...
```

logs after:

```
2024-08-29 08:39:45,019 INFO db.liquibase :: Running 83 migrations ...
...
2024-08-29 08:39:45,166 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:00::calherries ran successfully in 0ms
2024-08-29 08:39:45,182 INFO liquibase.changelog :: Columns is_defective_duplicate(boolean) added to metabase_field
2024-08-29 08:39:45,182 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:01::calherries ran successfully in 14ms
2024-08-29 08:39:45,199 INFO liquibase.changelog :: Custom SQL executed
2024-08-29 08:39:45,200 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:02::calherries ran successfully in 16ms
2024-08-29 08:39:45,202 INFO liquibase.changelog :: Foreign key fk_field_parent_ref_field_id dropped
2024-08-29 08:39:45,202 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:03::calherries ran successfully in 1ms
2024-08-29 08:39:45,204 INFO liquibase.changelog :: Foreign key constraint added to metabase_field (parent_id)
2024-08-29 08:39:45,205 INFO liquibase.changelog :: ChangeSet migrations/001_update_migrations.yaml::v49.2024-06-27T00:00:04::calherries ran successfully in 2ms
...
```

* empty commit to bump CI

* handle cache config overflowed

* fix root too

* do not comment out test

* make cljfmt happy

---------

Co-authored-by: Alexander Solovyov <alexander@solovyov.net>

Exclude inactive tables from the perms cache, and block queries over inactive tables in the QP (#47363)

* Make metric metadata keep the metric's name

This is a fix for the problem reported in #40355, which is fixed separately for v50 and backported to v49.

* Patch broken test

* Normalize dataset-query

* Normalize legacy queries only

* keep locale in url query params

* e2e tests

* ugly fix for the missing baseUrl error

* applies suggestion from Kelvin to make code more demure

* Update e2e/test/scenarios/sharing/public-question.cy.spec.js

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* remove it.skip added by mistake

* sort imports

* handle native question/ SyncedParametersList too

* shorter and more accurate comment 



* Update e2e/support/helpers/e2e-embedding-helpers.js

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

* Support multiple bucketings of a column in nested queries

Fixes #46644

* Fix e2e test

* Adding schema name to recents, search results in CP

* adding unit test

* fixup tests + clarify recent item schema

---------

Co-authored-by: Bryan Maass <bryan.maass@gmail.com>

* remove prevent default on CP links

* unit flakes?

* ask for tenancy isolation columns

* deny all permissions for all users group

* create new collections

* add jwt group mappings

* add the permissions step

* add multi-tenancy message in helper text format

* add permission graph

* wire together permissions

* use schema permissions

* use fields from table metadata from query_metadata

* add tenancy field reference

* remove log messages

* deny access to unsandboxed tables

* make permission graph more explicit

* deny access to sample database for customer groups

* add unit test for permission graph

* split permission groups and sandboxes

* jwt settings and hard-coded user attributes

* handle errors when updating sso mappings

* add express api and user switcher

* only fallback to api keys when license is invalid

* add util to sample tenancy column values

* conditional BASE_SSO_API imports

* improve embedding error message

* setup jwt configuration after license step

* setup permissions at the last step

* add missing import

* update steps that requires license

* fix incorrect imports

* add missing useContext

* handle permission update error

* remove tenancyIsolationEnabled field

* add tenancy column sampling

* differentiate tenancy column query error

* rename tenancyColumnValues to tenantIds

* assign sampled tenant ids to user attributes

* add tenant ids

* define collection permissions

* reference sandboxing group by name

* update snippet to be same as the README

* extract ask for tenancy columns to a separate step

* use the customer_id attribute

* query the table query metadata at origin

* append tables correctly

* improve error handling in table scanning

* add retry logic to metadata fetching

* only query metadata for selected fields

* fix race condition with retry

* update loading state and retries

* update comments on jwt license

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>

* filter the target table by id

* highlight last selected tenant column

* use breakout to get list of ids

* temporary workaround to reload the whole page

* update row value types

* update row value types

* block non-selected tables

* remove the source-field from sandboxing

* use the fk_target_field_id as instead of target.id

* update unit test

* remove source-field as we only reference our own column

* make native permission types more strict

---------

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: Oisin Coveney <oisin@metabase.com>

* ask for tenancy isolation columns

* deny all permissions for all users group

* create new collections

* add jwt group mappings

* add the permissions step

* add multi-tenancy message in helper text format

* add permission graph

* wire together permissions

* use schema permissions

* use fields from table metadata from query_metadata

* add tenancy field reference

* remove log messages

* deny access to unsandboxed tables

* make permission graph more explicit

* deny access to sample database for customer groups

* add unit test for permission graph

* split permission groups and sandboxes

* jwt settings and hard-coded user attributes

* handle errors when updating sso mappings

* add util to sample tenancy column values

* improve embedding error message

* setup jwt configuration after license step

* setup permissions at the last step

* handle permission update error

* add tenancy column sampling

* differentiate tenancy column query error

* rename tenancyColumnValues to tenantIds

* define collection permissions

* reference sandboxing group by name

* extract ask for tenancy columns to a separate step

* query the table query metadata at origin

* append tables correctly

* improve error handling in table scanning

* add retry logic to metadata fetching

* only query metadata for selected fields

* fix race condition with retry

* update loading state and retries

* filter the target table by id

* highlight last selected tenant column

* use breakout to get list of ids

* update row value types

* block non-selected tables

* remove the source-field from sandboxing

* use the fk_target_field_id as instead of target.id

* update unit test

* remove source-field as we only reference our own column

* make native permission types more strict

---------

Co-authored-by: Oisin Coveney <oisin@metabase.com>

This reverts commit 3759c8d4.

fix(admin/performance): Ensure that the strategy form is not considered dirty if the user changes an empty field to that field's default value, or vice versa (#46543)

Previously, truncating a `:type/DateTime` column by `:month` or `:day`
would return a `:type/Date`, which subtly broke the query.

In particular, if you try to order-by the breakout column `Created At (month)`
then it would not get de-duplicated, causing a SQL error about conflicting
ORDER BY clauses.

Fixes #46992.

* clarify caching doc

* edit

Previously, `visible-collection-ids` was effectively "free" in that we'd
cached `collection-id->collection` for *all* collections, within a
single request, and then locally filtered it for permissions without
needing to hit the database again.

To be honest, there is probably a better fix for this - we're repeatedly
calling `visible-collection-ids` when we probably could just save a
single copy of it and use it when calculating the effective location of
every collection.

However, this is a very quick and low-risk fix, and I want to prioritize
getting this done, and then we can improve it later.

Locally, I copied down the database from stats and timed the
`/api/search?model_ancestors=true` endpoint.

Before my "speedup" PR (https://github.com/metabase/metabase/pull/46942)
it took ~7 seconds to return results.

After my "speedup" PR, it took ~15s to return results. :grimace:

With this change, it takes 818ms to return results.