Previously, truncating a `:type/DateTime` column by `:month` or `:day`
would return a `:type/Date`, which subtly broke the query.

In particular, if you try to order-by the breakout column `Created At (month)`
then it would not get de-duplicated, causing a SQL error about conflicting
ORDER BY clauses.

Fixes #46992.

* clarify caching doc

* edit

Previously, `visible-collection-ids` was effectively "free" in that we'd
cached `collection-id->collection` for *all* collections, within a
single request, and then locally filtered it for permissions without
needing to hit the database again.

To be honest, there is probably a better fix for this - we're repeatedly
calling `visible-collection-ids` when we probably could just save a
single copy of it and use it when calculating the effective location of
every collection.

However, this is a very quick and low-risk fix, and I want to prioritize
getting this done, and then we can improve it later.

Locally, I copied down the database from stats and timed the
`/api/search?model_ancestors=true` endpoint.

Before my "speedup" PR (https://github.com/metabase/metabase/pull/46942)
it took ~7 seconds to return results.

After my "speedup" PR, it took ~15s to return results. :grimace:

With this change, it takes 818ms to return results. 

* Fix typo in expression-clause-normalization-test

s/time-interval/relative-time-interval/

* Add missing table-id to card def to fix failing legacy-ref-test

* Ensure that if yarn test-cljs fails it fails the CI checks

When configured to autorun, the command

   shadow-cljs compile test

will exit success even if test failures occur in the node test. See

   https://github.com/thheller/shadow-cljs/issues/425

In order to ensure our CI checks fail when cljs tests fail, we need to invoke
node directly. To prevent running the tests twice, remove `:autorun true` from
the `:test` target in shadow-cljs.edn.

fix(questions + admin/performance): Improve Clear cache button text and question refresh button tooltip (#46791)

* zoom-in-timeseries-drill: do not include :hour and :minute for DATE columns

When the user attempts to drill through a DATE column, we should not return :hour or :minute drill throughs from
available-drill-thrus. Previously we did, which resulted in an error being displayed when the user attempted to select
them.

Modify next-breakout-unit in zoom_in_timeseries.cljc to not return :hour or :minute units for :type/Date columns,
which prevents these from being returned by available-drill-thrus, which prevents them appearing in UI context menus.

* Use lib.temporal-bucket/available-temporal-buckets

Per review feedback, use lib.temporal-bucket/available-temporal-buckets to get the valid-current-units in
metabase.lib.drill-thru.zoom-in-timeseries.

* Do not construct zipmap just to lookup the next unit

Rather than constructing a zipmap from `valid-current-units', which requires iterating over the seq twice, just
traverse the seq once and return the next item in the seq (if any).

Constructing a zipmap made sense in the old code because it was constructed once from static data and used many times
for lookup. In the new version, the vec of `valid-current-units` is dynamic and dependent on the type of the breakout.

* PR suggestion: use threading macro rather than nested sexps

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* PR suggestion: use threading macro rather than transducer / eduction

* PR suggestion: reduce boilerplate in zoom-in-timeseries tests

Reduce code dup by looping over aggregation and bucketing options rather than having a separate deftest for each.

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>
Fixes #39366

* use loki to test png exports on questions

* try to put the img in the body to see if it fixes the scrollbar issues in ci

* fix linting issues

* let's try again in ci

* Attempt to make CI wait before taking a snapshot for downloaded PNGs

* Use the snapshots from CI for PNG downloads

* pdf export test thanks to kelvin suggestion about just testing the png

* refactored the code to extract a util function

* remove pdf tests as they broke in ci 



* Update frontend/src/metabase/env.ts

Co-authored-by: Phoomparin Mano <poom@metabase.com>

* Update frontend/src/metabase/lib/loki-utils.ts

---------

Co-authored-by: Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
Co-authored-by: Phoomparin Mano <poom@metabase.com>

Analyze queries synchronously on save

Lotsa fixes for this

* entity id translation + tests

* add api level test

* simplify definition of eid-table->model + add test

* update tests to take keywords

* improve comment

* generate the eid-table->model map

* delete now-obsolete test

* make it work in oss

* put the resulting response into a key, so we can add more information later

* formatting

* use model names without the model/ prefix as keys

* Creates list of `api/model->db-model`

- update keys for util/entity_id request
- update shape of util/entity_id response
- add test for not-found eids

* formatting

* Respond to code review feedback

---------

Co-authored-by: Oisin Coveney <oisin@metabase.com>

Co-authored-by: Metabase bot <metabase-bot@metabase.com>

* Convert e2e-custom-column-helpers to TS

* Refactor createTimeline, createTimelineEvent & createTimelineWithEvents to TS

* Fix circular imports

Failing after changes in 13983247

Per slack conversation, ok to disable:

https://metaboat.slack.com/archives/C013N8XL286/p1724711461313089?thread_ts=1724414568.662469&cid=C013N8XL286

* Don't encrypt boolean settings (by default)

We have tooling to disable encryption on settings even when the
`MB_ENCRYPTION_SECRET_KEY` is set. Turn it on by default for all boolean
settings, which don't need to be encrypted.

I also optimized the code that decrypts settings on startup because I
didn't want to delay startup if someone had set a bunch of boolean
settings. With 20 set, the old version added about 200+ ms to startup,
about 10ms per boolean setting, whether or not it was encrypted or in
the DB already. The optimized version selects all the never-encrypt
values from the database at once (a bit silly, but we also just exclude
raw `true` and `false` values so we don't bother checking them) and
updates them if they're encrypted - this adds ~40ms to startup with 20
encrypted boolean settings (about 2ms per boolean setting) and ~5ms to
startup on subsequent runs, when no encrypted values are in the DB.

This testing is too strict to support some legitimate use cases that
have no workaround at present. We can bring back this type-checking
eventually, once it's possible to correctly express things like
automatic coercion of strings to numbers in `SUM` aggregations.

Fixes #44431.

* fix import order in the files with disabled rules

* remove hack

* minor

* serialization clarification

* Update docs/installation-and-operation/serialization.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Update docs/installation-and-operation/serialization.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Update docs/installation-and-operation/serialization.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* Update docs/installation-and-operation/serialization.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

Seems that `glue:GetCatalogImportStatus` is also needed for the integration with Amazon Athena because there's multiple AccessDenied on cloudtrail if this permission is missing:
```
{
    "eventVersion": "1.09",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "***********************",
        "arn": "arn:aws:iam::*************:user/metabase",
        "accountId": "**********************",
        "accessKeyId": "*******************",
        "userName": "metabase"
    },
    "eventTime": "2024-04-11T08:29:00Z",
    "eventSource": "glue.amazonaws.com",
    "eventName": "GetCatalogImportStatus",
    "awsRegion": "eu-west-1",
    "sourceIPAddress": "***********",
    "userAgent": "DriverVersion/02.00.35.1001/JDBCVersion/4.2/PluginName/IAM, aws-sdk-java/1.12.339 Linux/5.10.213-201.855.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.22+7 java/11.0.22 clojure/1.11.1 vendor/Eclipse_Adoptium cfg/retry-mode/legacy",
    "errorCode": "AccessDenied",
    "errorMessage": "An unknown error occurred",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "*************************************",
    "eventID": "*************************************",
    "readOnly": true,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "*************************************",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "glue.eu-west-1.amazonaws.com"
    }
}
```

* Update appearance.md

Adding information about how colors are selected by Metabase because there is the confusion that the order in which you select colors is the order you get on the chart

* Update appearance.md

Removed code reference and tried to simplify the context

* Update docs/configuring-metabase/appearance.md

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

---------

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

fix(dashboard): In dashboard value source modal, after closing question picker modal, don't also close the first modal (#47242)

There are a few separate changes here:

- Migrations: add and populate indexed columns perm_value, perm_type, and collection_id to permissions

These fields allows us to efficiently run queries based on collection permissions in the DB without string manipulation. Keeping the table as permissions allows us to do this migration in-place. Note that in some cases collections may have been deleted from the database without deleting the associated permissions row (since there was no foreign key before). We need to be defensive here: if we have a permissions row without a corresponding collection, delete the row before running the rest of the migration.

- Write perm_value, perm_type, and collection_id for new collection permissions

A very simple before-insert method sets these fields before a collection permission is written to the DB.

- Replace collection/permissions-set->visible-collection-ids and collection/visible-collection-ids->honeysql-filter-clause with collection/honeysql-filter-clause

Previously, just about everywhere we used permissions-set->visible-collection-ids, what we were essentially doing was an in-app join: select all the collection IDs you have permission on, then convert it to a SQL clause like WHERE collection_id IN ( all of those collection IDs).

Replace both of these with honeysql-filter-clause, which uses the new fields we added to permissions above to construct a honeysql filter representing "all the collections I have permissions on", without needing to round-trip them to the application and back to the DB.

Of course, we can then write a function visible-collection-ids, which uses honeysql-filter-clause, for those cases where we do actually need the whole bunch in the application (we use this, for example, when constructing the effective-location for a collection).

I also added one more toggle to the VisibilityConfig that's passed into the honeysql-filter-clause (and used to be passed to permissions-set->visible-collection-ids), allowing you to select only the effective children of some collection.