* Support multiple bucketings of a column in nested queries

Fixes #46644

* Fix e2e test

* Adding schema name to recents, search results in CP

* adding unit test

* fixup tests + clarify recent item schema

---------

Co-authored-by: Bryan Maass <bryan.maass@gmail.com>

* Fix typo in expression-clause-normalization-test

s/time-interval/relative-time-interval/

* Add missing table-id to card def to fix failing legacy-ref-test

* Ensure that if yarn test-cljs fails it fails the CI checks

When configured to autorun, the command

   shadow-cljs compile test

will exit success even if test failures occur in the node test. See

   https://github.com/thheller/shadow-cljs/issues/425

In order to ensure our CI checks fail when cljs tests fail, we need to invoke
node directly. To prevent running the tests twice, remove `:autorun true` from
the `:test` target in shadow-cljs.edn.

* zoom-in-timeseries-drill: do not include :hour and :minute for DATE columns

When the user attempts to drill through a DATE column, we should not return :hour or :minute drill throughs from
available-drill-thrus. Previously we did, which resulted in an error being displayed when the user attempted to select
them.

Modify next-breakout-unit in zoom_in_timeseries.cljc to not return :hour or :minute units for :type/Date columns,
which prevents these from being returned by available-drill-thrus, which prevents them appearing in UI context menus.

* Use lib.temporal-bucket/available-temporal-buckets

Per review feedback, use lib.temporal-bucket/available-temporal-buckets to get the valid-current-units in
metabase.lib.drill-thru.zoom-in-timeseries.

* Do not construct zipmap just to lookup the next unit

Rather than constructing a zipmap from `valid-current-units', which requires iterating over the seq twice, just
traverse the seq once and return the next item in the seq (if any).

Constructing a zipmap made sense in the old code because it was constructed once from static data and used many times
for lookup. In the new version, the vec of `valid-current-units` is dynamic and dependent on the type of the breakout.

* PR suggestion: use threading macro rather than nested sexps

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* PR suggestion: use threading macro rather than transducer / eduction

* PR suggestion: reduce boilerplate in zoom-in-timeseries tests

Reduce code dup by looping over aggregation and bucketing options rather than having a separate deftest for each.

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>
Fixes #39366

Analyze queries synchronously on save

* entity id translation + tests

* add api level test

* simplify definition of eid-table->model + add test

* update tests to take keywords

* improve comment

* generate the eid-table->model map

* delete now-obsolete test

* make it work in oss

* put the resulting response into a key, so we can add more information later

* formatting

* use model names without the model/ prefix as keys

* Creates list of `api/model->db-model`

- update keys for util/entity_id request
- update shape of util/entity_id response
- add test for not-found eids

* formatting

* Respond to code review feedback

---------

Co-authored-by: Oisin Coveney <oisin@metabase.com>

Failing after changes in 13983247

Per slack conversation, ok to disable:

https://metaboat.slack.com/archives/C013N8XL286/p1724711461313089?thread_ts=1724414568.662469&cid=C013N8XL286

* Don't encrypt boolean settings (by default)

We have tooling to disable encryption on settings even when the
`MB_ENCRYPTION_SECRET_KEY` is set. Turn it on by default for all boolean
settings, which don't need to be encrypted.

I also optimized the code that decrypts settings on startup because I
didn't want to delay startup if someone had set a bunch of boolean
settings. With 20 set, the old version added about 200+ ms to startup,
about 10ms per boolean setting, whether or not it was encrypted or in
the DB already. The optimized version selects all the never-encrypt
values from the database at once (a bit silly, but we also just exclude
raw `true` and `false` values so we don't bother checking them) and
updates them if they're encrypted - this adds ~40ms to startup with 20
encrypted boolean settings (about 2ms per boolean setting) and ~5ms to
startup on subsequent runs, when no encrypted values are in the DB.

This testing is too strict to support some legitimate use cases that
have no workaround at present. We can bring back this type-checking
eventually, once it's possible to correctly express things like
automatic coercion of strings to numbers in `SUM` aggregations.

Fixes #44431.

There are a few separate changes here:

- Migrations: add and populate indexed columns perm_value, perm_type, and collection_id to permissions

These fields allows us to efficiently run queries based on collection permissions in the DB without string manipulation. Keeping the table as permissions allows us to do this migration in-place. Note that in some cases collections may have been deleted from the database without deleting the associated permissions row (since there was no foreign key before). We need to be defensive here: if we have a permissions row without a corresponding collection, delete the row before running the rest of the migration.

- Write perm_value, perm_type, and collection_id for new collection permissions

A very simple before-insert method sets these fields before a collection permission is written to the DB.

- Replace collection/permissions-set->visible-collection-ids and collection/visible-collection-ids->honeysql-filter-clause with collection/honeysql-filter-clause

Previously, just about everywhere we used permissions-set->visible-collection-ids, what we were essentially doing was an in-app join: select all the collection IDs you have permission on, then convert it to a SQL clause like WHERE collection_id IN ( all of those collection IDs).

Replace both of these with honeysql-filter-clause, which uses the new fields we added to permissions above to construct a honeysql filter representing "all the collections I have permissions on", without needing to round-trip them to the application and back to the DB.

Of course, we can then write a function visible-collection-ids, which uses honeysql-filter-clause, for those cases where we do actually need the whole bunch in the application (we use this, for example, when constructing the effective-location for a collection).

I also added one more toggle to the VisibilityConfig that's passed into the honeysql-filter-clause (and used to be passed to permissions-set->visible-collection-ids), allowing you to select only the effective children of some collection.

* [MBQL lib] Map dashboard filters to last stage w/aggregations

Originally we used the last stage's columns for dashboard filters, but
then if you eg. use a custom expression to clean up the results of the
aggregation, then you can't filter at the most useful spot - before the
aggregations.

Fixes #19744. Related to #26244, #33958.

* Add stage-number option to the dimension not to the field

* Remove FE changes

* Support negative stage numbers

* Add tests for multi-stage parameter expansion

* Specify schema for dimension options

* Fix legacy MBQL schema

* Append stage when running cards with top level summary

* Fix schema

* Only ensure filter stage when the parameters say it's used

* Implement ensure-filter-stage

* Format clj

* Add ensure-filter-stage test

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* Fix DateTimeRange substitution for open ranges

* Add E2E repro for #47172

* cljfmt

* Add test for datetime column

* Update range boundary for open range

* Update test names

* Apply suggestions from code review

Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

* Fix e2e

---------

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

* Fix #46846

* Don't use `-v` to check if script has arg since it doesn't work on all bash versions

* Fix typo

* feat: BigQuery nested fields

Updates dataset definitions to allow `nested-fields`

Adds the `::sql.qp/nfc-path` to include nfc-path in the field
identifier.

* Add bigquery to nested-fields feature. Rework geographical-tips to allow adding to bigquery

* Parse RECORD type into maps

* Fix tests

* Fix tests

* Fix formatting

* Handle arrays of records

* Fix formatting

* Conditionally add nested-fields

* Fix test

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Strip expression metadata for field expression

Fixes #44940.

* Fix binning matching

* Add tests

* Add tests

* Add tests

* typo

* Update src/metabase/lib/equality.cljc

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* Fix alignment

* Fix alignment

* Update src/metabase/lib/js.cljs

Co-authored-by: Braden Shepherdson <braden@metabase.com>

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>

* recents updates to better support metrics

allows updating metric views through the API
improves parent_collection calculation

* add 2 tests for metrics

* Use global search for aggregation picker

* Add names to aggregation picker items so they can be searched

* Render info icon metric descriptions

* Add unit tests for global aggregation picker search

* Add test for searching metrics

* Add a test for metrics descriptions

* Sort metrics

* Fix e2e test

* Use Flex from metabase/ui

* Handle markdown in metric description

* Add test for sorted metrics

* Update Markdown styling in metric description

* Fix clojure formatting

* Test for markdown in the Metric description

* Fix lint

* Fix test for markdown

* Use simpler equality test

* Fix overflow in test

* tests ensure blocked tables downgrade > blocked db

* implement the blocked table -> blocked db rollback

- move/add tests for it

Make sure we insert special segmented permission paths for sandboxed tables on downgrade to 49 (#47085)

* Cljfmt

* Fix new GH action

* Add update-stale-references and test

* Update test

* Kondo Config cleanup

* Restore warnings for with-log-messages-for-level until #28827 is merged

* Updated clojure.test hooks

* Fix kondo warning

* Test defn/defmacro exclamation point linter should also use :parallel/disallowed (part 1)

* WIP

* Time for me to learn to spell

* Finish the cleanup

* Remove code I was typing as an example

* Fix renamed var

* Fix the SAML tests

* Fix Kondo warning

* allow `:blocked` to be saved for table level perms

* Adds 2 tests for table level blocked permission settings

- N.B. these are NOT ENFORCED YET

* update test that asserted we cannot set block on tables (we can)

* WIP: Pairing on making perms checking less wild

* cleanup, update docs, and add a test for view-data perm only

- Added a test where we have data permissions, but not create query, and
  I think it is failing when we have create query and blocked data
  permissions.
- renamed some functions from check-x -> has-x? since they return a
  value instead of throwing now

* Revert "WIP: Pairing on making perms checking less wild"

Keep the same behavior, but stick with the saner flow control

This reverts commit 63bcb5b4.

* update docs

* update test to be passing

- TODO: make sure it's correct w.r.t. perm settings

* Allow schema level blocked setting in permgraph

* remove invalid test cases

- continue to have a forcing function to test newly added perms

* conform function output

* ensure a single blocked table blocks native queries to its DB

* update error message

- we now catch this error in `metabase.models.query.permissions/has-perm-for-query?`

* we now check for data permissions to process query for card

* add more explanation to what we are testing

- to help see why it fails on CI and passes locally

* remove excess `def`

* Add test for table-level data X collection perms

- update test found to be in-error

* update param values qp permission check style

* set view-data and create-query explicitly

* set viewdata and createquery explicitly in qp test

* Respond to review comments (which fixes a case)

* setting a table to blocked: leave other tables the same

* [Permissions] Add "No access" schema/table permission (#46509)

* first pass

* refactors downgrading native permission logic and updates calculation so that "No access" downgrades native permissions to "No"

* stub for permissions help info on table block

* modal changes wip, updates downgrading create queries permissions to all happen at a single call site

* clean up, sandboxing modal copy changes, removes rekoke/limit access modal changes to make the diff smaller and move code to a seperate PR

* updates permissions help section to contain the final copy

* sandboxing copy fix and remove modal that was dropped from requirements

* adds blocked at the schema level, updates no access copy to blocked, updates permissions help section to contain new blocked and schema level changes

* fixes failed unit and e2e tests after sandboxing copy changes

* improve the block e2e test to include table blocking

* fixes failing blocked test, fixes other schemas create queries permissions getting correct with one schema was droped to blocked view data access, fixes a bug that prevents the save bar from going away when all permissions for group are set to the default values

* clean up

* remove color changes

* prevents parent being set to blocked preventing edits for children entities

* add new hasPermissionValueInSubgraph fn, adds modal to warn users we have to upgrade the view data permissions when they upgrade create queries permissions when a child entity is set to blocked

* adds test coverage for new modal

* removes unused function, adds new updateEntityPermission fn to help consolidate some logic elsewhere

* unit test fix and type fix

* most pr feedback

* updates the confirmation modal copy when changing a parent entity that contains a child with blocked permissions and/or sandboxed children, adds test coverage for that, adds test coverage for permission view data column not appearing in oss

* type fix

* [Permissions] Add e2e test coverage for blocked permissions enforcements (#46663)

* adds test coverage for enforcement of blocked permissions

* moves tests around based on pr feedback

* copy changes

* adds fix to make sure that blocked permissions are not removed from sibling tables that have the create queries permissions upgraded (#46854)

* Fix table name lookup for dbs w/ 1 schema per db

* add test for blank schema identifiers

* Refine sandboxed user perms for query builder access (#46939)

* Refine sandboxed user perms for query builder access

- Limit create-queries permissions to unblocked tables only
- Check user permissions for each table before granting query builder access
- Prevent querying of blocked joined tables from query builder for sandboxed users

* Adjust permissions for sandboxed users

- Grant view-data permissions only for unblocked tables
- Revert create-queries permissions to all tables in sandbox
- Remove unnecessary intermediate variable

* when sandboxing we no longer grant unrestricted view perms for blocked tables

* Update enterprise/backend/src/metabase_enterprise/sandbox/query_processor/middleware/row_level_restrictions.clj

remove blank line

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* - make coalesce-test exhaustive (except for sandbox)

* Update enterprise/backend/src/metabase_enterprise/sandbox/query_processor/middleware/row_level_restrictions.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* t2/select ... -> database/table-id->database-id

* update comment

* [Permissions] Prevent "Granular" option in DB View Data options from changing permissions to unrestricted (#46976)

* fix

* adds back most of the code and limits it to only happen with impersonations, updates test to handle differing logic between the two flows

* removes test that is not longer needed

* more sandbox join table perms tests

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>