# Primary objective: From an indexed entity, located the items referencing that entity and make a dashboard for that entity. This will make several dashboards, one for each entity, so the strategy is to combine these all into a single dashboard with tabs. If there are no linked entities as simple dashboard indicating that nothing interesting is linked. If there is a single linked entity we create a single dashboard (no tab) but with the same linked entity title.

A few of the main nses that were affected and other bullet point changes:

* metabase.api.automagic-dashboards - The vast majority of the work to create the unified dashboard is found here.
* metabase.automagic-dashboards.core - Docs, cleanup, and destructuring for clarity
* metabase.api.search - Adding fix to realize search results
* metabase.automagic-dashboards.filters - Fix to build-fk-map so that parameters show up on model x-rays
* metabase.automagic-dashboards.populate - Fix typo
* metabase.api.search-test - Unit tests for search fix
* Brought over tab saving of transient x-ray dashboard code to save-transient-dashboard!

---------

## Overall summary

The primary entry point to these changes can be found in `metabase.api.automagic-dashboards` at:

```clojure
(api/defendpoint GET "/model_index/:model-index-id/primary_key/:pk-id" ...
```

A simple reproduction of dashboard creation from indexed entities is shown here:

```clojure
(let [model-index-id 54 ;; This and pk-id will be specific to some indexed entity in your system
      pk-id          1]
  (binding [api/*current-user-permissions-set* (delay #{"/"})
            api/*current-user-id*              1]
    (let [model-index       (t2/select-one ModelIndex :id model-index-id)
          model             (t2/select-one Card (:model_id model-index))
          model-index-value (t2/select-one ModelIndexValue
                                           :model_index_id model-index-id
                                           :model_pk pk-id)
          linked            (#'api.magic/linked-entities
                              {:model             model
                               :model-index       model-index
                               :model-index-value model-index-value})
          dashboard         (#'api.magic/create-linked-dashboard
                              {:model             model
                               :linked-tables     linked
                               :model-index       model-index
                               :model-index-value model-index-value})]
      dashboard)))
```

---------

## Fixed the query filter in `metabase.api.automagic-dashboards` so that `create-linked-dashboard` doe not produce bad queries.

We're no longer making joins back to the model's underlying
table. Recognizing that we were joining on linked table's fk to the
model's underlying table's pk and then searching where pk = <pk-value>,
we can just filter where fk = <pk-value>, omit the join.

So these tests just grab all of the linked tables and assert that one of
those filters is found.

eg, suppose a model based on products. the linked tables are orders and
reviews. And rather than the queries:

```sql
select o.*
from orders o
left join products p
on p.id = o.product_id
where p.id = <pk-value>
```

we can just do

```sql
select *
from orders
where product_id = <pk-value>
```

And similar for reviews. And for each query in the dashboard we're
looking for one of the following:

```clojure
, #{[:= $orders.product_id 1] [:= $reviews.product_id 1]}
```

---------

## Handle expression refs in indexed-models

Custom columns look like the following:

```clojure
{:expressions {"full-name" [:concat <first-name> <last-name>]}
```

To index values, we need a sequence of primary key and the associated
text value. So we select them from a nested query on the model. But the
model's way to refer to an expression and queries _on_ the model are
different. To the model, it's an expression. To queries based on the
model, it's just another field.

And have field refs in the result_metadata `[:expression
"full-name"]`. But when selecting from a nested query, they are just
string based fields: `[:field "full-name" {:base-type :type/Text}]`.

old style query we issued when fetching values:

```clojure
{:database 122
 :type :query
 :query {:source-table "card__1715"
         :breakout [[:field 488 {:base-type :type/Integer}]
                    [:expression "full name"]] ;; <-- expression
         :limit 5001}}
```

new style after this change:

```clojure
{:database 122
 :type :query
 :query {:source-table "card__1715"
         :breakout [[:field 488 {:base-type :type/Integer}]
                    [:field "full name" {:base-type :type/Text}]]
         :limit 5001}}
```

---------

## Normalize field references

The schema was expecting valid mbql field refs (aka vectors and
keywords) but was getting a list and string (`[:field 23 nil]`
vs ("field" 23 nil)`). So normalize the field ref so we can handle the
stuff over the wire

this nice little bit of normalization lived in models.interface and
comped two functions in mbql.normalize. A few places reused it so moved
it to the correct spot.

* Better error message in `fix-expression-refs`

handles [:field ...] and [:expression ...] clauses. Seems unlikely that
aggregations will flow through here as that's not a great way to label a
pk. But i'll add support for that in a follow up

* Move `normalize-field-ref` below definition of `normalize-tokens`

`normalize-tokens` is `declare`d earlier, but we aren't using the var as
a var, but in a def we derefence it. But that's nil as it hadn't been
defined yet. Caused lots of failures downstream, including in cljs land

```clojure
user=> (declare x)
user=> (def y x) ;; <- the use in a def gets its current value
user=> (def x 3) ;; <- and it's not reactive and backfilling
user=> y
```

* Don't capture `declare`d vars in a def

need late binding. when the function is called, those vars will have a
binding. But if we use a `def`, it grabs their current value which is an
unbound var.

---------

Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>

* Add caching
* Hand roll simplify-compound-filter

* Run slow E2E tests using faster runner

* Tag slow tests

* Add additional x-ray tests to the slow batch

* inital commit

* fix tests

* update test

* fix tests

* fix tests

* fix tests

* fix tests

* update test

* update tests

* update tests

* address changes

* fix tests

* fix test

* fix tess

---------

Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>

The build is failing on master due to :cause "No such var: lib.join/current-join-alias"
This breaking change was introduced in #33574.

Searching for current-join-alias shows that all other references use a different namespace. Namely, lib.join.util.
This PR uses that namespace instead.

The function was moved in #32698. Both PRs have been opened, and have had CI green before the merge, but got out of sync. A merge queue would've prevented this.

This log shows the output of the files changed:
https://github.com/metabase/metabase/actions/runs/6087912385/job/16537680230#step:3:602

```
Changes output set to ["shared_sources","frontend_sources","frontend_specs","frontend_all","backend_sources","backend_specs","backend_all","sources","e2e_specs","e2e_all","codeql","i18n","visualizations"]
```

Yet, the `download-uberjar` job ran, and the `build` never ran.
This resulted in E2E test failures due to missing FE changes.

The fault in the logic introduced in #33190 was thinking that
`needs.files-changed.outputs.e2e_specs == 'true'` means the only output,
when in reality it means one of many outputs.

This PR should fix this by making sure that `needs.files-changed.outputs.e2e_all != 'true'`
is also respected. This translates to - if there were no source code changes.

[ci skip]

* Extract ObjectDetailBody to a separate file

* Extract ObjectDetailHeader to a separate file

* Extract ObjectDetailView to a separate file
- extract ObjectDetailWrapper.styled.tsx

* Extract ObjectDetailHeader unit tests

* Extract ObjectDetailBody unit tests

* Rename ObjectDetail.unit.spec.tsx to ObjectDetailView.unit.spec.tsx

* Remove the non-null assertion operator

* Use relative import

* Introduce hasFks

* Do not loadFKReferences() when there was no previous zoomed row
- this function will be called in useMount hook's callback

* Use project-wide != null pattern

* Remove loadFKReferences() on mount because it is duplicated later by useEffect

* Extract ObjectRelationships.styled

* Extract ObjectDetailTable.styled

* Fix imports

* Add repro test for #32720

* Sync use Malli

* Test fixes

* One last test fix 

* Update Kondo config

* Test fixes

* Revert accidental whitespace

* Fix whitespace

* Fix more mystery whitespace

* Remove byte order marks

* Test fixes 

* Move MLv2 metadata schemas into a metabase.lib.schema namespace

* Fix schema ref

* Fix schema references

* parameterize collection and dashboard ids in e2e tests

* fix copyPasta errors

* appease the linter

* obey the linter

* First crack at drill-thru, adding quick-filter-drill

* Adding object detail drill

Needs tests for the multi-PK case

* Add distribution-drill and foreign-key-drill

* Add sort drill

* summarize-column and part of automatic-insights

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `drill-thru` implementations for all current drills

* small fixes

* overhaul of available-drill-thru

Fixes several issues with targeting, merges in Mode logic, fixes
pivot-drills

Still TODO: other drills' mode conditions; lots of checking and testing
the various cases of what should be shown; writing tests for that.

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Integrating the basic drill-thru ops, with some debugging hacks

* Fix bad spelling of parameters

* bunch of a progress, :null sentinel, refactor to context

* tests for lots more cases; two missing drills

* adding underlying records drills, not yet working

* fixing up to work with upstream changes

* fixes from Denis' PR

* more progress debugging FE drills and adding tests

* drill tests coming along

* cleanup, test fixes

* display-info-js for recursion

* Add TS typing for displayInfo of drill thrus

* partial changes for underlying records

* hack hack

* splitting out progression logic to a separate file

* First crack at drill-thru, adding quick-filter-drill

* Adding object detail drill

Needs tests for the multi-PK case

* Add distribution-drill and foreign-key-drill

* Add sort drill

* summarize-column and part of automatic-insights

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `drill-thru` implementations for all current drills

* small fixes

* overhaul of available-drill-thru

Fixes several issues with targeting, merges in Mode logic, fixes
pivot-drills

Still TODO: other drills' mode conditions; lots of checking and testing
the various cases of what should be shown; writing tests for that.

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Integrating the basic drill-thru ops, with some debugging hacks

* Fix bad spelling of parameters

* bunch of a progress, :null sentinel, refactor to context

* tests for lots more cases; two missing drills

* adding underlying records drills, not yet working

* fixing up to work with upstream changes

* fixes from Denis' PR

* more progress debugging FE drills and adding tests

* drill tests coming along

* cleanup, test fixes

* display-info-js for recursion

* Add TS typing for displayInfo of drill thrus

* partial changes for underlying records

* hack hack

* splitting out progression logic to a separate file

* Lint and test cleanup

* MLv2 pivot reorganization part 1

* MLv2 pivots reorganization

* Revert some unneeded changes

* Don't enable the histogram test

* Fix lint errors and add a drill-thru-method implementation for sort

* `drill-thru` should support > 3 args

* Implement timeseries zoom drill thru

* Fix circular refs

* Fix `mu/defmethod` for Cljs

* Fix reader error in commented-out code 



* appease Eastwood

* PR feedback

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

[ci skip]

* PoC: Download previously saved uberjar for E2E workflow

* Separate `e2e_specs` and `e2e_all` file path triggers

* Fix file path result oputput

* Refine E2E tests conditional trigger

* Further refine E2E tests conditional trigger

* Download the uberjar directly where it's needed

* Work around inability to dynamically define job ouputs

* First obtain the artifact, then run anything else

* Map job outputs

https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs

* Apply changes to visual regression tests

* Final logic cleanup

* Work around the API rate limit

* Don't decode SSO redirect URLs before extracting host

* clean ns

* re-add kondo ignore

* address feedback

* Remove external op on the FE

* Clean commit

* Extract ObjectDetailBody to a separate file

* Extract ObjectDetailHeader to a separate file

* Extract ObjectDetailView to a separate file
- extract ObjectDetailWrapper.styled.tsx

* Extract ObjectDetailHeader unit tests

* Extract ObjectDetailBody unit tests

* Rename ObjectDetail.unit.spec.tsx to ObjectDetailView.unit.spec.tsx

* Modify GET /api/user endpoint to only work for admins/group managers

* Restrict group managers to only see users of groups they manage

* Adjust segmented user tests and dedupe when user in multiple groups

* Revert a small change that broke other tests. Change wasn't strictly needed anyway.

* Properly enable premium feature :advanced-permissions in group-manager test

* Fix sandbox user list tests. Count query needs distinct now bc of how group manager user list is returned

* Add back the filter I forgot to include in the count query

* Unused require due to a change in one of the tests

* Fix group-manager permissions logic on the user endpoint

* Update src/metabase/api/user.clj

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* Changed endpoint's logic and impl slightly

 - cleaner group-ids-for-manager query
 - use existing validation/check-manager-of-group
 - simplify group-id-clause since at that point we know that the user is admin or group manager

Added a test case to ensure group-manager still only gets their groups users when group_id is not supplied

These scenarious should be accounted for in this setup now

;; ADMIN GM group_id  GM of group_id OUTCOME
;; t     t    t            t         can see users from that group
;; t     t    t            f         can see users from that group (bc admin can see)
;; t     f    t            f         can see users from that group (bc admin can see)
;; t     t    f            f         can see all (admin can see all)
;; f     t    f            f         can see users from all groups they manage
;; f     t    t            t         can see users from that group
;; f     t    t            f         cannot see (bc non admin, GM of different group)

* Alter tests to correctly check 403 in OSS setting

* Modify test to use generic verified by text since User endpoint is now more restricted.

* Strange pulse e2e test. Intercept the user api call to circumvent for now

* request recipients through /api/user/recipients instead of /api/user

* I don't even know anymore. Why did this change now?

Seems that there's a collection ID where before there was a null,
which caused redirect to collection/root

---------

Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Jesse Devaney <22608765+JesseSDevaney@users.noreply.github.com>

Migrate QP/drivers code to use MLv2 metadata directly; make another 100+ tests parallel and shave 12 seconds off test suite (#33429)

* QP MLv2 Metadata Provider follow-ons

* Update lots of stuff to use MLv2 metadata instead of legacy metadata

* Fix lint warnings

* B I G  cleanup

* Everything is neat

* Mention new test pattern in driver dev changelog

* Appease Cljs by renaming a bunch of namespaces

* Move more stuff over

* Fix kondo errors

* Fix even more things

* Test fixes

* Fix JS export

* Test fixes 

* Fix final MongoDB test

* Bump simple changes
* Account for changed exception message
* Add buddy-sign license
* Use maven coordinates to select tools.build version

* alternatives

* remove links

* remove mentions of eb

* remove eb from rds page

... you can reference when using native questions

Fixes #33490.

* First crack at drill-thru, adding quick-filter-drill

* Adding object detail drill

Needs tests for the multi-PK case

* Add distribution-drill and foreign-key-drill

* Add sort drill

* summarize-column and part of automatic-insights

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `drill-thru` implementations for all current drills

* small fixes

* overhaul of available-drill-thru

Fixes several issues with targeting, merges in Mode logic, fixes
pivot-drills

Still TODO: other drills' mode conditions; lots of checking and testing
the various cases of what should be shown; writing tests for that.

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Integrating the basic drill-thru ops, with some debugging hacks

* Fix bad spelling of parameters

* bunch of a progress, :null sentinel, refactor to context

* tests for lots more cases; two missing drills

* adding underlying records drills, not yet working

* fixing up to work with upstream changes

* fixes from Denis' PR

* more progress debugging FE drills and adding tests

* drill tests coming along

* cleanup, test fixes

* display-info-js for recursion

* Add TS typing for displayInfo of drill thrus

* partial changes for underlying records

* hack hack

* splitting out progression logic to a separate file

* First crack at drill-thru, adding quick-filter-drill

* Adding object detail drill

Needs tests for the multi-PK case

* Add distribution-drill and foreign-key-drill

* Add sort drill

* summarize-column and part of automatic-insights

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `drill-thru` implementations for all current drills

* small fixes

* overhaul of available-drill-thru

Fixes several issues with targeting, merges in Mode logic, fixes
pivot-drills

Still TODO: other drills' mode conditions; lots of checking and testing
the various cases of what should be shown; writing tests for that.

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Add `display-info` for drill-thrus and the special pivot functions

Clean up and should be ready for FE first crack

* Integrating the basic drill-thru ops, with some debugging hacks

* Fix bad spelling of parameters

* bunch of a progress, :null sentinel, refactor to context

* tests for lots more cases; two missing drills

* adding underlying records drills, not yet working

* fixing up to work with upstream changes

* fixes from Denis' PR

* more progress debugging FE drills and adding tests

* drill tests coming along

* cleanup, test fixes

* display-info-js for recursion

* Add TS typing for displayInfo of drill thrus

* partial changes for underlying records

* hack hack

* splitting out progression logic to a separate file

* Lint and test cleanup

* MLv2 pivot reorganization part 1

* MLv2 pivots reorganization

* Revert some unneeded changes

* Don't enable the histogram test

* Fix lint errors and add a drill-thru-method implementation for sort

* `drill-thru` should support > 3 args

* Fix `mu/defmethod` for Cljs

* Fix reader error in commented-out code 



* appease Eastwood

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Denis Berezin <denis.berezin@metabase.com>

* quote impersonation roles for postgres conn impersonation

* also quote snowflake roles

Fixes #21502.

* Fix flaky test
* Fix all occurrences of the typo replaced-indecies

* refactor use-list-select hook

- toggleAll(items) felt unintuitive as some items could be selected while simultaneously unselecting others
  - switched to selectOnlyTheseItems(items) to be more explicit on what the function is doing
- updated consumers of useListSelect to use the new API

* convert to typescript

* improve types

* add desired behavior verification for useListSelect

* throw exception if impersonated db doesn't have default role

* change text in snowflake plugin file

* fix bugs and add tests

* FE changes

* inline enabled? call

* add new driver feature connection-impersonation-requires-role

* use variant=brand

* break out impersonation-enabled-for-db? into a defenterprise helper function

* MLv2: add `findColumnForLegacyRef` and `findFilterableColumnForLegacyRef`

* Fix JS tests

* Fix typo

* Remove stray comment

* obey hidden widget settings in ColumnSettings widgets

* always show unambiguous sample dates