Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>
Co-authored-by: Sloan Sparger <sloansparger@users.noreply.github.com>
Co-authored-by: Sloan Sparger <sloansparger@gmail.com>

* Fix #39138 again

Fixes #40934

Got unfixed somehow in #40578. seems like some file renames caught it off guard

But the long and short is that its possible for a persisted model to end
up in the state "refreshing" (possible an instance restart during
refreshing). The refreshing job doesn't look for these so they become
effectively invisible.

Since the job to refresh them will only run one at a time cluster wide,
any jobs that are in the "refreshing" state when the refresher begins to
refresh are necessarilly stuck (no one else could currently be
refreshing them). So we can just add them to the queue of models to
refresh.

```clojure
(jobs/defjob ^{org.quartz.DisallowConcurrentExecution true ;; <----
               :doc "Refresh persisted tables job"}
  PersistenceRefresh [job-context]
  (refresh-job-fn! job-context))
```

* Ensure ee/oss pathways are taken in tests

Annoying little footgun here. CI does not run with an ee token, so in
order to ensure that test pathway goes through ee version of a
defenterprise we _must_ use the `mt/with-premium-features
{:cache-granular-controls}`,  but we also want to ensure that it goes
through the oss version.

So two options, a `doseq` on both features (empty set and the feature
that triggers this). But want a test in the enterprise folder as well to
ensure.

The real trickiness comes from running tests at a repl and CLI. My REPL
always has an ee token in it. My command line always lacks that as
well. So want to be explicit about the token features in effect at test
time.

That's why I'm essentially duplicating tests (ee in ee folder, oss in
regular pathway)

New endpoint accepts entity and entity ids in form of `database=1&dashboard=2&question=3`. If you don't supply
`&include=overrides`, then it tries to find configs directly referencing supplied entities and updates their
`invalidated_at`. If you supply `&include=overrides`, all the referenced cards are
updated (`report_card.cache_invalidated_at`).

Cache strategies then select the maximum `invalidated_at` in their logic. EE-only.

resolves #40548

Fixes #40328

[Our documentation](https://www.metabase.com/docs/latest/people-and-groups/managing#group-managers) states that:

> Group managers can:
>
> - View all people in the Admin settings > People tab.

This fixes enforcement to be aligned with the documentation. This behavior makes sense, because as the docs also state, Group Managers should be allowed to *add* people to the group they manage, which they can only do if they can see those people!

Initially, I also removed a faulty test, which was described as:
```
;; Non-segmented users are allowed to ask for a list of all of the users in the Metabase instance. Pulse email lists
;; are an example usage of this. Segmented users should not have that ability. Instead they should only see
;; themselves. This test checks that GET /api/user for a segmented user only returns themselves
```

but actually failed to test this in a relevant way (because it was testing the `GET /api/user` endpoint rather than the `GET /api/user/recipients` endpoint). The test had continued to pass only because the user was the only member of the group they managed.

I initially thought this behavior wasn't desired, but as it turns out, it *is* in fact documented behavior to disallow sandboxed users from seeing any email suggestions. Investigating, I found that a bug was allowing sandboxed users to see all users on the `/api/user/recipients` endpoint if the user-visibility setting was set to `:all`.

Thus, the second commit fixes the bug and re-adds the (fixed) test. A sandboxed user now only sees their own user when hitting `/api/user/recipients`.

* Split from #40146: only the filter changes

* Don't include format-rows changes here.

* Test fixes 

* Test fix 

* Misc test fixes 

* Test fix 

* Tweak SQL Server fix

* Often slo won't be setup, but we should still delete the session

* test that we delete the session even when "/handle_slo" is not called

* cleanup test

* take the cookie from the session, not as a parameter

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* E2E support for CRUD + running pMBQL queries; new MLv2 normalization

* More work on improved normalization

* Update schema references

* Test fixes 

* Move the MBQL parameters schema

* Performance improvements

* Improved pivot code etc.

* Clean namespaces

* Fixes 

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Split pivot stuff off

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Test fix

* Another fix

* Test fix 

* decrypt and read values from airgap token

- Adds notion of "AirgapToken"
- Adds max-users and company optional fields to `TokenStatus`
- Adds branch to `fetch-token-status*` for handling airgapped tokens

* Add support for handling airgap tokens

- validate airgap token valid-thru date

* enforce user creation limits using airgap token

* fix airgap token reads + fill in the token data

* in oss mode don't try to read the airgap_ token

* use airgap-token? helper

* tighten the user-creation logic

- make the check correct
- account for archived users

Co-authored-by: John Swanson <john.swanson@metabase.com>

* check airgap user count

setting premium-embedding-token
app startup

* move decryption code into ee namespace

* add some tests for token decryption

* fix tests and add tests

* move ee features into ee tests

* add typehint + warn on reflection

* add a check for missing public key resource

* respond to review

- add docstring to AirgapToken
- remove outdated comment
- fix off-by-1 error

* valid-now? takes a TokenStatus instead of a :map

* revert usage of mt/with-temporary-setting-values

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>

* Rename :model/Metric to :model/LegacyMetric
* Rename ::lib.schema.metadata/metric tp ::lib.schema.metadata/legacy-metric
* Rename models.metric/Metric to models.metric/LegacyMetric
* Move /api/metric endpoints to /api/legacy-metric

There were two problems here:

- one endpoint was sorting in the database, but not putting 'analytics'
last.

- one endpoint was sorting in the database, but then sorting again in
clojure, which had subtly different behavior for special characters

Fixes #39965

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* SQL Lineage: Create the QueryField model

This links Cards (with native queries) to Fields, showing us which
queries use which Fields.

QueryFields are automatically maintained as part of the Card lifecycle

Cards with stale queries (powered by QueryField info) are exposed via the `card?f=stale` API

Co-authored-by: Tim Macdonald <tim@metabase.com>

* first move to a db api

* mdb.spec into mdb

* mdb.u/isa -> mdb.query/isa

* mdb.u/qualify -> mdb.query/qualify

* last of mbql.u -> mbql.query and mbql.u

* remove last of the mdb.connection/db-type and use from mdb/db-type

* remove the last non-cmd usages of mdb.connection

had some circularity in

models.interface requires mdb/db-type for sql flavored "now"
mdb requires db/setup
which requires db/custom_migrations to ensure they are on cp
custom migration requires json functions in models.interface

so had to do a dynamic require for the db-type

* get db-type from mdb not mdb.env

* invoke `(mdb/app-db)` and not pass a var. lots of things fail

* mdb.setup/migrate! into mdb with potemkin

* remove get-connection. call .getConnection on the app-db

perhaps invites clumsy usage outside of `with-open` if it's a top level
function

* namespace docstrings

* last few and grab db.env/db-file into api ns

* lint cleanup

* addresses PR comments

* Fix card summarize endpoint

* Fix "AI features" setting section visibility

* Add  setting

* Use `ee-ai-features-enabled` setting

* Add test for new LLM defsetting. Add note in settings UI

---------

Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: adam-james <21064735+adam-james-v@users.noreply.github.com>

* Keep fetch_source_query.clj

* Copy fetch_source_query.clj into fetch_source_query_legacy.clj

* Set back fetch_source_query.clj file

* Convert QP to MLv2, part 1

* Keep legacy implementation around for time being

* Port thru convert-to-legacy

* expand-macros is converted to legacy

* Test fixes 

* Test fixes and better keys

* Test fixes

* Test fixes 

* More test fixes 

* Test fix? 

* Wow, maybe I finally fixed everything 

* Last few test fixes 

* PR feedback; lib.walk should support splicing in multiple replacement stages

* Update fetch-source-query to use improved lib.walk

* Have I fixed everything now? 

* Remove stray println

* Another MongoDB fix 

* Remove the changes to expand-macros for now, we can do that in a follow-on PR.

* Test fix

* Test fix

Co-authored-by: bryan <bryan.maass@gmail.com>
Co-authored-by: Bryan Maass <bryan.maass@gmail.com>
Co-authored-by: Jerry Huang <jhuang37050@gmail.com>
Co-authored-by: Noah Moss <noahbmoss@gmail.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: John Swanson <john.swanson@metabase.com>

* No more Schema, only Malli

* Remove the rest of Schema

* Fix Kondo warnings

* Port some decoding code

* More porting

* Almost all done?

* Parsing fixes 

* More fixes

* Fix Kondo errors

* Test & lint fixes 

* Test fix 

* PR feedback