* Enable not emptyable arguments for is-empty and not-empty clauses

* Add review remarks

* Shorten dataset name so it is usable with some drivers

* Disable test for Athena

* Handle possibly missing base type

* Update comment

* Missing base type field is treated as not emptyable

* Update desugar middleware

Now middleware adds base type to fields with integer id that are missing that first. This ensures correct function of is-empty expansion.

* Update desugar middleware

Cleanup base types necessary for desugaring is-empty

* Update desugar middleware cleanup

* Update test

* Adjust tests

* Remove now redundant e2e test

* Update comment

* Wow

* Test fix 

* Fixes

* Actions should use strings for column names (fix :update-row and :create-row normalization)

* MLv2 schema should check against keys for the other query type

* Ok, have I fixed things?

* More fixes 

* Fix indentation

* Another round of test fixes. 

* Hopefully the last few test fixes 

* We need to test normalization for queries that have keyword keys as well.

* Fix Cljs i18n namespaces

* Sort namespaces

* Only test against H2

* Rename `metabase.mbql` to `metabase.legacy-mbql`

* Fix Kondo warnings

* Test fixes 

* Register MBQL clause schemas and test fixes 

* Test fixes and PR feedback

* Test fix

* Remove the normalization tests

* Test fixes 

* Fix kondo

* Fix import

* Another fix 

* Merge

* FIXES

* Add another missing REQUIRE

* decrypt and read values from airgap token

- Adds notion of "AirgapToken"
- Adds max-users and company optional fields to `TokenStatus`
- Adds branch to `fetch-token-status*` for handling airgapped tokens

* Add support for handling airgap tokens

- validate airgap token valid-thru date

* enforce user creation limits using airgap token

* fix airgap token reads + fill in the token data

* in oss mode don't try to read the airgap_ token

* use airgap-token? helper

* tighten the user-creation logic

- make the check correct
- account for archived users

Co-authored-by: John Swanson <john.swanson@metabase.com>

* check airgap user count

setting premium-embedding-token
app startup

* move decryption code into ee namespace

* add some tests for token decryption

* fix tests and add tests

* move ee features into ee tests

* add typehint + warn on reflection

* add a check for missing public key resource

* respond to review

- add docstring to AirgapToken
- remove outdated comment
- fix off-by-1 error

* valid-now? takes a TokenStatus instead of a :map

* revert usage of mt/with-temporary-setting-values

---------

Co-authored-by: John Swanson <john.swanson@metabase.com>

* wip

* SLO works with auth slo handler route

* move slo handling endpoint to /auth/sso/handle_slo

* fix slo redirect url

* SLO works, and the sso-handle-slo for saml is where it belongs

- a ton of cleanup

* fix api/session namespace + add docstrings

* cleaning up logout action

* add slo logout test along with slo response xml

* whitespace + linter

* add docstring

* update exclusions in deps.edn

* un-require metabase-enterprise ns from oss ns

* add docs for how to setup SLO to metabase docs

* docs: clarify that setting up SLO is optional

* move slo logout endpoint into ee code

- removes sso-info defenterprise since it is no longer needed

* use current version of saml20-clj

---------

Co-authored-by: Nick Fitzpatrick <nickfitz.582@gmail.com>

* Move sample-dataset.edn into test-data.edn

Replace (mt/dataset sample-dataset with (mt/dataset test-data

Replace more references to sample-dataset with test-data

Fix the format of the combined data

Might revert: remove test that doesn't apply, now that there's 1 DB

Make api.database-test pass, though expected could be wrong

* Update unit tests for combined dataset

* Bump Bigquery version from v3_ to v4_

* Bump v3_test-data to v4_test-data

* Remove wrapping with mt/dataset

Alphabetize fk-mappings

* Remove needless whitespace edits

* Fix failed e2e_test by removing reference to sample-dataset

* Fix tests for Mongo, Oracle, Snowflake, and Presto

* Fix unit tests again for Mongo, Presto, and Snowflake

* Will revert: run driver tests on my fork

* Will revert: run driver tests on my draft fork

* Revert "Will revert: run driver tests on my draft fork"

This reverts commit 078c8af1.

* Revert "Will revert: run driver tests on my fork"

This reverts commit e3e2922f.

* Make the postgres driver test pass again by reverting a change

* Apply Case Nelson's patch to fix Athena test

https://github.com/metabase/metabase/pull/36064#issuecomment-1824837705
Props @snoe

* Remove (mt/dataset sample-dataset) from new tests

* Maybe fix Athena driver tests

Revert needless deletions of (mt/dataset test-data
Maybe those deletions caused the
failing Athena driver tests.
But those deletions are out of scope
for this PR either way.

* Remove sample-dataset reference from a test in the master merge

* In card_test.clj, replace sample-dataset with test-data

* Merge in master, resolve conflict in pivot-from-model-test

* Merge in master, resolve conflict in pivot-from-model-test

* Must revert: allow databse creation

To see if it will fix:
https://github.com/metabase/metabase/actions/runs/7120808552/job/19388845720?pr=36064#step:3:441


Athena database creation is disabled: not creating database v2_test_data. Tests will likely fail.

* Revert "Must revert: allow databse creation"

This reverts commit d4c8f129.

* Add back in (mt/dataset where I deleted it

But replace sample-dataset with test-data.
This might be wrong, or maybe it's not needed in
so many places.
But it could show if this is the problem.

* Fix the failed sync test

---------

Co-authored-by: Case Nelson <case@metabase.com>

* wip

* fix caching behavior for audit db in database-id->connection-pool

* test that caching behavior works as advertised

- db->pooled-connection-spec always returns the same pool for audit-db
- audit-db-id is not in the database-id->connection-pool cache

* DISABLE audit db exposure through GET api/database

* Lots of things

- automagically loads instance analytics on app startup
- TODO: logging in after that is bokred

* add some logs / fix comment

* remove internal_analytics resource

* fix linter issues + update logging

* add tests for existing and missing content

- add test replica of content into test-resources

* remove unused require

* delete sample db + personal collection from test-resources

* get instance_analytics resource off the resource path

* point it to the proper IA resource

* un-hotwire the change to show audit DB on GET /databases

* fix test + lookup the right resource

* remove creator_id references from instance_analytics export test data

- get test feedback

* Revert "remove creator_id references from instance_analytics export test data"

This reverts commit 1aa9d4b1dad304ac0c6d88221f81432d78fc88f0.

* fix tests

* lint

* fix tests

* more test fixing

* linter fix

* revert yarn.lock

* Look up the correct instance_analytics resource

* cleanup comment in serialization/cmd.clj

* silence noisy serialization import logs

* remove a bunch of unused serialization files

* rename Audit Database + docstring

* rename audit database everywhere

* Put the database and tables back

* constrain tests to postgres

* ignore ia data in a test

* initial changes

* add saml api

* address changes

* git restore enterprise/backend/test/metabase_enterprise/sso/api/saml_test.clj

* update comment

* update to have /api

* remove dbg

* fix alias

* change deps

* update test

* add test

* update test

* split tests

* add paren

Added `pretty` option to POST api/dataset/native route. Defaults to `true` such that invocations of this endpoint will return pretty-formatted SQL. In the case of previewing the native SQL editor with templating, we pass `false` so that what is typed is what is presented. In all other cases we accept the default.

* Use gvenzl/oracle-xe docker to test Oracle
* Configure SSL connection for Oracle 21.3
* Fix secret handling

Updates embedded H2 driver to v2.1.212. When a local H2 v1 file-based application database is detected, it will be
automatically updated in-place at startup. Migrations and tests have been updated to account for changes in H2
behavior. Sample data and test harness database files in the repo have all been updated to H2 v2 file format.

Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

We have already had support for server authentication based on custom
certificates. This change adds support for authenticating the client
based on custom client key and certificate.

* Adjust JWT and SAML fetch-and-update user to save new attributes

Before this change, JWT/SAML logins would attempt to update attributes, but never considered the first-name or
last-name attributes.

* Attempts to fix tests to prevent pulluting test users with "Unknown"

* No deleting users.

* Unit tests checking that first/last names are updated for SSO users

When an SSO user is first logged in, they might not have first_name and/or last_name keys. This is allowed, but the
names will be "Unknown" in the app-db. Subsequently, a User may log in again with SSO but have fisrt/last name
attributes, which should update the Metabase user data in the app-db.

These unit tests set up such a scenario to check that the :first_name and :last_name keys are indeed updated.

* Adjust Enterprise LDAP to also use SSO-UTILS

Trying to unify the LDAP implementation with JWT/SAML a bit here.

* Lint error

* Reverting LDAP ns changes to get the PR unstuck

This is to keep the ball rolling on SSO fixes. I'll add LDAP as an item in the Epic to address this separately.

* fix ldap requires uid to do group sync

* fix ee test

* fix failed to sync admin group

* address noah's comments and add migration script

* document for run-with-data-migration-index

* update comments

* fix name space

* adding data_migrations tests

* add docg

* make sure we don't remove admin group if sso and ldap are not configured

* fix tests for be-ee

* fix tests for oss

* misc docs updates

* remove data-migration-index

* return some newlines

* is it failling here?

* update data_migration docs

* update data_migration docs

* fix all styling comments

* make migration to run both in oss and enterprise and make sure the tests are accounted for that

* fix failed namespace checks

* Add a comment to the test

* fix per comments

* Update permissions.clj

* tweaking with the  icon

* refactor with-temporary-raw-setting-values

* update comments

* Add extension for cert file

* address Noah's comments

* Enable Postgres driver to use secrets for configuring SSL parameters

Adding secret related properties for SSL options to Postgres driver

Move `conn-props->secret-props-by-name` to secret.clj since it's needed directly there, too

Add `us-east-2-bundle.pem` to `test-resources/certificates` for testing Postgres with SSL connectivity to our RDS instance (and a README.md explaining how it differs from the existing `ssl` directory)

Updating `value->file!` to have better logic for building the error message when the existing file is not found (for better UX from the Database admin page)

Updating frontend to support the `visible-if` value being an array, in which case any value may match the current details value, in order for that field to be visible

Adding secret related properties for SSL options to Postgres driver

Updating CircleCI config.yml to refer to the RDS instance when running Postgres SSL test

Implement server side expansion of transitive visible-if dependency chain

Update shouldShowEngineProvidedField on client to consider multiple key/value pairs in the visible-if map, and only show the field if ALL are true

Adding new test to confirm transitive visible-if resolution, and cycle detection

Add Cypress test for SSL field visibility

`org.apache.sshd` requires `net.i2p.crypto/eddsa` dependency
to work with ED25519 keys.

This worked in 0.40 because `eddsa` was included as a transitive
dependency of another unrelated dependency (`buddy`) but stopped
working in 0.41 because the new version of `buddy` no longer depends
on `eddsa`. Thus, we must explicitly include the dependency.

To prevent this from breaking again, switched one of the test
keys to an ED25519 key.

* Fix some Eastwood failures

* Fix a lot of Eastwood errors now that it runs against test namespaces [ci skip]

* Run Eastwood against test namespaces [WIP]

* Bump Eastwood version to fix some errors in Eastwood itself

* Fix another lint warning

* Fix test failure

* More test fixes 

* Fix all the warnings!

* Test fix 

* Bump Eastwood GH action timeout to 20 minutes (!)

* adapt ldap test helpers to be able to test AD memberOf attribute

* fix ldap search to handle memberOf attributes with a single group

* use one-or-many instead of if statement

* fix tests

* Implement ssh tunnel reconnection

From the connection-with-timezone method of execute, check whether an ssh tunnel that should be open actually is not, and if so, mark the entire pool as invalid (thereby forcing the connection code to rebuild the source and open a new tunnel)

Fixing the create-pool! function so that the relevant ssh tunnel entries are kept (in addition to the :datasource)

Adding test in ssh-test namespace, which will test that the tunnel is reestablished (for now, running with Postgres driver)

* Responding to PR feedback from Dan

* Fixing test by adding AcceptAllForwardingFilter forwardingFilter to the mock password server instance

* Change with-driver to test-driver

* Add ssh tunnel reconnect test that can run against H2

* Implement ssh tunnel reconnection

From the connection-with-timezone method of execute, check whether an ssh tunnel that should be open actually is not, and if so, mark the entire pool as invalid (thereby forcing the connection code to rebuild the source and open a new tunnel)

Fixing the create-pool! function so that the relevant ssh tunnel entries are kept (in addition to the :datasource)

Adding test in ssh-test namespace, which will test that the tunnel is reestablished (for now, running with Postgres driver)

* Responding to PR feedback from Dan

* Fixing test by adding AcceptAllForwardingFilter forwardingFilter to the mock password server instance

* Rebase again to fix merge conflict

* Change test-driver to with-driver in hopes of making CodeCov finally happy

* Adding new multimethod to ssh namespace, called incorporate-ssh-tunnel-details, for accounting for the

Implementing incorporate-ssh-tunnel-details for h2 so update the URI string (:db key) to point to the tunnel entry point

Pulled logic for :sql-jdbc implementation of connection-with-timezone out to a new fn, so it can be called elsewhere

Updating H2 reconnection test imn light of the changes above

Added "!" suffix to name of include-ssh-tunnel to reflect the fact that it does modify global state

* Fixing NPE in incorporate-ssh-tunnel-details implementation for :h2

Moving multimethod declaration for incorporate-ssh-tunnel-details to driver namespace

* Fix :h2 implementation again

* Remove another errant extra line :(

* Rebase onto master

* Inline the private helper fn back into connection-with-timezone

* Remove dead code

* Quick sync

* Try reducing -Xmx for :ci profile a bit to prevent random test failures

* Update quick sync parameter name

* Update docstring for dataset

* Fix sync-database! schema

* Fix schema again

* Test fix 

* Fix noisy logs in tests

* Fix lint error

* Fix Google driver build [ci bigquery]

* Tweak chain filter tests

* Fix i18n tag

* Fix database sync test (maybe)

* Add u/decolorize for latest JUnit output tweak

* Fix 

* Test fix