* Don't use persisted model tables for segmented users

This actually isn't a bug, but due to very subtle and arbitrary reasons.

For background about why we need to ensure this never happens, we cannot
use persisted models when sandboxing is at play. A simple example is as
follows: make a model on a products table that does not select the
category. Have a sandbox on category such that someone can only see
products of category "Gizmo". the model lacks the category column but we
insert a where clause that still works. When the model is persisted,
there is no category column in the underlying table so sandboxing cannot
possibly work: the data necessary to filter is no longer associated with
the rest of the data in the model.

The fix for this is quite simple: in
`metabase.query-processor.middleware.fetch-source-query` we only splice
in the persisted query if the user is not a segmented user (product name
for sandboxing).

```clojure
(and persisted-info/*allow-persisted-substitution*
     (not (segmented-user?))  ;; <----- new check
     (:active card)
     (:definition card)
     (:query_hash card)
     (= (:query_hash card) (persisted-info/query-hash (:dataset_query card)))
     (= (:definition card) (persisted-info/metadata->definition (:result_metadata card)
                                                                (:table_name card)))
     (= (:state card) "persisted"))
```

Technical details about why this bug did not manifest

When swapping out a card__<id> to a source query, if its a model we will
see if it is persisted, and if so, we will use the native sql to select
from the persisted table. It does this by adding the native sql at a key
called `:persisted-info/native` and a middleware
`#'qp.persistence/substitute-persisted-query` walks the query replacing
the query with the native:

```clojure
;; metabase.query-processor.middleware.persistence

    (mbql.u/replace query
      (x :guard (every-pred map? :persisted-info/native))
      {:native (:persisted-info/native x)})
```

There is also a middleware that walks through the query looking for
tables with gtaps on them and replacing them. By change, the sandboxing
middleware runs immediately before the substitute-persisted middleware!

```clojure
   ;; literally the previous middleware
   (resolve 'ee.sandbox.rows/apply-sandboxing)
   #'qp.persistence/substitute-persisted-query
```

If you swap the order of these two sandboxing is broken. As is, it
"works" but not by design, just by happenstance. The sandboxing
middleware just did not know that the `:persisted-info/native` key meant
that a native query was to be substituted. In the reverse order, the
native query is already substituted and there is no change for the
sandboxing to occur.

The obvious fix is to ensure that we never even attempt to use the
persisted tables and that is what this PR does.

* Eastwood doesn't like shadowing like this

* Rearrange check order for tests

`segmented-user?` throws if there is no bound user. A test in
`fetch-source-query-test` was failing because there was no user bound,
but it wasn't attempting to swap out a persisted table, it just didn't
expect to need a user.

Moving it lower lets it short circuit on other bits that are bound to
fail (definition, query_hash, etc) requiring persistence before we check
for a bound user

* first pass at LDAP setup improvements

* fix lint errors

* fix LDAP api tests

* WIP test for new setting setter

* fix setting test

* set ldap-enabled to true in ldap server macro

* try to fix java11 tests

* Update src/metabase/integrations/ldap.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* add transaction

* remove ldap-ever-enabled? setting and revert some of the logic that is no longer necessary

* set ldap-enabled via the ldap api and add tests

* fix tests and lint

* fix error on settings save

* fix cypress test

* actually fix cypress

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Cards can depend on other Cards as their `:source-table`, but the code
to extract `serdes-dependencies` from the MBQL query did not
capture that case.

Many entities have `creator_id` and similar fields. `User`s are not
serialized. Foreign keys to users are serialized as email addresses.

During deserialization in a different instance (eg. a local dev instance
importing a dump from a prod instance) many such users may not exist.
This change creates new `User` entities on the fly with empty names,
generated passwords, and the email set.

The YAML file names have an optional `:label` portion that becomes the
latter part of the filename. Reconstructed paths (eg. from
`serdes-dependencies`) don't have those labels.

This change makes the YAML ingestion code able to find a file with a
human-readable label even if the request didn't include it.
No ambiguity results because the file names are always based on the
unique serdes `:id`, usually an `entity_id`.

- Limit the scanning of directories and files to those named after
  models; don't try to ingest `.git`, `README.md`, etc.
- `table_id` and `collection_id` are optional on Cards
- Deserialization was not resolving some deeply nested `:field`s inside MBQL
  queries.

Pass `--collections 123,456,789` to dump only these collections and
their transitive `serdes-descendants`.

* enforce import order

* reorder all imports

* more import updates

* Rename DashboardCard `sizeX` and `sizeY` to `size_x` and `size_y` (#16344)

* Fix migration

* Fix MySQL/MariaDB

This is a simple expedient: convert our entities to a `(sorted-map)` before
passing them to the YAML writer!
This gives a consistent, platform-agnostic order between dumps.
(It might have already been fine, since it's based on Clojure's `hash`?)

* Add `serdes-descendants` for "containment" to serialize a subtree

This allows naming eg. a Collection and will recursively serialize: all
dashboards, cards and dashcards it contains directly, plus all child
collections and everything they contain.

Currently this words on Collection, Dashboard, DashboardCard, and Card.

* lint

* Switch to plural `extract-subtrees` with a list of `targets`

* use login-attributes when hash advanced field-values if sandbox use native query

* add a e2e test

* Add repro for #24966

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* minor comment fixes

* fix error

* unskip cypress

* remove debugging code

* retrigger ci

* throw an exception when a recursion limit is hit

* [Toucan 2 prep] Don't invoke Toucan models as functions

* Some fixes

* Test fixes

* Test fix

* [Toucan 2 prep] Don't call `type` or `class` on Toucan models

* Test fixes

* More test fixes 

* Replace perms protocol with multimethods; derive models from perms policy keywords

* Test fixes 

* Appease Eastwood

* Fix errors now that App has been merged in

* Empty commit to trigger CI

* [Toucan 2 prep] Don't invoke Toucan models as functions

* Some fixes

* Test fixes

* Test fix

* [Toucan 2 prep] Don't call `type` or `class` on Toucan models

* Test fixes

* More test fixes 

* [Toucan 2 prep] Don't invoke Toucan models as functions

* Some fixes

* Test fixes

FieldValues can be rebuilt by the sync process, but they are portable
and including them aids in the plans for content moderation via git.

* Separate setting the timeout cookie's expires attribute from the session cookie's max-age

* Add back public-settings/session-cookies tests and add docstrings

* Fix setting cookies with full-app-embedding

* Add docstring

* Change logout to always delete the session

* Remove extra code

* Change session-cookie-name from multimethod to simple case expression

* Refactor: move use-permanent-cookies? closer to usage

* Fix FE unit test

* fix incorrectly hash a fieldvalues for field that is not the field we use to define sandbox rule

* appease clj-kondo

* make the doc clearer

* remove the _id destructring

Revert "Fix #23689: Sandboxed group managers can't see other users in the People tab (#23825)" (#24760)

* Fix some small things

* Add Kondo to deps.edn to be able to debug custom hooks from REPL

* Fix macroexpansion hook for with-temp* without values

* Test config (WIP)

* More misc fixes

* Disable :inline-def for tests

* More misc fixes

* Fix $ids and mbql-query kondo hooks.

* Fix with-temporary-setting-values with namespaced symbols

* More misc fixes

* Fix the rest of the easy ones

* Fix hook for mt/dataset

* Horrible hack to work around https://github.com/clj-kondo/clj-kondo/issues/1773 . Custom linter for mbql-query macro

* Fix places calling mbql-query with a keyword table name

* Fix the last few errors in test/

* Fix errors in enterprise/test and shared/test

* Fix driver test errors

* Enable linters on CI

* Enable unresolved-namespace linter for tests

* Appease the namespace linter again

* Test fixes

* Enable unused-binding linter for test/ => 293 warnings

* 259 warnings

* 234 warnings

* => 114 warnings

* Fix the rest of the unused binding warnings in test/

* Fix unused binding errors in enterprise/backend/test

* Fix unused binding lint errors in driver tests

* Test fix 

* Assure Kondo that something is in fact used

* Fix some small things

* Add Kondo to deps.edn to be able to debug custom hooks from REPL

* Fix macroexpansion hook for with-temp* without values

* Test config (WIP)

* More misc fixes

* Disable :inline-def for tests

* More misc fixes

* Fix $ids and mbql-query kondo hooks.

* Fix with-temporary-setting-values with namespaced symbols

* More misc fixes

* Fix the rest of the easy ones

* Fix hook for mt/dataset

* Horrible hack to work around https://github.com/clj-kondo/clj-kondo/issues/1773 . Custom linter for mbql-query macro

* Fix places calling mbql-query with a keyword table name

* Fix the last few errors in test/

* Fix errors in enterprise/test and shared/test

* Fix driver test errors

* Enable linters on CI

* Enable unresolved-namespace linter for tests

* Appease the namespace linter again

* Test fixes

* Move writeback migrations to 45

* Empty commit to trigger GitHub Actions

* [Actions] Simplify emitter schema model

emitter_action was dropped since emitters just have a singular action
and the join table was unecessary.

emitter_action.action_id columns moved onto emitter table.

Dropped CardEmitter and DashboardEmitter pre-insert, pre-update,
pre-delete since they were used in tests only and normal operation
would see the emitter inserted first.

Since previous code may have 'orphaned' emitters without an action, we
delete emitters without action to be safe.

* Handle flakiness with geojson java.net.UnknownHostException errors (#24523)

* Handle flakiness with geojson java.net.UnknownHostException errors

In CI seems like we are getting errant errors:

```clojure
geojson.clj:62
It validates URLs and files appropriately
http://0xc0000200
expected: (valid? geojson)
  actual: #error {
 :cause "Invalid IP address literal: 0xc0000200"
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "Invalid GeoJSON file location: must either start with http:// or https:// or be a relative path to a file on the classpath. URLs referring to hosts that supply internal hosting metadata are prohibited."
   :data {:status-code 400, :url "http://0xc0000200"}
   :at [metabase.api.geojson$valid_url_QMARK_ invokeStatic "geojson.clj" 62]}
  {:type java.net.UnknownHostException
   :message "0xc0000200"
   :at [java.net.InetAddress getAllByName "InetAddress.java" 1340]}
  {:type java.lang.IllegalArgumentException
   :message "Invalid IP address literal: 0xc0000200"
   :at [sun.net.util.IPAddressUtil validateNumericFormatV4 "IPAddressUtil.java" 150]}]
```

Not clear if this change has a hope of fixing it: if it doesn't resolve
once its possible it is cached somewhere in the network stack, or it
won't resolve if you ask again.

But gonna give it a shot.

Set the property `"networkaddress.cache.negative.ttl"` to `"0"`

> networkaddress.cache.negative.ttl (default: 10)
>    Indicates the caching policy for un-successful name lookups from the name service. The value is specified as an integer to indicate the number of seconds to cache the failure for un-successful lookups.

>    A value of 0 indicates "never cache". A value of -1 indicates "cache forever".

From
https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/net/InetAddress.html


in the hopes that we can try multiple times. Restores the original value
after the test completes so we don't inadvertently change behavior
elsewhere.

If we get an error of java.net.UnknownHostException we try again if we
have attempts remaining. If we get a boolean it means the ip resolution
worked so we can rely on the response (checking if it resolves locally
or not)

* add a delay

* comment out test

Co-authored-by: Cam Saul <github@camsaul.com>
Co-authored-by: dpsutton <dan@dpsutton.com>

* fix List.tsx in master merge

* fix accidental drag and drop

These fields hold JSON, which can contain field IDs in a few places.

Particularly nasty is the `:column_settings` subfield, which is a map
whose *keys* are JSON strings with field IDs.

This PR handles the other JSON-encoded MBQL snippets I was able to find.

These snippets contain `Table` and `Field` IDs, and so are not portable. These
fields are expanded during serialization and the IDs replaced with portable
references, then converted back in deserialization.

Note that the referenced field must already be loaded before it has a valid ID.
`serdes-dependencies` defines this order, therefore each entity depends on those
tables and fields referenced in its MBQL fragments.

The complete set of fields I found to convert:

- `Metric.definition`
- `Segment.definition`
- `DashboardCard.parameter_mappings`
- `Card.parameter_mappings`

These snippets contain raw `Database`, `Table`, and `Field` IDs, and so are not portable. This PR proves out the concept for `Card.dataset_query`, paving the way for the others in a follow-up PR.

* exception for none and block

* add howon's test

* open urls with the site-url origin in the same tab

* revoew

* Remove the `Dependency` model code

* Appease the namespace linter again.

* smarter hashing for advanced fieldvalues

* saving some db calls

* updates some docs and remove an unecessary macro

* Add a check to PUT /user/:id to disallow name edits if an SSO user

* Clean up After SAML SSO tests

The `:sso_source` key is set for the Rasta user in some SAML tests, but is expeted to be nil in subsequent tests, so
we clean up in the SAML test ns.

* Add a test to ensure SSO user names can't be changed via API

* Missed a change I had made while adjusting tests

* valid-name-update? take 1 name, to allow better error messages

Let's the user know that first or last name is the cause of a problem, rather than just 'names'.

* Remove unneeded thread macro

* Use partial=

* slight change to local fn for reusability

* Serdes v2: Add `--v2 true` flag to the `dump` and `load` commands

This is the last piece for end-to-end serialization and deserialization.
The results should be similar to v1, though probably not identical.

`PulseChannelRecipients` are a `:recipients` field on `PulseChannels`.

This requires some careful handling when inserting or updating a
`PulseChannel` to upsert the recipients.