* Drop old and not used  endpoint

* fixup! Drop old and not used  endpoint

* fixup! fixup! Drop old and not used  endpoint

Sanitize SVG before parsing with Batik to prevent render failures when invalid characters are in the SVG content (#44516)

* Add Temporal Units List to Dashboard Parameters Schema

Closes: #44361

This adds an entry to the Dashboard Parameter schema so that we can validate
the list of temporal_units that might be passed from the frontend when adding
or updating the `Unit of Time` type parameters.

* Sanitize the SVG before parsing it with Batik

WIP

Fixes: #43677

There are characters that are invalid according to the Batik XML Parser. This sanitizes the svg string to strip out
some of those characters so that the render can continue.

* Use the xml 1.0 spec allowed unicode chars list for the regex

* Remove accidentally merged code

* still miseed a line

* fix: dont always optimize between expressions

Fixes #42291

The frontend produces expressions like

```
[:between
 [+ [:field ... {:temporal-bucket :day] [:interval 2 :day]]
 [:relative-datetime -1 :week]
 [:relative-datetime 0 :week]]
```

This should not be optimized because of the mixed `:day` and `:week`
units. However, it was being optimized since the compatible units
weren't being properly picked up by the match.

* Disable suspcicios args eastwood lint, kondo does it better

* Stop autobucket to day when adding a time interval

* perf: disable view log and card used updates during metadata fetches

* Disable writes for view_count and last_used_at

* Disable view_count tests

* Disable last-used-at tests

* Disable view_count test

* Fix view count test

* Disable view_count e2e testing

* Fix the operator list for `type/TextLike` PKs

* Fix the bug with using wrong operators

* Fix the bug with using wrong operators

* Update test/metabase/lib/filter_test.cljc

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>

* Test FKs too

* Update test/metabase/lib/filter_test.cljc

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>

---------

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>

This not only does not make sense, but also fails for models with aggregations (and thus not having those fields for filtering available).

This PR disables the 'make exports look pivoted' post processing step for csv and xlsx exports.

This is in response to a reported issue that Large Pivot Table downloads are failing entirely: #44556

A more hollistic fix and improvement to the post processing feature will be worked on, so the post processing code is
left untouched in this PR.

The dynamic vars in the csv and xlsx namespaces each default `false` and disable the export from running any post
processing. The var should remain `false` until we've got a plan for these types of exports, and there is no way for
users (either through ENV or UI) to change this, so the feature is effectively disabled.

* Set `useLocalSessionState=true` for MySQL DW connections

* Fix GH issue number

* Test fixes 

* Lint fix 

Make the App DB transaction isolation level READ_COMMITTED for MySQL [MEGA PERFORMANCE BOOST] (#44505)

* Make the App DB transaction isolation level READ_COMMITTED for MySQL/MariaDB [MEGA PERFORMANCE BOOST]

* Add test

Add a new query parameter to `/api/collection/:id/items` and
`/api/collection/root/items`. If `official_collections_first` is passed,
we'll sort official collections first, or not, as requested. If it isn't
passed, we'll default to `false` for the trash collection, or `true` for
normal collections.

* Update e2e test

* Update e2e test

* fix: populate param-fields for named fields on public dashboards

* Add name to public hydration

* Add tests

* Add tests

* Fix tests

---------

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

Exclude native query matches in search scoring when the search should exclude native queries (#43982)

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* UserParameterValue transforms wrapped to properly escape string vals

The `json-in` and `json-out` functions used for toucan model transforms do not perform any string escaping logic. This
makes sense as we don't want to make assumptions about the shape of data flowing in/out of the db. But, this did mean
that for the User paramter values table, string values were causing an error to be logged. This doesn't break
anything, as the function will still correctly return the string, but it can clutter up logs.

So, this PR wraps all incoming values in a map and unwraps it for outgoing values.

* Add migration to wrap existing user param values with ::wrapper key

Since the in/out transform for UserParameterValues is updated, we need to migrate any existing values to have this wrapping.

* Simplify the solution to the problem.

Since we're already getting what we need from the json-in/json-out *except* that it's logging a parse error, we can
create a json-out that works the same way (tries to parse and returns the string as-is if it fails) without logging an
error at all.

* take away the arg and keywordize inside the json-out fn

* remove irrelevant comment

Signed-off-by: Adam James <adam.vermeer2@gmail.com>

* Add a little more test coverage for different value types

* faster test thanks to Dan!

* Add a comment about the data being tested

---------

Signed-off-by: Adam James <adam.vermeer2@gmail.com>

* XLSX Pivot Table Downloads have 'Native' Pivot Table

Addresses: #2473

The xlsx downloads work fine for regular tables. However, on Pivot Table downloads, the resulting file isn't pivoted
and is formatted somewhat strangely (it contains additional rows related to totals, has an extra 'pivot-grouping'
column).

Now, the downloaded xlsx file contains 2 sheets:
- the data sheet containing the question's rows unpivoted. If you were constructing a pivot table yourself, you would
probably start with this shape of data.
- the pivot sheet containing a 'native' Pivot table whose reference points to the data on the data sheet, and whose
columns, rows, and values align with the cols, rows, and measures set up in Metabase

* Silly typo!

* Exported Pivot uses correct aggregation (sum, count, stddev, avg, min, max)

* data for pivot exports now also uses the formatting

* Add a test asserting that a pivot table exists in the xlsx file

* add-row! is a method dispatching on sheet class

* remove unnecessary hint

* address review feedback:

 - consolidate the aggregation function key fns into one function in the .xlsx namespace
   this was moved out of postprocessing so that the xlsx specific fns don't litter the post processing namespace at
   this time

 - clean up an unnecessary destructiring inside the native-pivot function
 - don't shadow 'name' inside the body of a fn

* Add tests that confirm zero-col and zero-row situations work

* Consolidate the atoms used to store pivot rows/options

Fixes #43993 for real.

The earlier fix #44182 was needlessly restricting the
"nominal refs" check to fields which were using numeric IDs,
when really any match on nominal refs is valid.

Part of #44298.

Also OpenAPI generation now handles parameter renaming, like in `{c :count}` destructuring.

* Pivot Tables with no Pivot Columns should still download

Fixes #44159

Pivot Tables can still be valid without columns, and therefore the download should respect such a query and be
successfully exported.

In this PR, the postprocessing of pivot results into 'visual pivot' exports for csv takes into account the zero-column
configuration and the download no longer fails.

* add a test to confirm that zero cols and >1 measures works as well

* Handle the case where there are zero pivot-rows.

This is a similar problem to the zero pivot-cols case, just with rows.

The app actually doesn't render these tables correctly, but that's a separate frontend bug

I'd fixed the `can_restore` on non-collections to always be present, but
forgot about fixing it for collections as well.

* Convert drill-through (CLJ) values to JS

* Convert the drill-through `:null` to an actual `null`

Resolves #44232

* Use new function in `filter-drill-details`

* display-info->js should check for nil before seqable?

* fix js object comparison

* Add test for the `drill-value->js` function

* Simplify the test

* Fix the JS object equality in the test

---------

Co-authored-by: Alexander Solovyov <alexander@solovyov.net>

* Change run to make-run in process-query-for-card

Fixes #44160

There were two levers where a card could change how it was run.

`qp` this is basically `qp/process-query` with some optional userland
flags and constraints marked on the passed in query.
In the case of pivot tables in the
`/pivot/:card-id/query` endpoints it was `qp.pivot/process-query`

'run' this defaults to a streaming response that takes the `qp` param
and `export-format` and passes a query to qp. Most often this is
overriden to not stream the response.

Unfortunately, if `run` is overwritten, then `qp` is thrown away, and
the caller who overrides `run` must determine the correct `qp` to use.

However, when running `process-query-for-dashcard` it is difficult to properly set
`qp` for all of the dashcards without doing card lookups. So [this commit](https://github.com/metabase/metabase/blob/release-x.50.x/src/metabase/query_processor/card.clj#L230-L232) did a check to swap the runner in the presence of a pivot query that calls the **default** runner with a `qp.pivot/run-pivot-query`. The problem is that pulse needs to use a non-default runner, but there's no way for `process-query-for-card` to do both.

This commit changes `run` to `make-run` which raises the abstraction to
a function that produces a runer given a `qp`. This allows
`process-query-for-card` to change the `qp` as needed while keeping the
passed in runner.

There's really three things going on that could be teased apart
further/better.

1. `qp` - this conflates the processor (`qp/process-query` `qp.pivot/run-pivot-query`)
  and modifying query (`qp/userland-query`,
  `qp/userland-query-with-default-constraints` `(update query :info merge info)`)
2. `make-run` - How the above are processed: (streaming/realized)

* Fix tests

* Fix a potential bug in `hydrate-can-restore`

I had a bug here. I wanted to hydrate `:collection` on the items and
then return the original items, but instead I was `dissoc`ing
`:collection` from the items. If the original items passed in already
had `:collection` keys this would break them.

AFAIK this didn't cause any issues, but it could have - so fixing it here.

* Add a test

* Present collection items in the collection

When fetching collection items, we need to fetch the true
`collection_id` in order to figure out things like `can_restore` and
`can_delete`, but set it to the displayed/effective `collection_id` when
we actually send it out.

This fixes a bug where collections had their `collection_id` match their
`id` when returned by this endpoint.

Previously we were just adding this when the item was archived. Instead,
always send a value - if the item is not archived, it can't be restored,
so it will be `false` in this case.

Making this change will make fixing a frontend bug easier, because the
frontend will be able to just trust the `can_restore` value coming in
from the backend.

Fixes #43993.

* little bit of refactoring in sso test code

* use :once instead of :each

* Handle errors when inferring separator of csv

This file

```
"c1","c2"
"a,b,c","d"
```

was failing to upload for a silly reason: when we were checking for
which separator it used, we had a parse error when using a semi-colon
and that blew up the whole pipeline.

* Use cal's helpful test suggestions