* mark a defsetting as being :sensitive?

* add test

* Wrap non-latin characters in a span specifying working font

Fixes: #38753

CSSBox seems to have a bug where font fallback doesn't work properly. This is noticeable when a font does not contain
glyphs that are present in the string being rendered. For example, Lato does not have many international characters,
so the rendered version of tables (that show up in Slack messages) will not render properly, making the card
unreadable.

Since this appears to be a downstream bug, I've opted to work around this limitation by wrapping any non-latin
characters in a <span> that specifies the font family to be sans-serif, which should contain the glyphs to properly
render.

This leaves Lato in place for other characters.

For now, I figured it's worth trying this solution over using Noto for 2 reasons:
- we can keep Lato, which has been the decided font style for the app for some time (this keeps things consistent
where possible)
- the Noto font containing all glyphs is quite a large font (>20mb, I think) and it would be nice to avoid bundling
that if we can.

* stub installed fonts test

* typo

* Do wrapping, but now per-string, and in Clojure data not html string

I've decided that a reasonable solution is to still wrap strings containing non-lato characters. But it's not done
with str/replace to the html string but rather in a postwalk over the Hiccup data prior to rendering.

Seems like a decent compromise of issues without patching CSSBox or fixing upstream (may be good to do, but will take
longer to get a fix in).

* add test checking that glyphs render correctly

* Add a test that directly checks the wrapping fn

* Change the string to keep the linter quiet

* Change how we check if string can be rendered to faster method, new Lato Font

With Sanya's help, the way I check if a given string is renderable with Lato is now faster.

Also use the full Lato font, not just the 'latin' lato so we can cover more chars

* change lato so that it loads the fn even in a fresh test run

* Rather support old files, than duplicate columns

* Cope with driver specific failure modes

* Restore bytes comment

* Fix order by field refs

* Update test to handle all drivers

* Untangle fix-order-by-field-refs

* Drop old and not used  endpoint

* fixup! Drop old and not used  endpoint

* fixup! fixup! Drop old and not used  endpoint

Sanitize SVG before parsing with Batik to prevent render failures when invalid characters are in the SVG content (#44516)

* Add Temporal Units List to Dashboard Parameters Schema

Closes: #44361

This adds an entry to the Dashboard Parameter schema so that we can validate
the list of temporal_units that might be passed from the frontend when adding
or updating the `Unit of Time` type parameters.

* Sanitize the SVG before parsing it with Batik

WIP

Fixes: #43677

There are characters that are invalid according to the Batik XML Parser. This sanitizes the svg string to strip out
some of those characters so that the render can continue.

* Use the xml 1.0 spec allowed unicode chars list for the regex

* Remove accidentally merged code

* still miseed a line

* fix: dont always optimize between expressions

Fixes #42291

The frontend produces expressions like

```
[:between
 [+ [:field ... {:temporal-bucket :day] [:interval 2 :day]]
 [:relative-datetime -1 :week]
 [:relative-datetime 0 :week]]
```

This should not be optimized because of the mixed `:day` and `:week`
units. However, it was being optimized since the compatible units
weren't being properly picked up by the match.

* Disable suspcicios args eastwood lint, kondo does it better

* Stop autobucket to day when adding a time interval

* perf: disable view log and card used updates during metadata fetches

* Disable writes for view_count and last_used_at

* Disable view_count tests

* Disable last-used-at tests

* Disable view_count test

* Fix view count test

* Disable view_count e2e testing

* Fix the operator list for `type/TextLike` PKs

* Fix the bug with using wrong operators

* Fix the bug with using wrong operators

* Update test/metabase/lib/filter_test.cljc

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>

* Test FKs too

* Update test/metabase/lib/filter_test.cljc

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>

---------

Co-authored-by: lbrdnk <lbrdnk@users.noreply.github.com>

This commit adds `$size` operator wrapping to `$addToSet`. That enables use of `:distinct`
clause in expressions on Mongo.

This not only does not make sense, but also fails for models with aggregations (and thus not having those fields for filtering available).

This PR disables the 'make exports look pivoted' post processing step for csv and xlsx exports.

This is in response to a reported issue that Large Pivot Table downloads are failing entirely: #44556

A more hollistic fix and improvement to the post processing feature will be worked on, so the post processing code is
left untouched in this PR.

The dynamic vars in the csv and xlsx namespaces each default `false` and disable the export from running any post
processing. The var should remain `false` until we've got a plan for these types of exports, and there is no way for
users (either through ENV or UI) to change this, so the feature is effectively disabled.

* Set `useLocalSessionState=true` for MySQL DW connections

* Fix GH issue number

* Test fixes 

* Lint fix 

* Don't run sandboxing rollback test on mysql

* Update test/metabase/db/schema_migrations_test.clj

Make the App DB transaction isolation level READ_COMMITTED for MySQL [MEGA PERFORMANCE BOOST] (#44505)

* Make the App DB transaction isolation level READ_COMMITTED for MySQL/MariaDB [MEGA PERFORMANCE BOOST]

* Add test

Add a new query parameter to `/api/collection/:id/items` and
`/api/collection/root/items`. If `official_collections_first` is passed,
we'll sort official collections first, or not, as requested. If it isn't
passed, we'll default to `false` for the trash collection, or `true` for
normal collections.

* Update e2e test

* Update e2e test

* fix: populate param-fields for named fields on public dashboards

* Add name to public hydration

* Add tests

* Add tests

* Fix tests

---------

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>

Exclude native query matches in search scoring when the search should exclude native queries (#43982)

Co-authored-by: Chris Truter <crisptrutski@users.noreply.github.com>

* UserParameterValue transforms wrapped to properly escape string vals

The `json-in` and `json-out` functions used for toucan model transforms do not perform any string escaping logic. This
makes sense as we don't want to make assumptions about the shape of data flowing in/out of the db. But, this did mean
that for the User paramter values table, string values were causing an error to be logged. This doesn't break
anything, as the function will still correctly return the string, but it can clutter up logs.

So, this PR wraps all incoming values in a map and unwraps it for outgoing values.

* Add migration to wrap existing user param values with ::wrapper key

Since the in/out transform for UserParameterValues is updated, we need to migrate any existing values to have this wrapping.

* Simplify the solution to the problem.

Since we're already getting what we need from the json-in/json-out *except* that it's logging a parse error, we can
create a json-out that works the same way (tries to parse and returns the string as-is if it fails) without logging an
error at all.

* take away the arg and keywordize inside the json-out fn

* remove irrelevant comment

Signed-off-by: Adam James <adam.vermeer2@gmail.com>

* Add a little more test coverage for different value types

* faster test thanks to Dan!

* Add a comment about the data being tested

---------

Signed-off-by: Adam James <adam.vermeer2@gmail.com>

* XLSX Pivot Table Downloads have 'Native' Pivot Table

Addresses: #2473

The xlsx downloads work fine for regular tables. However, on Pivot Table downloads, the resulting file isn't pivoted
and is formatted somewhat strangely (it contains additional rows related to totals, has an extra 'pivot-grouping'
column).

Now, the downloaded xlsx file contains 2 sheets:
- the data sheet containing the question's rows unpivoted. If you were constructing a pivot table yourself, you would
probably start with this shape of data.
- the pivot sheet containing a 'native' Pivot table whose reference points to the data on the data sheet, and whose
columns, rows, and values align with the cols, rows, and measures set up in Metabase

* Silly typo!

* Exported Pivot uses correct aggregation (sum, count, stddev, avg, min, max)

* data for pivot exports now also uses the formatting

* Add a test asserting that a pivot table exists in the xlsx file

* add-row! is a method dispatching on sheet class

* remove unnecessary hint

* address review feedback:

 - consolidate the aggregation function key fns into one function in the .xlsx namespace
   this was moved out of postprocessing so that the xlsx specific fns don't litter the post processing namespace at
   this time

 - clean up an unnecessary destructiring inside the native-pivot function
 - don't shadow 'name' inside the body of a fn

* Add tests that confirm zero-col and zero-row situations work

* Consolidate the atoms used to store pivot rows/options

Fixes #43993 for real.

The earlier fix #44182 was needlessly restricting the
"nominal refs" check to fields which were using numeric IDs,
when really any match on nominal refs is valid.

Part of #44298.