* Log a warning when extraneous parameters are passed into a request

* Massive Metabase logging performance improvement: use configured log levels in logs API endpoint

---------

Co-authored-by: Cam Saul <github@camsaul.com>

insert next-gen permission paths alongside currently active permission paths + classification (#27911)

* implement move, which returns v2 paths

- TODO: insert these into the db

(move v1-path) => [v2 paths]

* cleanup + add some schemas

* generative tests 4 permission path classification

* whitespace lint

* detect data, query, and paths for v2

* calling move on v2 paths is a no-op

* differentiate between v1 and v2 permissions

quickchecking for move, classify-path, and classify-data-path

* fix tests + add idempotency test

* add tests for classification of permission paths

- rename move to ->v2-path
- move some fxns around
- ascii art in test

* making the legos line up

- need to insert both v1 and v2 versions of paths (of course)
- valid-path? has to allow v2 paths to be inserted

* replace mu/with-api-error-message

* linter + code quality fixes

* privatize rx->kind

* remove some changes that should be 4 anotherbranch

* revert ns

* delete v2 permissions in

- they aren't handled by the perm graph parser, so they don't get propagated into "old graph", so the diff between old and new indicates that they need to be rewritten.

* only delete v2 paths for the current group_id

- reorder declarations in models.permissions

* remove extra line break

* Update src/metabase/models/permissions.clj

fix typo

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Switch to the new Hawk test runner

* Fix some stuff

* Fix everything

* Commentz

* Bump init timeout?

* Fix Kondo config

* Remove unused

* Bump test runner version to get `:exclude-tags` support

Replaced, as needed, instances of str/lower|upper-case with the -en version. Also updated rules in our kondo config.

* Implicit action pre-fetch values (#25940)

* [Apps] Implicit action pre-fetch values

WIP To return a parameter-id: value map in order to prefill ui.

* Use jsonstring encoded parameters

* Change parameters exposed by implicit actions

Previously, we would look at a model's source-table to determine which
parameters were valid for implicit actions. So if an implicit action is
on a User model that hides the user.password column, you could still
update or insert the password.

With this commit, if the model doesn't expose a source table column, it
is not a valid parameter. In the example user.password could not be
inserted/updated or fetched for prefilling.

* Fixing tests

* Show data app page ID params in editing mode (#25943)

* Data apps layout updates (#25930)

* play with grid sizes to make apps feel appier [ci skip]

* move app nav to bar inside app

* only hide other header buttons outside of edit mode

* tweak overflow issue

* fix header width on data apps

* add control to see app contents

* set data apps layout constants contextually [ci skip]

* remove hardcoded value [ci skip]

* extract conextual header styles [ci skip]

* set max-width on paramaters for data apps [ci skip]

* move data apps nav deeper, only show if not editing [ci skip]

* add spacing to contents trigger, rm old code [ci skip]

* rm table thing for now [ci skip]

* Fixes for data app layout updates (#25962)

* Reorder import

* Fix type

* Fix missing translation, move out URL

* Remove not used import

* Rework `getSelectedItems` utility for data apps

* Fix selected data app page highlighting

* Don't reload app navbar on page change

* Change nav item links look

* Remove `DataAppPageSidebarLink`

* Turn nav into HTML list

* Use styled components

* Fix edit button covered by visualizations

* Fix opening homepage

* Remove redundant prop

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* First take at parameterized data app page titles (#25938)

* Bring `DataAppContext` back

* Pass `onFocus` and `onBlur` to `EditableText`

* Export `EditableTextProps`

* Add basic `DataAppPageTitle` component

* Use `DataAppPageTitle`

* Fix component name

* Add primitive autocompletion to page title input

* Add `title_template` to `DataAppNavItem` type

* Tweak value management in `DataAppPageTitle`

* Handle `null` values in titles more gracefully

* Don't show suggestions without detail cards

* Don't add whitespace when pasting a token

* Don't update app's collection all the time

* Add data app related selectors to dashboard selectors

* Add redux state to keep title template changes

* Update page title template on page save

* Wire up `DataAppPageTitle` with dashboard code

* Fix unit tests

* Reset state on exiting editing mode

* Add back button to template data picker

* Fix test that didn't include pk on insert

* Use the actual pk value

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>

* Revert "Remove data app permission code (#26016)"

This reverts commit b8c92cf1.

* Remove unused imports

* Don't require the table_id on result_metadata elements (#26073)

In some cases (which ones still has to be clarified) the result metadata doesn't
contain the table_id and so no parameters are provided. In fact the table ID is
probably not needed anyway because fields are already table specific.

* Support filtering cards by used model (#26088)

This change adds a new filter option for the GET /card endpoint.

* Support filtering apps using a given model (#26124)

* Handle type errors more cleanly (#26120)

* [Apps] Handle type errors more cleanly

Adds a number of tests for different db datatypes, these can be expanded
on as we solve.

Catches most exceptions during execution and logs then throws a generic
message that leaks less of the details.

* Make sure we order-by expected results

* Fix tests, allow purposeful exception through

* Accept model-ids for scaffolding apps (#26139)

* [Apps] Accept model-ids for scaffolding apps

WIP - accept model-ids but there's still some issues, might be able to
start building UI from this point though.

* Maybe working model

* Work with card__n table-ids passed in. Fix problems with detail page

* Dynamically add actions that exist on models

* Update tests for existing custom actions

* Can't def tru

* Address review comments

* Add required to action parameters (#26296)

Looks at `database_required` field values for implicit actions, and
`required` template-tags for custom query actions. Bubble those values
up onto `action.parameters`. Finally, look at `requires_pk` to see if pk
fields should be marked as required regardless.

* Cleanup scaffolded back button (#26317)

* Forward action execution error messages to the frontend (#26469)

This is a crude temporary solution necessary to be able to debug problems related to executing actions.
Although error parsing has been implemented for Postgres (only) with the goal to provide information for
the FE like `{"errors":{"id":"violates foreign key constraint note_subject_id_fkey"}}`. This code doesn't
care about the error message itself, it just forwards the error message from the DB to the FE.

* Actions for public pages (#26554)

* [Apps] Testing out public dashboards

* Add WIP endpoints for public execution

* Add throttle check to public execution route

* Bump attempts-threshold to 1000

* Bump threshold to 5000

* Add test

* Fix linter

* Review changes

* Fix lint

* Run publicly shared actions with full permissions (#26610)

* Support actions with MySQL and MariaDB (#26573)

Support actions with MySQL/MariaDB

* Check permissions for implicit actions (#26652)

* Check permissions when executing implicit actions

* Fix some action tests and reduce permissions boilerplate

* Revert "[Apps] Disable data-apps prior to release (#25982)"

This reverts commit 51147027.

* WIP: Unify action representation (#26717) (#27349)

* [Apps] WIP initial db migrations for giving implicit actions an action_id and removing card from query_action

* Make crud work for http actions

* Make implicit crud work

* Rename implicit_action.namespace to kind. Rewrite merged-model-actions to actions-with-implicit-parameters

* Fix query action creation

* Make actions.test-util/with-action create a holding model

* Fix unified-action-create-test

* Fix get-action-test

* Add tests for creating implicit and http dashcard actions

* Enable query action in dashcard-action-create-update-test

* Fix metabase.api.dashboard-test/dashcard-query-action-execution-test

* Fix metabase.api.dashboard-test/dashcard-http-action-execution-test

* Fix metabase.api.dashboard-test/dashcard-action-execution-auth-test

* Fix metabase.api.dashboard-test/dashcard-action-execution-granular-auth-test

* Make implicit action execution work

* Scaffolding implicit actions from table

* Fix metabase.api.action-test

* Make scaffolding work with existing models. Update with-action macro to allow multiple actions

* Fix metabase.api.public-test

* Remove unused names from metabase.models.action

* Remove some unused imports

* Remove an unused import

* Remove model_action and fix a bunch of tests

* Fix dashcard-implicit-action-only-expose-and-allow-model-fields

* Fix action type inheritance

* Fill in remarks and add description to action

* Add forward migration for existing actions and rollbacks that remove existing actions

* Prevent overwriting parameters of custom actions

* Support setting description on actions

* Update migration to work with mysql - rollback is still not quite working in mysql due to database_id fk

* Rename with-action macro to with-actions and document it

* Remove report_card.is_write

* Fix some more tests

* Fix rollback across all appdb drivers, add not null constraints, use action.name to hold implicit slugs

* Pull name and description from is_write card

* Add name to scaffolded implicit actions

* Remove ModelAction

* Remove is_write from comments too

* Different databases reference capture groups in regexp_replace differently. Also drop model_action table

* Update migrations to pass linter.

Introduce ${capturegroup.ref} to handle the different ways to reference
capture groups within a regexp_replace. This allows collapsing large
data migrations into a single `sql` element that the linter requires.

* Add kind to action-mapper so we can sort as we wish

* Add name to with-temp Action

* Adjust FE to query actions migration (#26764)

* Remove `is_write` card notion

* Remove action `slug` from action types

* Fix typo in `estimateCardSize` name

* Remove hanging card's `is_write` prop

* Remove not used writeback actions

* Update endpoints

* Remove not used prop

* Remove action `slug` usage

* Update actions entity

* Tweak action creator to handle new actions shape

* Fix saving action dashcards

* Fix actions API usage

How did that work before? 



* Pass `database_id` when creating query actions

* Remove custom list API implementation for actions

* Clean up types

Co-authored-by: Tamás Benkő <tamas@metabase.com>

* Change regexp_replace to substring to support mysql - this is ok because we have consistent json serialization.

* Check permissions on the model in the action API

* Remove unused required namespace

* Throw an exception when demoting a model with actions to question

* Allow saved parameters on implicit actions to add to implicit parameters

Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Case Nelson <case@metabase.com>

* empty commit to kick CI

* bump CI

* Fix down migrations

 change v45.00-002 sets up a fk fk_query_action_ref_action_id. h2 v2 for
 some strange reason requires us to remove the constraint before we can
 remove the primary key. But then we have to restore the state that
 v45.00-002 creates manualy

* Allow for version for `(migrate! :down 45)`

* Remove data apps parts not planned for 0.46

Removed app container, app scaffolding and pages (dashboards with is_app_page)

* Address linter issues

* Make some cosmetic changes to reduce the diff

* Make new permission API endpoints use defendpoint-schema

* Fix bad automerge

* Skip Mongo 4.2 instead of 4.0

Co-authored-by: Case Nelson <case@metabase.com>
Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>
Co-authored-by: dan sutton <dan@dpsutton.com>

- add the rest of malli reserved aliases
- malli.util :as [mu -> mut] (had collision with metabase.util.malli)

* Make Dimension unique on `field_id` and deduplicate

* Fix rollback for new migrations

* Don't require migration comments to contain 'added' anymore since it's part of the version ID now

* Don't require 'Added <version>' in migration comments anymore

* Fix SerDes test that created duplicate dimensions for one Field

* Silly fix to fix MySQL 5.7

* add the library, and dev/start-malli!

* Update deps.edn

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* enable the use of regexes as malli schemas

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* First pass at grouping axes

This PR uses the results_metadata key to look at fingerprints for numerical axes to try determine if each series on
the Y axis can be sanely represented on the same axis.

This is done by calculating an overlap (some value between 0 and 1) and grouping all axes on the LEFT if they pass the threshold, which is
some value between 0 and 1. The overlap is always calculated when there is SOME overlap between the ranges of each
axis, and is calculated as:

```clojure
(/ (- (max maximums) (min minimums))
   (- (min maxiumums) (max minimums)))
```

This is done to try catch situations where one column's range is entirely inside the other, but is much smaller (- max
min); such a case would have a small percent overlap by the above calculation, and implies that it might be better to
split the axes.

* Address feedback.

* Fix shape of data in tests of 2 private fns

* Add test for split axes

* Fixed error in test util

* render-utils does with-redef, disallows parallel tests. Makes sense

* Column settings correct keys. Show Axes labels logic defaults 'true'

These are two small changes related to static viz column formatting.

The show axes labels logic change fixes a faulty assumption from an earlier PR where it was assumed that if
:labels_enabled key was NOT present, then it was 'false'. The default however, is 'true', so the check had to change
to reflect the frontend.

Column settings have to be formatted so that the js code running in the backend looks in the right places. For
example, `:number-format` must become `:number_format`. I move the function that does that change up so that it can be
rolled up into `settings-from-column` to reduce the number of call sites for `update-col-for-js`.

* Adjust number format fn to consider :decimal_digits of currencies

* Simpler date-separator fix

* Prevent non-temporal labels from being passed to datetime formatter

* Table Temporal format columns are improved

The column headers will now be properly looked up, the key format was incorrect before, causing the `:column_title`
lookup to return nil, which ignores when a user changes the column name.

Table columns with temporal formatting settings come in with `:unit` `:default`, so that case was added to the
datetime formatter. Additionally, a helper function was added to pull the correct column viz settings to pass to the datetime.

* Change some table rendering to look for '_' keys instead of '-'

And also allow the field-id to take the form `[:field-id 10 nil]` OR `[:field-id 10]`

* Added test utils for static-viz rendering. Docstrings for details.

A big set of functions to enable cleaner static-viz rendering tests.

Next push will be thorough table tests, and some other static viz cleanup

* Add some viz-scenarios to help with table testing

* Lint

* Wrong map here

* Currently unused bindings

* Writing tests with render-test utils now, and catching new bugs

* Use column's field ref to maybe get the column settings

* Adjustments so that column-settings are properly grabbed

* Number formatter no longer assumes decimal places

The frontend will only display decimal places when strictly necessary OR when the column formatting dictates the
number of decimal places to show.

The number formatter on the backend made different assumptions about decimal places, so precision may be lost OR
needlessly displayed.

The formatter now checks the actual value passed in and adjusts the format string according to the decimal places
present in the value, not just via column-settings, though the column settings will still take priority.

* Shame on me. inline def is not needed here :/

* Adjust number format tests to assert that now we match FE behavior

* Really trying to get formatting buttoned up.

* Matching frontend's default format behavior for numbers

* Some tests failed because they assumed two decimal places always

* Values less than one still round after 2 sig-figs

* Column formatting on tables follows expected default behavior

* Getting public-settings included in number and datetime formatters

* Test for custom formatting public settings

* Removed unused require.

* address some PR feedback

* Try to clarify the logic in the number formatter

* Add some clarifying comments

* Use the existing viz tools to help create field-refs properly

* Cleaned up column-name impl. to use normed field-refs. Much nicer!

* Remove some unnecessary tests, and add a negative scientific num

* Simplify table columns test

* Simplify and add more format settings to column-formatting test

* Unused require

* Another one

* Fix format of column refs in test

* Remove unused thread

Previously, some automatic behavior was causing problems in serdes.
(The worst example is generating an `entity_id` on insert while
deserializing an entity we don't own - if deserialized again it would be
duplicated.)

A whole cascade of design choices fell out of this problem:
- `insert!` would generate `entity_id`s, so use `simple-insert!`
- But `simple-insert!` doesn't convert eg. MBQL `:definition` maps back
  into JSON strings
- We got the maps as Clojure data and not JSON strings because `select`
  and `simple-select` both run `post-select` and parse the JSON.
- So we ended up with a raw query on the select side and `simple-insert!`
  on the storage side.

This change unwinds that whole stack, and instead uses a dynamic
var to suppress the few pieces of `pre-insert` and `pre-update` logic
that causes problems.

The end result is much cleaner, and much more consistent with the rest
of Metabase's backend logic.

* Add helper function sending debug-qp events to portal
* Include the intended value in the transformed-result event

* Refactor debugger output

* adds ddiff2 as a dev dependency

- used in process-query-debug

* Update dev/src/dev/debug_qp.clj

use 0 arg version of println

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Fix in-memory logger

Our Admin > Troubleshooting > Logs page broke, just showing a spinner
and never showing the logs.

Don't quite understand why this fixes it. Javadocs are
https://logging.apache.org/log4j/2.x/log4j-api/apidocs/org/apache/logging/log4j/LogManager.html#getContext-boolean-

```clojure
logger=> (log/warn "test")
nil
logger=> (count @messages*)
0
;; no in-memory logs so page is empty
;; change `(LogManager/getContext true)` in the momoized ns-logger fn
;; and then observe:
logger=> (log/warn "test")
nil
logger=> (count @messages*)
4
```

Some explorations that might shine some light:

```clojure
logger=> (into {} (.getAppenders (.getLogger (LogManager/getContext false) (str *ns*))))
{}
logger=> (into {} (.getAppenders (.getLogger (LogManager/getContext true) (str *ns*))))
{"metabase-appender" #object[metabase.logger.proxy$org.apache.logging.log4j.core.appender.AbstractAppender$ff19274a
                             "0x4d680247"
                             "metabase-appender"]}
```

So something is not hooked up quite right.

* Add tests for metabase.logger

* Ensure `api/util/logs` returns logs

* tests

* Check for presence of `MetabaseLoggerFactory` rather than whole str

When in a namespace with a record, `Foo resolves to ns.Foo. But outside
it resolves to ns/Foo. When running tests from the command line *ns* is
user so it gets more complicated.

* Kinda playing whackamole™ with `(LogManager/getContext true)`

* Remove custom memoizing logger

History:

39.2 we set `Multi-Release: true` in our manifest file and query speed
drops like a stone. Jeff was able to track this to our logger calls in
tight loops.

We revert the multi-release and keep seeing the warning on startup

> WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

Benchmarking on 39.2
(bench (log/trace "hi")) -> 15383 ns

So we freaked out and set up a memoizing logger factory

(bench (log/trace "hi")) -> 141 ns

What a win.

But we never noticed that the default *logger-factory* being picked up
was slf4j ( `(log.impl/name log/*logger-factory*)` -> "org.slf4j" ). On
39.2 if you set the factory to the log4j2 version you get back to a
great number: `(bench (log/trace "hi"))` -> 25 ns

And thus ensuring that our logger is the default log4j2 version is even
faster than our memoizing logger-factory.

Memoizing factory: 141 ns
slf4j factory: 2269 ns
log4j2 factory: 31 ns

What does `(LogManager/getContext false)` mean versus using `true`? We
only need and want a single context. But log4j2 by default uses a
context selector called `ClassLoaderContextSelector`. We could put all
of this behind us if we used a context selector type
`BasicContextSelector` but this is surprisingly hard to do: you have to
set a system property. And since all of this stuff gets initialized in
static initializers, we don't have an opportunity to do this
programmatically. The way around this is to require people to pass this
system property on startup which is not acceptable.

So getContext true checks for a threadlocal context in a specific static
variable and falls back to a Default context. getContext false looks at
classloaders and ends up at a different context. BUT: the log.impl
version uses a closure over getContext false instead of getting it each
time. And I suspect that when it does this there is only one so it is
the default and continues to use this one. In our LoggerFactory
implementation we were looking up the context each time. This still
seems to work and everything is playing nice in our application
classloader but its totally possible that our drivers are not hitting
this. I'll have to investigate this.

Verification:
- build the uberjar locally (`bin/build`)
- copy to some temp directory and also copy criterium.jar

```shell
MB_JETTY_PORT=4000 java "$(socket-repl 4001)" -cp locally-built.jar:criterium.jar metabase.core
```

```clojure
/tmp/locally-built via  v17.30 on   metabase-query
❯ nc localhost 4001
user=> (doto 'metabase.logger require in-ns)
metabase.logger
metabase.logger=> (require '[criterium.core :refer [bench]])
nil
metabase.logger=> (bench (log/trace "hi"))
Evaluation count : 1686535500 in 60 samples of 28108925 calls.
             Execution time mean : 22.487972 ns
    Execution time std-deviation : 0.101004 ns
   Execution time lower quantile : 22.326806 ns ( 2.5%)
   Execution time upper quantile : 22.648368 ns (97.5%)
                   Overhead used : 6.924761 ns
nil
metabase.logger=> (count (messages))
358
metabase.logger=>
```

Verifies that we are on the order of 22 nanoseconds and the in-memory
logger has messages in it.

* Appease our namespace linters

* I'll unplug you ns linter

* Better tests and ns docstring

* Bootstrap to set system properties

New entrypoint for the application: metabase.bootstrap

sets two properties for logging (context selector, log4j2 factory) and
ensures those properties are set before any logging code is loaded.

* docstrings and clean ns

* metabase.logger ns docstring cleanup

* docstring

* rename a test now that there's no memoization

* add logger properties to :dev profile

* Revert "add logger properties to :dev profile"

This reverts commit 4f09fa3b631f882a3c5edcab4508769ffb20d4fa.

* deps

* Record startup times

updated `dev/start!`. `mb.core/init!` itself sets up the db, loads
plugins, and sets the init status so those were superfluous.

* clean up dev namespace

* testing doesn't call init so set a value

* Make namespace aliasing consistent everywhere; enforce with clj-kondo

See the table of aliases in .clj-kondo/config.edn

Notable patterns:
- `[metabase.api.foo :as api.foo]`
- `[metabase.models.foo :as foo]`
- `[metabase.query-processor.foo :as qp.foo]`
- `[metabase.server.middleware.foo :as mw.foo]`
- `[metabase.util.foo :as u.foo]`
- `[clj-http.client :as http]` and `[metabase.http-client :as client]`

Fixes #19930.

* Add aliases for H2 and Liquibase CLIs

* Use reflection to invoke Liquibase method so it doesn't bust stuff

Rework how the remappings middleware matches remapped to/from columns for explicit external (FK) remaps (#20009)

* Fix #9236 [WIP]

* Revert unneeded changes

* Only merge namespaced :options into result info

* Revert unneeded changes

* Everything is working 

* Revert unneeded change

* Clean namespaces

* Add `:test` profile to `namespace-checker` to suppress log messages.

* PR feedback

* Fix namespaces

Don't parse MB_DB_CONNECTION_URI; use javax.sql.DataSource everywhere for app DB connection sources (#19970)

* Use javax.sql.DataSource everywhere for app DB connection sources

* Sort namespaces

* Un-remove Kondo warning

* Minor tweaks to support 'postgres:' ...

* Remove ConnectionDataSource

* Add warning about PG without sslmode back

* PR feedback

* Sort namespaces

* Remove top-level calls to trs (not allowed because app DB isn't initialized yet anyway)

* Test fix  (MySQL)

`add-alias-info`: recognize `:field` clauses in join source queries to be the same if it has a `:temporal-unit` (#19789)

* Backport Debug QP improvements from #19754

* Enable test

* "Fuzzy" matching when looking for Fields in join source queries

* Add/update tests

* Test fix 

* Remove trailing whitespace

* Fix a few docstrings

* Debugging QP Improvements

* Appease namespace linter

* Backport additional debug QP changes from #19384

* Test fix 

* Pass along date formatting

* extract and rename some things

* Number formatting

* Update tests

* Improve test validation of svg labels

* Clean up tests

* Move static-viz interface file to custom file

* Backend SVG rendering proof of concept [ci skip]

* Update cssbox to 5.0.0

* Render bar, line, and pie charts in js to svg

sparkline is now done in js, bar is now recognized and done in js, new
:categorical/donut as well

* Remove api route for render

* pass along render-type, not hardcoded to :inline

* Move bar chart above sparkline and remove line check

In order to introduce the bar chart type need it above the sparkline
check since it is otherwise the same except for display property of
the card. But lots of tests assume that this will get hit with a nil
display type set in testing so remove checking for `:line` allows all
the testing cases to hit the right type

* Fix tests now that bar graphs aren't html but images

* Include attachments for bar charts

* Move over to in-tree bundle

* Force everything [ci noskip]

trying to ensure that the built jar includes the newer
"resources/frontend_client/app/dist/lib-static-viz.bundle.js"

* Run `yarn build-static-viz` in backend-deps in CI

this js file is now a hard dependency of the backend so it fits in
this tsk. All such things that depend on the backend sources will need
it. Makes me think perhaps we want a checked in version but i'm not
sure yet.

* Look on classpath not filesystem for js bundle [ci noskip]

* Move yarn build-static-viz into the checkout step

* License information for antlr4-runtime

* create attachment for categorical donuts

* add ordinal legend to donuts (#17177)

* set widths of html image and svg image to 1200

* Revert "add ordinal legend to donuts (#17177)"

This reverts commit 1eb81d2e.

* Helper functions to render html easily

* readme in dev

* readme ensure that static viz bundle exists

* Cleanup ns after removing proxy

* Donut chart colors and legend (#17251)

* use external color map for fill per dimension

* Add support new color legend for donut

* Ensure text doesn't appear as link

entire thing is actually the body of a link tag for emails but we want
a decent text color rather than a default link color

* use chart colors from https://stats.metabase.com/_internal/colors



* Make checkers happy

- remove unused imports
- add a docstring
- don't shadow fn with a local

* cleanup ns import

* Remove reflective call

* Cleanup ns on correct branch

Co-authored-by: dan sutton <dan@dpsutton.com>

* X-axis: just use (approx) 5 ticks to avoid overlapping labels (#17287)

* increase gap between arcs (#17271)

* Set rendering hints on html->image

* ignore width for now and make them larger

* Ns deprecation and some cleanup

* make namespace checker happy

* Simple tests for detecting chart type

* Rename from poc

* Tests for scalar/smartscalar

* cleanup js svg namespace a bit

* Tests of svg engine

* ns sorting after renaming

* Unify our two different js engine usages

settled on the js context. Has typed returns `(.asString ^Value ...)`
instead of perhaps capturing std out?
https://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/Value.html



Context is a bit more friendly for getting source into it. One
downside is that the invocable bit isn't quite as nice. The old way
would return a java.util.functionFunction but the difference is

(.apply function (object-array args))

vs

(.execute fn-ref (object-array args))

* Don't io/resource the io/resource

* js engine tests

* Ns cleanup in js-svg

type hints in the js-engine ns mean we don't need as many classes from
polyglot here

* Cleanup of text, ns docstrings, alignment

* Fix fill->fill-opacity with parsed doc, not regex

* Make a single helper that loads a static viz bundle context

* Docstrings and make private in js-svg

* Sort imported classes in js-svg

* Make width passed down through rendering aparatus

- svgs are always rendered at 1200 for quality
- slack images of html are rendered at 1200 so that they can be zoomed
  in in the ui but slack automatically scales down
- email sends html and the full svg but includes img width tags so
  that is handled appropriately

* docstring

Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>
Co-authored-by: Ariya Hidayat <ariya@metabase.com>

* Fix SQL saved question reference to Saved Questions with parameters

* Test fixes 

* Don't run test for drivers that don't support FKs

* Test fix 

* Test fix 

* Remove location sub-categories

These sub-categories are only for filtering the list of options
when mapping a parameter filter to a field. Since we are introducing
operator types as a sub-category of location, city/zip/etc. just get in
the way.

* add number section + number/string operator subtypes

Light refactor of meta/Dashboard changes

rmv 'all-options' options (for now)

* add/update parameter type icons

* pass operator to ParameterFieldWidget + show input per operator field

* Add operator helper fns that aren't dependent on fields/tables

* Make operator prop optional

fix date filter err

* add combined  name for native question filter widget type list

Otherwise, a field that matches both "Location" and "Category"
options will show duplicate "Starts with" options, etc. Now,
that'll look like "Category - Starts with" and "Location - Starts with"

* correct some unused prop/arg passing

* Convert location/category parameter types to string for query

location/category don't mean anything to BE but we use them for
"reasons" on the FE. Reasons are legacy reliance on unique-ness of
the parameter.type value, primarily.

* operators in backend

* Remove errant tap>

* Docstrings and differing numbers in tests in some dbs

* Make unary private so docstring checker ~shuts-up~ is satisfied

* Don't parse arguments to operators params

they were coming in just fine from the FE as numeric or string
types. no need to ensure strings everywhere and parse here

* add max-width to PopoverPicker

* rmv unused value

* use combinedName on dashboard parameters

* fix parameter to mbql code

* Ensure = operator filter popovers have no label

This is to match "old" style of parameter popover

* Update Cypress tests to reflect new parameter flow

fix cypress dashboard parameter tests

Fix more cy tests

* Don't call fk/joinAlias on ExpressionDimension

The methods don't exist on ExpressionDimension class.

This doesn't make them work (yet), but it prevents the app
from crashing.

* Namespace doc and remove unnecessary comment

* tap>-spy in dev

* first pass at substitution of new operators in native

* Docstring on wrap-value-literals-in-mbql to appease the gods

* variadic equality operators (string/= number/=)

* move functions out of component file

* Pass parameter object to tag editor for use in default input

We should inline this input eventually because it looks ugly.

* map parameters in Questions to correct type

* continue to pass janky fake parameter for text/number tags

* mongo native substitution

* variadic not-equals for string and number

* Docstring and use correct function to make errors

* add number/between dash param cy test

* Update function name to better reflect behavior

* Add unit tests for paramer/operator util fns

add unit tests for parameter util functions

add unit test for operator util fns

* add variadic string 'is not' param operator option

* Modify operator parameter display labels

don't append 'matches exactly' to location/category parameters

label tweaks

Update cy tests to reference correct label name

rmv it.only

* Desugar mongo parameters mbql

desugaring makes for a bit more verbose query but that's ok.This
change was done to ensure that we negated regexes in a correct way,
and to do so we always return the string version. This ensures that it
can be json/generate-string'd for native parameters or left as
datastructures and sent to monger

* Cleanup stale comments and fixup docstring for consistency

* Arglists metadata on defmulti and denude some threaded forms

* add single arity number tag predicate to variable filter

* add Location operators to fix parameter<->filter mapping

For question filters to work we need for the new parameter operators
to be supported by "location" fields in all areas of the app.

* Don't show coords for param number widgets

I don't think we want to support all the various number operators when
dealing with coordinates, so in order to avoid that I'm preventing the
mapping of number parameter operator to coord fields.

* prevent mapping of tags to non-equal operators

while possibly useful to end users, this needs more UI work on the
native question side of things.

* Ensure parameter values are wrapped in an array

When an = operator is mapped to a field AND a tag, it ends up not being
wrapped in an array due to the TextWidget (I think).

ensure parameter value is an array

ensure number params have an array value

* Sort imports correctly

clojure-lsp used to do this incorrectly (sorting `[` before `j`) and
that has now been fixed

Co-authored-by: dan sutton <dan@dpsutton.com>

* Fix pivot table queries w/o data permissions

* Text fix 

* Refactor

* Ns sorting

Switch query execution to use `Statement` instead of `PreparedStatement` when there are no params

Add new multimethod to create a `Statement`, rather than `PreparedStatement`, which works similarly, called `statement`

Add new multimethod to run a SQL query against a Statement, similar to `execute-query!`, called `execute-statement!`

For consistency, rename `execute-query!` to `execute-prepared-statement!`

Change `execute-reducible-query` to capture whether the query has params, and if not, using the new multimethods instead

Update fetch-results-metadata-test so it replaces execute-select! instead of prepared-statement

Adding `statement-supported?` multimethod (defaults to true), to let drivers control whether statements are used at all

Updating Oracle driver to not override holdability for `statement`, similar to `prepared-statement`, and bumping module version

Updating SparkSQL driver to indicate that statements are not supported, and bumping module version

Fixing Redshift test to also override `execute-statement!` so it can capture the SQL for a statement

Background

The original version of MBQL (known as the "Structured Query" language at the time, which was a little confusing) just referred to all fields by integer ID. e.g.

{:filter ["STARTS_WITH" 1 "abc"]}
That made clauses like this ambiguous:

{:filter ["=" 1 2]} ; is 2 a Field, or the number 2?
To clear up ambiguity and to let you filter Fields against Fields, we added the :field-id clause to make it explicit that you were referring to a Field, not an Integer:

{:filter [:= [:field-id 1] [:field-id 2]]} ; Field 1 == Field 2
We soon added a couple of new types of Field clauses, :fk-> and :datetime-field, both of which wrap the original :field-id:

;; refer to Field 2 from a different Table, use Field 1 from the current Table to perform the join
[:fk-> [:field-id 1] [:field-id 2]]

;; bucket Field 3 by month
[:datetime-field [:field 3] :month]
So far things were still pretty reasonable, the worst you'd have to do is something like

[:datetime-field [:fk-> [:field-id 1] [:field-id 2]] :month]
Things started to get out-of-hand IMO when we continued to add more Field clause types that just wrapped everything else. We added :binning-strategy:

[:binning-strategy <field> :num-buckets 10]
then :field-literal, to refer to a Field from a nested native source query:

[:field-literal "my_field" :type/Text]
then we added :joined-field to specify the Table you're explicitly joining against that is the source of a Field:

[:joined-field "my_join" [:field-id 4]]
This ends up getting really hairy when you combine things. This is an actual real clause you can use right now:

[:binning-strategy
 [:datetime-field
  [:joined-field "my_join"
   [:field-literal "my_field" :type/DateTimeWithLocalTZ]]
  :month]
 :num-bins 10]
And even with all of those clauses, we still can't pass around arbitrary extra information in a way that would make it easy to implement performance optimizations. If we ever try to add another new clause, the whole house of cards is going to come crashing down.

The proposal
Combine all of the Field clauses into a single new :field clauses with the schema

[:field id-or-name options-map]
Here are some before & after examples:

[:field-id 1] 
=> 
[:field 1 nil]

[:field-literal "my_field" :type/Text] 
=> 
[:field "my_field" {:base-type :type/Text}]

[:datetime-field [:field 1] :month] 
=> 
[:field 1 {:temporal-unit :month}]

[:binning-strategy [:field 1] :num-bins 10] 
=>
[:field 1 {:binning {:strategy :num-bins, :num-bins 10}}]

[:joined-field "my_join" [:field 1]] 
=> 
[:field 1 {:join-alias "my_join}]

[:fk-> [:field 1] [:field 2]] 
=> 
[:field 2 {:source-field 1}]

[:binning-strategy
 [:datetime-field
  [:joined-field "my_join"
   [:field-literal "my_field" :type/DateTimeWithLocalTZ]]
  :month]
 :num-bins 10]
=>
[:field "my_field" {:base-type :type/DateTimeWithLocalTZ, :join-alias "my_join", :binning {:strategy :num-bins, :num-bins 10}}]

Fixes #14724 × FK remaps
Fixes #14724 × inception query
Fixes #14724 × inception query × FK remaps
Fixes #14766

Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>

Note: UI-related problem (parent collection should be immediately open) will be addressed in a separate isue.

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>