* Fix in-memory logger

Our Admin > Troubleshooting > Logs page broke, just showing a spinner
and never showing the logs.

Don't quite understand why this fixes it. Javadocs are
https://logging.apache.org/log4j/2.x/log4j-api/apidocs/org/apache/logging/log4j/LogManager.html#getContext-boolean-

```clojure
logger=> (log/warn "test")
nil
logger=> (count @messages*)
0
;; no in-memory logs so page is empty
;; change `(LogManager/getContext true)` in the momoized ns-logger fn
;; and then observe:
logger=> (log/warn "test")
nil
logger=> (count @messages*)
4
```

Some explorations that might shine some light:

```clojure
logger=> (into {} (.getAppenders (.getLogger (LogManager/getContext false) (str *ns*))))
{}
logger=> (into {} (.getAppenders (.getLogger (LogManager/getContext true) (str *ns*))))
{"metabase-appender" #object[metabase.logger.proxy$org.apache.logging.log4j.core.appender.AbstractAppender$ff19274a
                             "0x4d680247"
                             "metabase-appender"]}
```

So something is not hooked up quite right.

* Add tests for metabase.logger

* Ensure `api/util/logs` returns logs

* tests

* Check for presence of `MetabaseLoggerFactory` rather than whole str

When in a namespace with a record, `Foo resolves to ns.Foo. But outside
it resolves to ns/Foo. When running tests from the command line *ns* is
user so it gets more complicated.

* Kinda playing whackamole™ with `(LogManager/getContext true)`

* Remove custom memoizing logger

History:

39.2 we set `Multi-Release: true` in our manifest file and query speed
drops like a stone. Jeff was able to track this to our logger calls in
tight loops.

We revert the multi-release and keep seeing the warning on startup

> WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.

Benchmarking on 39.2
(bench (log/trace "hi")) -> 15383 ns

So we freaked out and set up a memoizing logger factory

(bench (log/trace "hi")) -> 141 ns

What a win.

But we never noticed that the default *logger-factory* being picked up
was slf4j ( `(log.impl/name log/*logger-factory*)` -> "org.slf4j" ). On
39.2 if you set the factory to the log4j2 version you get back to a
great number: `(bench (log/trace "hi"))` -> 25 ns

And thus ensuring that our logger is the default log4j2 version is even
faster than our memoizing logger-factory.

Memoizing factory: 141 ns
slf4j factory: 2269 ns
log4j2 factory: 31 ns

What does `(LogManager/getContext false)` mean versus using `true`? We
only need and want a single context. But log4j2 by default uses a
context selector called `ClassLoaderContextSelector`. We could put all
of this behind us if we used a context selector type
`BasicContextSelector` but this is surprisingly hard to do: you have to
set a system property. And since all of this stuff gets initialized in
static initializers, we don't have an opportunity to do this
programmatically. The way around this is to require people to pass this
system property on startup which is not acceptable.

So getContext true checks for a threadlocal context in a specific static
variable and falls back to a Default context. getContext false looks at
classloaders and ends up at a different context. BUT: the log.impl
version uses a closure over getContext false instead of getting it each
time. And I suspect that when it does this there is only one so it is
the default and continues to use this one. In our LoggerFactory
implementation we were looking up the context each time. This still
seems to work and everything is playing nice in our application
classloader but its totally possible that our drivers are not hitting
this. I'll have to investigate this.

Verification:
- build the uberjar locally (`bin/build`)
- copy to some temp directory and also copy criterium.jar

```shell
MB_JETTY_PORT=4000 java "$(socket-repl 4001)" -cp locally-built.jar:criterium.jar metabase.core
```

```clojure
/tmp/locally-built via  v17.30 on   metabase-query
❯ nc localhost 4001
user=> (doto 'metabase.logger require in-ns)
metabase.logger
metabase.logger=> (require '[criterium.core :refer [bench]])
nil
metabase.logger=> (bench (log/trace "hi"))
Evaluation count : 1686535500 in 60 samples of 28108925 calls.
             Execution time mean : 22.487972 ns
    Execution time std-deviation : 0.101004 ns
   Execution time lower quantile : 22.326806 ns ( 2.5%)
   Execution time upper quantile : 22.648368 ns (97.5%)
                   Overhead used : 6.924761 ns
nil
metabase.logger=> (count (messages))
358
metabase.logger=>
```

Verifies that we are on the order of 22 nanoseconds and the in-memory
logger has messages in it.

* Appease our namespace linters

* I'll unplug you ns linter

* Better tests and ns docstring

* Bootstrap to set system properties

New entrypoint for the application: metabase.bootstrap

sets two properties for logging (context selector, log4j2 factory) and
ensures those properties are set before any logging code is loaded.

* docstrings and clean ns

* metabase.logger ns docstring cleanup

* docstring

* rename a test now that there's no memoization

* add logger properties to :dev profile

* Revert "add logger properties to :dev profile"

This reverts commit 4f09fa3b631f882a3c5edcab4508769ffb20d4fa.

* deps

* Record startup times

updated `dev/start!`. `mb.core/init!` itself sets up the db, loads
plugins, and sets the init status so those were superfluous.

* clean up dev namespace

* testing doesn't call init so set a value

* Make namespace aliasing consistent everywhere; enforce with clj-kondo

See the table of aliases in .clj-kondo/config.edn

Notable patterns:
- `[metabase.api.foo :as api.foo]`
- `[metabase.models.foo :as foo]`
- `[metabase.query-processor.foo :as qp.foo]`
- `[metabase.server.middleware.foo :as mw.foo]`
- `[metabase.util.foo :as u.foo]`
- `[clj-http.client :as http]` and `[metabase.http-client :as client]`

Fixes #19930.

* Add aliases for H2 and Liquibase CLIs

* Use reflection to invoke Liquibase method so it doesn't bust stuff

Rework how the remappings middleware matches remapped to/from columns for explicit external (FK) remaps (#20009)

* Fix #9236 [WIP]

* Revert unneeded changes

* Only merge namespaced :options into result info

* Revert unneeded changes

* Everything is working 

* Revert unneeded change

* Clean namespaces

* Add `:test` profile to `namespace-checker` to suppress log messages.

* PR feedback

* Fix namespaces

Don't parse MB_DB_CONNECTION_URI; use javax.sql.DataSource everywhere for app DB connection sources (#19970)

* Use javax.sql.DataSource everywhere for app DB connection sources

* Sort namespaces

* Un-remove Kondo warning

* Minor tweaks to support 'postgres:' ...

* Remove ConnectionDataSource

* Add warning about PG without sslmode back

* PR feedback

* Sort namespaces

* Remove top-level calls to trs (not allowed because app DB isn't initialized yet anyway)

* Test fix  (MySQL)

`add-alias-info`: recognize `:field` clauses in join source queries to be the same if it has a `:temporal-unit` (#19789)

* Backport Debug QP improvements from #19754

* Enable test

* "Fuzzy" matching when looking for Fields in join source queries

* Add/update tests

* Test fix 

* Remove trailing whitespace

* Fix a few docstrings

* Debugging QP Improvements

* Appease namespace linter

* Backport additional debug QP changes from #19384

* Test fix 

* Pass along date formatting

* extract and rename some things

* Number formatting

* Update tests

* Improve test validation of svg labels

* Clean up tests

* Move static-viz interface file to custom file

* Backend SVG rendering proof of concept [ci skip]

* Update cssbox to 5.0.0

* Render bar, line, and pie charts in js to svg

sparkline is now done in js, bar is now recognized and done in js, new
:categorical/donut as well

* Remove api route for render

* pass along render-type, not hardcoded to :inline

* Move bar chart above sparkline and remove line check

In order to introduce the bar chart type need it above the sparkline
check since it is otherwise the same except for display property of
the card. But lots of tests assume that this will get hit with a nil
display type set in testing so remove checking for `:line` allows all
the testing cases to hit the right type

* Fix tests now that bar graphs aren't html but images

* Include attachments for bar charts

* Move over to in-tree bundle

* Force everything [ci noskip]

trying to ensure that the built jar includes the newer
"resources/frontend_client/app/dist/lib-static-viz.bundle.js"

* Run `yarn build-static-viz` in backend-deps in CI

this js file is now a hard dependency of the backend so it fits in
this tsk. All such things that depend on the backend sources will need
it. Makes me think perhaps we want a checked in version but i'm not
sure yet.

* Look on classpath not filesystem for js bundle [ci noskip]

* Move yarn build-static-viz into the checkout step

* License information for antlr4-runtime

* create attachment for categorical donuts

* add ordinal legend to donuts (#17177)

* set widths of html image and svg image to 1200

* Revert "add ordinal legend to donuts (#17177)"

This reverts commit 1eb81d2e.

* Helper functions to render html easily

* readme in dev

* readme ensure that static viz bundle exists

* Cleanup ns after removing proxy

* Donut chart colors and legend (#17251)

* use external color map for fill per dimension

* Add support new color legend for donut

* Ensure text doesn't appear as link

entire thing is actually the body of a link tag for emails but we want
a decent text color rather than a default link color

* use chart colors from https://stats.metabase.com/_internal/colors



* Make checkers happy

- remove unused imports
- add a docstring
- don't shadow fn with a local

* cleanup ns import

* Remove reflective call

* Cleanup ns on correct branch

Co-authored-by: dan sutton <dan@dpsutton.com>

* X-axis: just use (approx) 5 ticks to avoid overlapping labels (#17287)

* increase gap between arcs (#17271)

* Set rendering hints on html->image

* ignore width for now and make them larger

* Ns deprecation and some cleanup

* make namespace checker happy

* Simple tests for detecting chart type

* Rename from poc

* Tests for scalar/smartscalar

* cleanup js svg namespace a bit

* Tests of svg engine

* ns sorting after renaming

* Unify our two different js engine usages

settled on the js context. Has typed returns `(.asString ^Value ...)`
instead of perhaps capturing std out?
https://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/Value.html



Context is a bit more friendly for getting source into it. One
downside is that the invocable bit isn't quite as nice. The old way
would return a java.util.functionFunction but the difference is

(.apply function (object-array args))

vs

(.execute fn-ref (object-array args))

* Don't io/resource the io/resource

* js engine tests

* Ns cleanup in js-svg

type hints in the js-engine ns mean we don't need as many classes from
polyglot here

* Cleanup of text, ns docstrings, alignment

* Fix fill->fill-opacity with parsed doc, not regex

* Make a single helper that loads a static viz bundle context

* Docstrings and make private in js-svg

* Sort imported classes in js-svg

* Make width passed down through rendering aparatus

- svgs are always rendered at 1200 for quality
- slack images of html are rendered at 1200 so that they can be zoomed
  in in the ui but slack automatically scales down
- email sends html and the full svg but includes img width tags so
  that is handled appropriately

* docstring

Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>
Co-authored-by: Ariya Hidayat <ariya@metabase.com>

* Fix SQL saved question reference to Saved Questions with parameters

* Test fixes 

* Don't run test for drivers that don't support FKs

* Test fix 

* Test fix 

* Remove location sub-categories

These sub-categories are only for filtering the list of options
when mapping a parameter filter to a field. Since we are introducing
operator types as a sub-category of location, city/zip/etc. just get in
the way.

* add number section + number/string operator subtypes

Light refactor of meta/Dashboard changes

rmv 'all-options' options (for now)

* add/update parameter type icons

* pass operator to ParameterFieldWidget + show input per operator field

* Add operator helper fns that aren't dependent on fields/tables

* Make operator prop optional

fix date filter err

* add combined  name for native question filter widget type list

Otherwise, a field that matches both "Location" and "Category"
options will show duplicate "Starts with" options, etc. Now,
that'll look like "Category - Starts with" and "Location - Starts with"

* correct some unused prop/arg passing

* Convert location/category parameter types to string for query

location/category don't mean anything to BE but we use them for
"reasons" on the FE. Reasons are legacy reliance on unique-ness of
the parameter.type value, primarily.

* operators in backend

* Remove errant tap>

* Docstrings and differing numbers in tests in some dbs

* Make unary private so docstring checker ~shuts-up~ is satisfied

* Don't parse arguments to operators params

they were coming in just fine from the FE as numeric or string
types. no need to ensure strings everywhere and parse here

* add max-width to PopoverPicker

* rmv unused value

* use combinedName on dashboard parameters

* fix parameter to mbql code

* Ensure = operator filter popovers have no label

This is to match "old" style of parameter popover

* Update Cypress tests to reflect new parameter flow

fix cypress dashboard parameter tests

Fix more cy tests

* Don't call fk/joinAlias on ExpressionDimension

The methods don't exist on ExpressionDimension class.

This doesn't make them work (yet), but it prevents the app
from crashing.

* Namespace doc and remove unnecessary comment

* tap>-spy in dev

* first pass at substitution of new operators in native

* Docstring on wrap-value-literals-in-mbql to appease the gods

* variadic equality operators (string/= number/=)

* move functions out of component file

* Pass parameter object to tag editor for use in default input

We should inline this input eventually because it looks ugly.

* map parameters in Questions to correct type

* continue to pass janky fake parameter for text/number tags

* mongo native substitution

* variadic not-equals for string and number

* Docstring and use correct function to make errors

* add number/between dash param cy test

* Update function name to better reflect behavior

* Add unit tests for paramer/operator util fns

add unit tests for parameter util functions

add unit test for operator util fns

* add variadic string 'is not' param operator option

* Modify operator parameter display labels

don't append 'matches exactly' to location/category parameters

label tweaks

Update cy tests to reference correct label name

rmv it.only

* Desugar mongo parameters mbql

desugaring makes for a bit more verbose query but that's ok.This
change was done to ensure that we negated regexes in a correct way,
and to do so we always return the string version. This ensures that it
can be json/generate-string'd for native parameters or left as
datastructures and sent to monger

* Cleanup stale comments and fixup docstring for consistency

* Arglists metadata on defmulti and denude some threaded forms

* add single arity number tag predicate to variable filter

* add Location operators to fix parameter<->filter mapping

For question filters to work we need for the new parameter operators
to be supported by "location" fields in all areas of the app.

* Don't show coords for param number widgets

I don't think we want to support all the various number operators when
dealing with coordinates, so in order to avoid that I'm preventing the
mapping of number parameter operator to coord fields.

* prevent mapping of tags to non-equal operators

while possibly useful to end users, this needs more UI work on the
native question side of things.

* Ensure parameter values are wrapped in an array

When an = operator is mapped to a field AND a tag, it ends up not being
wrapped in an array due to the TextWidget (I think).

ensure parameter value is an array

ensure number params have an array value

* Sort imports correctly

clojure-lsp used to do this incorrectly (sorting `[` before `j`) and
that has now been fixed

Co-authored-by: dan sutton <dan@dpsutton.com>

* Fix pivot table queries w/o data permissions

* Text fix 

* Refactor

* Ns sorting

Switch query execution to use `Statement` instead of `PreparedStatement` when there are no params

Add new multimethod to create a `Statement`, rather than `PreparedStatement`, which works similarly, called `statement`

Add new multimethod to run a SQL query against a Statement, similar to `execute-query!`, called `execute-statement!`

For consistency, rename `execute-query!` to `execute-prepared-statement!`

Change `execute-reducible-query` to capture whether the query has params, and if not, using the new multimethods instead

Update fetch-results-metadata-test so it replaces execute-select! instead of prepared-statement

Adding `statement-supported?` multimethod (defaults to true), to let drivers control whether statements are used at all

Updating Oracle driver to not override holdability for `statement`, similar to `prepared-statement`, and bumping module version

Updating SparkSQL driver to indicate that statements are not supported, and bumping module version

Fixing Redshift test to also override `execute-statement!` so it can capture the SQL for a statement

Background

The original version of MBQL (known as the "Structured Query" language at the time, which was a little confusing) just referred to all fields by integer ID. e.g.

{:filter ["STARTS_WITH" 1 "abc"]}
That made clauses like this ambiguous:

{:filter ["=" 1 2]} ; is 2 a Field, or the number 2?
To clear up ambiguity and to let you filter Fields against Fields, we added the :field-id clause to make it explicit that you were referring to a Field, not an Integer:

{:filter [:= [:field-id 1] [:field-id 2]]} ; Field 1 == Field 2
We soon added a couple of new types of Field clauses, :fk-> and :datetime-field, both of which wrap the original :field-id:

;; refer to Field 2 from a different Table, use Field 1 from the current Table to perform the join
[:fk-> [:field-id 1] [:field-id 2]]

;; bucket Field 3 by month
[:datetime-field [:field 3] :month]
So far things were still pretty reasonable, the worst you'd have to do is something like

[:datetime-field [:fk-> [:field-id 1] [:field-id 2]] :month]
Things started to get out-of-hand IMO when we continued to add more Field clause types that just wrapped everything else. We added :binning-strategy:

[:binning-strategy <field> :num-buckets 10]
then :field-literal, to refer to a Field from a nested native source query:

[:field-literal "my_field" :type/Text]
then we added :joined-field to specify the Table you're explicitly joining against that is the source of a Field:

[:joined-field "my_join" [:field-id 4]]
This ends up getting really hairy when you combine things. This is an actual real clause you can use right now:

[:binning-strategy
 [:datetime-field
  [:joined-field "my_join"
   [:field-literal "my_field" :type/DateTimeWithLocalTZ]]
  :month]
 :num-bins 10]
And even with all of those clauses, we still can't pass around arbitrary extra information in a way that would make it easy to implement performance optimizations. If we ever try to add another new clause, the whole house of cards is going to come crashing down.

The proposal
Combine all of the Field clauses into a single new :field clauses with the schema

[:field id-or-name options-map]
Here are some before & after examples:

[:field-id 1] 
=> 
[:field 1 nil]

[:field-literal "my_field" :type/Text] 
=> 
[:field "my_field" {:base-type :type/Text}]

[:datetime-field [:field 1] :month] 
=> 
[:field 1 {:temporal-unit :month}]

[:binning-strategy [:field 1] :num-bins 10] 
=>
[:field 1 {:binning {:strategy :num-bins, :num-bins 10}}]

[:joined-field "my_join" [:field 1]] 
=> 
[:field 1 {:join-alias "my_join}]

[:fk-> [:field 1] [:field 2]] 
=> 
[:field 2 {:source-field 1}]

[:binning-strategy
 [:datetime-field
  [:joined-field "my_join"
   [:field-literal "my_field" :type/DateTimeWithLocalTZ]]
  :month]
 :num-bins 10]
=>
[:field "my_field" {:base-type :type/DateTimeWithLocalTZ, :join-alias "my_join", :binning {:strategy :num-bins, :num-bins 10}}]

Fixes #14724 × FK remaps
Fixes #14724 × inception query
Fixes #14724 × inception query × FK remaps
Fixes #14766

Co-authored-by: Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>

Note: UI-related problem (parent collection should be immediately open) will be addressed in a separate isue.

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Split metabase.db into connection, connection-pool-setup, env, setup, and util

* Remove old logging string

* Test fixes 

* Test fix 

* Deprecate parse-connection-string

* Remove application-db-mock-id

* Full support for MB_DB_CONNECTION_URI for application DB without parsing

* Remove *allow-potentailly-unsafe-connections*

* Test fixes 

* Test fix 

* Test fixes 

* Refactor load/dump commands

* New metabase.cmd.copy namespace

* More tweaks etc.

* 

*  2000

* Test fix 

* 

* Make linter happy

* Test fix

* Fix typo

* Use correct skip key for backend tests

* Proposal: no more prefix namespaces

* Fix a few missing files

When using a service account, the project-id isn't provided by the user,
but instead comes from the credentials for the service account. Handle
this case by looking for a project-id if one is provided explicitly,
otherwise defer to the credential object.

Additional change - during development, if (dev/init!) isn't called, the
event listeners are never setup, meaning that the initial syncs don't
occur.

Resolves #12648

[ci bigquery]

* Remove QP context cancelf and metadataf functions which aren't needed/used

* Test fixes 

Works around upstream bug in Postgres JDBC driver pgjdbc/pgjdbc#425

Also includes a lot of test reformatting/cleanup

register fonts for use by cssbox
also tools for rendering pngs from the repl

*  Make sure report timezone is taken into account when converting bucketed datetime filter clauses to ranges
*  Properly handle results for H2 and Oracle TIMESTAMP WITH TIME ZONE columns and SQL Server DATETIMEOFFSET columns
*  Enable timezone tests for all DBs; failing ones disabled for now

Test improvements: 

*  with-temporary-setting-values macro will now throw an Exception if passed an incorrect number of args in binding form
*  Rework Oracle test extensions to use Oracle's wacky INSERT ALL syntax to load all rows for a test dataset at once. Oracle tests 10x faster 
*  Rework timezone tests to be more idiomatic and readable
*  Test fixes to make sure various drivers can properly load test data with timezone-aware columns
*  A few fixes for test utility functions, including making sure the dataset macro works properly when used at the top-level of a test with multiple drivers
*  Fix the random Vertica test failures by adding retry logic when loading data fails
*  Other code cleanup

The biggest performance improvement in history of Metabase since we switched to Clojure

Before when filtering by a "bucketed" datetime field we would wrap the field and the value(s) it was being compared against in casting/truncation functions. For example when compiling a query to find fields where month is September 2019, we previously generated SQL like this:

```sql
SELECT count(*)
FROM public.checkins
WHERE date_trunc('month', CAST(public.checkins.date AS timestamp)) 
    = CAST(timestamp '2019-09-01T00:00:00Z' AS date)
```

As mentioned in #4043, this is not good for performance! Metabase now generates logically equivalent SQL like

```sql
SELECT count(*)
FROM public.checkins 
WHERE (
    public.checkins.date >= timestamp '2019-09-01T00:00:00Z' AND
    public.checkins.date < timestamp '2019-10-01T00:00:00Z'
)
```

Fixes #4043
Fixes #11009 
Fixes #11010
Fixes #11011 
Fixes #11012
Fixes #11013

  

Fix how optional field filter params are substituted; fix support for nested optional params (#10947)

* Param fixes [ci skip]

* Almost all working [ci skip]

* Test fixes

* Test fixes

* Test / linter fixes

* Fix linter errors