Don't parse MB_DB_CONNECTION_URI; use javax.sql.DataSource everywhere for app DB connection sources (#19970)

* Use javax.sql.DataSource everywhere for app DB connection sources

* Sort namespaces

* Un-remove Kondo warning

* Minor tweaks to support 'postgres:' ...

* Remove ConnectionDataSource

* Add warning about PG without sslmode back

* PR feedback

* Sort namespaces

* Remove top-level calls to trs (not allowed because app DB isn't initialized yet anyway)

* Test fix  (MySQL)

* MBQL :expressions should use strings for keys

* Make some more tests ^:parallel

* Update dox

* Test fix

* Sort namespaces

Still pursuant to #18676

1. Funnel was previously not doing lato font for the bold bit because visx's text doesn't take weights, it takes styles
2. Columns can be nontrivial with respect to funnels, which was an oversight
3. For some reason the first try for fixing 2 also impinged upon values being strings for funnels, which breaks things at SVG level
4. And let's just shove in a nit for the display of trends if the previous value and current value are the same

CI was broken inasmuch as it wasn't running, so master was broken without us knowing it. Thankfully, to the best of our knowledge, it was a couple of nits. Fixes master.

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

Only attempt to read values that were actually set on the client side

Add test to confirm specific combination from reported issue works

* Improve lifecycle of metadata

Previously we have assumed that if valid metadata is passed into the
card create and udpate, that that is valid metadata for the query. There
are a few scenarios where this can fail, but they all boil down to
editing the query after running it. Examples:

- when creating a query, come up with a query, visualize, realize you
wanted to not select all columns, edit to not select those, possibly
select a few extra, and then save. The saved metadata might only have
the columns from when you ran the query and not after you saved it.
- edit an existing query and save without rerunning. Again, adding and
removing some columns.

How it affects:
- if you remove columns, the preserved metadata can cause us to attempt
to select columns that are no longer in that question, leading to errors
for nested queries
- if you add columns, the query builder won't show the new columns until
that information is populated in the metadata by running the query.

Lifecycle now:
- should be encapsulated in `api.card/result-metadata-async`. But we
only allow metadata to remain if is is valid metadata and the query has
not been edited. If the query has been edited (and new queries always
satisfy this since the previous query will be nil) we rerun the query to
get the metadata. No more drift. However, when dealing with datasets we
will attempt to blend the existing metadata into the newly computed
metadata so that metadata edits can persevere.

Other changes:

- removed an extra indirection in the card api that made it harder to
reason about when to recompute metadata
- moved the `combine-metadata` function into a util namespace and gave
some more generic parameter names
- deleted two tests that are no longer relevant after removing the
checksum on metadata. One had to do with how we hash floats vs ints
which is no longer relevant since we aren't hashing at all (RIP
checksum) and one checked that we accepted "valid" checksum-ed
metadata.
- new test that updating a card will update its metadata by rerunning
the query
- when validating the checksum, do not call s/validate since this WILL
THROW. Instead check `(nil? (s/check ...))`.

* Normalize query before comparing for changes

When comparing db vs api-sent query, need to normalize otherwise they
will always be different and our optimization will never kick in

* Fix ns

* Cleanup after ourselves

* Move `combine-metadata` util functions to a better namespace

* Allow for no query in api/card/:id patch

Cypress tests would do a

```javascript
  describe("simple mode", () => {
    beforeEach(() => {
      cy.request("PUT", "/api/card/1", {
        name: "Orders Model",
        dataset: true,
      });
    });
```

The metadata testing stuff would always use the query from the PUT to
check whether to dump or retain the metadata. In these instances, it
would dump the metadata and then i guess ask the qp for the metadata for
a nil query. Similar scenario if anyone uses the API to update
descriptions, etc.

The method is PUT not PATCH, but in the saving function we have

```clojure
(u/select-keys-when
 card-updates
 :present #{:collection_id :collection_position :description :cache_ttl :dataset}
 :non-nil #{:dataset_query :display :name :visualization_settings
            :archived :enable_embedding :embedding_params
            :result_metadata})
```

So we kinda have a PATCH anyways.

* Normalize metadata before blending

* Ensure api blends metadata correctly

When blending the metadata, need to update the field_ref to keyword the
original. Otherwise we are attempting to match up the field with ref
["field" "subtotal"] against [:field "subtotal"] and they don't match.

* Normalize metadata from api correctly

if you just call normalize/normalize it doesn't know not to change
`:field_ref` -> `:field-ref` so it gets out of whack. Need to pass it
the function appropriate for that level.

Also add a test that you can update _just_ the metadata. This actually
already worked for a subtle reason, and that is possibly buggy. The
reasoning is that the result_metadata is clobbered conditionally now
instead of unconditionally. If the async metadata stuff returns nil it
won't clobber and then whatever was sent over the wire goes in.

* Ensure previous metadata is there

When updating the query of a dataset, handle the case where just the
query is sent, and not the whole card. If the client didn't send the
previous metadata, we did not look it up in the database

* Card api always uses metadata from `result-metadata-async`

Originally this did

    (assoc card-data :result_metadata (a/<! result-metadata-chan))

But I changed that so it only optionally did so when that channel had
information. The problem with this is that if there was :result_metadata
on the original request but the channel returned nil for some reason,
whatever was in the request was going in the db. This includes invalid
metadata, non-json strings, what ever. Obviously bad.

But we need to handle PATCHs where clients don't send metadata or they
_ONLY_ send metadata and no query, etc. So this always uses the results
of `result-metadata-async` so it can go through that logic and we always
clobber.

And remember that if we only update the result_metadata when it is
non-nil (`update-card-async!`). The end result is that switching from a
PATCH model where you expect the full object to a PUT where things can
be optional has lots of footguns. Some examples:
- only updating the query
- updating the query along with edits to the metadata on a card (throw
the metadata away)
- updating the query along with edits on a dataset (save the metadata
but blend it into the fresh computation of the query's metadata)
- updating the query but not sending along the metadata. Need to grab
from the db and blend in.
- Sending invalid metadata, changing the description. We want to toss
away the metadata but save the description.

Thanks for the review/approvals.

Adds an endpoint to the cards which are basically materially like dashboard/:id/copy, only it's for card/:id/copy - POST to it, get a copy of that entity.

* Add has-user-setup setting

- iff no user is setup aka has-user-setup == false, then redirect to /setup
- when set from the env variable the setup-token is completely
  immutable, so can remove clear-token!

* make /api/setup only setup accounts when not has-user-setup

- react router changes to respect 'has-user-setup' setting

- include docstring for print-setup-url

* allow tests to setup multiple users via POST /api/setup

* remove unused function hasSetupToken

* Adds a test to enforce a single creation only

- we now throw an ex-info with status-code 403 when has-user-setup and
  /api/setup route is hit (with error message mentioning the route)
- more code review responses

- fixup test to not delete users, (or set locale to spanish...)

* addressing code review concerns

- *disallow... -> *allow... for the dynamic var name
- has-user-settings test: do not assume the test db has users in it
- update test

* Add question link to the 'stopped working' email template.

* Adjusted copy from alert email to correctly indicate how to unsubscribe

* Trim whitespace around commas in schema filtering code

Change impl of `schema-pattern->re-pattern` so that strings are trimmed before being joined on `|`

Add test to confirm

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Support overriding ROWCOUNT for SQL Server

Add new "ROWCOUNT Override" connection property for `:sqlserver`, which will provide a DB-level mechanism to override the `ROWCOUNT` session level setting as needed for specific DBs

Change `max-results-bare-rows` from a hardcoded constant to a setting definition instead, which permits a DB level override, and move the former constant default to a new def instead (`default-max-results-bare-rows`)

For `:sqlserver`, set the DB-level setting override (if the connection property is set), via the `driver/normalize-db-details` impl

Add test to confirm the original scenario from #9940 works using this new override (set to `0`)

Move common computation function of overall row limit to the `metabase.query-processor.middleware.limit` namespace, and invoke it from execute now, called `determine-query-max-rows`

Add new clause to the `determine-query-max-rows` function that preferentially takes the value from `row-limit-override` (if defined)

Waterfall can now have pretty colors and goals now. Y-axis positions reverted back to the way Dan liked less but is more congruent to the dynamic visualizations.

* Add program_name connectionAttribute for MySQL

Set the program_name connection attribute when creating a MySQL connection, to the same value that we do for other drivers - `config/mb-version-and-process-identifier`

Updating tests

Add test for mysql to ensure `:connectionAttributes` aren't emitted when they are given in `additional-options`

Set `:connectionAttributes` directly in the `jdbc-spec` map (which in turn becomes a property for the JDBC connection) instead of putting in `additional-options`

Fixing some bugs in `additional-options->map`

Relevant links:

* https://dev.mysql.com/doc/refman/8.0/en/performance-schema-connection-attribute-tables.html#performance-schema-connection-attributes-available
* https://mariadb.com/kb/en/performance-schema-session_connect_attrs-table/
* http://mysqlblog.fivefarmers.com/2013/03/26/connection-attributes-in-mysql-5-6/

* Apply schema inclusion/exclusion filtering to sql-jdbc drivers

Update `sql-jdbc` namespaces to handle schema inclusion/exclusion patterns when filtering schemas

Add new generic schema inclusion/exclusion test for sql-jdbc drivers that define the property

Update Snowflake and Redshift driver manifests to include schema filtering property

Create `db-details->schema-filter-patterns` util fn to turn DB details into the inclusion/exclusion patterns

Move schema inclusion/exclusion filtering code to new namespace (since it's not strictly used by `:sql-jdbc` derived drivers)

Move existing tests accordingly

Add schema inclusion/exclusion check to the new `filtered-syncable-schemas` multimethod (and updating docstring)

Change `:redshift` impl of `filtered-syncable-schemas` to call the `:sql-jdbc` version instead

Use new multimethod instead for `filtered-syncable-schemas`, and have default impl of `syncable-schemas` call that

Mark `syncable-schemas` as deprecated and include notes on the new method (and update driver markdown file accordingly)

* Linting improvements: 0 errors in src + shared/src

* satisfy other linter

* Change label for "enable-nested-queries"

* Fix query types compatibility

* Add utility to check if DB supports nested queries

* Accept store initial state for `renderWithProviders`

* Hide model button when nested queries disabled

* Handle nullish database in modeling utils

* Fix var

Static viz errors are of basically two kinds:
1. Someone futzed up their query, which we can't do anything about
2. We futzed up the rendering, which is our fault.
Display those two errors differently.

`add-alias-info`: recognize `:field` clauses in join source queries to be the same if it has a `:temporal-unit` (#19789)

* Backport Debug QP improvements from #19754

* Enable test

* "Fuzzy" matching when looking for Fields in join source queries

* Add/update tests

* Test fix 

* Remove trailing whitespace

* Fix a few docstrings

* Postgres: avoid unnecessary casts of DATE or TIMESTAMPTZ to TIMESTAMP

* Update src/metabase/util/honeysql_extensions.clj

* Update test/metabase/driver/postgres_test.clj

* Don't add `:join-alias` to `:field` clauses for fields that came from source query's source table

* Some test fixes 

* Test fixes 

* Test fix 

* Test fix 

* Add test for #7487

* Fix error message

* Update test for new error message

* RIP checksum

Now that users can edit the metadata the checksum prevents that and
would throw it away when edited. We'll treat the metadata as valid if it
conforms. The UI isn't really expected to modify type information anyways.

* Ns checker is a harsh disciplinarian

* Extract `fields` variable in dataset editor

* Allow extra onChange callback to form fields

* Only show metadata sidebar when field is selected

* Add onChange callback to semantic type picker

* Add action for overwriting results metadata

* Connect essential fields to form

* display_name
* description
* semantic_type

TBD: mapped database column for native datasets

* Fix missing import

* Fix sidebar inputs loosing focus while typing

* Ensure focused field is in sync

* Refactor FK and currency to be form fields

* Use `field.settings` to keep formatting changes

* Remove "display_as" from form definition

* Connect remaining fields

* Remove CurrencyPicker in favor of formatting input

* Fix incomplete field metadata

Results metadata doesn't contain everything we need for the editor.

'Global' metadata is the most complete, though it shouldn't be muted while in the editor, only once a user saves the changes

Now we only keep focused field ref in state and derive the metadata by merging global metadata and query's results metadata. Query results metadata is mutated by the editor, so merging them both will do the job

* Fix ViewSidebar prop type

* Fix prop type warnings

* Add default values for some fields

* Connect MappedFieldPicker

* Fix MappedFieldPicker closes after loading fields

Sidebar dependant on IDFields was rerendering once additional metadata was loaded, causing DataSelector to unmount

Fixed by moving IDFields subscription to FKTargetPicker directly (the only component that actually needs them)

* Hide metadata popovers in dataset editor

* Disable saving invalid dataset editor changes

Disables "Save changes" button if:
* there is at least one field with empty display name
* the underlying query is empty

* Add `onChangeSetting` prop to ColumnSettings

The existing `onChange` passes the whole settings object as an argument. On the other hand, `onChangeSetting` passes only the changed setting which is useful for storing metadata diff

* Keep field metadata diff in redux

* Ignore column width reset events in metadata editor

* Preserve metadata diff between query edits

* A few important changes:

- ensure that we don't throw away metadata and return an empty channel
in the card api if the query hasn't changed
- ensure api/dataset uses `:metadata/dataset-metadata`
- hydrate fields on api/table/card__id/query_metadata so that it has
target and fk information
- preserve id so this information can be recovered†

†: When a user enters an id we preserve it, but it also gets confusing
if that's the actual field's id or the user-identified id. This is
important when we drill down (and probably lots of other places too if
they start assuming an id means they know exactly what is going on).

A situation based on the orders table:

Suppose you make a native question

```sql
SELECT "PUBLIC"."ORDERS"."ID" AS "ID",
  "PUBLIC"."ORDERS"."USER_ID" AS "user_id_bro",  -- note this is renamed
  "PUBLIC"."ORDERS"."PRODUCT_ID" AS "PRODUCT_ID",
  "PUBLIC"."ORDERS"."SUBTOTAL" AS "SUBTOTAL",
  "PUBLIC"."ORDERS"."TAX" AS "TAX",
  "PUBLIC"."ORDERS"."TOTAL" AS "TOTAL",
  "PUBLIC"."ORDERS"."DISCOUNT" AS "DISCOUNT",
  "PUBLIC"."ORDERS"."CREATED_AT" AS "CREATED_AT",
"PUBLIC"."ORDERS"."QUANTITY" AS "QUANTITY"
FROM "PUBLIC"."ORDERS"
LIMIT 15
```

In my db, the "USER_ID" column is field 3 and points at field 22. Our db
knows that field 3 is called "USER_ID". But in the source query that
column is actually called "user_id_bro". And lots of places assume that
if there's a field's `:id` it knows what the name is. This is obviously
incorrect here. It is a very important distinction that this column "is
basically field 3 USER_ID" and this is literally the field "USER_ID".

So if you do a drilldown based on this information, the FE will issue a
query to "select field 3 from card__127" which expands to roughly
"select USER_ID from (select ... USER_ID as user_id_bro ...)" and this
query obviously will fail.

* Don't put nil on a channel, clean ns

i honestly don't know why the results-metadata and encrypt deps were
there. I didn't add them but they just got caught in the linter for some
reason. I don't understand it.

* Update tests to include target and has_field_values

* Correct final test

* Select enough information for FK to work in UI

* Pass mapped column's properties to BE

* Qualify which semantic_type FK/PK we remove

Previously always removed these semantic types. Now only remove when
they do not have an id there. One thing I'm confused about though is
that native fields should never have a semantic type anyways. There's no
way for one to end up there, certainly not FK or PK right? So i want to
delete this entire method because as I understand it it's not possible
to trigger its conditions except for the new user-entered information,
in which case we want to preserve it. But I have no idea why
Mr. Chesterton built this function so I don't want to delete it.

Bug reports and connecting them back to this change is so hard. Because
it can be things like "native query with an aggregation named "blah"
bucketed by month break when you drilldown into this column" and that's
because it includes a semantic_type that breaks something. It's all very
at-a-distance so hard to trace back to this.

* Fix failed queries handling

* Support custom column metadata editing

* Hide empty sections in semantic type picker

* Remove redundant import

* Fetch database ID fields in FKTargetPicker

* Allow aliasing EntityObjectLoader's entity prop

* Reset DataSelector's state when ID props change

* Fix MappedFieldPicker

* Fixed saved question picker regression

* Select description and settings when merging result_metadata

Annotate columns allows for settings to be preserved from
datasets. Example

```clojure
;; shows a little bar in the UI. quite nice
:settings {:show_mini_bar true, :decimals 0},
```

These settings were correctly preserved in annotate.clj but dropped in
the final merging in result_metadata.clj. Fun times in metadata land

* Let dataset revisions preserve metadata

* Only wrap form fields with custom onChange handler

This prevents form fields from extra re-renders

* Fix summarize sidebar behavior

* Fix e2e test

* Update cypress expectations

Previously expected really generic placeholder text in search boxes. But
in `api/table/card__id/query_metadata` we hydrate native columns with
`:has_field_values` so that when a user identifies an underlying field
on datasets, we have extra information.

This change comes about because the hydrate for `:has_field_values`
doesn't hit the db and look for field values but uses a heuristic.  And
this heuristic works for native fields and finds that they are
searchable because when they are text or textlike and improves the
search bar.

Code from models/field.clj below

```clojure
(defn- is-searchable?
  "Is this `field` a Field that you should be presented with a search widget for (to search its values)? If so, we can
  give it a `has_field_values` value of `search`."
  [{base-type :base_type}]
  ;; For the time being we will consider something to be "searchable" if it's a text Field since the `starts-with`
  ;; filter that powers the search queries (see `metabase.api.field/search-values`) doesn't work on anything else
  (or (isa? base-type :type/Text)
      (isa? base-type :type/TextLike)))

(defn- infer-has-field-values
  "Determine the value of `has_field_values` we should return for a `Field` As of 0.29.1 this doesn't require any DB
  calls! :D"
  [{has-field-values :has_field_values, :as field}]
  (or
   ;; if `has_field_values` is set in the DB, use that value; but if it's `auto-list`, return the value as `list` to
   ;; avoid confusing FE code, which can remain blissfully unaware that `auto-list` is a thing
   (when has-field-values
     (if (= (keyword has-field-values) :auto-list)
       :list
       has-field-values))
   ;; otherwise if it does not have value set in DB we will infer it
   (if (is-searchable? field)
     :search
     :none)))

(defn with-has-field-values
  "Infer what the value of the `has_field_values` should be for Fields where it's not set. See documentation for
  `has-field-values-options` above for a more detailed explanation of what these values mean."
  {:batched-hydrate :has_field_values}
  [fields]
  (for [field fields]
    (when field
      (assoc field :has_field_values (infer-has-field-values field)))))
```

* Ignore `has_field_values` for field literals

Field literals are field using a column title instead of a numeric ID. They're usually coming from native or nested queries.

Metadata propagation changes introduced for data models, made the BE send a guessed `has_field_values` property for field literals too. But searching through possible field values isn't properly supported yet and it's difficult to prevent `has_field_values` being sent to the FE, so this makes the FE ignore it for field literals

* Fix E2E test

* Fix field literal check

* Ensure `:has_field_values` only goes on fields with ids

when a user selects an id to back a native field, we want to hydrate the
field_value information and the target info. But _ONLY_ when they have
ids. The hydration for `:has_field_values` works on a heuristic, not by
hitting the db and seeing if we have field values and will give
information about native fields that do not have backing field information.

* Ensure no annotations on numeric field ids

we put the vector id on each field when there's no numeric id field. I
have no idea why and the comment `;; TODO -- what????` explains the
thinking there. So only annotate where we have numeric ids (fields that
we know how to identify fields in the db) and not when we have a made up
vector id

* Revert changes to FieldValuesWidget

* Fix character case

* Show `has_field_values` selector when ID is known

* Undo the testing for indiscriminant hydration

previously had put target and has_field_values on every native field
which doesn't work well for the UI. Had to clean up the remaining tests
to no longer expect it

* Select `:dataset` when creating a new question

when duplicating a question, we send over this info and need to select
it from the existing question.

* Ensure card is saved with boolean dataset info

largely just for testing which does not send a `:dataset false` property
in some cases. In regular usage there should always be an initial value

* Remove dead code

* Replace TODO with a meaningful comment

* Type Field type

* Select only what we need on the source card

Co-authored-by: dan sutton <dan@dpsutton.com>

* Better error messages on collection permissions errors

* Test fixes 

* Add test for #15538

* Add extra validation for native source queries

* Add test for sql-source-query validation

* Fix Migration number 420.00-070 => 42.00-070

* Remove onUpdateSQL

* Log a shortened version of the enterprise token

- when checking that an enterprise token is valid
- test captures the logs, and checks for no token
- in the test, regrettably we print the logs we need to capture

* fix namespace requires order

* remove "Using this URL to check ..." log line

- and update test

* respond to review comments

- no circular dependency with mt, so use that
- "fa3e"*16 -> random token

* print the shortened token one time, and remove let