* Remove :bigquery driver

Add "migration" to convert existing Database instances from :bigquery to :bigquery-cloud-sdk (with error log if using outdated OAuth mechanisms), by way of `normalize-db-details`

Removing references to :bigquery from various places in the code

* Remove from modules/drivers/deps.edn

* Remove stuff from CircleCI that is running the old driver

Remove `driver-switch-test` since there is no practical way to run this anymore (since we can't initialize the old driver)

Remove buggy redef of `isa?` from `semantic-type-migration-tests`

Co-authored-by: Cam Saul <github@camsaul.com>

Pursuant to issue #20332 - it was blanking out without settings set before

Rework how the remappings middleware matches remapped to/from columns for explicit external (FK) remaps (#20009)

* Fix #9236 [WIP]

* Revert unneeded changes

* Only merge namespaced :options into result info

* Revert unneeded changes

* Everything is working 

* Revert unneeded change

* Clean namespaces

* Add `:test` profile to `namespace-checker` to suppress log messages.

* PR feedback

* Fix namespaces

* Retain property file path when revisiting database edit page

Ensuring that the file path (if that is the secret type) is returned via the API response to the admin, so the UX on the form edit page is better

This involved a lot of refactoring to clean up some stuff:

* removed the population of inferred secret properties (ex: last updated timestamp and creator ID) from `handle-db-details-secret-prop!`, since that stores those values into the details blob, defeating some of the purpose of normalizing secrets into a separate table
* moved the logic for injecting such inferred values to a new function, `expand-inferred-secret-values`, which lives within the secret model namespace
* delegating existing calls that require such expansion to the new function
* adding a new helper function, `reduce-over-details-secret-values`, to handle some common logic that is now showing up in a few places (reducing over db-details on secret values and doing some kind of manipulation)

Adding new database API test to ensure that the file path value is, in fact, returned from the API

*** Code:
- ignore hidden tables and archived cards/dashboards
- correct docstring to indicate we are only getting 5 not 10 as
previously claimed
- in query, set limit to 8 in the case we get archived or non-existing
things and then take 5 in memory
- select extra properties from models that we need for this
determination

*** Tests:
- for mysql we were inserting views with a thread/sleep of one second
between each. Just compute timestamps seconds apart and do them all at
once. Should save 7 seconds off mysql test runs. verified temporal
spacing works with

```
 MB_DB_CONNECTION_URI="mysql://username:password@localhost:3306/cli" \
 clj -X:dev:test :only metabase.api.activity-test
```
- cleanup these activities we were creating with `mt/with-model-cleanup`
which introduces a big looking diff. But checking diff with whitespace
shows its just an indentation and then add the extra keys needed for
identifying which are archived (:archived and :visibility_type for cards
and tables respectively)

* Rework QP.dashboard to expect dashcard-id

* Update /api/dashboard endpoints

* Rework public endpoints for running queries for Dashcards

* Tweak parameter order for Dashboard endpoints for consistency

* Pulse should pass Dashcard ID

* Embed should pass Dashcard ID

* Unskip cypress test

* Update API documentation

* Update endpoints and pass dashcardId to endpoint fns

* Update cypress tests that reference previous endpoint

* Update placement of pivot in endpoint

* Update another endpoint + more e2e fixes

* update QB to hold onto dashcardId like it does with dashboardId

* Address PR feedback

Co-authored-by: Dalton Johnson <daltojohnso@users.noreply.github.com>

Right now, datasets are all cards. In the future this could change so
record the original model of the dataset as a card. In the future, maybe
we care about the underlying model and this allows us to at least query
and distinguish

Metabase 43 QP middleware overhaul part 3: convert all pre-processing middleware to the new pattern (#19937)

* Metabase 43 QP middleware overhaul (part 1)

* Make linter happy

* Test fix

* Fix merge

* Remove perms key after normalization

* Remove unused namespace

* Metabase 43 QP middleware overhaul part 2

* Sort namespaces

* Convert all pre-processing middleware to the new pattern

* Sort namespace

* Minor code cleanup

* Address PR feedback

* Sort namespaces

* Split QP middleware into different groups

* Sort namespaces

* Fix the helper `question` function

* Wait for the card query to load before typing

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Fix regression breaking support for user:password@host Postgres connection strings

* Actually we need to handle either Postgres OR MySQL

* Fix the helper `question` function

* Wait for the card query to load before typing

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

Disable remapping when syncing Field values. Handle human readable values in `GET /api/field/:id/values` (#19901)

* Disable remapping when syncing Field values. Handle human readable values in GET /api/field/:id/values

* Don't require `:has_field_values` to be list to actually return anything.

* Make sure GET /api/field/:id/values returns :field_id or it breaks the FE

* Fix add-dimension-projections middleware not respecting disable-remaps? key

More stuff in #18676. Supposed to be the BE for #20004 Just continuing static viz work. Stacked graphs only have a simple viz-settings flag turned on so making sure that gets to FE bit.

No implementation of normalized stacks.

* Unused bindings cleanup

`clj-kondo --lint src:shared/src > lint` and then just work my way
through it. Note that there are warnings about deprecations that we will
need to get rid of as well. Most likely by turning off that warning
until we are ready to tackle the project.

* More unused and a sort ns

* Removing unused svg helpers that eastwood is complaining about

* Restore accidentally deleted `begin!` impl for xlsx

* clean ns

* clean ns

* Last few unused bindings

* kondo checks for empty docstrings

* Little more cleanup

Don't parse MB_DB_CONNECTION_URI; use javax.sql.DataSource everywhere for app DB connection sources (#19970)

* Use javax.sql.DataSource everywhere for app DB connection sources

* Sort namespaces

* Un-remove Kondo warning

* Minor tweaks to support 'postgres:' ...

* Remove ConnectionDataSource

* Add warning about PG without sslmode back

* PR feedback

* Sort namespaces

* Remove top-level calls to trs (not allowed because app DB isn't initialized yet anyway)

* Test fix  (MySQL)

* MBQL :expressions should use strings for keys

* Make some more tests ^:parallel

* Update dox

* Test fix

* Sort namespaces

Still pursuant to #18676

1. Funnel was previously not doing lato font for the bold bit because visx's text doesn't take weights, it takes styles
2. Columns can be nontrivial with respect to funnels, which was an oversight
3. For some reason the first try for fixing 2 also impinged upon values being strings for funnels, which breaks things at SVG level
4. And let's just shove in a nit for the display of trends if the previous value and current value are the same

CI was broken inasmuch as it wasn't running, so master was broken without us knowing it. Thankfully, to the best of our knowledge, it was a couple of nits. Fixes master.

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

Only attempt to read values that were actually set on the client side

Add test to confirm specific combination from reported issue works

* Improve lifecycle of metadata

Previously we have assumed that if valid metadata is passed into the
card create and udpate, that that is valid metadata for the query. There
are a few scenarios where this can fail, but they all boil down to
editing the query after running it. Examples:

- when creating a query, come up with a query, visualize, realize you
wanted to not select all columns, edit to not select those, possibly
select a few extra, and then save. The saved metadata might only have
the columns from when you ran the query and not after you saved it.
- edit an existing query and save without rerunning. Again, adding and
removing some columns.

How it affects:
- if you remove columns, the preserved metadata can cause us to attempt
to select columns that are no longer in that question, leading to errors
for nested queries
- if you add columns, the query builder won't show the new columns until
that information is populated in the metadata by running the query.

Lifecycle now:
- should be encapsulated in `api.card/result-metadata-async`. But we
only allow metadata to remain if is is valid metadata and the query has
not been edited. If the query has been edited (and new queries always
satisfy this since the previous query will be nil) we rerun the query to
get the metadata. No more drift. However, when dealing with datasets we
will attempt to blend the existing metadata into the newly computed
metadata so that metadata edits can persevere.

Other changes:

- removed an extra indirection in the card api that made it harder to
reason about when to recompute metadata
- moved the `combine-metadata` function into a util namespace and gave
some more generic parameter names
- deleted two tests that are no longer relevant after removing the
checksum on metadata. One had to do with how we hash floats vs ints
which is no longer relevant since we aren't hashing at all (RIP
checksum) and one checked that we accepted "valid" checksum-ed
metadata.
- new test that updating a card will update its metadata by rerunning
the query
- when validating the checksum, do not call s/validate since this WILL
THROW. Instead check `(nil? (s/check ...))`.

* Normalize query before comparing for changes

When comparing db vs api-sent query, need to normalize otherwise they
will always be different and our optimization will never kick in

* Fix ns

* Cleanup after ourselves

* Move `combine-metadata` util functions to a better namespace

* Allow for no query in api/card/:id patch

Cypress tests would do a

```javascript
  describe("simple mode", () => {
    beforeEach(() => {
      cy.request("PUT", "/api/card/1", {
        name: "Orders Model",
        dataset: true,
      });
    });
```

The metadata testing stuff would always use the query from the PUT to
check whether to dump or retain the metadata. In these instances, it
would dump the metadata and then i guess ask the qp for the metadata for
a nil query. Similar scenario if anyone uses the API to update
descriptions, etc.

The method is PUT not PATCH, but in the saving function we have

```clojure
(u/select-keys-when
 card-updates
 :present #{:collection_id :collection_position :description :cache_ttl :dataset}
 :non-nil #{:dataset_query :display :name :visualization_settings
            :archived :enable_embedding :embedding_params
            :result_metadata})
```

So we kinda have a PATCH anyways.

* Normalize metadata before blending

* Ensure api blends metadata correctly

When blending the metadata, need to update the field_ref to keyword the
original. Otherwise we are attempting to match up the field with ref
["field" "subtotal"] against [:field "subtotal"] and they don't match.

* Normalize metadata from api correctly

if you just call normalize/normalize it doesn't know not to change
`:field_ref` -> `:field-ref` so it gets out of whack. Need to pass it
the function appropriate for that level.

Also add a test that you can update _just_ the metadata. This actually
already worked for a subtle reason, and that is possibly buggy. The
reasoning is that the result_metadata is clobbered conditionally now
instead of unconditionally. If the async metadata stuff returns nil it
won't clobber and then whatever was sent over the wire goes in.

* Ensure previous metadata is there

When updating the query of a dataset, handle the case where just the
query is sent, and not the whole card. If the client didn't send the
previous metadata, we did not look it up in the database

* Card api always uses metadata from `result-metadata-async`

Originally this did

    (assoc card-data :result_metadata (a/<! result-metadata-chan))

But I changed that so it only optionally did so when that channel had
information. The problem with this is that if there was :result_metadata
on the original request but the channel returned nil for some reason,
whatever was in the request was going in the db. This includes invalid
metadata, non-json strings, what ever. Obviously bad.

But we need to handle PATCHs where clients don't send metadata or they
_ONLY_ send metadata and no query, etc. So this always uses the results
of `result-metadata-async` so it can go through that logic and we always
clobber.

And remember that if we only update the result_metadata when it is
non-nil (`update-card-async!`). The end result is that switching from a
PATCH model where you expect the full object to a PUT where things can
be optional has lots of footguns. Some examples:
- only updating the query
- updating the query along with edits to the metadata on a card (throw
the metadata away)
- updating the query along with edits on a dataset (save the metadata
but blend it into the fresh computation of the query's metadata)
- updating the query but not sending along the metadata. Need to grab
from the db and blend in.
- Sending invalid metadata, changing the description. We want to toss
away the metadata but save the description.

Thanks for the review/approvals.

Adds an endpoint to the cards which are basically materially like dashboard/:id/copy, only it's for card/:id/copy - POST to it, get a copy of that entity.

* Add has-user-setup setting

- iff no user is setup aka has-user-setup == false, then redirect to /setup
- when set from the env variable the setup-token is completely
  immutable, so can remove clear-token!

* make /api/setup only setup accounts when not has-user-setup

- react router changes to respect 'has-user-setup' setting

- include docstring for print-setup-url

* allow tests to setup multiple users via POST /api/setup

* remove unused function hasSetupToken

* Adds a test to enforce a single creation only

- we now throw an ex-info with status-code 403 when has-user-setup and
  /api/setup route is hit (with error message mentioning the route)
- more code review responses

- fixup test to not delete users, (or set locale to spanish...)

* addressing code review concerns

- *disallow... -> *allow... for the dynamic var name
- has-user-settings test: do not assume the test db has users in it
- update test

* Add question link to the 'stopped working' email template.

* Adjusted copy from alert email to correctly indicate how to unsubscribe

* Trim whitespace around commas in schema filtering code

Change impl of `schema-pattern->re-pattern` so that strings are trimmed before being joined on `|`

Add test to confirm

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Support overriding ROWCOUNT for SQL Server

Add new "ROWCOUNT Override" connection property for `:sqlserver`, which will provide a DB-level mechanism to override the `ROWCOUNT` session level setting as needed for specific DBs

Change `max-results-bare-rows` from a hardcoded constant to a setting definition instead, which permits a DB level override, and move the former constant default to a new def instead (`default-max-results-bare-rows`)

For `:sqlserver`, set the DB-level setting override (if the connection property is set), via the `driver/normalize-db-details` impl

Add test to confirm the original scenario from #9940 works using this new override (set to `0`)

Move common computation function of overall row limit to the `metabase.query-processor.middleware.limit` namespace, and invoke it from execute now, called `determine-query-max-rows`

Add new clause to the `determine-query-max-rows` function that preferentially takes the value from `row-limit-override` (if defined)

Waterfall can now have pretty colors and goals now. Y-axis positions reverted back to the way Dan liked less but is more congruent to the dynamic visualizations.