* Add has-user-setup setting

- iff no user is setup aka has-user-setup == false, then redirect to /setup
- when set from the env variable the setup-token is completely
  immutable, so can remove clear-token!

* make /api/setup only setup accounts when not has-user-setup

- react router changes to respect 'has-user-setup' setting

- include docstring for print-setup-url

* allow tests to setup multiple users via POST /api/setup

* remove unused function hasSetupToken

* Adds a test to enforce a single creation only

- we now throw an ex-info with status-code 403 when has-user-setup and
  /api/setup route is hit (with error message mentioning the route)
- more code review responses

- fixup test to not delete users, (or set locale to spanish...)

* addressing code review concerns

- *disallow... -> *allow... for the dynamic var name
- has-user-settings test: do not assume the test db has users in it
- update test

* Trim whitespace around commas in schema filtering code

Change impl of `schema-pattern->re-pattern` so that strings are trimmed before being joined on `|`

Add test to confirm

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Support overriding ROWCOUNT for SQL Server

Add new "ROWCOUNT Override" connection property for `:sqlserver`, which will provide a DB-level mechanism to override the `ROWCOUNT` session level setting as needed for specific DBs

Change `max-results-bare-rows` from a hardcoded constant to a setting definition instead, which permits a DB level override, and move the former constant default to a new def instead (`default-max-results-bare-rows`)

For `:sqlserver`, set the DB-level setting override (if the connection property is set), via the `driver/normalize-db-details` impl

Add test to confirm the original scenario from #9940 works using this new override (set to `0`)

Move common computation function of overall row limit to the `metabase.query-processor.middleware.limit` namespace, and invoke it from execute now, called `determine-query-max-rows`

Add new clause to the `determine-query-max-rows` function that preferentially takes the value from `row-limit-override` (if defined)

* Add program_name connectionAttribute for MySQL

Set the program_name connection attribute when creating a MySQL connection, to the same value that we do for other drivers - `config/mb-version-and-process-identifier`

Updating tests

Add test for mysql to ensure `:connectionAttributes` aren't emitted when they are given in `additional-options`

Set `:connectionAttributes` directly in the `jdbc-spec` map (which in turn becomes a property for the JDBC connection) instead of putting in `additional-options`

Fixing some bugs in `additional-options->map`

Relevant links:

* https://dev.mysql.com/doc/refman/8.0/en/performance-schema-connection-attribute-tables.html#performance-schema-connection-attributes-available
* https://mariadb.com/kb/en/performance-schema-session_connect_attrs-table/
* http://mysqlblog.fivefarmers.com/2013/03/26/connection-attributes-in-mysql-5-6/

* Apply schema inclusion/exclusion filtering to sql-jdbc drivers

Update `sql-jdbc` namespaces to handle schema inclusion/exclusion patterns when filtering schemas

Add new generic schema inclusion/exclusion test for sql-jdbc drivers that define the property

Update Snowflake and Redshift driver manifests to include schema filtering property

Create `db-details->schema-filter-patterns` util fn to turn DB details into the inclusion/exclusion patterns

Move schema inclusion/exclusion filtering code to new namespace (since it's not strictly used by `:sql-jdbc` derived drivers)

Move existing tests accordingly

Add schema inclusion/exclusion check to the new `filtered-syncable-schemas` multimethod (and updating docstring)

Change `:redshift` impl of `filtered-syncable-schemas` to call the `:sql-jdbc` version instead

Use new multimethod instead for `filtered-syncable-schemas`, and have default impl of `syncable-schemas` call that

Mark `syncable-schemas` as deprecated and include notes on the new method (and update driver markdown file accordingly)

* Add test for #15863

* Remove dead code

Static viz errors are of basically two kinds:
1. Someone futzed up their query, which we can't do anything about
2. We futzed up the rendering, which is our fault.
Display those two errors differently.

`add-alias-info`: recognize `:field` clauses in join source queries to be the same if it has a `:temporal-unit` (#19789)

* Backport Debug QP improvements from #19754

* Enable test

* "Fuzzy" matching when looking for Fields in join source queries

* Add/update tests

* Test fix 

* Remove trailing whitespace

* Fix a few docstrings

* Postgres: avoid unnecessary casts of DATE or TIMESTAMPTZ to TIMESTAMP

* Update src/metabase/util/honeysql_extensions.clj

* Update test/metabase/driver/postgres_test.clj

* Don't add `:join-alias` to `:field` clauses for fields that came from source query's source table

* Some test fixes 

* Test fixes 

* Test fix 

* Test fix 

* Add test for #7487

* Fix error message

* Update test for new error message

* The return of parallel tests

* Cleanup

* `partial=`

* Diff improvements

* Update test/metabase/test_runner/effects.clj

* RIP checksum

Now that users can edit the metadata the checksum prevents that and
would throw it away when edited. We'll treat the metadata as valid if it
conforms. The UI isn't really expected to modify type information anyways.

* Ns checker is a harsh disciplinarian

* Extract `fields` variable in dataset editor

* Allow extra onChange callback to form fields

* Only show metadata sidebar when field is selected

* Add onChange callback to semantic type picker

* Add action for overwriting results metadata

* Connect essential fields to form

* display_name
* description
* semantic_type

TBD: mapped database column for native datasets

* Fix missing import

* Fix sidebar inputs loosing focus while typing

* Ensure focused field is in sync

* Refactor FK and currency to be form fields

* Use `field.settings` to keep formatting changes

* Remove "display_as" from form definition

* Connect remaining fields

* Remove CurrencyPicker in favor of formatting input

* Fix incomplete field metadata

Results metadata doesn't contain everything we need for the editor.

'Global' metadata is the most complete, though it shouldn't be muted while in the editor, only once a user saves the changes

Now we only keep focused field ref in state and derive the metadata by merging global metadata and query's results metadata. Query results metadata is mutated by the editor, so merging them both will do the job

* Fix ViewSidebar prop type

* Fix prop type warnings

* Add default values for some fields

* Connect MappedFieldPicker

* Fix MappedFieldPicker closes after loading fields

Sidebar dependant on IDFields was rerendering once additional metadata was loaded, causing DataSelector to unmount

Fixed by moving IDFields subscription to FKTargetPicker directly (the only component that actually needs them)

* Hide metadata popovers in dataset editor

* Disable saving invalid dataset editor changes

Disables "Save changes" button if:
* there is at least one field with empty display name
* the underlying query is empty

* Add `onChangeSetting` prop to ColumnSettings

The existing `onChange` passes the whole settings object as an argument. On the other hand, `onChangeSetting` passes only the changed setting which is useful for storing metadata diff

* Keep field metadata diff in redux

* Ignore column width reset events in metadata editor

* Preserve metadata diff between query edits

* A few important changes:

- ensure that we don't throw away metadata and return an empty channel
in the card api if the query hasn't changed
- ensure api/dataset uses `:metadata/dataset-metadata`
- hydrate fields on api/table/card__id/query_metadata so that it has
target and fk information
- preserve id so this information can be recovered†

†: When a user enters an id we preserve it, but it also gets confusing
if that's the actual field's id or the user-identified id. This is
important when we drill down (and probably lots of other places too if
they start assuming an id means they know exactly what is going on).

A situation based on the orders table:

Suppose you make a native question

```sql
SELECT "PUBLIC"."ORDERS"."ID" AS "ID",
  "PUBLIC"."ORDERS"."USER_ID" AS "user_id_bro",  -- note this is renamed
  "PUBLIC"."ORDERS"."PRODUCT_ID" AS "PRODUCT_ID",
  "PUBLIC"."ORDERS"."SUBTOTAL" AS "SUBTOTAL",
  "PUBLIC"."ORDERS"."TAX" AS "TAX",
  "PUBLIC"."ORDERS"."TOTAL" AS "TOTAL",
  "PUBLIC"."ORDERS"."DISCOUNT" AS "DISCOUNT",
  "PUBLIC"."ORDERS"."CREATED_AT" AS "CREATED_AT",
"PUBLIC"."ORDERS"."QUANTITY" AS "QUANTITY"
FROM "PUBLIC"."ORDERS"
LIMIT 15
```

In my db, the "USER_ID" column is field 3 and points at field 22. Our db
knows that field 3 is called "USER_ID". But in the source query that
column is actually called "user_id_bro". And lots of places assume that
if there's a field's `:id` it knows what the name is. This is obviously
incorrect here. It is a very important distinction that this column "is
basically field 3 USER_ID" and this is literally the field "USER_ID".

So if you do a drilldown based on this information, the FE will issue a
query to "select field 3 from card__127" which expands to roughly
"select USER_ID from (select ... USER_ID as user_id_bro ...)" and this
query obviously will fail.

* Don't put nil on a channel, clean ns

i honestly don't know why the results-metadata and encrypt deps were
there. I didn't add them but they just got caught in the linter for some
reason. I don't understand it.

* Update tests to include target and has_field_values

* Correct final test

* Select enough information for FK to work in UI

* Pass mapped column's properties to BE

* Qualify which semantic_type FK/PK we remove

Previously always removed these semantic types. Now only remove when
they do not have an id there. One thing I'm confused about though is
that native fields should never have a semantic type anyways. There's no
way for one to end up there, certainly not FK or PK right? So i want to
delete this entire method because as I understand it it's not possible
to trigger its conditions except for the new user-entered information,
in which case we want to preserve it. But I have no idea why
Mr. Chesterton built this function so I don't want to delete it.

Bug reports and connecting them back to this change is so hard. Because
it can be things like "native query with an aggregation named "blah"
bucketed by month break when you drilldown into this column" and that's
because it includes a semantic_type that breaks something. It's all very
at-a-distance so hard to trace back to this.

* Fix failed queries handling

* Support custom column metadata editing

* Hide empty sections in semantic type picker

* Remove redundant import

* Fetch database ID fields in FKTargetPicker

* Allow aliasing EntityObjectLoader's entity prop

* Reset DataSelector's state when ID props change

* Fix MappedFieldPicker

* Fixed saved question picker regression

* Select description and settings when merging result_metadata

Annotate columns allows for settings to be preserved from
datasets. Example

```clojure
;; shows a little bar in the UI. quite nice
:settings {:show_mini_bar true, :decimals 0},
```

These settings were correctly preserved in annotate.clj but dropped in
the final merging in result_metadata.clj. Fun times in metadata land

* Let dataset revisions preserve metadata

* Only wrap form fields with custom onChange handler

This prevents form fields from extra re-renders

* Fix summarize sidebar behavior

* Fix e2e test

* Update cypress expectations

Previously expected really generic placeholder text in search boxes. But
in `api/table/card__id/query_metadata` we hydrate native columns with
`:has_field_values` so that when a user identifies an underlying field
on datasets, we have extra information.

This change comes about because the hydrate for `:has_field_values`
doesn't hit the db and look for field values but uses a heuristic.  And
this heuristic works for native fields and finds that they are
searchable because when they are text or textlike and improves the
search bar.

Code from models/field.clj below

```clojure
(defn- is-searchable?
  "Is this `field` a Field that you should be presented with a search widget for (to search its values)? If so, we can
  give it a `has_field_values` value of `search`."
  [{base-type :base_type}]
  ;; For the time being we will consider something to be "searchable" if it's a text Field since the `starts-with`
  ;; filter that powers the search queries (see `metabase.api.field/search-values`) doesn't work on anything else
  (or (isa? base-type :type/Text)
      (isa? base-type :type/TextLike)))

(defn- infer-has-field-values
  "Determine the value of `has_field_values` we should return for a `Field` As of 0.29.1 this doesn't require any DB
  calls! :D"
  [{has-field-values :has_field_values, :as field}]
  (or
   ;; if `has_field_values` is set in the DB, use that value; but if it's `auto-list`, return the value as `list` to
   ;; avoid confusing FE code, which can remain blissfully unaware that `auto-list` is a thing
   (when has-field-values
     (if (= (keyword has-field-values) :auto-list)
       :list
       has-field-values))
   ;; otherwise if it does not have value set in DB we will infer it
   (if (is-searchable? field)
     :search
     :none)))

(defn with-has-field-values
  "Infer what the value of the `has_field_values` should be for Fields where it's not set. See documentation for
  `has-field-values-options` above for a more detailed explanation of what these values mean."
  {:batched-hydrate :has_field_values}
  [fields]
  (for [field fields]
    (when field
      (assoc field :has_field_values (infer-has-field-values field)))))
```

* Ignore `has_field_values` for field literals

Field literals are field using a column title instead of a numeric ID. They're usually coming from native or nested queries.

Metadata propagation changes introduced for data models, made the BE send a guessed `has_field_values` property for field literals too. But searching through possible field values isn't properly supported yet and it's difficult to prevent `has_field_values` being sent to the FE, so this makes the FE ignore it for field literals

* Fix E2E test

* Fix field literal check

* Ensure `:has_field_values` only goes on fields with ids

when a user selects an id to back a native field, we want to hydrate the
field_value information and the target info. But _ONLY_ when they have
ids. The hydration for `:has_field_values` works on a heuristic, not by
hitting the db and seeing if we have field values and will give
information about native fields that do not have backing field information.

* Ensure no annotations on numeric field ids

we put the vector id on each field when there's no numeric id field. I
have no idea why and the comment `;; TODO -- what????` explains the
thinking there. So only annotate where we have numeric ids (fields that
we know how to identify fields in the db) and not when we have a made up
vector id

* Revert changes to FieldValuesWidget

* Fix character case

* Show `has_field_values` selector when ID is known

* Undo the testing for indiscriminant hydration

previously had put target and has_field_values on every native field
which doesn't work well for the UI. Had to clean up the remaining tests
to no longer expect it

* Select `:dataset` when creating a new question

when duplicating a question, we send over this info and need to select
it from the existing question.

* Ensure card is saved with boolean dataset info

largely just for testing which does not send a `:dataset false` property
in some cases. In regular usage there should always be an initial value

* Remove dead code

* Replace TODO with a meaningful comment

* Type Field type

* Select only what we need on the source card

Co-authored-by: dan sutton <dan@dpsutton.com>

* Better error messages on collection permissions errors

* Test fixes 

* Add test for #15538

* Add extra validation for native source queries

* Add test for sql-source-query validation

* Remove hardcoded project-id from `:bigquery-cloud-sdk` tests

Pull out common helper fn to extract project-id from credentials

Update `bigquery_cloud_sdk_test.clj` to have a var that loads the service-account-json from the env var, then uses aforementioned function to determine project-id

Change sensitive-data-redacted-test to use an arbitrary/fake project-id to remove any sort of confusion

* Log a shortened version of the enterprise token

- when checking that an enterprise token is valid
- test captures the logs, and checks for no token
- in the test, regrettably we print the logs we need to capture

* fix namespace requires order

* remove "Using this URL to check ..." log line

- and update test

* respond to review comments

- no circular dependency with mt, so use that
- "fa3e"*16 -> random token

* print the shortened token one time, and remove let

Nested Queries Overhaul 2022: Split logic for determining appropriate table & column aliases out of SQL QP (#19384)

* big query schemas settings

* fix be specs

Trivial trends previously broke. Now they default to basically the ordinary single value viz.

Our email tests can be a bit brittle. Sometimes we assert the first
email in the email inbox is some password reset email, but a race makes
it such that its a "logged in from new ip" email. Othertimes, we might
assert that there are only emails to a particular user but maybe another
test might have triggered an email to another user.

We often might just want two predicates: did the user receive an email
with this subject, or did the user receive an email with this body.

This simply adds those.

It should be safe from races as it can handle emails to multiple users
and multiple emails to the same user.

* Comparison for goals depends on alert_above_goal and if progressbar.

This commit sets up the timeseries predicate and writes tests for what I think the correct behaviours should be:

timeseries, alert above -> (<= goal-value value)
 - (<= 5.9 6) true  -> alert sent
 - (<= 6 6)   true  -> alert sent
 - (<= 6.1 6) false -> alert not sent

progressbar, alert above -> (<= goal-value value)
 - (<= 5.9 6) true  -> alert sent
 - (<= 6 6)   true  -> alert sent
 - (<= 6.1 6) false -> alert not sent

timeseries, alert below -> (>= goal-value value)
 - (>= 6.1 6) true   -> alert sent
 - (>= 6 6)   true   -> alert sent. NOTE: this is to match prior behaviour, but I don't understand why it is this way?
 - (>= 5.9 6) faluse -> alert not sent

progressbar, alert below -> (> goal-value value)
 - (> 6.1 6) true   -> alert sent
 - (> 6 6)   false  -> alert not sent. NOTE: this is what should fix the bug (#10899)
 - (> 5.9 6) faluse -> alert not sent

I think there may be a cleaner way to write and present the tests, so the next commit(s) will address that.

* Simplified Tests for goal-met? predicate.

The goal-met predicate is tested directly with mock data to make the tests more readable.

The change to `metabase.pulse/goal-met?` fixes #10899 while preserving the goal-met behaviour for any non-progress
goal. The behaviour should now be as follows:

```
| Timeseries? | alert_above? | goal | val | goal-met? |
+-------------+--------------+------+-----+-----------+
|     f       |      t       |  5   |  4  |     f     |
|     f       |      t       |  5   |  5  |     t     |
|     f       |      t       |  5   |  6  |     t     |
|     f       |      f       |  5   |  4  |     t     |
|     f       |      f       |  5   |  5  |     f     | <--- this is new behaviour
|     f       |      f       |  5   |  6  |     f     |

|     t       |      t       |  5   |  4  |     f     |
|     t       |      t       |  5   |  5  |     t     |
|     t       |      t       |  5   |  6  |     t     |
|     t       |      f       |  5   |  4  |     t     |
|     t       |      f       |  5   |  5  |     t     |
|     t       |      f       |  5   |  6  |     f     |
```

Otherwise, alerts should be triggered as before.

* added issue number to test

* fixed alignment issue

* swapped goal and value around in comparison for readability

* Fixing nits

* Timeseries and progress bar now follow same logic

I asked for clarification about when alerts fire in #ama-product in Metabase's Slack. There was consensus around the
idea that 'reaches the goal' does generally imply 'meets the goal', so we can use the same comparators between
progress bar and timeseries goals now.

This simplifies the goal-met? function again because it's no longer necessary to check if the given data is a
timeseries goal.

* Adjusted text in goal line alert modal

* ran prettier

Add functions to describe-database namespace for performing inclusive/exclusive schema filtering

Add tests for these functions in isolation

Update `:bigquery-cloud-sdk` driver and QP code to remove the single `dataset-id` conn-param

Changing :bigquery-cloud-sdk` driver to use dataset inclusion/exclusion filters instead

Updating `add.cy.spec.js` Cypress test to account for new structure

Add data upgrade (via `driver/normalize-db-details` implementation) to change `dataset-id` property to inclusion filter

Add test to confirm data upgrade

Adding server side expansion of `:schema-filters` type connection property

* Misc code cleanup

* PR feedback

* Debugging QP Improvements

* Appease namespace linter

* Backport additional debug QP changes from #19384

* Test fix 

* Exclude archived collections from tree view

This is a bit more nuanced that it might first appear. The bug report is
the permissions editor for collections is showing child collections
which are archived. But the api sends back ALL collections, regardless
of archival state. So the FE knows to omit top-level archived
collections but not archived child collections.

So we thought about just having the FE remove all archived ones, but
best to remove it from the response. One worry is that we're not sure if
any callers want archived collections in the tree view or not. So right
now they are removed with a `?exclude-archived=true` query param, but if
we can demonstrate that no callers of the tree api ever want archived
items we can make this the behavior, rather than an optional behavior.

To verify that the tree api is sending back all collections, you can
check

```clj
collection=> (collection/collections->tree
               {}
               [{:archived false
                 :name "foo"
                 :id 1
                 :location "/"}
                {:archived true
                 :name "foo"
                 :id 2
                 :location "/1/"}
                {:archived true
                 :name "its archived"
                 :location "/"
                 :id 3}])
;; we have always sent back archived collections, both top level and children
({:archived false,
  :name "foo",
  :id 1,
  :location "/",
  :children ({:archived true,
              :name "foo",
              :id 2,
              :location "/1/",
              :children ()})}
 {:archived true,
  :name "its archived",
  :location "/",
  :id 3,
  :children ()})
```

* hide archived collections from the collections permissions page (#19616)

* fix incorrect issue number

Co-authored-by: Alexander Lesnenko <alxnddr@users.noreply.github.com>
Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>