Updating schema connection property to have proper display name

Updating display name for database to be "Schema"

Fixing `driver/can-connect?` impl in `presto_jdbc.clj` to actually use the correct driver, and dissoc `:engine` before calling the `sql-jdbc.conn` method

Adding test for the `test-database-connection` method, which is what led to the error from the UI

* Fix some Eastwood failures

* Fix a lot of Eastwood errors now that it runs against test namespaces [ci skip]

* Run Eastwood against test namespaces [WIP]

* Bump Eastwood version to fix some errors in Eastwood itself

* Fix another lint warning

* Fix test failure

* More test fixes 

* Fix all the warnings!

* Test fix 

* Bump Eastwood GH action timeout to 20 minutes (!)

Implement JDBC based Presto driver

Adding new Presto JDBC driver using the PrestoDB JDBC driver from `https://github.com/prestodb/presto`

Marking the old Presto driver as being `superseded-by` the new one

Pulling out common Presto code into new presto-common driver (modeled after the relationship between, ex: `googleanalytics` and `google)`

Putting common QP/HoneySQL logic into the new (abstract) :presto-common driver

Updating :presto driver to extend from the new common driver and only adding HTTP/REST API related methods

Adding implementation of Presto JDBC driver, named :presto-jdbc, extending from :presto-common and :sql-jdbc

Using com.facebook.presto/presto-jdbc as underlying JDBC driver dependency (since this is explicitly for Presto clusters, as opposed to Trino)

Adapting code from the existing Presto driver where appropriate

Adding new dependency-satisfied? implementation for :env-var, to allow for a plugin to require an env var be set, and making the Presto JDBC driver depend on that (specifically: `mb-enable-presto-jdbc-driver`)

Adding CircleCI configuration to run against the newer Presto (0.254) Docker image

Adding explicit ordering in a few tests where it was missing

Fixing presto-type->base-type (timestamps were being synced as :type/Time because the regex pattern was wrong)

Add tx/format-name test implementation for :presto-jdbc to lowercase table name

Make modified test Oracle friendly

Fixing bug parsing the `[:zone :time]` case within `metabase.util.date-2.parse/parse-with-formatter`; the offset is nil so it can't be passed directly in this case, so use the `standard-offset` fn (which was moved from `date-2` to `common` to get a standard offset for that zone

Fixing more test failures by adding explicit ordering

Changing sync to check whether the driver supports foreign keys before attempting to sync those (since drivers might throw an exception if attempting to check)

Moving some common test dataset functionality between :presto and :presto-jdbc to a new test.data ns for :presto-common

Adding HoneySQL form for :count-where, since we have to explicitly give a higher precision Decimal in order for Presto to not reduce the precision in results

Put limit within subquery for `expression-using-aggregation-test` (since ordering from subquery is not guaranteed in Presto)

Adding impls for ->prepared-substitution to handle substitutions for native query params

Adding HoneySQL impls for `mod` (to do as "mod(x,y)" and `timestamp` (since it's a function with no parens to invoke) functions

Adding various `sql.qp/date` impls that use the `AT TIME ZONE` operator to account for report tz, and make bucketing tests happy

Remove regionid from plugin YAML file

Add new driver multimethod to `normalize db-details`, and implementing to handle this particular "migration" within Snowflake

Adding hook from database model post-select to invoke the new multi

Adding test

Update Oracle driver to set the v$session.program property on the connection, which appears in the PROGRAM column of the v$session table

Update tests accordingly

* Fix Oracle/Vertica comparison against empty string

* Remove extra println

* Add test for ::empty-string-is-null

* Cleanup from dump-load-entities-test (#16281)

Updating dump-load-entities-test test so that dump dir is actually deleted at the end

* Revamp Cypress custom commands (#16292)

* Extract UI custom commands

* Extract user related custom commands

* Extract overwrites

* Extract permissions related custom commands

* Extract API custom commands

* Extract helpers custom commands

* Extract database related custom commands

* Rename `helpers` group to `visibility`

* Rename custom command `isInViewport` to `isRenderedWithinViewport`

* Unskip repro for #15119 (#16297)

* Create composite Cypress custom command `cy.createQuestionAndDashboard` (#16294)

* Add composite Cypress custom command `createQuestionAndDashboard`

* Refactor the existing test to use new Cypress custom command

* Refactor repro for #13062 using the new Cypress custom command

* Use the `describe` block instead of using `forEach` for complex repro

* Add space between test cases

* #12629 Repro: Human-readable number formatting not working properly (#16231)

* Fix `nosql` user's email address (#16306)

* Generate correct SQL for columns from joins inside other joins or source queries (#16254)

* Fix joining a join inside a nested queries (#12928)

* Code cleanup & dox

* Tests for #13649

* Test fixes 



* Remove unit tests snapshots (#16321)

* Remove snapshots from `Calendar.unit.spec.js`

* Remove snapshots from `DashCard.unit.spec.js`

* Remove snapshots generated for "internal" components

* Remove unused import

* Remove snapshot related command from `scripts`

* Remove `noSnapshotTest` from internal components

* Remove `react-test-renderer`

* filter nulls out of histograms (#16345)

* #15993 Repro: Click behavior with filter pass-thru does not show filters defined on question (#16342)

* #16334 Repro: Click Behavior targeting a question with filter mapped causes the visualization type to change (#16343)

* Add repro for #16334

* Expand test assertion

* #16327 Repro: Double binning menu for date fields when using Saved Question (Native) (#16359)

* #16322 Repro: "Custom mapping" is only available, when "A list of all values" is set (and after a browser refresh) (#16361)

* #16226 Repro: LDAP/Email settings gets cleared if validation fails (#16364)

* Add repro for #16226

* Extract email settings into a separate file

* Add repro for #16226 (Email)

* switch to column settings when sidebar is already open (#16368)

* align labels on non-bars in combo charts (#16369)

* #16378 Repro: Cannot enable JWT authentication (#16384)

* Fix JS error on JWT settings form (#16394)

* fix default updateSettings function in SettingsBatchForm

* unskip repro

* #12128 Repro: Can't change label on empty row value (#16426)

* Post-merge fix

* Post-merge fix 2
I didn't solve this merge conflict properly in the original merge.

Co-authored-by: Jeff Evans <jeff303@users.noreply.github.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>
Co-authored-by: Paul Rosenzweig <paulrosenzweig@users.noreply.github.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>

* MBQL: support case statements inside arithmetic operators (#15107)

* SQLite test fix 

* More test fixes 

* Another test fix

* Presto test fix

* Fix #16299 MongoDB empty results when filtering against array

* Ok, we still need to use $expr if the VALUE in the expression isn't simple

* More test fixes 

* More fixes 

Previously just used JDBC. However, JDBC will merrily execute queries on empty file locations for SQLite, because it'll create a new SQLite file. This one checks that there exists a SQLite file already to be querying in the first place

* Fix joining a join inside a nested queries (#12928)

* Code cleanup & dox

* Tests for #13649

* Test fixes 

* Add yyyymmddhhss coercions to type system

* Implementations for h2/mysql/postgres for yyyymmddhhss bytes and strings

* Mongo and oracle

* Adding checksum on migration it said was bad

* import OffsetDateTime

* Redshift and bigquery

* snowflake expectations

* sql server yyyymmddhhmmss. have to format the string then parse

sqlserver lacks a parse function that takes a format string, they just
take an integer that specifies a predefined format string. So we have
to make the string into the right format then parse.

* presto yyyymmddhhmmss

* Test byte conversions

* Remove errant `mt/set-test-drivers!`

* Remove sqlite, change keyword for multiple native types in def

the spec couldn't handle different shapes under the same keyword. so
just use :natives {:postgres "BYTEA"} :native "BYTEA"

* Make schema work with different shape maps

* hx/raw "'foo'" -> hx/literal "foo" and remove checksums

* _coercion_strategy -> _coercion-strategy

* Handle coercion hierarchy for :Coercion/YYYYMMDDHHMMSSBytes->Temporal

We made fieldname friendliness, people didn't like it, we tried to fix it, people still didn't like it, now we're just making the fieldnames all mildly unfriendly again. Not like, snake_case unfriendly, but Pascal Case With Spaces unfriendly

* Fix Druid queries with multiple filters if some but not all are temporal

* Fix Druid queries with temporal Fields that don't have a :temporal-unit

* Test fix 

* Fix syntax errors

* Query result metadata should return the :unit from the source query when applicable

* Test fix 

* Clarify comment in test

* More test fixes 

* Test fix 

* Port legacy data type migrations -> Liquibase

* Fix migration IDs

* Field type validation/error handling

* Have semantic type fallback to nil

* Fix semantic-type-migrations-test

* Fix migrations

* Revert accidental changes

* Semantic/* & Relation/* ancestor types

* Fix stray Relation/PK and Relation/FKs

* Semantic/* and Relation/* ancestor types

* cljs test fix

* Fix :require

* FE test fixes 

* Test fixes 

* prettier

* PR  f e e d b a c k

* Use medium size CircleCI image for Presto to prevent all the OOMs

* Backport dir-locals tweaks from hierarchy PR

* Redshift: only sync the test schema (faster CI and fix failure)

* Better error handling for sync in tests

* Revert accidental commit

* Redshift test fixes 

* Fix Druid queries with multiple filters if some but not all are temporal

* Fix syntax errors

* Druid Docker image [WIP]

* More Druid Docker stuff [WIP]

* Test fixes 

* Remove maxRowsQueuedForOrdering from query, don't think we need it

* Force test Druid host -> localhost for now

* Update sync tests for updated test data

* Revert unused changes

Similar to #15664, update the applicationName to include the local process identifier

Updating test accordingly

Add new namespace to manage diagnostic info about the DW DB connection pools

Adding new atom/fn/macro to wrap the capture of diagnostic info, modeled after the call counting functionality of toucan.db

Capturing diagnostic info from a new `datasource-with-diagnostic-info!` fn in the execute namespace, and calling that from the various drivers (instead of `datasource`)

Adding to log message generated for the "stats" portion of the line logged via middleware to output this information

Set c3p0 dataSourceName property to include Metabase specific info

Change data-warehouse-connection-pool-properties multimethod to include database as 2nd param

Setting the dataSourceName c3p0 property to have the format db-<N>-<D>-<DB> where <N> is the DW database ID, <D> is the driver name, and <DB> is the database name from the db details

Adding test

Setting c3p0 dataSourceName for the app DB pool as well

* Backend feature flag for new field filters

* Feature flag new parameter options

When the "field-filter-operators-enabled?" flag is disabled we do the following:
1. Replace new operator options with old category and location/city, etc., options
   in the PARAMETER_OPTIONS list found in metabase/meta/Parameter.js
2. Hide numbers section in the PARAMETER_SECTIONS list found in
   metabase/meta/Dashboard.js
3. Return args as-is in the mapUIParameterToQueryParameter function
   found in metabase/meta/Parameter.js

React/UI code handles both old options and new options so doesn't need
to change. Old parameter types like "category" and "location/city" are
treated like "string/=" in the UI but retain their own parameter type
when used to send a new query.

* Fix FE issues caused by meta/Parameter refactor

* mock the field operator param flag to make tests pass

* add/fix cypress tests

* fix import in ParametersPopover

* update widget tag type

* Enable field filter operators for cypress tests

* Question marks are questionable

* Conditionally use category or string/= if field filters are enabled

* rmv mocks where we don't need them

* rmv mock from chained-filters test

* env vars as string in project.clj, alignment

Co-authored-by: dan sutton <dan@dpsutton.com>

* Revert name case changes

* Test fix 

* Test fix 

Only attempt to sync schemas that have some/any possible permissions

Make syncable-schemas multimethod, whose base implementation is the same as the previous private fn

Override syncable-schemas for Redshift to add a filtering step to the transducer pipeline to check for the schema privilege, by calling the HAS_SCHEMA_PRIVILEGE Redshift function to check for USAGE permission

Making all-schemas function public since it's now invoked from the Redshift driver

Fixing redshift-types-test to remove order flakiness

Adding test that confirms that a real schemas with no permissions aren't synced

Fix druid throwing "Invalid type marker byte 0x3c for expected value token" errors if limit is 1048575  (#15452)

* Better error logging for druid

* Apparently Druid queries don't like the number 1058575

* Test fix 

* Bump a few dep versions and add missing exclusions

* Fix export timezones; fix Excel time values

* Text fixes & other improvements

* Test fix 

* Move some XLSX/CSV specific tests into appropriate test namespaces

* Fix SQLite parsing of date-only values

* address PR feedback

* Add missing "is" assertions to various tests

* Fixing save-card-with-empty-result-metadata-test by simply asserting on API response

* Fix more failing tests

* Switch row-type-agnostic-test back to strings for expected

* Modify check for convert-id-to-string to also include keys whose :base_type is :type/Number, in addition to :type/Integer to fix failures in Oracle and Snowflake

Fix Oracle SSL tests

Define new test for Oracle SSL connectivity, in oracle_test.clj, similar to how things work in mysql_test.clj

Add new test util macro, with-env-keys-renamed-by, to support running tests with environ keys temporarily renamed

Using new test macro from both MySQL and Oracle SSL connectivity tests

Removing now unneeded be-tests-oracle-ssl-ee CircleCI job

Removing now unneeded test-selector parameter for test-driver orb in CircleCI config.yml

Updating JVM_OPTS to use a trust store that starts with cacerts and adds the RDS root CA, rather than one only
containing the RDS root CA

* No more old-style tests

* Presto test fix 

* Remove location sub-categories

These sub-categories are only for filtering the list of options
when mapping a parameter filter to a field. Since we are introducing
operator types as a sub-category of location, city/zip/etc. just get in
the way.

* add number section + number/string operator subtypes

Light refactor of meta/Dashboard changes

rmv 'all-options' options (for now)

* add/update parameter type icons

* pass operator to ParameterFieldWidget + show input per operator field

* Add operator helper fns that aren't dependent on fields/tables

* Make operator prop optional

fix date filter err

* add combined  name for native question filter widget type list

Otherwise, a field that matches both "Location" and "Category"
options will show duplicate "Starts with" options, etc. Now,
that'll look like "Category - Starts with" and "Location - Starts with"

* correct some unused prop/arg passing

* Convert location/category parameter types to string for query

location/category don't mean anything to BE but we use them for
"reasons" on the FE. Reasons are legacy reliance on unique-ness of
the parameter.type value, primarily.

* operators in backend

* Remove errant tap>

* Docstrings and differing numbers in tests in some dbs

* Make unary private so docstring checker ~shuts-up~ is satisfied

* Don't parse arguments to operators params

they were coming in just fine from the FE as numeric or string
types. no need to ensure strings everywhere and parse here

* add max-width to PopoverPicker

* rmv unused value

* use combinedName on dashboard parameters

* fix parameter to mbql code

* Ensure = operator filter popovers have no label

This is to match "old" style of parameter popover

* Update Cypress tests to reflect new parameter flow

fix cypress dashboard parameter tests

Fix more cy tests

* Don't call fk/joinAlias on ExpressionDimension

The methods don't exist on ExpressionDimension class.

This doesn't make them work (yet), but it prevents the app
from crashing.

* Namespace doc and remove unnecessary comment

* tap>-spy in dev

* first pass at substitution of new operators in native

* Docstring on wrap-value-literals-in-mbql to appease the gods

* variadic equality operators (string/= number/=)

* move functions out of component file

* Pass parameter object to tag editor for use in default input

We should inline this input eventually because it looks ugly.

* map parameters in Questions to correct type

* continue to pass janky fake parameter for text/number tags

* mongo native substitution

* variadic not-equals for string and number

* Docstring and use correct function to make errors

* add number/between dash param cy test

* Update function name to better reflect behavior

* Add unit tests for paramer/operator util fns

add unit tests for parameter util functions

add unit test for operator util fns

* add variadic string 'is not' param operator option

* Modify operator parameter display labels

don't append 'matches exactly' to location/category parameters

label tweaks

Update cy tests to reference correct label name

rmv it.only

* Desugar mongo parameters mbql

desugaring makes for a bit more verbose query but that's ok.This
change was done to ensure that we negated regexes in a correct way,
and to do so we always return the string version. This ensures that it
can be json/generate-string'd for native parameters or left as
datastructures and sent to monger

* Cleanup stale comments and fixup docstring for consistency

* Arglists metadata on defmulti and denude some threaded forms

* add single arity number tag predicate to variable filter

* add Location operators to fix parameter<->filter mapping

For question filters to work we need for the new parameter operators
to be supported by "location" fields in all areas of the app.

* Don't show coords for param number widgets

I don't think we want to support all the various number operators when
dealing with coordinates, so in order to avoid that I'm preventing the
mapping of number parameter operator to coord fields.

* prevent mapping of tags to non-equal operators

while possibly useful to end users, this needs more UI work on the
native question side of things.

* Ensure parameter values are wrapped in an array

When an = operator is mapped to a field AND a tag, it ends up not being
wrapped in an array due to the TextWidget (I think).

ensure parameter value is an array

ensure number params have an array value

* Sort imports correctly

clojure-lsp used to do this incorrectly (sorting `[` before `j`) and
that has now been fixed

Co-authored-by: dan sutton <dan@dpsutton.com>

Bump mssql-jdbc version from 7.4.1.jre8 to 9.2.1.jre8

Bump plugin version accordingly

Override prepared-statement and statement multimethods for :sqlserver to not set holdability at the statement level

Fixing inaccurate log statements

Port the metabase.mbql utility namespaces to ./shared/ so they can be used on both the frontend and backend.

* First pass using coercions

* Coercions

* Handle effective_type coercion_strategy in test data sets

* special-type -> semantic type in sample db

```clojure
user> (def config (metabase.db.spec/h2 {:db (str "/Users/dan/projects/clojure/metabase/resources/sample-dataset.db"
                                                 ";UNDO_LOG=0;CACHE_SIZE=131072;QUERY_CACHE_SIZE=128;COMPRESS=TRUE;"
                                                 "MULTI_THREADED=TRUE;MVCC=TRUE;DEFRAG_ALWAYS=TRUE;MAX_COMPACT_TIME=5000;"
                                                 "ANALYZE_AUTO=100")}))
user> (jdbc/execute! config ["UPDATE _metabase_metadata
                        SET keypath = 'PEOPLE.ZIP.semantic_type'
                        WHERE keypath = 'PEOPLE.ZIP.special_type'" ])
[1]
user> (jdbc/execute! config ["UPDATE _metabase_metadata
                        SET keypath = 'REVIEWS.BODY.semantic_type'
                        WHERE keypath = 'REVIEWS.BODY.special_type'" ])
[1]
```

* Correct mismatch in validation preventing sync

* fixing up alternative date tests

* More passing tests

* Tests for values, nested queries, fetch metadata

* tests

* tests passing

* Fixup mongo qp for coercions

locally i have some failing tests that are off by 1 errors:

Fail in compile-time-interval-test

[36m:mongo[0m Make sure time-intervals work the way they're supposed to. [:time-interval $date -4 :month] should give us something like Oct 01 2020 - Feb 01 2021 if today is Feb 17 2021

expected: [{$match {$and [{:$expr {$gte [$date {:$dateFromString {:dateString 2020-10-01T00:00Z}}]}} {:$expr {$lt [$date {:$dateFromString {:dateString 2021-02-01T00:00Z}}]}}]}} {$group {_id {date~~~day {:$let {:vars {:parts {:$dateToParts {:date $date}}}, :in {:$dateFromParts {:year $$parts.year, :month $$parts.month, :day $$parts.day}}}}}}} {$sort {_id 1}} {$project {_id false, date~~~day $_id.date~~~day}} {$sort {date~~~day 1}} {$limit 1048576}]

  actual: [{"$match"
            {"$and"
             [{:$expr {"$gte" ["$date" {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
              {:$expr {"$lt" ["$date" {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}
           {"$group"
            {"_id"
             {"date~~~day"
              {:$let
               {:vars {:parts {:$dateToParts {:date "$date"}}},
                :in {:$dateFromParts {:year "$$parts.year", :month "$$parts.month", :day "$$parts.day"}}}}}}}
           {"$sort" {"_id" 1}}
           {"$project" {"_id" false, "date~~~day" "$_id.date~~~day"}}
           {"$sort" {"date~~~day" 1}}
           {"$limit" 1048576}]
    diff: - [{"$match"
              {"$and"
               [{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-10-01T00:00Z"}}]}}
                {:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-02-01T00:00Z"}}]}}]}}]
          + [{"$match"
              {"$and"
               [{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
                {:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}]

* ee fixes

* UI to set coercion type

* Don't need to populate effective-type here

it actually has knock on effects:
- does more work now as almost every field has an update to do in
`add-extra-metadata`
- we have databases that have state that we don't create. druid for
example has stuff to mimic the dataset in tqpt/with-flattened-dbdef on
checkins but we don't actually create this. And our dbdef has a field
called "date" that is not present in the druid db, so if we attempt to
add metadata it fails and kills the rest of the metadata that we add.
- tests need this metadata to be present and the error causes field
visibilities (for example) to not be set

* Docstrings on shared lib

* Add effective and coercion to redshift expectations

* Fixup google analytics

* Derecordize instead of recordize the expectation

object details didn't work out well here. they added way more stuff
from the db than what is flowing through here.

```clojure
  actual: {:field
           {:name "DATE",
            :parent_id nil,
            :table_id 69,
            :base_type :type/Date,
            :effective_type :type/Date,
            :coercion_strategy nil,
            :semantic_type nil},
           :value {:type :date/all-options, :value "past5days"}}
    diff: - {:field
             {:description nil,
              :database_type "VARCHAR",
              :fingerprint_version 0,
              :has_field_values nil,
              :settings nil,
              :caveats nil,
              :fk_target_field_id nil,
              :custom_position 0,
              :active true,
              :last_analyzed nil,
              :position 1,
              :visibility_type :normal,
              :preview_display true,
              :database_position 0,
              :fingerprint nil,
              :points_of_interest nil}}
```

Object defaults adds quite a bit of stuff such that we'd be dissoc'ing
more than we are currently adding in

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Optimize SQL Server GROUP BY and ORDER BY with :year, :month, or :day bucketing

* Avoid unneeded CAST() statements

* More tests 

* Revert unneeded changes

* Test fixes 

* Don't count MySQL and Postgres driver namespaces for test coverage

* Address feedback from @dpsutton; test fixes

* Test fixes