* Add SSL keystore and truststore secret properties to Oracle driver YAML files


Update Oracle driver to set keystore and truststore connection options from secret values

Adding new `select-keys-sequentially` helper function for dealing with assertions on transformed conn props

Add new function to the secret namespace to return a lighter weight "secret map", and remove that functionality from the `handle-db-details-secret-prop!`, so that it can be reused from connection testing

Modifying CircleCI to set the corresponding truststore secret conn prop vars instead of JVM-level truststore settings, as the test had been doing previously

Expand Oracle test connection details to incorporate all SSL related properties, and also make it a function instead of delayed def (so changes can be picked up later)

Misc. fixes in Oracle driver YAML files

Update `exception-classes-not-to-retry` to include SSLHandshakeException

Fix misc issues in secret and database model code

Update CircleCI config to use correct secret based env var keys and values

Add *database-name-override* dynamic var in `metabase.test.data.interface` to allow for DB name overriding

Fix up ssl connectivity test so all parts pass successfully

Get rid of crazy with-redefs to swap in the existing test-data's schema, and instead, just dropping the DB if it was created under a different schema (similar to what H2 does)

Co-authored-by: Gustavo Saiani <gustavo@poe.ma>

.git is necessary so that some later steps (determining Uberjar version)
can work correctly.

Make sure that the built Uberjar contains translations etc.
Also, build with "large" resource_class on Circle CI.

This reverts commit adb2f715 as it broke Uberjar builds on CircleCI.

Make sure that the built Uberjar contains translations etc.

* Extract cc reproductions from `question` group

* Extract custom-column spec

* Extract custom column help-text spec

* Extract custom column typing-suggestion spec

* Extract custom column error-feedback spec

* Expand typing-suggestions spec

* Consolidate repros

* Extract custom column data-type spec

* Remove references to `cy.server` from custom-column spec

* Extract custom column expression-editor spec

* Remove duplicate tests

* Extract and unskip repro for #13289

* Register custom column helpers

* Use CC helper in tests

* Move related tests to help-text spec

* Enable "custom-column" e2e group in CI

* Delete a duplicate repro for #13289

* Remove obsolete test

* Expand error-feedback spec

* Extract and expand repro for #14843

* Fix linter errors

* Check also for master branch

* Create `mysql-8` snapshot

* Use `mysql-8` snapshot in tests

* Extract repro for #15342

* Relocate mysql `add` spec

* Extract mysql GUI query tests

* Relocate mysql `query` spec

* Extract repro for #12445

* Remove `metabase-db` from Cypress

* Update Cypress README

* Create `mongo-4` snapshot

* Use `mongo-4` snapshot in tests

* Restore and authenticate

* Relocate mongo `line` spec

* Relocate mongo `native` spec

* Relocate mongo `add` spec

* Extract repro for #13097

* Relocate mongo `query` spec

* Remove mongo from `metabase-db`

- a56b640d broke Percy workflow on master and introduced a major inconvenience for developers who wish to run Cypress locally. In both cases, it's required to have all supported QA DB docker images running at all times. Otherwise, the Cypress snapshot creation phase breaks.

This PR:
- uses the existing ENV var to conditionally skip qa db snapshots
- this doesn't affect CircleCI workflow - it merely makes life easier for developers running Cypress locally and for a Percy workflow that runs on GitHub Actions (where we still don't have QA docker images available)

* Create `postgres-12` snapshot

* Use `postgres-12` snapshot in tests

* Relocate postgres `add` spec

* Relocate postgres `sandboxes` spec

* Relocate postgres `native` spec

This actually didn't have anything to do with the native questions.
We simply used native editor because it was easier to reproduce the issue.

* Relocate postgres `permissions` spec

* Relocate postgres `query` spec

* Relocate postgres `custom-column` spec

* Remove postgres from `metabase-db`

* Run percy (sanity check) visual tests in CircleCI

* Update path for visual tests

* Make `--folder` flag optional

* visual tests docs

Co-authored-by: Alexander Lesnenko <alxnddr@users.noreply.github.com>

The node_modules folder was being persisted in the first step (checkout) as it's doing the building of the static viz package. This had an impact in the fe-deps step, so wiping the folder and doing a clean install should make the trick

* Rename file to make it more specific

* Extract dashboard sql filters into separate folder and CI group

* Add short descriptions to dashboard filters repros

* Extract dashboard sql filters helpers (data objects)

* Try reverting mongo QA image back to 4.0

* Update mongo query assertion

Mongo custom expressions now are turned on for mongo 5.0 and after only. One idiosyncrasy is that we don't really support the BSON ID format at all in our typesystem despite having it in there, but both of my attempts to get them to work with the native mongo pipeline commands bounced because they really aren't strings, they're BSON ID's. We do rely on the pipeline commands heavily which is the reason for the version requirement.

New BigQuery Driver

Create new :bigquery-cloud-sdk driver using the google-cloud-bigquery library instead, and whose source is adapted from the :bigquery driver

https://cloud.google.com/bigquery/docs/reference/libraries

Marking existing :bigquery driver as deprecated, and superseded-by the new one (bigquery-cloud-sdk)

Update new driver and query processor code to use newer Google SDK

Switch test data loading over to use new API

Add project-id connection property to override the value from the service account JSON, and use it as part of qualified names in the query processor if set

Updating google driver so its libraries are compatible with the newer ones used in BigQuery

Update date bucketing tests to skip :bigquery-cloud-sdk (new driver) where :bigquery is skipped

Update `with-bigquery-fks` to take in the driver, since there are now multiple ones

Adding test to confirm that overriding project-id for a public BQ project works (sync and query)

Fixing a bunch of whitespace alignment errors in tests

* Backend SVG rendering proof of concept [ci skip]

* Update cssbox to 5.0.0

* Render bar, line, and pie charts in js to svg

sparkline is now done in js, bar is now recognized and done in js, new
:categorical/donut as well

* Remove api route for render

* pass along render-type, not hardcoded to :inline

* Move bar chart above sparkline and remove line check

In order to introduce the bar chart type need it above the sparkline
check since it is otherwise the same except for display property of
the card. But lots of tests assume that this will get hit with a nil
display type set in testing so remove checking for `:line` allows all
the testing cases to hit the right type

* Fix tests now that bar graphs aren't html but images

* Include attachments for bar charts

* Move over to in-tree bundle

* Force everything [ci noskip]

trying to ensure that the built jar includes the newer
"resources/frontend_client/app/dist/lib-static-viz.bundle.js"

* Run `yarn build-static-viz` in backend-deps in CI

this js file is now a hard dependency of the backend so it fits in
this tsk. All such things that depend on the backend sources will need
it. Makes me think perhaps we want a checked in version but i'm not
sure yet.

* Look on classpath not filesystem for js bundle [ci noskip]

* Move yarn build-static-viz into the checkout step

* License information for antlr4-runtime

* create attachment for categorical donuts

* add ordinal legend to donuts (#17177)

* set widths of html image and svg image to 1200

* Revert "add ordinal legend to donuts (#17177)"

This reverts commit 1eb81d2e.

* Helper functions to render html easily

* readme in dev

* readme ensure that static viz bundle exists

* Cleanup ns after removing proxy

* Donut chart colors and legend (#17251)

* use external color map for fill per dimension

* Add support new color legend for donut

* Ensure text doesn't appear as link

entire thing is actually the body of a link tag for emails but we want
a decent text color rather than a default link color

* use chart colors from https://stats.metabase.com/_internal/colors



* Make checkers happy

- remove unused imports
- add a docstring
- don't shadow fn with a local

* cleanup ns import

* Remove reflective call

* Cleanup ns on correct branch

Co-authored-by: dan sutton <dan@dpsutton.com>

* X-axis: just use (approx) 5 ticks to avoid overlapping labels (#17287)

* increase gap between arcs (#17271)

* Set rendering hints on html->image

* ignore width for now and make them larger

* Ns deprecation and some cleanup

* make namespace checker happy

* Simple tests for detecting chart type

* Rename from poc

* Tests for scalar/smartscalar

* cleanup js svg namespace a bit

* Tests of svg engine

* ns sorting after renaming

* Unify our two different js engine usages

settled on the js context. Has typed returns `(.asString ^Value ...)`
instead of perhaps capturing std out?
https://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/Value.html



Context is a bit more friendly for getting source into it. One
downside is that the invocable bit isn't quite as nice. The old way
would return a java.util.functionFunction but the difference is

(.apply function (object-array args))

vs

(.execute fn-ref (object-array args))

* Don't io/resource the io/resource

* js engine tests

* Ns cleanup in js-svg

type hints in the js-engine ns mean we don't need as many classes from
polyglot here

* Cleanup of text, ns docstrings, alignment

* Fix fill->fill-opacity with parsed doc, not regex

* Make a single helper that loads a static viz bundle context

* Docstrings and make private in js-svg

* Sort imported classes in js-svg

* Make width passed down through rendering aparatus

- svgs are always rendered at 1200 for quality
- slack images of html are rendered at 1200 so that they can be zoomed
  in in the ui but slack automatically scales down
- email sends html and the full svg but includes img width tags so
  that is handled appropriately

* docstring

Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>
Co-authored-by: Ariya Hidayat <ariya@metabase.com>

* Add "moderation" folder to e2e CI groups

* fix question-moderation tests

Co-authored-by: Dalton Johnson <daltojohnso@users.noreply.github.com>

* Tweak Codecov settings to account for front-end coverage
* Scope back-end coverage with the correct codecov flag
* Add enterprise/frontend

* Remove `project.clj` from Percy workflows

* Replace `project.clj` with `deps.edn` in `build-for-test` script

* Update reference to the `project.clj` in the comment

Implement JDBC based Presto driver

Adding new Presto JDBC driver using the PrestoDB JDBC driver from `https://github.com/prestodb/presto`

Marking the old Presto driver as being `superseded-by` the new one

Pulling out common Presto code into new presto-common driver (modeled after the relationship between, ex: `googleanalytics` and `google)`

Putting common QP/HoneySQL logic into the new (abstract) :presto-common driver

Updating :presto driver to extend from the new common driver and only adding HTTP/REST API related methods

Adding implementation of Presto JDBC driver, named :presto-jdbc, extending from :presto-common and :sql-jdbc

Using com.facebook.presto/presto-jdbc as underlying JDBC driver dependency (since this is explicitly for Presto clusters, as opposed to Trino)

Adapting code from the existing Presto driver where appropriate

Adding new dependency-satisfied? implementation for :env-var, to allow for a plugin to require an env var be set, and making the Presto JDBC driver depend on that (specifically: `mb-enable-presto-jdbc-driver`)

Adding CircleCI configuration to run against the newer Presto (0.254) Docker image

Adding explicit ordering in a few tests where it was missing

Fixing presto-type->base-type (timestamps were being synced as :type/Time because the regex pattern was wrong)

Add tx/format-name test implementation for :presto-jdbc to lowercase table name

Make modified test Oracle friendly

Fixing bug parsing the `[:zone :time]` case within `metabase.util.date-2.parse/parse-with-formatter`; the offset is nil so it can't be passed directly in this case, so use the `standard-offset` fn (which was moved from `date-2` to `common` to get a standard offset for that zone

Fixing more test failures by adding explicit ordering

Changing sync to check whether the driver supports foreign keys before attempting to sync those (since drivers might throw an exception if attempting to check)

Moving some common test dataset functionality between :presto and :presto-jdbc to a new test.data ns for :presto-common

Adding HoneySQL form for :count-where, since we have to explicitly give a higher precision Decimal in order for Presto to not reduce the precision in results

Put limit within subquery for `expression-using-aggregation-test` (since ordering from subquery is not guaranteed in Presto)

Adding impls for ->prepared-substitution to handle substitutions for native query params

Adding HoneySQL impls for `mod` (to do as "mod(x,y)" and `timestamp` (since it's a function with no parens to invoke) functions

Adding various `sql.qp/date` impls that use the `AT TIME ZONE` operator to account for report tz, and make bucketing tests happy

* Update Webpack to the latest version (5.37.0)

* Update `webpack-dev-server` to the latest version (3.11.2)

* Install `webpack-cli`

* Replace `CommonsChunkPlugin` with `splitCunks`

* Remove `NamedModulesPlugin`

* Remove `UnusedFilesWebpackPlugin`

* Replace `extract-text-webpack-plugin` with `mini-css-extract-plugin`

* Upgrade `html-webpack-plugin` to the latest version (5.3.1)

* Upgrade `html-webpack-harddisk-plugin` to the latest version (2.0.0)

* Remove `banner-webpack-plugin`

* Replace deprecated `[hash]` in the config

* Add `mode` to the config

* Upgrade `file-loader` to the latest version (6.2.0)

* Don't include polyfills for `path` and `crypto`

* Remove `uglifyjs-webpack-plugin` and use terser instead

* Workaround CLJS_GLOBAL needs top-level `process` object

* yarn add node-polyfill-webpack-plugin

* Use some polyfills

Apparently, the polyfill for crypto is still necessary since the front-end
code isn't migrated yet to the browser's native Crypto API
(https://www.w3.org/TR/WebCryptoAPI/)

See also https://www.npmjs.com/package/node-polyfill-webpack-plugin

* Admin smoke test: fix getting the empty DB path

* Onboarding setup test: fix getting the empty DB path

* Replace file-loader with type: "asset/resource"

See https://webpack.js.org/guides/asset-modules/

* upgrade react-markdown (#16934)

* Use large CI resource since webpack consumes a lot of memory [ci nocache]

https://circleci.com/docs/2.0/configuration-reference/#resourceclass



* Tidy up `NODE_ENV` references

* Remove Webpack `DefinePlugin`

Co-authored-by: Ariya Hidayat <ariya@metabase.com>
Co-authored-by: Dalton Johnson <daltojohnso@users.noreply.github.com>
Co-authored-by: Alexander Lesnenko <alxnddr@users.noreply.github.com>

Related issue: https://github.com/metabase/metabase/issues/17005

* Add `arg` library

* Rewrite the main Cypress runner logic

* Destructure Cypress snapshot results

* Rename Cypress snapshots runner

* Update CircleCI config to reflect `folder` changes

* Update readme for smoke tests

* Run `metabase-db` tests conditionally based on env `QA_DB_ENABLED`

* Extract repro for #15279 into a separate file

* Extract repro for #15689 into a separate file

* Extract repro for #15695 into a separate file

* Extract repro for #15119 into a separate file

* Extract repro for #12614 into a separate file

* Extract repro for #12720 into a separate file

* Extract repro for #6660 into a separate file

* Extract repro for #16103 into a separate file

* Extract repro for #13150 into a separate file

* Move existing tests related to the dashboard filters into a new folder

* Create new CI group `dashboard-filters`

This group already takes close to 10 minutes to finish.

We need to have an additional extended coverage for dashboard filters and for query builder filters.
It doesn't make sense to keep them all together under the `filters` CI group because that group on its own could take more than half an hour to finish.

* Add initial set of e2e tests around sql basic filter types

- Reproduces #16811

* Add new `filters` e2e group to the CI

* Move smoke tests to the CircleCi

Closes #16741.

* Remove smoketest script

* Add new 'binning' e2e group to the CI

* Add initial set of tests for the binning options (buckets)

Point CircleCI presto test to the tagged older version (0.186)

* Port legacy data type migrations -> Liquibase

* Fix migration IDs

* Field type validation/error handling

* Have semantic type fallback to nil

* Fix semantic-type-migrations-test

* Fix migrations

* Revert accidental changes

* Semantic/* & Relation/* ancestor types

* Fix stray Relation/PK and Relation/FKs

* Semantic/* and Relation/* ancestor types

* cljs test fix

* Fix :require

* FE test fixes 

* Test fixes 

* prettier

* PR  f e e d b a c k

* Use medium size CircleCI image for Presto to prevent all the OOMs

* Backport dir-locals tweaks from hierarchy PR

* Redshift: only sync the test schema (faster CI and fix failure)

* Better error handling for sync in tests

* Revert accidental commit

* Redshift test fixes 

- Cherry picks 274be949 from `master`, which extracted unit and timezone tests from CircleCi and made them run using GitHub actions
- Synchronizes the "state" between the `master` and `release-x.39.x` branches

Co-authored-by: Ariya Hidayat <ariya@metabase.com>

* Druid Docker image [WIP]

* More Druid Docker stuff [WIP]

* Test fixes 

* Remove maxRowsQueuedForOrdering from query, don't think we need it

* Force test Druid host -> localhost for now

* Update sync tests for updated test data

* Revert unused changes