* Add direction arg to orderByClause

* Update order_by.ts

* Fix TypeScript error

* Update order_by.ts

---------

Co-authored-by: Anton Kulyk <kuliks.anton@gmail.com>

* update and test number display for gauge viz

* update and test numeric display for progress viz

* display null in scalar viz

* Rework repro for 16382

* Merge multiple "connection settings" tests together

* Merge multiple "scheduling settings" tests together

* Merge tiny test with the main body

* Remove redundant tests

* Rework the sample database action sidebar

* Handle database exceptions in one spec

* Move stray database test to exceptions spec

* Move Postgres SSL test to external

* Group together Google service account JSON tests

* Custom caching

* Move remaining pieces of add to external

* Rename spec to `add-new-database`

* Move repro for 20471 to exceptions

* Add segments and metrics to the database we want to delete

* Add `visitDatabase` helper

* MLv2 Multimethod hierarchies

* Remove test

Added both FE and BE tooling for integration with LLMs, called "metabot". FE adds prompts for querying models and databases. BE takes these prompts, calls a LLM, parses the response, and returns the result.

* MLv2 implement temporal extract

* Test fix 

* Disallow duplicate UUIDs in a query

* Update test

* Fixes 

* WIP Initial commit, not sure of difference between Clause and Field

* Finish wrappers for remove-clause and replace-clause

* Increase the BE timeout

* Increase the drivers timeout [ci skip]

* Clean cache after branch closes

* Add whitespace

* Add whitespace

* Finalize caches cleanup workflow

---------

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Update uberjar.yml

Added `pretty` option to POST api/dataset/native route. Defaults to `true` such that invocations of this endpoint will return pretty-formatted SQL. In the case of previewing the native SQL editor with templating, we pass `false` so that what is typed is what is presented. In all other cases we accept the default.

* Move `SortStep` to its own directory

* Add tests for `SortStep`

* Clean up `SortStep`

* Improve a11y

* test case

* fix query and split out test case

* Adapt unique schema logic in CI from Snowflake to Redshift

* Ok don't stomp on all the old schemas just yet.

* Fix missing vars

* Test fixes 

* Test fixes 

* Revert `^:parallel` on tests that shouldn't be allowed to use it

* Metadata/refs overhaul

* Fix Kondo error

* Fix #29702

* Fix #29701

* Revert silly changes

* Fix syntax error

* MLv2 metadata calculation: expressions SHOULD be included in default fields

* Metadata/refs overhaul

* Fix Kondo error

* Fix #29702

* Fix #29701

* Revert silly changes

* Fix syntax error

* Create `cache_info` table in our persisted schemas

```sql
-- postgres/redshift
test-data=# select * from metabase_cache_424a9_379.cache_info ;
       key        |                value
------------------+--------------------------------------
 settings-version | 1
 created-at       | 2023-03-28
 instance-uuid    | 407e4ba8-2bab-470f-aeb5-9fc63fd18c4e
 instance-name    | Metabase Test
(4 rows)
```

```sql
--mysql
mysql> select * from metabase_cache_424a9_435.cache_info;
+------------------+--------------------------------------+
| key              | value                                |
+------------------+--------------------------------------+
| settings-version | 1                                    |
| created-at       | 2023-03-28                           |
| instance-uuid    | 407e4ba8-2bab-470f-aeb5-9fc63fd18c4e |
| instance-name    | Metabase Test                        |
+------------------+--------------------------------------+
```

our key values in v1:
```clojure
(defn kv-table-values
  "Version 1 of the values to go in the key/value table `cache_info` table."
  []
  [{:key   "settings-version"
    :value "1"}
   {:key   "created-at"
    ;; "2023-03-28"
    :value (.format (LocalDate/now) DateTimeFormatter/ISO_LOCAL_DATE)}
   {:key   "instance-uuid"
    :value (public-settings/site-uuid)}
   {:key   "instance-name"
    :value (public-settings/site-name)}])
```

This will enable us to delete cached schemas in shared databases (cloud
versions like redshift) without risking wiping out a cached schema
during an active test run.

The code to delete the schemas will be something like,

```clojure
(def spec (sql-jdbc.conn/connection-details->spec
           :redshift
           {:host "xxx.us-east-1.reshift.amazonaws.com"
            :db "db"
            :port 5439
            :user "user"
            :password "password"}))

(let [days-prior 1 ;; probably 2 to handle crossing day time. Can also adjust to 6 hours, etc
      threshold (.minus (java.time.LocalDate/now)
                        days-prior
                        java.time.temporal.ChronoUnit/DAYS)]
  (doseq [schema (->> ["select nspname from pg_namespace where nspname like 'metabase_cache%'"]
                      (jdbc/query spec)
                      (map :nspname))]
    (let [[{created :value}] (jdbc/query spec (sql/format {:select [:value]
                                                           :from [(keyword schema "cache_info")]
                                                           :where [:= :key "created-at"]}
                                                          {:dialect :ansi}))]
      (when created
        (let [creation-day (java.time.LocalDate/parse created)]
          (when (.isBefore creation-day threshold)
            (jdbc/execute! spec [(format "drop schema %s cascade" schema)])))))))
```

or if we want to do it in a single query:

```clojure
schemas=> (let [days-prior 1
                threshold  (.minus (java.time.LocalDate/now)
                                   days-prior
                                   java.time.temporal.ChronoUnit/DAYS)]
            (let [schemas (->> ["select nspname
                         from pg_namespace
                         where nspname like 'metabase_cache%'"]
                               (jdbc/query spec)
                               (map :nspname))]
              (jdbc/with-db-connection [conn spec]
                (jdbc/execute! conn [(format "set search_path= %s;" (str/join ", " schemas))])
                (let [sql               (sql/format {:select [:schema :created-at]
                                                     :from   {:union-all
                                                              (for [schema schemas]
                                                                {:select [[[:inline schema] :schema]
                                                                          [{:select [:value]
                                                                            :from   [(keyword schema "cache_info")]
                                                                            :where  [:= :key [:inline "created-at"]]}
                                                                           :created-at]]})}}
                                                    {:dialect :ansi})
                      old?              (fn [{create-str :created-at}]
                                          (let [created (java.time.LocalDate/parse create-str)]
                                            (.isBefore created threshold)))
                      schemas-to-delete (->> (jdbc/query spec sql)
                                             (filter old?)
                                             (map :schema))]
                  schemas-to-delete))))
("metabase_cache_424a9_503") ;; when there are any, string join them and delete them as above
```

* Use Instants so we can compare by hours instead of just days

* Don't log in bigquery results hotpath

Right now we log in the parser of bigquery results for unrecognized
types. But the problem is that we log for each _value_ and not each
column. This is causing an _enormous_ amount of logs and performance
penalty.

See
- https://github.com/metabase/metabase/issues/29118 (performance)
- https://github.com/metabase/metabase/issues/28868 (filling disk space)

This log was added between 45.1 and 45.3

```diff
❯ git diff v0.45.1..v0.45.3 modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/**
diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
index a0d8081c30..f367199b55 100644
--- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
+++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
@@ -22,7 +22,7 @@
             [metabase.util :as u]
             [metabase.util.date-2 :as u.date]
             [metabase.util.honeysql-extensions :as hx]
-            [metabase.util.i18n :refer [tru]]
+            [metabase.util.i18n :refer [trs tru]]
             [pretty.core :refer [PrettyPrintable]]
             [schema.core :as s])
   (:import [com.google.cloud.bigquery Field$Mode FieldValue]
@@ -88,7 +88,8 @@
     (parse-fn v)))

 (defmethod parse-result-of-type :default
-  [_ column-mode _ v]
+  [column-type column-mode _ v]
+  (log/warn (trs "Warning: missing type mapping for parsing BigQuery results of type {0}." column-type))
   (parse-value column-mode v identity))
```

The result is that selecting 50,000 rows for download in excel:

| version                      | time       |
|------------------------------|------------|
| 0.45.1                       | 28 seconds |
| 0.45.3                       | 52 seconds |
| 0.45.3 with logging disabled | 30 seconds |

(disable logging by adding `<Logger
name="metabase.driver.bigquery-cloud-sdk.query-processor"
level="ERROR"/>` and `-Dlog4j2.configurationFile=log4j2.xml` to jar
startup)

For the query (3 string columns, 5 rows):

```sql
SELECT game_id, first_name, last_name
FROM `bigquery-public-data.ncaa_basketball.mbb_players_games_sr`
LIMIT 5
```

BEFORE:

```
```
2023-03-31 17:17:52,146 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
2023-03-31 17:17:52,155 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 795.2 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (2 idle, 0 queued) (192 total active threads) Queries in flight: 0 (0 queued)
```

Note this is 15 logs (3 columns x 5 rows)

AFTER:

```
2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column game_id of type STRING.
2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column first_name of type STRING.
2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column last_name of type STRING.
2023-03-31 17:19:15,757 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 973.5 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (3 idle, 0 queued) (193 total active threads) Queries in flight: 0 (0 queued)
```

* unused require to appease our overlords

* Update Clojure CLI

Most important change: using stable tools.deps

https://clojure.org/releases/tools

This is a great way to make sure the conversion in both directions
is robust; every feature of MBQL is exercised somewhere in the BE tests.

In a few cases the `mbql-query` is called from a context where a
`(is ...)` test won't work - those places can use a `mbql-query-no-test`
instead. There were actually only two of these.

This exposed several issues in `->legacy-MBQL` which have now been fixed.

This reverts commit 25a1b259.

* Add more MLv2 wrappers

* Don't use MLv2 directly

* Update tests

* Add dedicated `OrderByClause` type

* Rename `query` to `fromLegacyQuery`

* Remove too precise `orderableColumns` test

Agreed that should be better covered on the CLJS side

This reverts commit a817ed8a.

* Use type/JSON instead of type/SerializedJSON

* Tidy migration

* Update migration

* Fix rollback migration for h2

* whitespace

* Add test for migration

* Fix test

* Add rollback

* whitespace

* Test JSONB type as well as JSON

* Don't fingerprint JSON columns

* Remove comment, that might be wrong in the future

* Update test

* Add tests for fingerprinting

* Use base-type JSON for fingerprinting base query

* Add test for visibility-type=details-only

* undo .

* Change migration id

* Fix migration test

* Merge master

* Exclude mariadb from tests

* Make is-mariadb? public

* Migration for adding nfc_enabled

* Add nfc_enabled to field settings

* Update describe-nested-field-columns to only unfold fields that have not been disabled

* Remove spy

* Tidy

* Fix

* Clear nested fields immediately if folding is disabled

* Clear nested fields on nfc_enabled change

* Trim trailing whitespace

* Tidy

* Fix

* Add enable-json-unfolding-test

* Move to field api test

* nfc_enabled -> json_unfolding

* Tidy test

* Make json-unfolding in database details just the default for new settings

* Restore original visibility-type logic

* Fix

* Sync field json_unfolding according to db json_folding

* Rename to json-unfolding-default

* Add test for the case when the json-unfolding is false for the database

* Implement default json unfolding for first sync

* Update comment

* Update comment

* Restore PUT field

* Migration for populating json_unfolding for mysql and postgres

* Remove migration and use default on the frontend instead

* Update json-unfolding database setting copy

* Move Unfold JSON setting under semantic type setting

* Separate sentences with spaces

* Capitalize

* Restore handleChangeSemanticType

* Use base_type TYPE.JSON instead

* Change order of migrations

* Add h2 migration

* Add test for migration

* Update test description

* Remove validCheckSum

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Use ?? instead of ||

* Fix tests

* Remove outdated serdes stuff

* Remove unnecessary and

* Fix mysql migration

* Remove unused require

* Fix test

* Add false default value for json_unfolding

* Fix json-unfolding nil case

* Other suggestions

* whitespace

* Tidy describe-nested-field-columns

* Update comment

* Remove unused clear-nested-fields!

* Remove unused return value

* Fix H2 migration to use base_type not database_type

* Always set json-unfolding during sync

* Fix test

* Fix test

* Fix test

* Add comment explaining nested-field-column support for MySQL

* Fix tests

* Fix tests

* Fix test

* Fix test

* Default json_unfolding to false for new fields

* Fix merge

* Add json_unfolding to mock tables

* Don't capitalize prepositions

* Update setting description

* whitespace

* whitespace

* Fix fetch_metadata

* Fix fetch_metadata

* Fix tests

* Fix test

* Fix clj-kondo

* Remove postgres database-supports test

* Fix postgres test

* Fix postgres test

* Fix mysql migration

* Fix clj-kondo

* Don't test mariadb

* Fix mysql test

* Fix mysql json-unfolding nil case

* Add comments to test

* Add upterm step to mariadb test

* Move upterm step before tests

* Fix postgresql migration

* move upterm step to test-driver action

* Remove upterm step from drivers.yml

* Comment out everything else in test-driver

* Fix mariadb migration

* Remove upterm action

* Whitespace

* Only handle JSONObjects with JSON_VALUE, not JSONArray

* Fix fields.sync_metadata/update-field-metadata-if-needed!

* Add comment explaining `json-unfolding-default`

* Add comment for JSON_VALUE test

* Update json-unfolding-default-true-test to use fresh db

* Fix test descriptions

* Don't capitalize setting display name

* Fix update json_unfolding

* Fix unfolding json for only one JSON column, not all

* Add e2e test for JSON unfolding setting

* Fix test

* whitespace

* Fix test

* Fix json-unfolding for nested field columns

* Fix mysql and postgres tests for nested field columns

* Coerce nil json-unfolding to false for sync_metadata

* Add test for nil json-unfolding for sync_metadata

* Fix test

* Don't update json_unfolding from intial sync

* Add json_unfolding to update-field-test

* Fix tests

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* shared.ns/import-fn(s) emit optimized defn forms

* Code simplification