Co-authored-by: Bobby Iliev <bobby@bobbyiliev.com>

* drop unused computation_job table

* drop unused computation_job_result table

This reverts commit 9da0bd76.

* initial prototype w/out statement count parameter

* new approach

* default-database-role driver method

* migration for connection_impersonations table

* conn impersonation model, API and tests

* impersonation fetch and deletion endpoints

* switch test to t2 with-temp

* read conn impersonation settings from DB

* fix merge issue

* add impersonated key to data perms graph and treat it the same as full self-service access

* include impersonated key in returned permissions graph

* make sure impersonated graph passes StrictDataPerms validation

* fix boolean logic

* make sure impersonated keyword doesnt cause error when deleting gtaps

* clear impersonations as necessary when perms graphs changes

* add impersonation support for postgres

* fix typo

* make sure impersonation updates are a non-lazy seq

* add impersonated-user? fn

* fix impersonation api tests

* fix snippet tests

* fix build & exclude connection impersonations from serialization

* switch a test to use t2.with-temp

* add with-impersonations helper and util tests

* move macro and add a connection impersonation driver-level test for postgres

* fix rebase issue

* more tests and code reorganization

* add snowflake test

* clarify comment

* fix lint errors

* fix final kondo error

* reorganization

* fix one test

* fix lint errors

* revert change to sql_jdbc.execute from bad merge

* make sure perms for all users gets reset after conn impersonation tests

* ignore exceptions when restoring perms

* fix postgres test

* refactor to address bryan's comment

* add note about new methods to database changelog

* driver method refactor

* update audit log view migration

* separate mysql and fix h2

* typo

* fix views

---------

Co-authored-by: Luiz Arakaki <luiz.arakaki@metabase.com>

* add the migrations

* edit comment

* add 'if exists' to drop view statements

* update yaml: move sql to the right spot

* fix migrations and add new test

* fix migration IDs in test

* dont fail migration if FK already exists

* rewrite without table aliases to try to fix error on mysql/mariadb

* try to rewrite again

* revert to prior version of query

* another attempt at a rewrite

* fix typo

* fix typo

* bump ci

* replace LIKE with =

* forward migration for size_x and col

* done?

* test fixes

* adds some document

* migration should not contains comment

* mysql and postgres use different syntax in casting

* makes it work on mariadb

* update grid columns

* update specs

* update specs

* use float for postgres

* postgresql not postgres

* add migration for revision with model=Dashboard

* row is reserved in h2 too

* handle edge case with rollback migration where we have a 1x1 card that located at the shrinkage line

* kondo

* Cal's feedback

* add the code we used to generate random array

* clean up

* use the algorithm to dynamicall generate test data

* no more height migrations

* grid ratio

* unmigrate height in cypress specs

* update specs

---------

Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

* Adds audit v2 view migrations to liquibase

* bit of cleanup

* Add reverse migration for AddJoinAliasToVisualizationSettingsFieldRefs

* Limit selections to query_type="query"

* Make like strings more specific

* Fix where clause

* Further optimize where clause

* Rename and refactor

* Update test name

* Fix indentation and naming consistency

* add migration for typed collections

* add hydration of collections to card+dashboard api

- adds it for:
  - GET /api/question/:id
  - GET /api/dashboard/:id

* fix test that checks for collection shape

* get tests working with hydrated collection

* typo fixes

* fix more tests

- PUT requests don't hydrate :collection

* update tests to fix param_values {} vs nil inconsistencies

* Update underlying records drill - bring to top, label changes (#29984)

* Zoom in Drill (#30208)

* Add updated zoom drill title

* Fix e2e tests

* Add title unit tests

* Fix for Records drill UI when it is the only one displayed

* Review fixes

* Add pivot drill common popover (#30275)

* Add grouping for Pivot drill options

* Reuse filterField type

* Fix date formatting after rebase

* Move Automatic insight action to a common popover (#30370)

* Refactor ChartClickActionsView, reuse it component in drills (#30706)

* 26836 separate views for quick cell click filters (#30532)

* Added separate views for QuickFilterDrill

* Add tests, fix styles issues

* 26836 contains/does not contain quick filter for tables (#30778)

* Add contains/does not contain quick filter for tables

* Self review fixes

* Fix for column header filter search options cropped

* Fix for icon margin difference for contains / does not contain filters

* Fix contains filter actions icon styles

* Add quick filters support for aggregated queries

* Not show contains filter for joined fields

* Fix typecheck

* Adjust string field quick filter section title to have column name (#30888)

* 26836 Add View details action for quick filters (#30967)

* Add View details action to quick filters

* Self review fixes

* Fix e2e tests

* Fix review issues

* indexed entities schema

* schema

* endpoint with create and delete

* hooked up tasks

* first tests and hooking it all together

* some e2e tests

* search

* search

had to rename value -> name. The search stuff works well for stuff that
looks the same. And previously was renaming `:value` to name in the
search result, but it uses the searchable-columns-for-model for which
values to score on and wasn't finding :value. So easier to just let name
flow through.

* include model_pk in results

* send pk back as id, include pk_ref

* rename fk constraint (had copied it)

* populate indexed values with name not value

* GET http://localhost:3000/api/model_index?model_id=135

* populate indexed values on post

done here for ease of demo. followup work will schedule the task
immediately, but for now, do it on thread and return info

* test search results

* fix t2 delete syntax on h2

* fix h2

* tests work on h2 and postgres app dbs

* Fix after insert method

after method lets you change the data. and i was returning the results
of the refresh job as the model-index. not great

* unify some cross-db logic for inserting model index values

* Extract shared logic in populating indexed values; mysql support

* clean ns

* I was the bug (endpoint returns the item now)

* fix delete to check perms on model not model-index

* fix tests

* Fix tests

* ignore unused private var

* Add search tests

* remove tap>

* Tests for simple mbql, joined mbql, native

* shutdown scheduler after finishing

* Entity Search + Indexing Frontend (#30082)

* add model index types and mocks

* add integer type check

* add new entities for model indexes

* expose model index toggle in model metadata

* add search description for indexed entities

* add an error boundary to the app bar

* e2e test entity indexes

* update tests and types

* first tests and hooking it all together

* Renumber migrations

i forgot to claim them and i am a bad person

* Restore changes lost in the rebase

* add sync task to prevent errors in temp

without this you get errors when the temp db is created as it wants to
set the sync and analyze jobs but those jobs haven't been started.

* Restore and test GET model-index/:model_id

* clean up api: get by model id or model-index-id

* update e2e tests

* ensure correct types on id-ref and value-ref

* simplify lookup

* More extensive testing

* update types

* reorder migrations

* fix tests

* empty to trigger CI

* update types

* Bump clj-kondo

old version reports

/work/src/metabase/models/model_index.clj:27:6: error: #'metabase.models.interface/add-created-at-timestamp is private

on source:

```clojure
(t2/define-before-insert ::created-at-timestamp
  [instance]
  #_{:clj-kondo/ignore [:private-call]} ;; <- ignores this override
  (#'mi/add-created-at-timestamp instance))
```

* Move task assertions to model namespace

the task system is initialized in the repl so these all work
locally. But in CI that's not necessarily the case. And rather than
mocking the task system again I just leave it in the hands of the other
namespace.

* Don't serialize ModelIndex and ModelIndexValue

seems like it is a setting on an individual instance. No need to have
that go across boundaries

* Just test this on h2.

chasing names across the different dbs is maddening. and the underlying
db doesn't really matter

* indexes on model_index tables

- `model_index.model_id` for searching by model_id
- `model_index_value.model_index_id` getting values for a particular
  index

* copy/paste error in indexing error message

* `mt/with-temp` -> `t2.with-temp/with-temp`

* use `:hook/created-at-timestamped?`

* move field-ref normalization into models.interface

* nit: alignment

* Ensure write access to underlying model for create/delete

* Assert more about pk/value refs and better error messages

* Don't search indexed entities when user is sandboxed

Adds a `= 1 0` clause to indexed-entity search if the user is
sandboxed.

Also, got tired of the pattern:

```clojure
(if-let [segmented-user? (resolve 'metabase-enterprise.sandbox.api.util/segmented-user?)]
  (if (segmented-user?)
    :sandboxed-user
    :not-sandboxed-user)
  :not-sandboxed-user)
```

and now we have a tasty lil ole'

```clojure
(defenterprise segmented-user?
  metabase-enterprise.sandbox.api.util
  []
  false)
```

that you can just

```clojure
(ns foo
  (:require
   [metabase.public-settings.premium-features :as premium-features]))

(if (premium-features/segmented-user?)
  :sanboxed
  :not-sandboxed)
```

* redef premium-features/segmented-user?

* trigger CI

* Alternative GEOJSON.io fix (#30675)

* Alternative GEOJSON.io fix

rather than not testing, just accept that we might get an expected
location back or we might get nil. This actually represents how the
application will behave and will seamlessly work as the service improves
or continues throwing errors:

```shell
❯ curl "https://get.geojs.io/v1/ip/geo.json?ip=8.8.8.8,136.49.173.73,185.233.100.23"
<html>
<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>openresty</center>
</body>
</html>

❯ curl "https://get.geojs.io/v1/ip/geo.json?ip=8.8.8.8,136.49.173.73"
[{"area_code":"0","organization_name":"GOOGLE","country_code":"US",
  "country_code3":"USA","continent_code":"NA","ip":"8.8.8.8",
  "region":"California","latitude":"34.0544","longitude":"-118.2441",
  "accuracy":5,"timezone":"America\/Los_Angeles","city":"Los Angeles",
  "organization":"AS15169 GOOGLE","asn":15169,"country":"United States"},
 {"area_code":"0","organization_name":"GOOGLE-FIBER","country_code":"US",
 "country_code3":"USA","continent_code":"NA","ip":"136.49.173.73",
 "region":"Texas","latitude":"30.2423","longitude":"-97.7672",
 "accuracy":5,"timezone":"America\/Chicago","city":"Austin",
 "organization":"AS16591 GOOGLE-FIBER","asn":16591,"country":"United States"}]
```

Changes are basically

```clojure
(schema= (s/conditional some? <original-expectation-schema>
                        nil?  (s/eq nil)
         response-from-geojson)
```

* Filter to login error messages

```clojure
(into [] (map (fn [[log-level error message]] [log-level (type error) message]))
      error-messages-captured-in-test)
[[:error
  clojure.lang.ExceptionInfo
  "Error geocoding IP addresses {:url https://get.geojs.io/v1/ip/geo.json?ip=127.0.0.1}"]
 [:error clojure.lang.ExceptionInfo "Authentication endpoint error"]]
 ```

* check timestamps with regex

seems like they fixed the service

```shell
curl "https://get.geojs.io/v1/ip/geo.json?ip=8.8.8.8,136.49.173.73,185.233.100.23

"
[{"country":"United States","latitude":"34.0544","longitude":"-118.2441","accuracy":5,"timezone":"America\/Los_Angeles","ip":"8.8.8.8","organization":"AS15169 GOOGLE","country_code3":"USA","asn":15169,"area_code":"0","organization_name":"GOOGLE","country_code":"US","city":"Los Angeles","continent_code":"NA","region":"California"},{"country":"United States","latitude":"30.2423","longitude":"-97.7672","accuracy":5,"timezone":"America\/Chicago","ip":"136.49.173.73","organization":"AS16591 GOOGLE-FIBER","country_code3":"USA","asn":16591,"area_code":"0","organization_name":"GOOGLE-FIBER","country_code":"US","city":"Austin","continent_code":"NA","region":"Texas"},{"country":"France","latitude":"48.8582","longitude":"2.3387","accuracy":500,"timezone":"Europe\/Paris","ip":"185.233.100.23","organization":"AS198985 AQUILENET","country_code3":"FRA","asn":198985,"area_code":"0","organization_name":"AQUILENET","country_code":"FR","continent_code":"EU"}]
```

whereas a few hours ago that returned a 500. And now the timestamps are different

```
;; from schema
"2021-03-18T19:52:41.808482Z"
;; results
"2021-03-18T20:52:41.808482+01:00"
```

kinda sick of dealing with this and don't want to deal with a schema
that matches "near" to a timestamp.

* Restore these

i copied changes over from tim's fix and it included these fixes. But
now the service is fixed and these are the original values from cam's
original fixes

* Separate FK reference because of index

ERROR in metabase.db.schema-migrations-test/rollback-test (ChangeLogIterator.java:126)
Uncaught exception, not in assertion.

   liquibase.exception.LiquibaseException: liquibase.exception.RollbackFailedException: liquibase.exception.DatabaseException: (conn=637) Cannot drop index 'idx_model_index_model_id': needed in a foreign key constraint [Failed SQL: (1553) DROP INDEX `idx_model_index_model_id` ON `schema-migrations-test-db-32952`.`model_index`]
   liquibase.exception.RollbackFailedException: liquibase.exception.DatabaseException: (conn=637) Cannot drop index 'idx_model_index_model_id': needed in a foreign key constraint [Failed SQL: (1553) DROP INDEX `idx_model_index_model_id` ON `schema-migrations-test-db-32952`.`model_index`]
   liquibase.exception.DatabaseException: (conn=637) Cannot drop index 'idx_model_index_model_id': needed in a foreign key constraint [Failed SQL: (1553) DROP INDEX `idx_model_index_model_id` ON `schema-migrations-test-db-32952`.`model_index`]
   java.sql.SQLTransientConnectionException: (conn=637) Cannot drop index 'idx_model_index_model_id': needed in a foreign key constraint
     SQLState: "HY000"
    errorCode: 1553

* delete cascade on model_index fk to report_card

* reorder fk and index migrations

* break apart FK from table definition for rollbacks

* update types and appease the new lint rules

* define models in a toucan2 way

* remove checksum: ANY from migrations

* api permissions 403 tests

* test how we fetch values

* remove empty line, add quotes to migration file

* cleanup from tamas's comments

- indention fix
- inline an inc
- add values in a tx (also catch errors and update model_index)
- clarify docstring on `should-deindex?`
- don't use declare, just define fn earlier

* update after merge

* update to new model name

* only test joins for drivers that support it

* don't test mongo, include scenario in test output

* sets use `disj` not `dissoc`

* handle oracle primary id type

`[94M "Awesome Bronze Plate"]`
(type 94M) -> java.math.BigDecimal

* remove magic value of 5000 for threshold

* Reorder unique constraint and remove extra index

previously had a unique constraint (model_pk, model_index_id) and an
index on model_index_id. If we reorder the unique constraint we get the
index on the model_index_id for free. So we'll take it.

* remove breakout.

without breakout

```sql
 SELECT "source"."id"    AS "ID",
       "source"."title" AS "TITLE"
FROM   (SELECT "PUBLIC"."products"."id"    AS "ID",
               "PUBLIC"."products"."title" AS "TITLE"
        FROM   "PUBLIC"."products") AS "source"
ORDER  BY "source"."title" DESC
```

with breakout

```sql
 SELECT "source"."id"    AS "ID",
       "source"."title" AS "TITLE"
FROM   (SELECT "PUBLIC"."products"."id"    AS "ID",
               "PUBLIC"."products"."title" AS "TITLE"
        FROM   "PUBLIC"."products") AS "source"
GROUP  BY "source"."id",
          "source"."title"
ORDER  BY "source"."title" DESC,
          "source"."id" ASC
```

* restore breakout

caused some tests with joins to fail

* update model-index api mock

* Simpler method for indexing

Remove the `generation` part. We'll do the set operations in memory
rather than relying on db upserts. Now all app-dbs work the same and we
get all indexed values, diff against current values from the model, and
retract and add the appropriate values. "Updates" are counted as a
retraction and then addition rather than separately trying to update a
row.

Some garbage generation stats:

Getting garbage samples with

```clojure
(defn tuple-stats
  []
  (let [x (-> (t2/query ["select n_live_tup, n_dead_tup, relname from pg_stat_all_tables where relname = 'model_index_value';"])
              (first))]
    {:live (:n_live_tup x)
     :dead (:n_dead_tup x)}))
```

And using a simple loop to index the values repeatedly:

```clojure
(reduce (fn [stats _]
          (model-index/add-values! model-index)
          (conj stats (tuple-stats)))
        []
        (range 20))
```

With generation style:

```clojure
[{:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 400}
 {:live 200, :dead 400}
 {:live 200, :dead 400}
 {:live 200, :dead 400}
 {:live 200, :dead 770}
 {:live 200, :dead 770}
 {:live 200, :dead 787}
 {:live 200, :dead 787}
 {:live 200, :dead 825}
 {:live 200, :dead 825}
 {:live 200, :dead 825}
 {:live 200, :dead 825}
 {:live 200, :dead 818}
 {:live 200, :dead 818}
 {:live 200, :dead 818}
 {:live 200, :dead 818}
 {:live 200, :dead 854}]
```

With "dump and reload":

```clojure
[{:live 200, :dead 0}
 {:live 200, :dead 200}
 {:live 200, :dead 200}
 {:live 200, :dead 600}
 {:live 200, :dead 600}
 {:live 200, :dead 600}
 {:live 200, :dead 800}
 {:live 200, :dead 800}
 {:live 200, :dead 800}
 {:live 200, :dead 800}
 {:live 200, :dead 800}
 {:live 200, :dead 1600}
 {:live 200, :dead 1600}
 {:live 200, :dead 1600}
 {:live 200, :dead 2200}
 {:live 200, :dead 2200}
 {:live 200, :dead 2200}
 {:live 200, :dead 2200}
 {:live 200, :dead 2200}
 {:live 200, :dead 3600}]
```

And of course now that it's a no-op,

```clojure
[{:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}
 {:live 200, :dead 0}]
```

Since no db actions have taken when reindexing. We've traded memory set
operations for garbage in the database. Since we're capped at 5000 rows
it seems fine for now.

* add analogs to `isInteger` to constants.cljc and isa.cljc files

* clarity and do less work on set conversion

* exclude clojure.core/integer? from isa.cljc

* Rename `state_change_at` -> `indexed_at`

* breakout brings along an order-by and fields clause

---------

Co-authored-by: dan sutton <dan@dpsutton.com>

Fix column settings being duplicated when fields have the same `id-or-name`, but different `join-alias` (#27487)

* Migrate legacy field refs in result_metadata

* Add expression and field_ref

* Update testing description

* Fix clj-kondo

* Exclude empty result_metadata from the query

Removed colloquial use of "your" in x-rays where appropriate.

* add column for marking the audit database

* create audit db on app startup

On app startup, in enterprise mode, always check that there is an
up-to-date audit database with is_audit set to true. up-to-date means
that it matches the app-db. If the audit db doesnt exist, or doesn't match up with
the app-db, it will be created. otherwise no-op

- refactor GET /api/database/ into defendpoint

* add a few tests

* add more tests + return kw for no-op

* Merge branch 'master' into bcm-add-app-db-as-audit-db

* fix migrations file

* lint fix + remove def

* keep the old audit db around

- nil -> ::no-op

* small macro fix

* Merge branch 'master' into bcm-add-app-db-as-audit-db

* Merge branch 'master' into bcm-add-app-db-as-audit-db

* move ensure-db-exists! into its own namespace

- avoids circlar dependency on app startup

* fix tests

* lint fix

* inspect test values

* use mdb.env/env to initialize audit db

* try returning the app-db connection pool

- instead of making a doppelganger in metabase database

* get the is_audit column when looking up dbs in qp

- this key is needed to return the appdb for the "Audit DB"'s connection

* clean up audit-db install/update/noop logic

* remove audit db from GET api/database/

* Merge branch 'master' of github.com:metabase/metabase into bcm-add-app-db-as-audit-db

* fix linter + inline extra function

* fix a test

* Merge branch 'master' into bcm-add-app-db-as-audit-db

* don't test actions on audit db 

* Revert "don't test actions on audit db "

This reverts commit 581ed1ab9495de7b46bccb8b1ab19f66a8fd106d.

* for tests to work low IDs are basically reserved

- use a higher id when installing audit db

* use defenterprise for ensure-audit-db-installed!

* Merge branch 'master' into bcm-add-app-db-as-audit-db

* Dashboard Tab entity (#29802)

* add dashboard tab entity

* Dashboard tabs CRUD (#29861)

* Dashboard tabs frontend (#30189)

* rename `DashboardHeader.tsx` to `DashboardHeaderView` to avoid collision with `DashboardHeader.jsx`

commit-id:7cdfd86a

* add tabs to dash

commit-id:d3c2fa74

* improve code structure and other fixes

* rename `dashboardtab_id` to `dashboard_tab_id`

* remove explicit `position` logic

* improve code quality

* add renaming functionality

* add two new tabs when creating first tab

* hide tabs if there is only one

* add unit test for `DashboardTabs` component

* refactor

* fix type errors

* add `tabId` to text cards

* add `tabId` to link cards

* add `tabId` to action button

* use disabled state when only one tab remains

* refetch cards with filter value when changing tabs

* add e2e test

* fix broken e2e tests

* add horizontal scrolling

* fix unit test

* fix create tab button being on right

* fix performance issue when switching tabs

* add `ttag` to `tabs.ts`

* use `aria-label` in e2e test helper

* fix type error after rebasing

* making ordered_tabs optional for tests purposes

* fix failing unit tests

* fix failing e2e tests

* fix flaky revision history e2e test

* fix type error after merging feature branch with master and rebasing

* fix revisions unit test

* fix `ActionParmatersInputsForm.tsx` to stop `actions-on-dashboards.cy.spec.js` from flaking

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

---------

Co-authored-by: Emmad Usmani <emmadusmani@berkeley.edu>

* Create a migration for updating column_settings

* Make fn private

* Update test description

* Fix clj-kondo

* Migrate the entire vizualization_settings

* Add test for normalization and migration

* Update migration class name

* Use generate-string to improve key readability

* Add testing strings

* Fix migration

* transduce

* Add another test checking that `expected` is actually the up-to-date version

* Rework migration to only update field refs

* Use run!

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* Add CSV uploads feature and have postgres and H2 support it (#29636)

* Add csv uploads feature and have postgres support it

* Add h2 support

* CSV: schema detection (#29650)

* CSV / Admin Upload Settings (#29709)

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* Add CSV uploads feature and have postgres and H2 support it (#29636)

* Add csv uploads feature and have postgres support it

* Add h2 support

* Add defsettings

* Add table prefix setting

* add data upload settings

* Add newline

* update settings mock

* Whether or not uploads are enabled

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>

* Create table and insert rows from CSV uploads for postgres (#29677)

* WIP create table and insert rows

* Add newline

* Use multimethods

* WIP

* Fix postgres

* Fix postgres impl

* Use detect-schema and add test

* Fix load-from-csv arglists

* Undo whitespace change

* Use empty db instead

* Add table and field tests

* Remove prn

* Uniquify names

* newline

* Add test for float

* Add a test for the table contents

* Tidy load-from-csv

* Drop the table if loading the csv fails

* Drop the table only if loading the csv fails

* whitespace

* Address SQL injection vulnerability

* Use the slugified file-name as the table-name

* Fix kondo

* Using absolute path instead of relative

* Fix kondo

* Detect types for empty CSV files appropriately (#29754)

* Create models from CSV uploads (#29966)

* Wire up CSV uploads, sync the new table

Still need to create a model

* Check database support for CSV uploads better

* Another test

* Create model after sync

* Fix request with formData

* Fix tests

* Fix tests

* Revert "Fix request with formData"

This reverts commit 66f146a41cf42feb34dbb03ed8566fa4220d0642.

* Add whitespace

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix postgres support for uploads (#29922)

* Check write perms for collection (#29976)

* Rename metabase.csv -> metabase.upload (#29975)

* Rename metabase.csv -> metabase.upload

* Add docstring

* Rename alias for release.common.upload

---------

Co-authored-by: Callum Herries <hi@callumherries.com>

* WIP - 29598 csv status toast (#29757)

* temp

* generalize StatusListing types

* Putting a pin in this for now

* poorly done typing

* generalizing upload function names

* point to real API

* Fixing Storybook

* Permissions for Status Listing

* use formdata and fix types

---------

Co-authored-by: Ryan Laurie <iethree@gmail.com>

* Don't support redshift (#29981)

* Handle uploads to the root collection (#29985)

* Allow the uploads schema to be nil (for MySQL, etc.) (#29987)

* CSV upload: fix stack overflow in detect-schema (#29983)

* Fix stack overflow error on detect schema

* map + vec -> mapv

---------

Co-authored-by: Tim Macdonald <tim@metabase.com>

* reload collection data after upload (#30009)

* MySQL support for uploads (#29829)

* Add MySQL support for uploads

* Rename arg to upload-type

* Use new ns name

* Allow no table-name prefix to be specified and fix tests

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix setting descriptions (#30023)

* Batch CSV insertion for better performance (#30028)

* CSV upload e2e tests (#30005)

* add CSV upload e2e tests

* update tests

* H2 support for uploads (#29828)

* Add uploads support for H2

* Rename arg

* whitespace

* Use new metabase.upload ns name

* Fix tests for H2

* Make table and field matching case insensitive

* Fix kondo error

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Clear CSV upload state after a while (#30014)

* clear upload status after 5s

* make async behavior clearer

* CSV Upload Dates and Datetimes Types (#30076)

* Simple dates

* Simple datetimes

* Fix coalesce for date and datetime

* Add mysql and h2 types

* Fix test

* Misc cleanup

* Fix test

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix Kondo (rebase gone wrong)

* MySQL test fix

* Fix order of Metabot setting

* adding unit tests to upload actions and reducers (#30161)

* adding unit tests to upload actions and reducers

* use fetch-mock

* Adding component unit test

* Check CSV Upload permissions on the frontend (#30094)

* CSV / Admin Upload Settings (#29709)

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* Add CSV uploads feature and have postgres and H2 support it (#29636)

* Add csv uploads feature and have postgres support it

* Add h2 support

* Add defsettings

* Add table prefix setting

* add data upload settings

* Add newline

* update settings mock

* Whether or not uploads are enabled

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>

* properly check if user can upload to a collection

* do some db checking

* don't check admin status for upload notification

* misc cleanup

* fix rebase

* add loggedin check to statusListing

also combines container + view component into a single file

* update unit tests

* update tests

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>

* Csv/misc bugfixes (#30219)

* Support empty header names

* Handle long CSV names more appropriately

Handle duplicate column names in CSVs

* Test fix

* Partition CSV inserts a little more intelligently

* Test fix

* Remove 'duplicate' from duplicated column names

* More intelligent sampling

* Add comment re. Postgres table-name-length-limit

* Quote column names, clean up tests

When unquoted, we couldn't use reserved words (true, false, group, etc.)
as column names

* Fix test

Now that we quote things, H2 is case-sensitive

* Use dialect-appropriate quoting

Also clean up some card-filtering tests so they work with a dirty DB

* Fix deduplication

* Remove use of setting/get

* Remove unneeded migration

---------

Co-authored-by: Callum Herries <hi@callumherries.com>

* Fix test (#30383)

---------

Co-authored-by: Ryan Laurie <iethree@gmail.com>
Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>

* initial changes

* fix test

* add ldap

* fix tests

* update migrations

* swap

* swap ldap

* fix test

* update test

* fix test

* fix test

* address comments

* update

* Use type/JSON instead of type/SerializedJSON

* Tidy migration

* Update migration

* Fix rollback migration for h2

* whitespace

* Add test for migration

* Fix test

* Add rollback

* whitespace

* Test JSONB type as well as JSON

* Don't fingerprint JSON columns

* Remove comment, that might be wrong in the future

* Update test

* Add tests for fingerprinting

* Use base-type JSON for fingerprinting base query

* Add test for visibility-type=details-only

* undo .

* Change migration id

* Fix migration test

* Merge master

* Exclude mariadb from tests

* Make is-mariadb? public

* Migration for adding nfc_enabled

* Add nfc_enabled to field settings

* Update describe-nested-field-columns to only unfold fields that have not been disabled

* Remove spy

* Tidy

* Fix

* Clear nested fields immediately if folding is disabled

* Clear nested fields on nfc_enabled change

* Trim trailing whitespace

* Tidy

* Fix

* Add enable-json-unfolding-test

* Move to field api test

* nfc_enabled -> json_unfolding

* Tidy test

* Make json-unfolding in database details just the default for new settings

* Restore original visibility-type logic

* Fix

* Sync field json_unfolding according to db json_folding

* Rename to json-unfolding-default

* Add test for the case when the json-unfolding is false for the database

* Implement default json unfolding for first sync

* Update comment

* Update comment

* Restore PUT field

* Migration for populating json_unfolding for mysql and postgres

* Remove migration and use default on the frontend instead

* Update json-unfolding database setting copy

* Move Unfold JSON setting under semantic type setting

* Separate sentences with spaces

* Capitalize

* Restore handleChangeSemanticType

* Use base_type TYPE.JSON instead

* Change order of migrations

* Add h2 migration

* Add test for migration

* Update test description

* Remove validCheckSum

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Use ?? instead of ||

* Fix tests

* Remove outdated serdes stuff

* Remove unnecessary and

* Fix mysql migration

* Remove unused require

* Fix test

* Add false default value for json_unfolding

* Fix json-unfolding nil case

* Other suggestions

* whitespace

* Tidy describe-nested-field-columns

* Update comment

* Remove unused clear-nested-fields!

* Remove unused return value

* Fix H2 migration to use base_type not database_type

* Always set json-unfolding during sync

* Fix test

* Fix test

* Fix test

* Add comment explaining nested-field-column support for MySQL

* Fix tests

* Fix tests

* Fix test

* Fix test

* Default json_unfolding to false for new fields

* Fix merge

* Add json_unfolding to mock tables

* Don't capitalize prepositions

* Update setting description

* whitespace

* whitespace

* Fix fetch_metadata

* Fix fetch_metadata

* Fix tests

* Fix test

* Fix clj-kondo

* Remove postgres database-supports test

* Fix postgres test

* Fix postgres test

* Fix mysql migration

* Fix clj-kondo

* Don't test mariadb

* Fix mysql test

* Fix mysql json-unfolding nil case

* Add comments to test

* Add upterm step to mariadb test

* Move upterm step before tests

* Fix postgresql migration

* move upterm step to test-driver action

* Remove upterm step from drivers.yml

* Comment out everything else in test-driver

* Fix mariadb migration

* Remove upterm action

* Whitespace

* Only handle JSONObjects with JSON_VALUE, not JSONArray

* Fix fields.sync_metadata/update-field-metadata-if-needed!

* Add comment explaining `json-unfolding-default`

* Add comment for JSON_VALUE test

* Update json-unfolding-default-true-test to use fresh db

* Fix test descriptions

* Don't capitalize setting display name

* Fix update json_unfolding

* Fix unfolding json for only one JSON column, not all

* Add e2e test for JSON unfolding setting

* Fix test

* whitespace

* Fix test

* Fix json-unfolding for nested field columns

* Fix mysql and postgres tests for nested field columns

* Coerce nil json-unfolding to false for sync_metadata

* Add test for nil json-unfolding for sync_metadata

* Fix test

* Don't update json_unfolding from intial sync

* Add json_unfolding to update-field-test

* Fix tests

* Use cypress-testing-library commands

* Fix e2e test

* Fix e2e test

* Add json-simple to deps

* Update uberjar.yml

* import org.json.simple

* Replace SQL migrations and H2 UDF with a clojure migration

* Remove json-simple from deps

* Restore uberjar.yml

* Rename to set-json-unfolding!

* Make the DB JSON unfolding setting override the field setting, instead of being a default.

* Remove unused var

* Fix error

* Remove custom clojure migration

* Fix migration

* Fix migration again

* Fix migration and kondo

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Backend for allow disabling auto-apply filters in slow dashboards (#30069)

* Dumb BE change before proper BE code is implemented

* Add some backend tests for auto_apply_filters

- included in revisions
- included in serdes

fixup some tests that didn't expect it

* Fix BE for public dashboards

---------

Co-authored-by: dan sutton <dan@dpsutton.com>

* Prevent auto apply filters on slow dashboards (#29569)

* Able to toggle auto-apply dashboard filters

Make FormField be able to show title with new style

* Only apply filters when clicking apply button

* Add E2E test

* Fix dashboard double fetch cards onload with parameters query

* Remove puzzle driven development comments

* Move logic from HoC to redux 1/x

* Make apply filter button works on embeddings

* Fix x-ray dashboard parameters not working

* Fix unit tests

* Add E2E tests to public dashboard sharing

* Add signed embedding E2E tests

* Fix the apply button transition not working when clicked

Since there's another CSS applied when the button is hovered.

* Reviews: Fix x-ray dashboard actions

Remove the need of 3-states boolean in `setParameterValue` action for
x-ray dashboards. This fixes so that BE sends x-ray dashboards with
`auto_apply_filters: true`.

* Review: remove duplicate `aria-label`

* Review: standadize `draftParameterValues` default values

* Review: Fix `aria-label` to not change when filter values change

* Review: avoid passing `ReactNode` as a prop

Also fix transition not working and apply button shift when adding a
filter.

* Move selectors and actions to their own file

* Review: Don't animate (un)mount of FilterApplyButton

* Address: Clean up eslint ignore on tests

* Review: Only test 1 scenario in E2E

Since testing other scenarios are unnecessary since they're repeat the
same thing

* Review: Use E2E helper to clean up test

* Review: Reuse selector

* Review: Use e2e helpers to clean up the test

* Fix typo

* Ensure Apply button won't appear unnecessarily (#30255)

---------

Co-authored-by: dan sutton <dan@dpsutton.com>

Better metabot logging, especially around generated token counts. This will be particularly useful as we determine what size of prompts we can generate.