* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* Add CSV uploads feature and have postgres and H2 support it (#29636)

* Add csv uploads feature and have postgres support it

* Add h2 support

* CSV: schema detection (#29650)

* CSV / Admin Upload Settings (#29709)

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* Add CSV uploads feature and have postgres and H2 support it (#29636)

* Add csv uploads feature and have postgres support it

* Add h2 support

* Add defsettings

* Add table prefix setting

* add data upload settings

* Add newline

* update settings mock

* Whether or not uploads are enabled

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>

* Create table and insert rows from CSV uploads for postgres (#29677)

* WIP create table and insert rows

* Add newline

* Use multimethods

* WIP

* Fix postgres

* Fix postgres impl

* Use detect-schema and add test

* Fix load-from-csv arglists

* Undo whitespace change

* Use empty db instead

* Add table and field tests

* Remove prn

* Uniquify names

* newline

* Add test for float

* Add a test for the table contents

* Tidy load-from-csv

* Drop the table if loading the csv fails

* Drop the table only if loading the csv fails

* whitespace

* Address SQL injection vulnerability

* Use the slugified file-name as the table-name

* Fix kondo

* Using absolute path instead of relative

* Fix kondo

* Detect types for empty CSV files appropriately (#29754)

* Create models from CSV uploads (#29966)

* Wire up CSV uploads, sync the new table

Still need to create a model

* Check database support for CSV uploads better

* Another test

* Create model after sync

* Fix request with formData

* Fix tests

* Fix tests

* Revert "Fix request with formData"

This reverts commit 66f146a41cf42feb34dbb03ed8566fa4220d0642.

* Add whitespace

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix postgres support for uploads (#29922)

* Check write perms for collection (#29976)

* Rename metabase.csv -> metabase.upload (#29975)

* Rename metabase.csv -> metabase.upload

* Add docstring

* Rename alias for release.common.upload

---------

Co-authored-by: Callum Herries <hi@callumherries.com>

* WIP - 29598 csv status toast (#29757)

* temp

* generalize StatusListing types

* Putting a pin in this for now

* poorly done typing

* generalizing upload function names

* point to real API

* Fixing Storybook

* Permissions for Status Listing

* use formdata and fix types

---------

Co-authored-by: Ryan Laurie <iethree@gmail.com>

* Don't support redshift (#29981)

* Handle uploads to the root collection (#29985)

* Allow the uploads schema to be nil (for MySQL, etc.) (#29987)

* CSV upload: fix stack overflow in detect-schema (#29983)

* Fix stack overflow error on detect schema

* map + vec -> mapv

---------

Co-authored-by: Tim Macdonald <tim@metabase.com>

* reload collection data after upload (#30009)

* MySQL support for uploads (#29829)

* Add MySQL support for uploads

* Rename arg to upload-type

* Use new ns name

* Allow no table-name prefix to be specified and fix tests

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix setting descriptions (#30023)

* Batch CSV insertion for better performance (#30028)

* CSV upload e2e tests (#30005)

* add CSV upload e2e tests

* update tests

* H2 support for uploads (#29828)

* Add uploads support for H2

* Rename arg

* whitespace

* Use new metabase.upload ns name

* Fix tests for H2

* Make table and field matching case insensitive

* Fix kondo error

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Clear CSV upload state after a while (#30014)

* clear upload status after 5s

* make async behavior clearer

* CSV Upload Dates and Datetimes Types (#30076)

* Simple dates

* Simple datetimes

* Fix coalesce for date and datetime

* Add mysql and h2 types

* Fix test

* Misc cleanup

* Fix test

---------

Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix Kondo (rebase gone wrong)

* MySQL test fix

* Fix order of Metabot setting

* adding unit tests to upload actions and reducers (#30161)

* adding unit tests to upload actions and reducers

* use fetch-mock

* Adding component unit test

* Check CSV Upload permissions on the frontend (#30094)

* CSV / Admin Upload Settings (#29709)

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* allow formData uploads

* basic CSV uploads from collections

* add dropzone support for collections

* Add CSV uploads feature and have postgres and H2 support it (#29636)

* Add csv uploads feature and have postgres support it

* Add h2 support

* Add defsettings

* Add table prefix setting

* add data upload settings

* Add newline

* update settings mock

* Whether or not uploads are enabled

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>

* properly check if user can upload to a collection

* do some db checking

* don't check admin status for upload notification

* misc cleanup

* fix rebase

* add loggedin check to statusListing

also combines container + view component into a single file

* update unit tests

* update tests

---------

Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>

* Csv/misc bugfixes (#30219)

* Support empty header names

* Handle long CSV names more appropriately

Handle duplicate column names in CSVs

* Test fix

* Partition CSV inserts a little more intelligently

* Test fix

* Remove 'duplicate' from duplicated column names

* More intelligent sampling

* Add comment re. Postgres table-name-length-limit

* Quote column names, clean up tests

When unquoted, we couldn't use reserved words (true, false, group, etc.)
as column names

* Fix test

Now that we quote things, H2 is case-sensitive

* Use dialect-appropriate quoting

Also clean up some card-filtering tests so they work with a dirty DB

* Fix deduplication

* Remove use of setting/get

* Remove unneeded migration

---------

Co-authored-by: Callum Herries <hi@callumherries.com>

* Fix test (#30383)

---------

Co-authored-by: Ryan Laurie <iethree@gmail.com>
Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: Ryan Laurie <30528226+iethree@users.noreply.github.com>
Co-authored-by: Callum Herries <hi@callumherries.com>
Co-authored-by: Nick Fitzpatrick <nick@metabase.com>

* MLv2: use kebab-case keys in Clojure & camelCase in JS

* Revert changes to tests

* MLv2: port time interval description code

* Port more description logic

Look for options at the last position in MBQL clauses

Fixes #30280.

* initial changes

* test fix

* update

* update

* update

* update

* update

* address comment

* add other sso methods

* notify slack of failing CI on master or release branches

* review comment updates

* Fix typo in password_reset.mustache

* Rename "becase" to "because"

* Do not record revision if the object is not changed

* [MLv2] Fix inner native query key

* Unskip tests

* Move rename-keys because source-query uses :native and native query uses :query to point at sql, so we have to see how the stage is used

* Don't touch parameters in conversion

Implement breakoutable-columns

* initial changes

* fix test

* add ldap

* fix tests

* update migrations

* swap

* swap ldap

* fix test

* update test

* fix test

* fix test

* address comments

* update

* [MLv2] Remove invalid query parts when summarizing

* Address PR feedback

Adding better handling for 429s due to billing issues. Probably worth investigating all of the openai failure modes in the future for even better UI ergonomics. We could also just return the OpenAI message, but that would feel kind of non-Metabasey since we're exposing a 3rd party error message.

Fixes #30262.

* Use type/JSON instead of type/SerializedJSON

* Tidy migration

* Update migration

* Fix rollback migration for h2

* whitespace

* Add test for migration

* Fix test

* Add rollback

* whitespace

* Test JSONB type as well as JSON

* Don't fingerprint JSON columns

* Remove comment, that might be wrong in the future

* Update test

* Add tests for fingerprinting

* Use base-type JSON for fingerprinting base query

* Add test for visibility-type=details-only

* undo .

* Change migration id

* Fix migration test

* Merge master

* Exclude mariadb from tests

* Make is-mariadb? public

* Migration for adding nfc_enabled

* Add nfc_enabled to field settings

* Update describe-nested-field-columns to only unfold fields that have not been disabled

* Remove spy

* Tidy

* Fix

* Clear nested fields immediately if folding is disabled

* Clear nested fields on nfc_enabled change

* Trim trailing whitespace

* Tidy

* Fix

* Add enable-json-unfolding-test

* Move to field api test

* nfc_enabled -> json_unfolding

* Tidy test

* Make json-unfolding in database details just the default for new settings

* Restore original visibility-type logic

* Fix

* Sync field json_unfolding according to db json_folding

* Rename to json-unfolding-default

* Add test for the case when the json-unfolding is false for the database

* Implement default json unfolding for first sync

* Update comment

* Update comment

* Restore PUT field

* Migration for populating json_unfolding for mysql and postgres

* Remove migration and use default on the frontend instead

* Update json-unfolding database setting copy

* Move Unfold JSON setting under semantic type setting

* Separate sentences with spaces

* Capitalize

* Restore handleChangeSemanticType

* Use base_type TYPE.JSON instead

* Change order of migrations

* Add h2 migration

* Add test for migration

* Update test description

* Remove validCheckSum

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Use ?? instead of ||

* Fix tests

* Remove outdated serdes stuff

* Remove unnecessary and

* Fix mysql migration

* Remove unused require

* Fix test

* Add false default value for json_unfolding

* Fix json-unfolding nil case

* Other suggestions

* whitespace

* Tidy describe-nested-field-columns

* Update comment

* Remove unused clear-nested-fields!

* Remove unused return value

* Fix H2 migration to use base_type not database_type

* Always set json-unfolding during sync

* Fix test

* Fix test

* Fix test

* Add comment explaining nested-field-column support for MySQL

* Fix tests

* Fix tests

* Fix test

* Fix test

* Default json_unfolding to false for new fields

* Fix merge

* Add json_unfolding to mock tables

* Don't capitalize prepositions

* Update setting description

* whitespace

* whitespace

* Fix fetch_metadata

* Fix fetch_metadata

* Fix tests

* Fix test

* Fix clj-kondo

* Remove postgres database-supports test

* Fix postgres test

* Fix postgres test

* Fix mysql migration

* Fix clj-kondo

* Don't test mariadb

* Fix mysql test

* Fix mysql json-unfolding nil case

* Add comments to test

* Add upterm step to mariadb test

* Move upterm step before tests

* Fix postgresql migration

* move upterm step to test-driver action

* Remove upterm step from drivers.yml

* Comment out everything else in test-driver

* Fix mariadb migration

* Remove upterm action

* Whitespace

* Only handle JSONObjects with JSON_VALUE, not JSONArray

* Fix fields.sync_metadata/update-field-metadata-if-needed!

* Add comment explaining `json-unfolding-default`

* Add comment for JSON_VALUE test

* Update json-unfolding-default-true-test to use fresh db

* Fix test descriptions

* Don't capitalize setting display name

* Fix update json_unfolding

* Fix unfolding json for only one JSON column, not all

* Add e2e test for JSON unfolding setting

* Fix test

* whitespace

* Fix test

* Fix json-unfolding for nested field columns

* Fix mysql and postgres tests for nested field columns

* Coerce nil json-unfolding to false for sync_metadata

* Add test for nil json-unfolding for sync_metadata

* Fix test

* Don't update json_unfolding from intial sync

* Add json_unfolding to update-field-test

* Fix tests

* Use cypress-testing-library commands

* Fix e2e test

* Fix e2e test

* Add json-simple to deps

* Update uberjar.yml

* import org.json.simple

* Replace SQL migrations and H2 UDF with a clojure migration

* Remove json-simple from deps

* Restore uberjar.yml

* Rename to set-json-unfolding!

* Make the DB JSON unfolding setting override the field setting, instead of being a default.

* Remove unused var

* Fix error

* Remove custom clojure migration

* Fix migration

* Fix migration again

* Fix migration and kondo

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

[ci skip]

Renamed Label to meet best practices.

[ci skip]

* add metabot tracking

* remove redundant metabot_page_view event

* mock metabot feedback endpoint

* feedback, add one prop

* Hide column types and expose a function (isa) to work with them

Part of #30054.

Effective and semantic type information are removed from display-info.
This forces the user of the API to use explicit type checks using the
new function isa.

Co-authored-by: Jeff Bruemmer <jeff.bruemmer@gmail.com>

* First pass at recent_views stored in user local setting

This modifies the view log event handler to write a user's views to their local settings map (stored as json in the
appdb).

some notes on what causes a view

:card-query
 - occurs when collection with pinned cards/models is opened. maybe shouldn't count as a view

:card-read
 - happens when you view a card, like, load it in the UI and are looking at it

:card-read AND :card-query
 - happens when you view a model, like, load it in the UI and are looking at it

:dashboard-read
 - happens when you view a dashboard

:card-create
 - when you save a card
 - this will probably cause double counts because it will save and then trigger a :card-read right away too

:table-read
 - when viewing a table.
 - when selecting a db/table in dropdowns in the Question editor

I think we need the view log topics to be much smaller: #{:card-read :dashboard-read :table-read}

* WIP test for the recent_views user local setting.

This isn't totally correct yet. I think it doesn't clean up after itself correctly. That's pending.

And, I have to investigate because :table-read isn't working either. Still worth pushing this progress!

* Test that works! All potential view events need :actor_id

* Working test for /api/recent_views

* Order of views in user-local recent-views works now

If a user views something they've recently viewed, the homepage order doesn't change. Now it will, as in, the most
recently viewed item will always be the first item in the vector.

* One time recent_views expensive query done in the user-recent-views getter

The logic of filling the recent-views setting was first written in the activity api namespace. Now it's handled where
the setting is defined, which lets anyone else consuming the setting not worry about filling it if its empty.

The setting can be reset with `nil` which will cause the query to run again. Otherwise, if the list of recent items is
an empty vector, that is considered valid and will not re-run the query.

Updating the setting per view is handled still in the view log handler, implemented in the private function `update-users-recent-views!`.

* Add `most-recently-viewed-dashboard` setting

The most recently viewed dashboard for that user, in the past 24 hours.

This is not meant to be a permanent solution to providing this bit of info for the user, rather, it's a 'stop gap'
while the audit tables are reworked.

* Add test for most recent dashboard setting

* Pass :context to the view log fns

This lets us filter out views triggered by pinned cards.

* Fix tests failing after adding :context to view metadata

* See if card-query can help us filter out pinned item views

* Tests should send :card-query events

* Since we use :card-query, we have to filter out dashboard context too

This should prevent views being recorded if someone just opens a dashboard.

* ViewLog Cleanup to see if it'll let other tests pass

* drop non-viz percy tests, add a workflow that assigns visual label which triggers percy tests runs

* remove push trigger in require-percy-test workflow

* fix typo

* driveby type improvement

* Fix not showing single or multiple values options

When a parameter is connected to a question with template tags

* Review: unit tests instead

* Fixes inlining of numeric params in native queries
* Updates namespaces to use sql.qp/with-driver-honey-sql-version

Inferencing for models was switched from an LLM to an embedding selector. Unfortunately, this did not do a very good job as too much context was not retained. This change goes back to using the LLM for model inferencing.