Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* feat: BigQuery nested fields

Updates dataset definitions to allow `nested-fields`

Adds the `::sql.qp/nfc-path` to include nfc-path in the field
identifier.

* Add bigquery to nested-fields feature. Rework geographical-tips to allow adding to bigquery

* Parse RECORD type into maps

* Fix tests

* Fix tests

* Fix formatting

* Handle arrays of records

* Fix formatting

* Conditionally add nested-fields

* Fix test

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* highlight 1

* highlight 2

* Add Kondo linter to disallow hardcoded driver names in tests

* Fix all tests using hardcoded driver names

* Test fixes 

* Appease Kondo

* Remove normal-drivers-except

* Test fix 

* Test fixes 

* MongoDB test fixes

* link fixes

* more links

Add Azure managed identity support 

---------

Co-authored-by: Case Nelson <case@metabase.com>

* Post sync hook stub for implicit joins testing stub

* Add join alias to field lvalues to enable sorting on joined fields

* Disable :foreign-keys on Mongo

* Require :left-join support for implicit joins instead of :foreign-keys

* Update implicit joins tests

* Adjust sync-fields-test

* Update implicit joins feature check test

* Transform post-sync-hook to normal function

* Add foreign key relationships only for dbmses without :foreign-keys feature

* Update test to handle Oracle correctly

* Split convoluted fn

* Avoid unnecessary computations for datasets with no fks

* Update docstring

* Fix driver usage

* Transform :foreign-keys to :metadata/key-constraints in test data loading code

* Update sync_test.clj

* Update driver_test.clj

* Update moviedb.clj

* Update dataset_definition_test.clj

* Update fetch_metadata.clj

* Update fields_test.clj

* Update driver.clj

* Update driver/sql.clj

* Set sql driver join support to true for all joins

Deriving drivers are expected to set to false where applicable.

* Update sqllite.clj

* Remove foreign-keys from spark

* Remove :foreign-keys from presto

* Remove :foreign-keys from Athena

* Remove foreign-keys from big query

Reading docs it seems fk inference should be ok. Let's see the test results. Act based on that.

* Update test_metadata.cljc

* Set key-constraints to false for bigquery

* Add foreign keys to sqlite manually until sync is fixed

* Return driver require to Athena

* Correct typo

* Add naive primary key heuristic

* Update pk fk logic to handle name components correctly

* Add alias escaping to presto

* Add ordering to test

* Add order by to test

* Update test

* Remove use of rewrite-fields-to-force-using-column-aliases in order by fields

* Add exception to alias forcing

* Different approach to exception from alias forcing

* Alternative approach for prefixing idents in bigq

* All seelcted fields by desired alias

* Rewrite only fields not from this source table

* Update test

* Enable breakout-on-fk-field-test for :left-join drivers

* Add feature comment

* Explicit joins tests foreign-keys removal

* Update nested_queries_test.clj

* Update remapping tests

* Update tests

* Update tests to handle sqlite results format

* Disable metadata/key-constraints on sqlite during tests until

* Address remarks

* Remove mt/with-mock-fks-for-drivers-without-fk-constraints

* Update bigquery test

* Update tests

* Adjust row level restrictions

* Add parameterized-sql feature

* Update comment

* Update leftovers

* Order keys

* Remove foreign keys from frontend

* Fix FE unit

* Update driver changelog

* Address review remark

* Update test/metabase/query_processor/test_util.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update docs/developers-guide/driver-changelog.md

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Update test

* Update comment and reduce expression

* Update comment

* Address remarks

* Fix merge

---------

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* tests for drivers with spaces in their identifiers

* Add driver feature

* Fix tests

* Remove ambiguous alias for sqlite

* Try to fix bigquery and sqlite tests not seeing left-join support

* Add explicit support for left-join because of foreign-keys during test hacks

* Handle left-join support weirdness

* Fix test

* Reuse connection for test data loading & JDBC data loading overhaul

* Revert silly change

* Remove unneeded logic for recording which datasets are loaded or not-yet-loaded

* Ok so I guess we do need the code to detect duplicate dataset loading after all

* B I G  improvements

* Appease Kondo

* Test fixes 

* More test fixes 

* Fix dataset-already-loaded?-test

* Fix test

* Fix Presto JDBC and greatly improve test data load speed for it

* Postgres test fix 

* Fix compilation errors

* MySQL fixes 

* Fix syntax error

* MySQL fixes 

* Implement `dataset-already-loaded?` for Athena and cache the set of existing databases

* Fix Redshift

* Fix Presto JDBC

* Vertica test fix 

* Fix Oracle dataset-already-loaded?

* Test fixes and parallelize some API tests.

* Test fixes 

* Oracle test fix? 

* Another test fix 

* Stop putting random spaces in code bodies this is not JavaScript code

* with-temp-vals-in-db should merge in original value of Database settings

* The most test fixes 

* 3 or 4 test fixes 

* Add schemas around stuff to catch errors

* Appease Kondo

* Implement dataset-already-loaded? for vertica

* Implement dataset-already-loaded? for SQLite and Snowflake

* [WIP] Testing driver filters

* Only test returned values in CI

* Add :uuid-type for dbs that support that type (test oracle's support)

* Try again for oracle test, redshift does not support uuid-type

* Add tests, support where available

* Add empty/null tests

* Adjust test query for athena

* Reset logging

* Fix comments

* Add Athena uuid conversion

* Address PR review

* Add comments

* Address PR comments

* Use varchar for portability. Add inline uuid casting for athena

---------

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* Support compiling SQL with inlined parameters directly

* Test fix

* Make `format-honeysql` a multimethod

* Add test for #32543

* Test also fixes #44915

* Add explicit test for #44915

* Better test for #44915

* More test fixes 

* Fix Kondo error

* Calculate metadata without running queries [JDBC]

* Appease Kondo

* Test fixes 

* Overhauled Card metadata calculation code

* Test fixes 

* Kondo fix 

* Ok maybe I fixed stuff

* Work around https://github.com/aws/amazon-redshift-jdbc-driver/issues/118

* Maybe fixx the Cypress tests

* Experimental possible e2e test fix 

* Test fixes 

* Do things a little differently.

* Try re-enabling the field refs

* Migrate brand

* Migrate brand-light, brand-lighter

* Fix unit tests

* Code review fixes

* Fix CI

* Remove Google Analytics driver

* Remove more GA-related tests

* Un-remove tests that aren't related to GA

These were added in #42602 but I neglected to update the changelog.
Their defaults should serve for most drivers.

* Remove unused `metabase.api.transform`

* Copy the deps graph improvements from my other PR

* New combined X-Rays API namespace

* Fix kondo error

* [MLv2] Add `preview-expression` to eval an expression on sample data

This is needed to show the previews in the UX for the combining columns
epic #39977.

Fixes #39979.

* [MBQL lib] Add "Combine columns" drill-thru

This is following the original plan of using a drill for this. It's hard
to combine that with the "preview" functionality.

See [thread](https://metaboat.slack.com/archives/C06P22KS4JH/p1712264174056569)
for discussion of how we might approach that issue.

Fixes #39978.

* [MBQL lib] Add "Combine columns" drill-thru

This is following the original plan of using a drill for this. It's hard
to combine that with the "preview" functionality.

See [thread](https://metaboat.slack.com/archives/C06P22KS4JH/p1712264174056569

)
for discussion of how we might approach that issue.

Fixes #39978.

* [MLv2] Add `preview-expression` to eval an expression on sample data

This is needed to show the previews in the UX for the combining columns
epic #39977.

Fixes #39979.

* [MLv2] Add `preview-expression` to eval an expression on sample data

This is needed to show the previews in the UX for the combining columns
epic #39977.

Fixes #39979.

* Revert "Merge branch 'mblib-preview-expressions' into combine-columns"

This reverts commit dac17f84aaae09906a206106290f7f58dea659bd, reversing
changes made to 2934fab19db495ca8104801d41bc8aba457e401f.

* Revert "Merge branch 'mblib-preview-expressions' into combine-columns"

This reverts commit 2934fab19db495ca8104801d41bc8aba457e401f, reversing
changes made to 6a43c22e78fb616c7ac0b024811b2989a1ae0f20.

* Revert "Merge branch 'mblib-preview-expressions' into combine-columns"

This reverts commit 6a43c22e78fb616c7ac0b024811b2989a1ae0f20, reversing
changes made to 2257333a8ae69f0c9a6f6fdf0c8f5d837263f97a.

* [FE] "Combine columns" drill thru (#40082)

* Type Section['icon'] properly, including the hack for summarize section

* Add combine columns section

* Add types for combine columns drill

* Add combine columns drill component stub

* Mock combine columns drill

* Add component stub

* Encapsulate typing hacks

* Extract CombineColumnsDrill

* Extract ColumnAndSeparatorRow

* Improve naming, add button to edit separators

* Translate string

* Improve styling

* Separator input

* Handle removing columns

* Rename lib.ts to utils.ts

* Implement onSubmit

* Add getNextColumnAndSeparator

* Use form for a11y

* Handle vertical overflow

* Handle horizontal overflow

* Extract formatSeparator

* Use proper translations

* Adjust min/max sizes

* Fix scrollbar being unnecessarily shown

* Display source column name

* Avoid serializing the entire column and putting it into DOM

* Use module.css extension

* Format code

* Improve a11y

* Mock combineColumnsDrillExpression and previewExpression

* Add Preview component

* Move preview logic to Preview component

* Style Preview

* Format code

* Move styles to CSS modules

* Update UI to latest designs
- move "Add another column" button
- remove "Separated by X" button
- always show separator inputs

* Remove displayInfo mock

* Add asReturned

* Update CombineColumnsDrillThruInfo

* Integrate new way of working with combine columns drill

* Fix generating expression names

* Integrate drill with click actions

* Integrate new preview workflow

* Revert "Update UI to latest designs"

This reverts commit cda039dffe9d452c0866f63800e2e887a540fe67.

* Use previewExpression interface correctly

* Use new preview_expression API

* Make sure columns are created with original query and stage index

* Extract usePreview and handle preview errors

* Update popover title

* Move add column button

* Fix outline being cut off

* Style preview label as per design

* Format empty separator

* Make preview scrollable

* Use project convention

* Simplify ScrollArea usage

* Use ScrollArea in CombineColumnsDrill

* Update comment

* Remove Lib.previewExpression

* Clean up the preview after removing Lib.previewExpression

* Implement getPreview

* Change default column preview to 'text'

* Remove usePreview hook which is no longer in use

* Rename Preview to Example

* Use different placeholder for separator input

* Add whitespace placeholder

* Add ColumnPicker boilerplate

* Export Input

* Use QueryColumnPicker for ColumnPicker

* Pass width to AccordionList

* Add sequences popover events with tippy

* Remove unused ColumnOption helpers

* Add testid to Example

* Add e2e test for combine column in header

* Select content of separator input on focus

* Reduce padding on example

* Add monospace variant to Text

* Use monospace variant in Example

* Use monospace font in textinput

* Match faux-select label styles to our TextInput label styles

* Add missing color

* Remove !important

* Use pre whitespace so consecutive spaces show up

* Remove empty default

* Add example for boolean

* Remove !important

* Use should.have text over contain

---------

Co-authored-by: Romeo Van Snick <romeo@romeovansnick.be>

* Remove duplicated code

* Export useMantineTheme from metabase/ui

* Use useMantineTheme instead of reading theme directly

* Remove setTimeout in popover

* Use plain t instead of jt

* Move Popover dropdown sequence hack to metabase/ui

---------

Co-authored-by: Braden Shepherdson <braden@metabase.com>
Co-authored-by: Kamil Mielnik <kamil@kamilmielnik.com>

Previously they could only be added to the query with `Lib.extract`, but
this returns the expression the extract would add.

* Fix add-alias-info matching for nominal field literal refs

* Minor cleanup

* Minimize diff

* A bit more cleanup

* Cumulative aggregations w/ multiple breakouts should reset counts (#2862)

* Update dox

* Do the nesting FIRST before adding alias info.

* Remove unused

* FE for Offset()

* Cleanup stray code

* Prettier

* Kondo

* Test fixes 

* Better error message when no breakout

* Support offset as an expression with no breakouts. Still require an ORDER BY tho.

* Fix most stuff

* Hack city??????

* Revert nest-query changes and disable the offset-no-breakout test for now.

* Appease TypeScript

* Test fix 

* Oracle & MySQL test fixes

* Test fixes 

* Remove maxWorkers for FE unit tests

Column extractions are "canned" expressions based on a column's
type. For example, we might extract the weekday from a temporal column,
or the domain from an email or URL column.

This logic already existed inside the `column-extract` drill; this pulls
it out as a top-level concept, since extractions are also being
integrated into the notebook editor apart from drills.

Part of the follow-up for Extract Column epic #38964.