* Fix bug where grouping strategy would NPE when fingerprint was `nil`

The axes grouping relies on fingerprint data existing, AND for now only works on :type/Number columns, so this bugfix
guards against those situations where the logic would fail.

The bug that brought this to my attention came up when there was a Custom column. This set the col-meta count to be
one greater than the count of the 'real' columns, which threw off my old assumptions, where if the grouped-columns
count was less than the total col-meta count, you had a potential grouping. This broke in part because some custom
columns are essentially derived from a real column, and so will most likely be groupable with the column they're
derived from.

Anyway, this filters out columns that won't work, so we hopefully eliminate the problem.

* Update src/metabase/pulse/render/body.clj

Co-authored-by: Tim Macdonald <tim@metabase.com>

* Fix some broken logic

Co-authored-by: Tim Macdonald <tim@metabase.com>

* First pass at grouping axes

This PR uses the results_metadata key to look at fingerprints for numerical axes to try determine if each series on
the Y axis can be sanely represented on the same axis.

This is done by calculating an overlap (some value between 0 and 1) and grouping all axes on the LEFT if they pass the threshold, which is
some value between 0 and 1. The overlap is always calculated when there is SOME overlap between the ranges of each
axis, and is calculated as:

```clojure
(/ (- (max maximums) (min minimums))
   (- (min maxiumums) (max minimums)))
```

This is done to try catch situations where one column's range is entirely inside the other, but is much smaller (- max
min); such a case would have a small percent overlap by the above calculation, and implies that it might be better to
split the axes.

* Address feedback.

* Fix shape of data in tests of 2 private fns

* Add test for split axes

* Fixed error in test util

* render-utils does with-redef, disallows parallel tests. Makes sense

Card, Dashboard, Dimension, Metric, NativeQuerySnippet, Segment and
Timeline all now using their `:name` fields for the file label.

File names should now be more human-friendly.

* Initial support for pg for `date-diff`

* Make the useful-dates closer to one day

* Add simple FE stuff

* shorter test bodies

* Ensure we can use datediff functions in arithmetic expressions

* Correctly disable datediff for redshift

* simplify var names

* Support week

* cleanup test

* :datediff -> :datetimediff

* ngoc's suggestions

* Better acceptance test for datetimediff

* sort ns

* embrace the different cases for results

* bigquery day month year

* Reverse args

* Update test

* Centralize tests

* Change postgres day, month, year behaviour

* Refactor keep identity

* Tidy tests

* Tidy

* Fix bigquery week

* Add week tests

* Fix bigquery week

* Change mysql day, month, year behaviour

* Add test for hour, minute, second

* Fix postgres hour minute second

* Formatting

* Fix bigquery hour, minute, second

* Formatting

* Fix postgres timestamptz

* WIP

* Allow literals in datediff clauses

* Uncomment tests

* Fix bigquery when reporting timezone is not UTC

* Linting

* Moving away from dataset based tests

* Add timezone tests for week and tidy

* Remove unused import

* Consolidate tests

* Remove with-time-column dataset

* Remove more-useful-dates defdataset

* Remove redshift driver WIP

* Typo

* Move DatetimeLiteral clause into DateTimeExpressionArg

* Try changing test order

* Remove mt/with-report-timezone-id nil

* Add year report timezone tests

* Rename

* Rename

* Remove unused tables from useful-dates

* Remove useful-dates

* Update helper-text-strings

* Tidy

* Swap order of mt/with-report-timezone-id

* Change with-report-timezone-id; notify databases after running test

* Use temp setting for report-timezone instead

* Update helper-test-strings

* Handle literals in `datetimediff-base-base`

* Update src/metabase/driver/postgres.clj

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>

* Simplify postgres second

* Tidy: prefer hx arithmetic functions

* Drop coercion for string timestamp args for now

* Extract helper

* Revert "Change with-report-timezone-id; notify databases after running test"

This reverts commit 7abb543bed13d9d13295d0e00ad1293a40e40d53.

* Use ->timestamptz

* Use hx/->timestamp

* Rename dataset

* Undo changes to datetime-arithmetics? as these will not match by default

* Use proper format for offset datetime literals

* Shorten datetime literals in tests

* Removing notify-all-databases-updated from report-timezone setting

* Fix datetime-arithmetics test

* Revert "Fix datetime-arithmetics test"

This reverts commit 9141582508170bdbe99c4aa74bc4444e41475be6.

* Revert "Undo changes to datetime-arithmetics? as these will not match by default"

This reverts commit 9cb05f5475b24a78cb89601c918f3440606cc525.

* Add UTC timezone to tests

* Coerce strings to datetimes for ISO formats

* Revert "Removing notify-all-databases-updated from report-timezone setting"

This reverts commit 37356435a2e3981deca9ae76118d857babb2aada.

* Add comment to postgres driver implementation

* Formatting

* Disable datediff from redshift for the moment (for ever?)

* Override redshift driver/database-supports?

* Fix comments mixed up by refactoring

* Fix comments mixed up by refactoring 2

* Tidy comment

* Rename datetimediff to datetime-diff/datetimeDiff

* date-add -> datetime-add

* Linting

* add datetime-subtract to `datetime-arithmetics?`

rework tests a bit as well.

* Update docstring

* Use ->temporal-type and trunc to handle report-timezone for bigquery

* Log errors caught during sync steps (#26306)

* log errors caught during sync steps

* remove accidental extra parens

* Errors combining datetime interval addition with datetime functions (#26279)

* Add failing tests

* Fix failing tests

* Update shared/src/metabase/mbql/util.cljc

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* datetime-add and datetime subtract should annotate type by col type

* Fix infer-expression-type for datetime-add/subtract with second, minute, hour

* Undo last commit; they actually always return :type/DateTime

* Fix test based on last commit

* Undo unrelated refactor

* Only test drivers that support expressions

* Only test drivers that support expressions, again

* Update tests from legacy mbql

* Change infered-col-type to be a function again, not macro

* Fix test

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* whitespace

* Remove comments

* Add explanation for datetime_diff

* Refactor: replace cast and add `mt/with-driver :bigquery-cloud-sdk` where report-timezone is relevant

* Add failing tests

* Fix failing tests

* Technically it should be bigquery-type

* Update modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk.clj

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Move documentation to metabase.mbql.schema

* Update DatetimeDiffUnits

Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* Add error type and optimize case expressions to driver implementations

* refactor for brevity

* Fix error with postgres

* Fix error with postgres

* Handle string literal parsing in wrap-value-literals

* Remove ->timestamptz

* Add comments + TODOs showing arithmetic expressions should return numeric values

* Add explanation of arithmetic expression as docstring

* Add test for normalize-mbql-clause-tokens

* Remove unused import

* Remove unused form

* Switch tests to use attempted-murders dataset

* Undo optimization that broke tests

* Validate non-temporal types for bigquery

* Add error handling for incorrect types

* Fix mysql type checking

* Fix mysql type checking

* invalid-parameter -> invalid-query

* Use date-trunc and extract

* Remove unit error handling

* DatetimeLiteral -> DateOrDatetimeLiteral

* Remove unused binding

* Fix mysql type checking

Co-authored-by: Callum Herries <hi@callumherries.com>
Co-authored-by: Cal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Noah Moss <32746338+noahmoss@users.noreply.github.com>
Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>
Co-authored-by: Cam Saul <1455846+camsaul@users.noreply.github.com>

* add app DB and version to common snowplow event context

* use .getDatabaseProductName for db type

* fix kondo error

* add new instance schema with app db type and version fields

* add maxLength to new fields

* address cam's feedback

- Add tests to check when bucketing by week and week-of-year behave consistently, Closes #4910
- While doing the above, found that sqlserver does not respect `start-of-week` setting, so fix that. Fix #25356

This reverts commit 2a857d7d.

The service provider's output has changed back again.

* realize lazy seqs during exports

* add test & helper

* update based on tamas's comment

* use mapv

* remove unused import

* try to fix test

* switch to using just a LocalDate for the second test since the ZonedDateTime test was failing for certain drivers and not others, and that isnt relevant to this test

* Add failing tests

* Fix failing tests

* Update shared/src/metabase/mbql/util.cljc

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* datetime-add and datetime subtract should annotate type by col type

* Fix infer-expression-type for datetime-add/subtract with second, minute, hour

* Undo last commit; they actually always return :type/DateTime

* Fix test based on last commit

* Undo unrelated refactor

* Only test drivers that support expressions

* Only test drivers that support expressions, again

* Update tests from legacy mbql

* Change infered-col-type to be a function again, not macro

* Fix test

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* [:field-id id] => [:field id nil] or $name. Except places where tests
are meant to check normalization

* [:datetime-field [:field id] unit] -> [:field id {:temporal-unit unit}]
or !unit.name

* [:fk-> [:field 1] [:field 2]] => [:field 2 {:source-field 1}]

* Uses all text scorers in the final result

- instead of just the maximum one
- add tests

* add prefix scorer test + fix text-score-with

* linter fixes

* pass in number of results to find

* refactor test function

* fix linter by removing unused namespace: metabase.util

* limit arity of serialize to 3

* make oss-score and ee-score different things

- They were defined to be exactly the same, but should be different!
- Update some tests that broke when a test function was fixed

* remove extra let

* move rseq back out of sorted-take

* improve test feedback

* force weight of text based scorers always weigh 10

* handle 0 score/weights when normalizing scores

* add nil check

* fix more subtle test differences

* more test fiddling

- still test that :offset and :limit respect limits

* reuse bit->boolean from api collection

* clean up some tests

- filter -> remove
- replace some magic numbers
- revert to testing entire maps instead of names of sorted items

* add test, docstring, and weight

* sort ns requires

* responding to most of the review comments

* start our zero-score sum check with 0

* do not tokenize / normalize nil raw-search-string

* force equality in basic search test

* modify test to work in dev and test environments

* use display_name in results when appropriate

- This was looking for the _first_ column that had a non-zero score, but
actually we need to consider all relevant columns.
- Uses them to figure out if there is a display name, and if there is,
to use it.
- Coppied over the logic about showing :context from the prior approach

* rename dateAdd to datetimeAdd and dateSubtract to datetimeSubtrac

* Config from file: fix User upsert not hashing password

* Move metabase.config.file code => enterprise directory

* Support setting Settings in config.yml

* Setting name has to be a keyword since that's how YAML gets parsed

* Test fix 

* Fix Kondo error

* Fix Kondo errors

* Remove unused namespace

* Move metabase.config.file code => enterprise directory

* Test fix 

* Fix Kondo error

* Fix Kondo errors

* Remove unused namespace

This makes them less likely to collide, which was a problem on some
larger instances.

A following PR will generate entity_id from the hashes before
(de)serialization, allowing for easier merging later (since entity_id is
indexed in the database).

* Add collection to card tag autocomplete results

* Remove comment

* Tidy test

* Remove unused import

* Bump width

* static pie labels

* visual tests

* Quick fix to prevent non-temporal labels from datetime formatting

* show values pie BE

* Text fix 

* add specs

* renamt show_labels to show_values

* Test fix 



* update spec

Co-authored-by: Adam James <adam.vermeer2@gmail.com>
Co-authored-by: Tim Macdonald <tim@tsmacdonald.com>

* Fix Collection created_at migrations for H2

* Fix MySQL/MariaDB set created_at for empty collections

* Fix MySQL/MariaDB one last time

* Add created_at to Collection

* Simplified migrations [ci skip]

* Add SQL for H2 and MySQL; add tests

* Sort namespaces

* Test fixes 

* Test fixes

* Test fixes 

* Make sure MySQL/MariaDB Liquibase generator sets the current timestamp function for ADD COLUMN

* Column settings correct keys. Show Axes labels logic defaults 'true'

These are two small changes related to static viz column formatting.

The show axes labels logic change fixes a faulty assumption from an earlier PR where it was assumed that if
:labels_enabled key was NOT present, then it was 'false'. The default however, is 'true', so the check had to change
to reflect the frontend.

Column settings have to be formatted so that the js code running in the backend looks in the right places. For
example, `:number-format` must become `:number_format`. I move the function that does that change up so that it can be
rolled up into `settings-from-column` to reduce the number of call sites for `update-col-for-js`.

* Adjust number format fn to consider :decimal_digits of currencies

* Simpler date-separator fix

* Prevent non-temporal labels from being passed to datetime formatter

* Table Temporal format columns are improved

The column headers will now be properly looked up, the key format was incorrect before, causing the `:column_title`
lookup to return nil, which ignores when a user changes the column name.

Table columns with temporal formatting settings come in with `:unit` `:default`, so that case was added to the
datetime formatter. Additionally, a helper function was added to pull the correct column viz settings to pass to the datetime.

* Change some table rendering to look for '_' keys instead of '-'

And also allow the field-id to take the form `[:field-id 10 nil]` OR `[:field-id 10]`

* Added test utils for static-viz rendering. Docstrings for details.

A big set of functions to enable cleaner static-viz rendering tests.

Next push will be thorough table tests, and some other static viz cleanup

* Add some viz-scenarios to help with table testing

* Lint

* Wrong map here

* Currently unused bindings

* Writing tests with render-test utils now, and catching new bugs

* Use column's field ref to maybe get the column settings

* Adjustments so that column-settings are properly grabbed

* Number formatter no longer assumes decimal places

The frontend will only display decimal places when strictly necessary OR when the column formatting dictates the
number of decimal places to show.

The number formatter on the backend made different assumptions about decimal places, so precision may be lost OR
needlessly displayed.

The formatter now checks the actual value passed in and adjusts the format string according to the decimal places
present in the value, not just via column-settings, though the column settings will still take priority.

* Shame on me. inline def is not needed here :/

* Adjust number format tests to assert that now we match FE behavior

* Really trying to get formatting buttoned up.

* Matching frontend's default format behavior for numbers

* Some tests failed because they assumed two decimal places always

* Values less than one still round after 2 sig-figs

* Column formatting on tables follows expected default behavior

* Getting public-settings included in number and datetime formatters

* Test for custom formatting public settings

* Removed unused require.

* address some PR feedback

* Try to clarify the logic in the number formatter

* Add some clarifying comments

* Use the existing viz tools to help create field-refs properly

* Cleaned up column-name impl. to use normed field-refs. Much nicer!

* Remove some unnecessary tests, and add a negative scientific num

* Simplify table columns test

* Simplify and add more format settings to column-formatting test

* Unused require

* Another one

* Fix format of column refs in test

* Remove unused thread

Previously, some automatic behavior was causing problems in serdes.
(The worst example is generating an `entity_id` on insert while
deserializing an entity we don't own - if deserialized again it would be
duplicated.)

A whole cascade of design choices fell out of this problem:
- `insert!` would generate `entity_id`s, so use `simple-insert!`
- But `simple-insert!` doesn't convert eg. MBQL `:definition` maps back
  into JSON strings
- We got the maps as Clojure data and not JSON strings because `select`
  and `simple-select` both run `post-select` and parse the JSON.
- So we ended up with a raw query on the select side and `simple-insert!`
  on the storage side.

This change unwinds that whole stack, and instead uses a dynamic
var to suppress the few pieces of `pre-insert` and `pre-update` logic
that causes problems.

The end result is much cleaner, and much more consistent with the rest
of Metabase's backend logic.

* Ensure static visualizations with multiple cards are rendered

[Fixes #21559]

* [static viz] Render multiple scalars as a bar chart

* add static viz build watch command

* reload graalvm context each time in dev mode

* run static viz with yarn:dev

* address comments

* use watch webpack env var

Co-authored-by: Noah Moss <noahbmoss@gmail.com>

* Initialize Metabase from a file

* Code cleanup

* Support optional template tags; more dox

* Finish documentation for ns str

* Force the first User to be an admin

* Tests for DB creation

* Add test to ensure that you can set User password with an {{env ...}} template

* Remove data app permission code (#26016)

* Revert "Use "apps" collection namespace for app collections (#25963)"

This reverts commit a1894202.

* Revert "Implement EE/Pro app permissions (#25764)"

This reverts commit cf9b1bf5.

* Revert "Implement global app permissions for the "All Users" group (#25679)"

This reverts commit 63e950f2.

* Revert "Require data permissions for executing actions (#25784)"

This reverts commit 732bf6b1.

* Revert "Prevent modifying app permissions via the collection endpoint (#25684)"

This reverts commit 5d3f8279.

* Revert "Implement DB specific execution permissions (#25629)"

This reverts commit 141a13ce.

* Revert "Introduce /execution/ permission (#25529)"

This reverts commit 57cbb921.

* Keep the app permission revision table

* [Apps] Remove emitters (#25999)

* [Apps] Remove emitters

Emitters were created when we thought that we would be adding actions to
multiple places in metabase. Instead we've coalesced to actions being a
core feature of data-apps and are executed from within them. Attaching
actions to models, as a form of re-use rather than a form of execution,
means that actions can still be executed directly from a model, if we
want to allow that in the future.

* Fix lint errors

* Move perform-action! tests out of the api.

Since we want to keep the implicit bulk functionality but remove those
endpoints move the tests to the actions namespace.

* Fix linter

* Change driver test to use function instead of endpoint

Co-authored-by: metamben <103100869+metamben@users.noreply.github.com>
Co-authored-by: Case Nelson <case@metabase.com>

* fix detection of backend files when generation i18n edn

* appease whitespace linter

* tweak comment

* fix cypress test

* appease linter

* Use eduction instead of threading

* Update bin/i18n/test/i18n/create_artifacts/backend_test.clj

Co-authored-by: Tim Macdonald <tim@metabase.com>

Co-authored-by: Tim Macdonald <tim@metabase.com>

* [Apps] Remove emitters

Emitters were created when we thought that we would be adding actions to
multiple places in metabase. Instead we've coalesced to actions being a
core feature of data-apps and are executed from within them. Attaching
actions to models, as a form of re-use rather than a form of execution,
means that actions can still be executed directly from a model, if we
want to allow that in the future.

* Fix lint errors

* Move perform-action! tests out of the api.

Since we want to keep the implicit bulk functionality but remove those
endpoints move the tests to the actions namespace.

* Fix linter

* Change driver test to use function instead of endpoint

* Revert "Use "apps" collection namespace for app collections (#25963)"

This reverts commit a1894202.

* Revert "Implement EE/Pro app permissions (#25764)"

This reverts commit cf9b1bf5.

* Revert "Implement global app permissions for the "All Users" group (#25679)"

This reverts commit 63e950f2.

* Revert "Require data permissions for executing actions (#25784)"

This reverts commit 732bf6b1.

* Revert "Prevent modifying app permissions via the collection endpoint (#25684)"

This reverts commit 5d3f8279.

* Revert "Implement DB specific execution permissions (#25629)"

This reverts commit 141a13ce.

* Revert "Introduce /execution/ permission (#25529)"

This reverts commit 57cbb921.

* Keep the app permission revision table

This ensures that the api endpoints and the relevant models cannot be
created.

We enable during testing, so that we can ensure the integration points
like permissions, search, collection items, etc... will continue to be tested.