Type hierarchy except for dates and arrays fixed. Visibility setting mutation on original columns and field values on deck. Still whacking away at #21534.

Add `exclude_uneditable_data_model` and `exclude_uneditable_details` flags to database list API (#21556)

* add exclude-uneditable-data-model and exclude-uneditable-details flags to GET /api/database

* remove debugging code

* catch remove last admin exception during sync group

* lint and make sure the log are catched

* fix weird problem when try to re-def a macro in test and update log message

* count warn log only

* force log in tests

* Add Schema Filtering to Postgres dbs.

Adjust the Postgres Driver to add the Schema filtering settings to the UI in the admin page.

Fix a bug where "all" schema filter case is ignored, defaulting (incorrectly) to use only the "public" schema. This
would lead to potentially filtering away schemas that the user expects, if some tables exist inside schemas that
aren't public.

* Add Schema filtering to mysql driver

* Remove mysql to see if CI works otherwise

* Removed tap>

* Add `:is_installer` key to users returned from api/user/current

* Add `:has_question_and_dashboard` Key to current users via hydration

This is also part of the starting page project, it's a key that helps the frontend serve more specific data to the
user.

* Moved `permissions-set` function up so I can use it in hydration

* Adjust recents query and first pass at popular items query

Recents:

Before, the recent query would look at the ViewLog, which records a view on cards even when it was only 'viewed' via a
dashboard.

For dashboards and tables, we don't need to make a change.

For cards, we now query the QueryExecution table and use started_at as a proxy for the last viewed timestamp, and can
then only grab cards with an execution context of 'question'.

Popular:

We now want to have a notion of 'popular' items as well, which can look different than 'recents'. This is the first
attempt at a reasonable query with scoring system.

The score takes into account:
 - recency of view
 - count of views total
 - if the card/dashboard is 'verified' (enterprise)
 - if the cad/dashboard is inside an 'official' Collection (enterprise)

The popular score currently uses only the *current-user-id* to fetch items, so popularity is (in this commit, at
least) a per-user concept.

* Fixed mistake and renamed endpoint

* Typo fix

* Clean up QueryExecution in tests

* Indent, naming, and some simple defs

* try to reduce db calls for :has_question_and_dashboard

I've moved the fn out of the hydration system to guarantee that it only runs for a single user, and is only used in
`GET /api/user/current` (and not accidentally used in other spots via hydration mechanism).

Still don't know how to check Card and Dashboard tables more efficiently

* ViewLog and QueryExecution have different timestamps. Can't compare

Pushing this for some review, but might want to build a proper way to compare, so that recent cards and recent
dashboards/tables are sorted equally

* fix namespace sorting

* Fix the endpoint name

* Sorting mixed date time formats 'works' but not ideal

This formats the timestamps into strings and uses those for comparison. Not perfect.

Pushing in case people are trying the branch

* Use simpler db function

* Let views and runs query work with one user or all users

Popular_items are an all-users notion, but recent_views applies only to the current user.

* Unify view_log.timestamp to query_execution.started_at type

these used to both be `DATETIME` in the migration file. Then migration
168 bumped up the type of the view_log.timestamp column:

```

  - changeSet:
      id: 168
      author: camsaul
      comment: Added 0.36.0
      changes:
        - modifyDataType:
            tableName: query_execution
            columnName: started_at
            newDataType: ${timestamp_type}
```

So these were no longer the same on h2 (and possibly mysql). But sorting
viewlogs and query_executions would fail.

```clojure
activity=> (->> (ViewLog) first :timestamp #_type)
        "0x6a33e42e"
        "2022-04-04T21:57:07.471"]
activity=> (->> (ViewLog) first :timestamp type)
java.time.LocalDateTime
activity=> (->> (QueryExecution) first :started_at #_type)
        "0x7af249ac"
        "2022-04-04T21:57:07.625738Z"]
activity=> (->> (QueryExecution) first :started_at type)
java.time.OffsetDateTime
```

The LocalDateTime and OffsetDateTime were not comparable. So make them
identical types again.

* Bookmarked results should not show up in recents/ popular.

This is done in a db query to still try fetch enough items from the db to present to the user.

Filtering bookmarked items later may result in an empty list. It's possible that the firt N results are all
bookmarked, and then the frontend would have no items to show. Filtering bookmarked results out from the beginning
increases the chances of a non-empty result.

* :first_login populated with the earliest login timestamp.

If there is a first login timestamp, that is used. If one does not exist, we assume it's the first time logging in and
use an OffsetDateTime (now). This is possible since the login_history is done off thread, so on a real first login, we
might hit the db before the login is logged, resulting in an empty list returned.

On subsequent checks, we should see a proper timestamp from the db.

Since this is used to check if a user is 'new' (within 7 days of first logging in), the accuracy of the timestamp
matters on the order of days, not milliseconds, so this should be ok.

* Passing test

* Popular_items test

Tweak the create-views! function to more consistently order items. And creates views for dashboards/tables (ViewLog)
and cards (QueryExecution) in a unified way, meaning we can reliably order views, and write tests more easily.

Note that the popular_items test may have to change if the scoring math changes.

* Fix e2e test

* Fix nit and bug

- forgot to remove '0' from the and clause, so we didn't get the expected boolean
- popular_items not WIP

* another nit

* Fix popular items on the frontend

Co-authored-by: Alexander Polyankin <alexander.polyankin@metabase.com>
Co-authored-by: dan sutton <dan@dpsutton.com>

* settings global permission

* fix specs

* Enforce Setting permissions (cont) (#21464)

* settings global permission

* more api permissions enforcement

* only admin could call token checks

* address Noah's comments

* clean ns

* clean ns 



Co-authored-by: Aleksandr Lesnenko <alxnddr@gmail.com>

* hide subscriptions buttons for users with no permissions

* fix specs

* review fixes

* update spec

Co-authored-by: Ngoc Khuat <qn.khuat@gmail.com>

* first pass

* a few more endpoints

* add can_access_db_details to /api/user/common

* handle unset last-updated times
* magic number -> zoned-time-epoch
* test that slack cache gets filled when
slack-channels-and-usernames-last-update is nil.

- remove a comment

Co-authored-by: Ryan Laurie <iethree@gmail.com>

* add can_access_data_model key to api/user/current

* add exclude_uneditable flag to /api/database/:id/metadata

* clean ns

* WIP figuring out how to update perm checks for Field model

* fix errors

* fix more errors

* tests for field APIs

* table perms changes

* tests for table API

* fix function call

* clean ns

* perm enforcement for other table APIs

* perm enforcement for other field APIs

* address comments

* fix failed to sync admin group

* address noah's comments and add migration script

* document for run-with-data-migration-index

* update comments

* fix name space

* adding data_migrations tests

* add docg

* make sure we don't remove admin group if sso and ldap are not configured

* fix tests for be-ee

* fix tests for oss

* misc docs updates

* remove data-migration-index

* return some newlines

* is it failling here?

* update data_migration docs

* update data_migration docs

* fix all styling comments

* make migration to run both in oss and enterprise and make sure the tests are accounted for that

* fix failed namespace checks

* Add a comment to the test

* fix per comments

* Update permissions.clj

* tweaking with the  icon

* refactor with-temporary-raw-setting-values

* update comments

* Add extension for cert file

* address Noah's comments

* fix bug

* address Dan's feedback

* move the real-number? helper to metabase.utils and use it here

* oops, fix build

* second try at fixing build

* third try 

Intended design of the JSON thing entails backing off on specific types if we find conflicting ones up to a point. Also there has to be a limit on the cardinality of the generated JSON fields. This one puts it in

* Use collection permissions fns to select correct timelines

The Timelines permissions are 1:1 with collection permissions, so on the GET timelines endpoint, use the existing
collection permissions functions to help efficiently check timeline permissions.

* Add permissions tests to timeline api

* Root lacks most collection properties

no need to hydrate parent id, effective location, etc

Co-authored-by: dan sutton <dan@dpsutton.com>

* Create truly rebindable `metabase.db.connection/*application-db*`

* call-on-change-fn -> call-on-change

* Namespace cleanup

* Add missing docstrings

* Appease linters again

* Fix handler stats logging middleware

* PR feedback

* PR feedback: slight optimization

* add API to fetch general permisisons graph

* add API to update general permissionns

* change author of migration

* update documents

* misc fixes to applease the CIs

* Add tests for general permission APIs and models

* linting and fix a failed test case

* fix some failed tests

* update docs and change /subscription/ to /general/subscription/ for consistency

* Hook and migration to make sure subscription are created for new groups by default

* add schema migrations tests

* set for the win

* address noah's comments

* Parse number as is in http-client for tests

* address Cam's comments

* revert the last commit about parsing API response in tests

* change fk name

* delete a comment

* Changes:
- Rename `changes` column to `after` to keep things consistent
- If a group  has no General Permisions, it'll not be included in the graph
- Update tests and some docs

* fix failing tests in ee

* add some tests and make docstring completes

* polishing comments

* namespaces

* fix namespaces

* Add general permisison flags to `GET /api/user/current` (#21250)

* return general permisison flags for /api/current

* namespaces

* move permission flags to under

* Enforce Subscription permissions (#21285)

* return general permisison flags for /api/current

* namespaces

* enforce general permissions for subscription and tests

* update check-has-general-permisison to add option to require superuser

* adding arity

* add tests for permissions helper function

* Move advanced permissions check funcs to ee namespace

* unpushed changes

* namespaces

* ignore exception when load namespaces

* change helper fn name

* Enforce Monitoring Permissions (#21321)

* return general permisison flags for /api/current

* namespaces

* enforce general permissions for subscription and tests

* update check-has-general-permisison to add option to require superuser

* adding arity

* enforce permissions to call /api/dataset for internal queries

* enforce monitoring permissions for api/task and api/util

* add tests for OSS

* add tests for db-connection-info endpoint

* change test schema

* update name func and fix ns

* whydon't CI run ?

* Enforce Setting Permissions (#21386)

* return general permisison flags for /api/current

* namespaces

* enforce general permissions for subscription and tests

* update check-has-general-permisison to add option to require superuser

* adding arity

* enforce permissions to call /api/dataset for internal queries

* enforce monitoring permissions for api/task and api/util

* add tests for OSS

* add tests for db-connection-info endpoint

* change test schema

* update name func and fix ns

* whydon't CI run ?

* Enforce Setting permissions

* fix failing test

* make sure we could run slack test twice

* make the mock consistent

* address Noah's comments

* shorter permissions check

* Remove errant :bookmarked hydration key

This was left over from an incorrect implementation of :bookmarked status. No :bookmarked hydration function exists
anyway, the key does nothing.

* Remove old favorites models that are superceded by bookmarks

* Remove dashboard favorite require

* Added bookmarked-score to enable boosting of bookmarked items

* Collection Search Query doesn't error with bookmark left join

* adjust collection id AS in base query

* Set proper bookmark status in collection for search tests

* Actually score bookmarks when bookmarked

* Added a bookmark scoring test

* Remove old todo since it does work. (yay)

All under the grand path to #708.

Changes field API to be compatible with nfc-paths. Allows joining of nested field columns by shoving the full identifier into the postgres function call. Also fixes arcane macro test bug by not using the macro.

* Revert "Log database connection error before humanizing it (#10695)"

This reverts commit 426df163.

This is no longer needed. The original exception message is kept
in the inner exception and everywhere this is used, the whole
stack trace is logged anyway.

* Don't log exception when opportunistically trying SSL

To be usable in the postgres driver, the _uploaded_ root cert needs to:

  - have its prefix stripped
  - and be base64 decoded

(note: this was not an issue when pointing to a local file)

* add API to fetch general permisisons graph

* add API to update general permissionns

* change author of migration

* update documents

* misc fixes to applease the CIs

* Add tests for general permission APIs and models

* linting and fix a failed test case

* fix some failed tests

* update docs and change /subscription/ to /general/subscription/ for consistency

* address noah's comments

* Parse number as is in http-client for tests

* revert the last commit about parsing API response in tests

* change fk name

* Changes:
- Rename `changes` column to `after` to keep things consistent
- If a group  has no General Permisions, it'll not be included in the graph
- Update tests and some docs

* fix failing tests in ee

* add some tests and make docstring completes

* fix namespaces

Add new `valid-path-format?` predicate for schema validation to fix downgrading issues generically (#21255)

* Add :dataset key to card bookmarks

The :dataset key is boolean and indicates when a card (question) has been turned into a model.
The frontend will use this key to determine which icon is used to display the bookmark in the sidebar.

* Reduce some duplication in this test

* Fix Postgres filter by quarter

* Fix test for MongoDB to work around #5419

* Spark SQL doesn't allow quarters in intervals either

* Hydrate Collections on Timelines for card/collection endpoints

* Add tests to check that collection is hydrated

* Add missing docstring