-
- Downloads
Semantic types 2 effective type (#15022)
* First pass using coercions
* Coercions
* Handle effective_type coercion_strategy in test data sets
* special-type -> semantic type in sample db
```clojure
user> (def config (metabase.db.spec/h2 {:db (str "/Users/dan/projects/clojure/metabase/resources/sample-dataset.db"
";UNDO_LOG=0;CACHE_SIZE=131072;QUERY_CACHE_SIZE=128;COMPRESS=TRUE;"
"MULTI_THREADED=TRUE;MVCC=TRUE;DEFRAG_ALWAYS=TRUE;MAX_COMPACT_TIME=5000;"
"ANALYZE_AUTO=100")}))
user> (jdbc/execute! config ["UPDATE _metabase_metadata
SET keypath = 'PEOPLE.ZIP.semantic_type'
WHERE keypath = 'PEOPLE.ZIP.special_type'" ])
[1]
user> (jdbc/execute! config ["UPDATE _metabase_metadata
SET keypath = 'REVIEWS.BODY.semantic_type'
WHERE keypath = 'REVIEWS.BODY.special_type'" ])
[1]
```
* Correct mismatch in validation preventing sync
* fixing up alternative date tests
* More passing tests
* Tests for values, nested queries, fetch metadata
* tests
* tests passing
* Fixup mongo qp for coercions
locally i have some failing tests that are off by 1 errors:
Fail in compile-time-interval-test
[36m:mongo[0m Make sure time-intervals work the way they're supposed to. [:time-interval $date -4 :month] should give us something like Oct 01 2020 - Feb 01 2021 if today is Feb 17 2021
expected: [{$match {$and [{:$expr {$gte [$date {:$dateFromString {:dateString 2020-10-01T00:00Z}}]}} {:$expr {$lt [$date {:$dateFromString {:dateString 2021-02-01T00:00Z}}]}}]}} {$group {_id {date~~~day {:$let {:vars {:parts {:$dateToParts {:date $date}}}, :in {:$dateFromParts {:year $$parts.year, :month $$parts.month, :day $$parts.day}}}}}}} {$sort {_id 1}} {$project {_id false, date~~~day $_id.date~~~day}} {$sort {date~~~day 1}} {$limit 1048576}]
actual: [{"$match"
{"$and"
[{:$expr {"$gte" ["$date" {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
{:$expr {"$lt" ["$date" {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}
{"$group"
{"_id"
{"date~~~day"
{:$let
{:vars {:parts {:$dateToParts {:date "$date"}}},
:in {:$dateFromParts {:year "$$parts.year", :month "$$parts.month", :day "$$parts.day"}}}}}}}
{"$sort" {"_id" 1}}
{"$project" {"_id" false, "date~~~day" "$_id.date~~~day"}}
{"$sort" {"date~~~day" 1}}
{"$limit" 1048576}]
diff: - [{"$match"
{"$and"
[{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-10-01T00:00Z"}}]}}
{:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-02-01T00:00Z"}}]}}]}}]
+ [{"$match"
{"$and"
[{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
{:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}]
* ee fixes
* UI to set coercion type
* Don't need to populate effective-type here
it actually has knock on effects:
- does more work now as almost every field has an update to do in
`add-extra-metadata`
- we have databases that have state that we don't create. druid for
example has stuff to mimic the dataset in tqpt/with-flattened-dbdef on
checkins but we don't actually create this. And our dbdef has a field
called "date" that is not present in the druid db, so if we attempt to
add metadata it fails and kills the rest of the metadata that we add.
- tests need this metadata to be present and the error causes field
visibilities (for example) to not be set
* Docstrings on shared lib
* Add effective and coercion to redshift expectations
* Fixup google analytics
* Derecordize instead of recordize the expectation
object details didn't work out well here. they added way more stuff
from the db than what is flowing through here.
```clojure
actual: {:field
{:name "DATE",
:parent_id nil,
:table_id 69,
:base_type :type/Date,
:effective_type :type/Date,
:coercion_strategy nil,
:semantic_type nil},
:value {:type :date/all-options, :value "past5days"}}
diff: - {:field
{:description nil,
:database_type "VARCHAR",
:fingerprint_version 0,
:has_field_values nil,
:settings nil,
:caveats nil,
:fk_target_field_id nil,
:custom_position 0,
:active true,
:last_analyzed nil,
:position 1,
:visibility_type :normal,
:preview_display true,
:database_position 0,
:fingerprint nil,
:points_of_interest nil}}
```
Object defaults adds quite a bit of stuff such that we'd be dissoc'ing
more than we are currently adding in
Co-authored-by:
Cam Saul <1455846+camsaul@users.noreply.github.com>
Showing
- backend/mbql/src/metabase/mbql/normalize.clj 1 addition, 1 deletionbackend/mbql/src/metabase/mbql/normalize.clj
- enterprise/backend/test/metabase_enterprise/sandbox/query_processor/middleware/row_level_restrictions_test.clj 14 additions, 8 deletions...uery_processor/middleware/row_level_restrictions_test.clj
- frontend/src/metabase/admin/datamodel/containers/FieldApp.jsx 32 additions, 0 deletions...tend/src/metabase/admin/datamodel/containers/FieldApp.jsx
- frontend/test/__runner__/test_db_fixture.db.mv.db 0 additions, 0 deletionsfrontend/test/__runner__/test_db_fixture.db.mv.db
- modules/drivers/googleanalytics/src/metabase/driver/googleanalytics/execute.clj 4 additions, 1 deletion...analytics/src/metabase/driver/googleanalytics/execute.clj
- modules/drivers/googleanalytics/test/metabase/driver/googleanalytics_test.clj 14 additions, 11 deletions...leanalytics/test/metabase/driver/googleanalytics_test.clj
- modules/drivers/mongo/src/metabase/driver/mongo/parameters.clj 3 additions, 3 deletions...es/drivers/mongo/src/metabase/driver/mongo/parameters.clj
- modules/drivers/mongo/src/metabase/driver/mongo/query_processor.clj 4 additions, 4 deletions...ivers/mongo/src/metabase/driver/mongo/query_processor.clj
- modules/drivers/oracle/src/metabase/driver/oracle.clj 2 additions, 2 deletionsmodules/drivers/oracle/src/metabase/driver/oracle.clj
- modules/drivers/redshift/test/metabase/driver/redshift_test.clj 6 additions, 2 deletions...s/drivers/redshift/test/metabase/driver/redshift_test.clj
- modules/drivers/sqlite/src/metabase/driver/sqlite.clj 3 additions, 3 deletionsmodules/drivers/sqlite/src/metabase/driver/sqlite.clj
- modules/drivers/sqlserver/src/metabase/driver/sqlserver.clj 1 addition, 1 deletionmodules/drivers/sqlserver/src/metabase/driver/sqlserver.clj
- modules/drivers/sqlserver/test/metabase/driver/sqlserver_test.clj 1 addition, 1 deletion...drivers/sqlserver/test/metabase/driver/sqlserver_test.clj
- resources/migrations/000_migrations.yaml 71 additions, 2 deletionsresources/migrations/000_migrations.yaml
- resources/sample-dataset.db.mv.db 0 additions, 0 deletionsresources/sample-dataset.db.mv.db
- shared/src/metabase/types.cljc 61 additions, 2 deletionsshared/src/metabase/types.cljc
- src/metabase/api/field.clj 15 additions, 3 deletionssrc/metabase/api/field.clj
- src/metabase/driver/common/parameters/values.clj 10 additions, 10 deletionssrc/metabase/driver/common/parameters/values.clj
- src/metabase/driver/mysql.clj 1 addition, 1 deletionsrc/metabase/driver/mysql.clj
- src/metabase/driver/sql/parameters/substitution.clj 1 addition, 3 deletionssrc/metabase/driver/sql/parameters/substitution.clj
Loading
Please register or sign in to comment