Skip to content
Snippets Groups Projects
Unverified Commit 9e508654 authored by dpsutton's avatar dpsutton Committed by GitHub
Browse files

Allow text strings to special type to date [ci drivers] (#13860)

* Allow text strings to special type to date [ci drivers]

* Allow for timestamps/time/date for iso8601 strings [ci drivers]

* First pass at tests for :type/ISO8601DateTimeString [ci drivers]

* Only run ISO8601datetimestring tests on sql dbs

* Make the default a timestamp and mysql the exception [ci drivers]

* Fixes for sqlite and sqlserver

i'm not happy with sqlite yet. they are returning strings not
java.dates. But this is the same with a field who's base type is a
datetime. However, the UI allows for ordering on the date times but
not on the :type/ISO8601 fields and i'm not sure why

* Handle oracle [ci drivers]

oracle doesn't have a time type so use a similar edn dataset that
doesn't have the time column

* Move dataset definitions inline [ci drivers]

* correct oracle datetime format [ci drivers]

sparksql seems to have issues with time so just punt on that for now
by running in the test that doesn't have the time column

* wack-a-mole [ci drivers]

* Fix tests for redshift, oracle, and sparksql [ci drivers]

* Oracle is funky [ci drivers]

* need to investigate what's going on with oracle here [ci drivers]

text fields with special_type :type/ISO8601DateTimeString are queryable as dates a datetime field
:oracle
expected: (= 1 (count (mt/rows (mt/dataset string-times (mt/run-mbql-query times {:filter [:= [:datetime-field $ts :day] "2008-10-19"]})))))
  actual: java.lang.NullPointerException: null
 at oracle.jdbc.driver.T4CTTIdcb.fillupAccessors (T4CTTIdcb.java:1041)
    oracle.jdbc.driver.T4CTTIdcb.receiveCommon (T4CTTIdcb.java:221)
    oracle.jdbc.driver.T4CTTIdcb.receive (T4CTTIdcb.java:161)
    oracle.jdbc.driver.T4C8Oall.readDCB (T4C8Oall.java:991)
    oracle.jdbc.driver.T4CTTIfun.receive (T4CTTIfun.java:560)
    oracle.jdbc.driver.T4CTTIfun.doRPC (T4CTTIfun.java:252)
    oracle.jdbc.driver.T4C8Oall.doOALL (T4C8Oall.java:612)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8 (T4CPreparedStatement.java:226)
    oracle.jdbc.driver.T4CPreparedStatement.doOall8 (T4CPreparedStatement.java:59)
    oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe (T4CPreparedStatement.java:747)
    oracle.jdbc.driver.OracleStatement.executeMaybeDescribe (OracleStatement.java:904)
    oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout (OracleStatement.java:1082)
    oracle.jdbc.driver.OraclePreparedStatement.executeInternal (OraclePreparedStatement.java:3780)
    oracle.jdbc.driver.T4CPreparedStatement.executeInternal (T4CPreparedStatement.java:1343)
    oracle.jdbc.driver.OraclePreparedStatement.executeQuery (OraclePreparedStatement.java:3822)
    oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery (OraclePreparedStatementWrapper.java:1165)
    com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery (NewProxyPreparedStatement.java:431)
    metabase.driver.sql_jdbc.execute$eval71770$fn__71771.invoke (execute.clj:267)
    clojure.lang.MultiFn.invoke (MultiFn.java:234)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invokeStatic (execute.clj:392)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invoke (execute.clj:377)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invokeStatic (execute.clj:386)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invoke (execute.clj:377)
    metabase.driver.sql_jdbc$eval102870$fn__102871.invoke (sql_jdbc.clj:49)
    metabase.driver.oracle$eval187218$fn__187219.invoke (oracle.clj:268)
    clojure.lang.MultiFn.invoke (MultiFn.java:244)
    metabase.query_processor.context$executef.invokeStatic (context.clj:59)
    metabase.query_processor.context$executef.invoke (context.clj:48)
    metabase.query_processor.context.default$default_runf.invokeStatic (default.clj:69)
    metabase.query_processor.context.default$default_runf.invoke (default.clj:67)
    metabase.query_processor.context$runf.invokeStatic (context.clj:45)
    metabase.query_processor.context$runf.invoke (context.clj:39)
    metabase.query_processor.reducible$pivot.invokeStatic (reducible.clj:34)
    metabase.query_processor.reducible$pivot.invoke (reducible.clj:31)

* sparksql has some issues querying on datetime [ci drivers]

text fields with special_type :type/ISO8601DateTimeString are queryable as dates a datetime field
:sparksql: org.apache.spark.sql.catalyst.parser.ParseException:
DataType time() is not supported.(line 2, pos 178)

== SQL ==
-- Metabase
SELECT `t1`.`id` AS `id`, `t1`.`name` AS `name`, CAST(CAST(`t1`.`ts` AS timestamp) AS timestamp) AS `ts`, CAST(CAST(`t1`.`d` AS date) AS timestamp) AS `d`, CAST(CAST(`t1`.`t` AS time) AS timestamp) AS `t` FROM `string_times`.`times` `t1` WHERE (CAST(CAST(`t1`.`ts` AS timestamp) AS timestamp) >= CAST(to_utc_timestamp('2008-10-19 00:00:00', 'UTC') AS timestamp) AND CAST(CAST(`t1`.`ts` AS timestamp) AS timestamp) < CAST(to_utc_timestamp('2008-10-20 00:00:00', 'UTC') AS timestamp)) LIMIT 1048576
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------^^^

expected: (= 1 (count (mt/rows (mt/dataset string-times (mt/run-mbql-query times {:filter [:= [:datetime-field $ts :day] "2008-10-19"]})))))
  actual: java.sql.SQLException: org.apache.spark.sql.catalyst.parser.ParseException:
DataType time() is not supported.(line 2, pos 178)

== SQL ==
-- Metabase
SELECT `t1`.`id` AS `id`, `t1`.`name` AS `name`, CAST(CAST(`t1`.`ts` AS timestamp) AS timestamp) AS `ts`, CAST(CAST(`t1`.`d` AS date) AS timestamp) AS `d`, CAST(CAST(`t1`.`t` AS time) AS timestamp) AS `t` FROM `string_times`.`times` `t1` WHERE (CAST(CAST(`t1`.`ts` AS timestamp) AS timestamp) >= CAST(to_utc_timestamp('2008-10-19 00:00:00', 'UTC') AS timestamp) AND CAST(CAST(`t1`.`ts` AS timestamp) AS timestamp) < CAST(to_utc_timestamp('2008-10-20 00:00:00', 'UTC') AS timestamp)) LIMIT 1048576
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------^^^

 at org.apache.hive.jdbc.HiveStatement.execute (HiveStatement.java:297)
    org.apache.hive.jdbc.HiveStatement.executeQuery (HiveStatement.java:392)
    org.apache.hive.jdbc.HivePreparedStatement.executeQuery (HivePreparedStatement.java:109)
    com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery (NewProxyPreparedStatement.java:431)
    metabase.driver.sql_jdbc.execute$eval71770$fn__71771.invoke (execute.clj:267)
    clojure.lang.MultiFn.invoke (MultiFn.java:234)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invokeStatic (execute.clj:392)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invoke (execute.clj:377)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invokeStatic (execute.clj:386)
    metabase.driver.sql_jdbc.execute$execute_reducible_query.invoke (execute.clj:377)
    metabase.driver.sql_jdbc$eval102870$fn__102871.invoke (sql_jdbc.clj:49)
    metabase.driver.sparksql$eval187174$fn__187176.invoke (sparksql.clj:142)
    clojure.lang.MultiFn.invoke (MultiFn.java:244)
    metabase.query_processor.context$executef.invokeStatic (context.clj:59)
    metabase.query_processor.context$executef.invoke (context.clj:48)
    metabase.query_processor.context.default$default_runf.invokeStatic (default.clj:69)
    metabase.query_processor.context.default$default_runf.invoke (default.clj:67)
    metabase.query_processor.context$runf.invokeStatic (context.clj:45)
    metabase.query_processor.context$runf.invoke (context.clj:39)
    metabase.query_processor.reducible$pivot.invokeStatic (reducible.clj:34)
    metabase.query_processor.reducible$pivot.invoke (reducible.clj:31)
    metabase.query_processor.middleware.mbql_to_native$mbql__GT_native$fn__57878.invoke (mbql_to_native.clj:26)
    metabase.query_processor.middleware.check_features$check_features$fn__57065.invoke (check_features.clj:42)
    metabase.query_processor.middleware.optimize_datetime_filters$optimize_datetime_filters$fn__58077.invoke (optimize_datetime_filters.clj:133)
    metabase.query_processor.middleware.auto_parse_filter_values$auto_parse_filter_values$fn__55756.invoke (auto_parse_filter_values.clj:44)
    metabase.query_processor.middleware.wrap_value_literals$wrap_value_literals$fn__59813.invoke (wrap_value_literals.clj:149)
    metabase.query_processor.middleware.annotate$add_column_info$fn__55502.invoke (annotate.clj:575)
    metabase.query_processor.middleware.permissions$check_query_permissions$fn__56924.invoke (permissions.clj:70)
    metabase.query_processor.middleware.pre_alias_aggregations$pre_alias_aggregations$fn__58633.invoke (pre_alias_aggregations.clj:40)
    metabase.query_processor.middleware.cumulative_aggregations$handle_cumulative_aggregations$fn__57154.invoke (cumulative_aggregations.clj:61)
    metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__60343.invoke (row_level_restrictions.clj:292)
    metabase.query_processor.middleware.resolve_joined_fields$resolve_joined_fields$fn__58912.invoke (resolve_joined_fields.clj:36)
    metabase.query_processor.middleware.resolve_joins$resolve_joins$fn__59239.invoke (resolve_joins.clj:183)
    metabase.query_processor.middleware.add_implicit_joins$add_implicit_joins$fn__50719.invoke (add_implicit_joins.clj:254)
    metabase.query_processor.middleware.large_int_id$convert_id_to_string$fn__57823.invoke (large_int_id.clj:44)
    metabase.query_processor.middleware.limit$limit$fn__57856.invoke (limit.clj:38)
    metabase.query_processor.middleware.format_rows$format_rows$fn__57795.invoke (format_rows.clj:84)
    metabase.query_processor.middleware.desugar$desugar$fn__57226.invoke (desugar.clj:22)
    metabase.query_processor.middleware.binning$update_binning_strategy$fn__56123.invoke (binning.clj:229)
    metabase.query_processor.middleware.resolve_fields$resolve_fields$fn__56714.invoke (resolve_fields.clj:24)
    metabase.query_processor.middleware.add_dimension_projections$add_remapping$fn__50239.invoke (add_dimension_projections.clj:318)
    metabase.query_processor.middleware.add_implicit_clauses$add_implicit_clauses$fn__50462.invoke (add_implicit_clauses.clj:141)
    metabase_enterprise.sandbox.query_processor.middleware.row_level_restrictions$apply_row_level_permissions$fn__60343.invoke (row_level_restrictions.clj:292)
    metabase.query_processor.middleware.add_source_metadata$add_source_metadata_for_source_queries$fn__50882.invoke (add_source_metadata.clj:105)
    metabase_enterprise.sandbox.query_processor.middleware.column_level_perms_check$maybe_apply_column_level_perms_check$fn__60025.invoke (column_level_perms_check.clj:24)
    metabase.query_processor.middleware.reconcile_breakout_and_order_by_bucketing$reconcile_breakout_and_order_by_bucketing$fn__58858.invoke (reconcile_breakout_and_order_by_bucketing.clj:98)

* Remove sparksql can't query on date [ci drivers]
parent 3fe5fb07
Branches
Tags
No related merge requests found
Showing with 290 additions and 7 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment