Skip to content
Snippets Groups Projects
Commit 8ee6d574 authored by Ryan Senior's avatar Ryan Senior Committed by Simon Belak
Browse files

Add support for classification of NaN and -Infinity/Infinity doubles (#7914)

* Add support for classification of NaN and -Infinity/Infinity doubles

These values are read correctly from the database, but after they are
serialized to JSON, don't get deserialized to a double object value,
but rather stay a string and cause failures.

This commit recognizes those values and will parse them as doubles.

Fixes #7779
parent bbd268b2
No related branches found
No related tags found
No related merge requests found
(ns metabase.models.field
(:require [clojure.core.memoize :as memoize]
[clojure.string :as str]
[medley.core :as m]
[metabase.models
[dimension :refer [Dimension]]
[field-values :as fv :refer [FieldValues]]
......@@ -119,6 +120,22 @@
;; otherwise we need to fetch additional info about Field's Table. This is cached for 5 seconds (see above)
(perms-objects-set* table-id)))
(defn- maybe-parse-special-numeric-values [maybe-double-value]
(if (string? maybe-double-value)
(u/ignore-exceptions (Double/parseDouble maybe-double-value))
maybe-double-value))
(defn- update-special-numeric-values
"When fingerprinting decimal columns, NaN and Infinity values are possible. Serializing these values to JSON just
yields a string, not a value double. This function will attempt to coerce any of those values to double objects"
[fingerprint]
(u/update-in-when fingerprint [:type :type/Number]
(partial m/map-vals maybe-parse-special-numeric-values)))
(models/add-type! :json-for-fingerprints
:in i/json-in
:out (comp update-special-numeric-values i/json-out-with-keywordization))
(u/strict-extend (class Field)
models/IModel
......@@ -129,7 +146,7 @@
:visibility_type :keyword
:description :clob
:has_field_values :keyword
:fingerprint :json
:fingerprint :json-for-fingerprints
:settings :json})
:properties (constantly {:timestamped? true})
:pre-insert pre-insert
......
(ns metabase.sync.analyze.classify-test
(:require [expectations :refer :all]
[metabase.models
[database :refer [Database]]
[field :refer [Field]]
[table :refer [Table]]]
[metabase.sync.analyze.classify :as classify]
......@@ -33,3 +34,43 @@
:last_analyzed (du/->Timestamp #inst "2017-08-09")}]]
(for [field (#'classify/fields-to-classify table)]
(:name field)))))
;; Check that we can classify decimal fields that have specially handled NaN values
(expect
[nil :type/Income]
(tt/with-temp* [Database [db]
Table [table {:db_id (u/get-id db)}]
Field [field {:table_id (u/get-id table)
:name "Income"
:base_type :type/Float
:special_type nil
:fingerprint_version i/latest-fingerprint-version
:fingerprint {:type {:type/Number {:min "NaN"
:max "NaN"
:avg "NaN"}}
:global {:distinct-count 3}}
:last_analyzed nil}]]
[(:special_type (Field (u/get-id field)))
(do
(classify/classify-fields-for-db! db [table] (constantly nil))
(:special_type (Field (u/get-id field))))]))
;; Check that we can classify decimal fields that have specially handled infinity values
(expect
[nil :type/Income]
(tt/with-temp* [Database [db]
Table [table {:db_id (u/get-id db)}]
Field [field {:table_id (u/get-id table)
:name "Income"
:base_type :type/Float
:special_type nil
:fingerprint_version i/latest-fingerprint-version
:fingerprint {:type {:type/Number {:min "-Infinity"
:max "Infinity"
:avg "Infinity"}}
:global {:distinct-count 3}}
:last_analyzed nil}]]
[(:special_type (Field (u/get-id field)))
(do
(classify/classify-fields-for-db! db [table] (constantly nil))
(:special_type (Field (u/get-id field))))]))
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment