Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Oct 19, 2021
  2. Oct 11, 2021
    • dpsutton's avatar
      Two column table (#18392) · 57506cc0
      dpsutton authored
      * Don't render bar charts are sparklines
      
      https://github.com/metabase/metabase/issues/18352
      
      A simple select `SELECT 'a', 1 UNION ALL SELECT 'b', 2` would trigger as
      a sparkline and blow up when comparing. There are two issues:
      - don't render tables as a sparkline even if it only has two columns
      - don't compare things that aren't comparable.
      
      this solves the first issue, ensuring that tables aren't matched as
      sparklines
      
      * Sparkline should only compare comparable values
      
      Check that the columns are temporal or numeric before comparing them. It
      is only to optionally reverse the order of the results which seems
      questionable in itself, but we can at least check that they are
      comparable before attempting to
      
      * Ensure tests render as sparklines
      
      default was a :table and that adds csv attachments which throw off our
      tests
      
      * unskip repro
      
      * remove unnecessary extra line
      Unverified
      57506cc0
  3. Oct 05, 2021
  4. Oct 01, 2021
  5. Sep 14, 2021
  6. Sep 09, 2021
  7. Aug 10, 2021
  8. Aug 09, 2021
  9. Jul 30, 2021
  10. Jul 29, 2021
  11. Jul 27, 2021
  12. Jul 22, 2021
    • Jeff Evans's avatar
      Fix serialization: Visualization column settings lost (#17127) · eb59ab27
      Jeff Evans authored
      Update visualization_settings.cljc to handle the table.columns submap
      
      Update dump and load to handle :visualization_settings on a Card (basically identically to how they are handled for a DashboardCard)
      
      Updating test to have a card include :visualization_settings and ensure they fields are properly handled
      
      Updating visualization_settings test case to incorporate table.columns as well as :show_mini_bar (under ::column_settings)
      Unverified
      eb59ab27
  13. Jun 11, 2021
  14. May 27, 2021
    • Jeff Evans's avatar
      Serialization fixes for x.39 (#15858) · 598a1124
      Jeff Evans authored
      Serialization refactoring and fixes
      
      *********************************************************
      * Support loading linked cards in different collections *
      *********************************************************
      
      Architectural changes:
      Lay foundation for more generic retry mechanism
      Establish a keyword to represent a failed name->id lookup that can happen during load, and map that to information about which part of the card resolution failed (for logging purposes)
      Similar to dashboards, add a separate fn for performing a load of cards
      Invoking the load-cards fn from the load multimethod implementation when some cards failed name resolution
      
      Test enhancements:
      Add new multimethod to perform more specific assertions on the loaded entities
      For cards - check that the native SQL of the loaded card matches that of the original (pre-dump) instance
      For dashboards - check that the linked card for each dashboard card series matches
      
      Bug fixes:
      Apply the mbql-id->fully-qualified-name fn on serialize to filter, breakout, and aggregation clauses
      Updating "My Test" card to include filter clause so that fix is exercised
      Instead of upserting users with a "changeme" password, simply do not attempt to change the password
      Adding :engine to the identity condition of Database (so unrelated test-data instances don't get trashed on load)
      
      Cleanup:
      Remove unused specs namespace
      
      ****************************************
      * Support dashboard card click actions *
      ****************************************
      
      Architectural changes:
      Add visualization settings shared model namespace with utility functions
      
      Adding tests for viz settings
      
      Changes to serialization:
      Update visualization_settings in dump to convert column_settings keys from field ID to fully qualified name, and update the click action target with the fully qualified name
      Also converting targetId in top level click action to fully qualified name
      
      Doing the reverse in load, using the same utility code
      
      Changes to serialization tests:
      Updating the existing dashcard in the "world" roundtrip test to include the click action
      
      Adding assertion for the click action in the assert-loaded-entity implementation for Dashboard
      
      ******************************************************************
      * Stop nesting cards and check for unresolved query dependencies *
      ******************************************************************
      
      In names.clj, stop qualifying a card's name by its dataset query's source card ID (this is what caused the "nested" directory structure)
      
      In load.clj, start checking that either the source query [:dataset_query :query] is resolved properly as a card ID or table ID, and if neither, then propagate an ::unresolved-names entry up into the card (causing it to be reloaded on second pass)
      
      Fix logging of unresolved name info
      
      Test changes:
      Add two new cards to the "world", one that lives in the root collection but depends on a card within a collection for its query, and one opposite (i.e. lives in a collection but depends on a root card for its query)
      
      Adding new check that collection names match after loading
      
      ************************
      * Logging improvements *
      ************************
      
      Include causes when logging errors to aid with diagnosis
      
      Add BEGIN and END messages for both dump and load commands
      
      Add dynamic var to suppress printing of name lookup error on first pass load
      
      *************************************************
      * Handle collection namespaces and retry pulses *
      *************************************************
      
      Architectural changes:
      Considering a collection namespace to be part of its fully qualified name (putting in a subdirectory called :namespace when writing to disk)
      Accounting for that change when dumping and loading, and upsert (i.e. namespace is part of unique columns)
      Bumping serialization format version to 2 because of this
      
      Changes to load:
      Add load-pulses fn that works similarly to others, to facilitate reloading
      Add similar logic as other entities to reload a pulse if the :card_id is not found
      
      Model changes:
      Add hack fn to compare namespace values for the "namespace does not change" assertion with appropriate comment
      
      Test changes:
      Added test for pulse with reload
      
      ***********************************************
      * Handle dimension entry within template-tags *
      ***********************************************
      
      Add :field to the recognized clauses under mbql-entity-reference? (since that is how field refs are now represented in MBQL)
      
      Add temp-field helper fn in test_util.clj to reduce some of the "world" verbosity
      
      Adding new test card to world that uses template-tags with a field reference under dimension
      
      ************************
      * Handle textbox cards *
      ************************
      
      Add support for passing textbox (virtual) cards through visualization_settings.cljc
      
      Updating load test to add a textbox type card to the dashboard and ensuring it is persisted
      
      ***************************************
      * Handle dimension parameter mappings *
      ***************************************
      
      In visualization_settings.cljc:
      Adding handling for parameterMapping
      Adding new roundtrip test for parameter mappings
      Some refactoring to split up larger fns
      
      In serialize.clj, convert all parts of the map that contain the dimension key to use fully qualified names for the respective fields
      
      In load.clj, do the opposite (convert back to new IDs)
      
      Updating test to add parameter mapping to an existing click action, and adding new assertions for that to ensure it is preserved
      
      Fixing serialize test now that the "ids" can themselves be maps (i.e within dimension vector)
      
      In visualization_settings_test.cljc
      Adding new roundtrip test for problem dashboard
      
      ***********************************
      * Handle field literals correctly *
      ***********************************
      
      On load, only attempt to translate field name to ID if it's actually a fully qualified field name
      
      Adding new fn to names.clj to provide that logic
      
      Adding new test card that uses such a field reference
      
      **************************************
      * Accept only-db-ids option for dump *
      **************************************
      
      Accept options map for the dump command.  Currently, the only supported option is `:only-db-ids`, which is expected to be a set of DB IDs to dump.  If set, then only these databases (and their respective tables, fields, and segments) will be dumped to the dump-dir.  If not set, then the previous behavior takes effect (all DBs).
      
      Update the load_test.clj to set this to be only the new temporary DB to avoid tramping on existing `sample-dataset` and other instances
      
      *****************************************************
      * Don't include personal collections of other users *
      *****************************************************
      
      Use the `select-collections` fn to filter only collections that are public, or owned bythe given user (by way of email), plus all collections with those as an ancestor
      
      Updating test to assert a non-owned personal collection is not persisted (including nested), but all nested personal collections are
      
      Adding new wrapped macro to also bind collection/*allow-deleting-personal-collections* true around the with-temp* call so that personal collections can legitimately be deleted (since they're temporary); the name of this macro was chosen to preserve the same amount of spacing within `with-world`
      
      *********************************
      * Support native query snippets *
      *********************************
      
      Adding support for NativeQuerySnippet:
      
      - in names
      - in dump (putting into subdirectory of collection, as cards)
      - in upsert (identity condition of :name :collection_id)
      - in load (no retry fn necessary since they should not have any dependencies themselves)
      
      Adding a snippet and a card referencing it to roundtrip test (including a deeply nested snippet)
      
      Fixing up handling of snippet namespaced collections and adding relevant name tests
      
      *********************************
      * Handle source-table for joins *
      *********************************
      
      Adding `mbql-fully-qualified-names->ids*` for recursive calls, which does not normalize its args, keeping mbql-fully-qualified-names->ids` as a wrapper to that
      
      Adding clause to `mbql-fully-qualified-names->ids*` to resolve `:source-table` when it's a fully qualified table name
      
      Adding util fn to names.clj to support that scenario
      
      Updating "My Card" test card to include a join
      
      Skipping two more drivers from `dump-load-entities-test` - SQLite and Spark because they don't support FKs
      
      ***************************
      * Cleanup and refactoring *
      ***************************
      
      Responding to PR feedback
      
      Updating upsert_test.clj to check that the identity-condition keys actually exist for the models
      
      Removing legacy MBQL syntax in a couple places
      
      Adding ordering by `:id` to serialization load command for predictability
      
      Lots of fixes and refactorings in visualization_settings.cljc:
       - added lots of specs and fdefs
       - added test fixture that instruments all the speced fns and required it from both visualization_settings.cljc and load_test.clj
       - renamed a bunch of fns for more consistency
       - added and fixed some docstrings
       - fix implementation of `parse-json-string` for cljs
      
      *************************************************
      * Preserve temporal-unit in source-field clause *
      *************************************************
      
      Update serialize and match macros to keep the rest of the match, when modifying the source field ID
      
      Adding a bunch of new tables in order to perform a "bucketing on implicit join" type query
      
      Updating test_util.clj to initialize the query processor with all the new temp tables so that queries actually work
      
      **************************
      * Misc cleanup and fixes *
      **************************
      
      Fixing the implementation of `fully-qualified-name->context` to work properly when the entity name itself is also a model name
      
      ***************************
      * Handle joining to cards *
      ***************************
      
      Recognize :source-table pointing to a fully qualified card name in `mbql-fully-qualified-names->ids*`, and adding the unresolved name capture to that lookup
      
      Adding new util fns to capture the unresolved named entries from anywhere within the MBQL query tree: `paths-to-key-in` and `gather-all-unresolved-names`
      
      Updating `resolve-card-dataset-query` to call this new `gather-all-unresolved-names` to pull all unresolved names from the MBQL query tree into card itself
      
      Renaming `fully-qualified-name->id-rec` to `resolve-source-query` for clarity, and removing now unneeded clause
      
      To the test case, adding a card that joins to another card
    • Cam Saul's avatar
      Fix expansion of :metric macros inside nested queries (#16241) · 8f18c64d
      Cam Saul authored
      * Failing test
      
      * Fix expansion of :metric macros inside nested queries
      
      * Unskip Cypress test
      
      * Test fix :wrench:
      
      
      
      * Rewrite (fix) repro for #12507
      
      Co-authored-by: default avatarNemanja <31325167+nemanjaglumac@users.noreply.github.com>
      Unverified
      8f18c64d
  15. May 24, 2021
    • dpsutton's avatar
      Yyyymmddhhmmss date strings (#15790) · 6a632764
      dpsutton authored
      * Add yyyymmddhhss coercions to type system
      
      * Implementations for h2/mysql/postgres for yyyymmddhhss bytes and strings
      
      * Mongo and oracle
      
      * Adding checksum on migration it said was bad
      
      * import OffsetDateTime
      
      * Redshift and bigquery
      
      * snowflake expectations
      
      * sql server yyyymmddhhmmss. have to format the string then parse
      
      sqlserver lacks a parse function that takes a format string, they just
      take an integer that specifies a predefined format string. So we have
      to make the string into the right format then parse.
      
      * presto yyyymmddhhmmss
      
      * Test byte conversions
      
      * Remove errant `mt/set-test-drivers!`
      
      * Remove sqlite, change keyword for multiple native types in def
      
      the spec couldn't handle different shapes under the same keyword. so
      just use :natives {:postgres "BYTEA"} :native "BYTEA"
      
      * Make schema work with different shape maps
      
      * hx/raw "'foo'" -> hx/literal "foo" and remove checksums
      
      * _coercion_strategy -> _coercion-strategy
      
      * Handle coercion hierarchy for :Coercion/YYYYMMDDHHMMSSBytes->Temporal
      Unverified
      6a632764
  16. May 19, 2021
    • Cam Saul's avatar
      Coercion strategy hierarchies [PoC] (#16088) · 5caf4de8
      Cam Saul authored
      
      * Coercion strategy hierarchies
      
      * Don't rebuild hierarchy on every change
      
      * Emacs config tweaks
      
      * clojurescript-mode is derived from clojure-mode, combine the dir locals
      
      * Remove log statements and improve dir locals
      
      * Use new api from app
      
      - add coercions for text (iso stuff)
      - function to get the effective type for a strategy
      - extra validation in saving field to ensure that the triple
         `[coercion base effective]` is valid
      
      * remove unnecessary `coercions` map
      
      Co-authored-by: default avatardan sutton <dan@dpsutton.com>
      Unverified
      5caf4de8
  17. May 17, 2021
    • Cam Saul's avatar
      Add Semantic/* and Relation/* ancestor types (#15994) · 9700fe5b
      Cam Saul authored
      * Port legacy data type migrations -> Liquibase
      
      * Fix migration IDs
      
      * Field type validation/error handling
      
      * Have semantic type fallback to nil
      
      * Fix semantic-type-migrations-test
      
      * Fix migrations
      
      * Revert accidental changes
      
      * Semantic/* & Relation/* ancestor types
      
      * Fix stray Relation/PK and Relation/FKs
      
      * Semantic/* and Relation/* ancestor types
      
      * cljs test fix
      
      * Fix :require
      
      * FE test fixes :wrench:
      
      * Test fixes :wrench:
      
      * prettier
      
      * PR  f e e d b a c k
      
      * Use medium size CircleCI image for Presto to prevent all the OOMs
      
      * Backport dir-locals tweaks from hierarchy PR
      
      * Redshift: only sync the test schema (faster CI and fix failure)
      
      * Better error handling for sync in tests
      
      * Revert accidental commit
      
      * Redshift test fixes :wrench:
      Unverified
      9700fe5b
  18. May 12, 2021
  19. Apr 13, 2021
    • dpsutton's avatar
      Remove unixtimestamp type (#15533) · 4910a3e2
      dpsutton authored
      * Remove type/UNIX* and type/ISO8601* from frontend
      
      * Set effective-type and coercion strategy when syncing
      
      these values will most likely only be available when using tests, as
      metadata is very unlikely to have this. As far as I can tell the
      toucannery apparatus is the only bit that has this. Its quite
      artificial. I'm not sure if this is a good idea.
      
      * :type/UNIXTimestampSeconds and type/ISO8601DateTimeString out of type
      
      remove the coercions from the type hierarchy. This brought up a
      strange issue that has been present for a while: all liquidbase
      migrations run and then all data migrations run. They are not
      interleaved. This allows for the following scenario:
      
      - data migration migrates all X values to Y in version 26
      - liquibase migration migrates all Y values to Z in version 28
      
      But these are not run in version order, but all liquibase migrations
      are run and then all data migrations are run. If you were on an old
      version for a while, you could end up with Y values even though the
      liquibase migration which comes "after" the data migration turns all Y
      values into Z values.
      
      This impacts this change because a data migration in this way:
      - liquibase changesets 287, 288, 289, and 290 remove all 'type/UNIX*'
      and 'type/ISO8601*' semantic types. These were in 0.39
      - data migration `migrate-field-types` added in 0.20.0 was looking for
      special_type's of "timestamp_milliseconds" and migrating them to
      "type/UNIXTimestampMilliseconds".
      
      Since the liquibase runs before the migrate-field-types, it's possible
      for a 'type/UNIX*' semantic type to reappear. And since these were
      removed from the type system it would fail validation and blow up. In
      this case it actually can't happen since the field its attempting to
      migrate (special_type) no longer exists. But since the migrations are
      not interleaved this problem still exists.
      
      * whatever prettier
      
      * Appease the machines
      
      * Restore the unix and iso semantic types to hierarchy
      
      migration tests still use these
      `(impl/test-migrations [283 296] [migrate!] ...)`
      
      A few things in tension: the field requires valid semantic_types,
      these _were_ valid semantic_types until v39.
      
      * Remove old "coercion" semantic types
      
      * Migrate the v20 version semantic types for unix epoch
      
      * Time travelling migrations
      
      these target v20 and v29 which at the time they existed had a column
      special type not semantic type. But these run after all of the schema
      migrations, not interleaved, so they will have a semantic_type not
      special_type when they actually run even though they target v20 or v29
      data. strange world
      
      * add migration's new checksum
      Unverified
      4910a3e2
  20. Apr 09, 2021
  21. Apr 07, 2021
  22. Apr 02, 2021
    • dpsutton's avatar
      Put schema forms in meta for schemas (#15325) · b7ca7e5d
      dpsutton authored
      ```clojure
      schema=> (->> (ns-publics 'metabase.mbql.schema)
                    (keep (fn [[s v]] (:clause-form (meta v))))
                    (run! pprint))
      
      [:not-null :field Field]
      [:interval :n s/Int :unit RelativeDatetimeUnit]
      [:regex-match-first :s StringExpressionArg :pattern s/Str]
      [:var :field-or-expression FieldOrExpressionDef]
      [:does-not-contain
       :field
       StringExpressionArg
       :string-or-field
       StringExpressionArg
       :options
       (optional StringFilterOptions)]
      [:= :field EqualityComparable :value-or-field EqualityComparable :more-values-or-fields (rest EqualityComparable)]
      [:log :x NumericExpressionArg]
      [:< :field OrderComparable :value-or-field OrderComparable]
      [:floor :x NumericExpressionArg]
      [:metric :metric-id (s/cond-pre helpers/IntGreaterThanZero helpers/NonBlankString)]
      [:ends-with :field StringExpressionArg :string-or-field StringExpressionArg :options (optional StringFilterOptions)]
      [:relative-datetime :n (s/cond-pre (s/eq :current) s/Int) :unit (optional RelativeDatetimeUnit)]
      [:sum :field-or-expression FieldOrExpressionDef]
      [:time-interval
       :field
       field
       :n
       (s/cond-pre s/Int (s/enum :current :last :next))
       :unit
       RelativeDatetimeUnit
       :options
       (optional TimeIntervalOptions)]
      [:rtrim :s StringExpressionArg]
      [:ceil :x NumericExpressionArg]
      [:starts-with :field StringExpressionArg :string-or-field StringExpressionArg :options (optional StringFilterOptions)]
      [:<= :field OrderComparable :value-or-field OrderComparable]
      [:upper :s StringExpressionArg]
      [:* :x NumericExpressionArg :y NumericExpressionArg :more (rest NumericExpressionArg)]
      [:min :field-or-expression FieldOrExpressionDef]
      [:inside
       :lat-field
       OrderComparable
       :lon-field
       OrderComparable
       :lat-max
       OrderComparable
       :lon-min
       OrderComparable
       :lat-min
       OrderComparable
       :lon-max
       OrderComparable]
      [:ltrim :s StringExpressionArg]
      [:desc :field FieldOrAggregationReference]
      [:contains :field StringExpressionArg :string-or-field StringExpressionArg :options (optional StringFilterOptions)]
      [:expression :expression-name helpers/NonBlankString]
      [:is-empty :field Field]
      [:substring :s StringExpressionArg :start NumericExpressionArg :length (optional NumericExpressionArg)]
      [:stddev :field-or-expression FieldOrExpressionDef]
      [:> :field OrderComparable :value-or-field OrderComparable]
      [:replace :s StringExpressionArg :match s/Str :replacement s/Str]
      [:sqrt :x NumericExpressionArg]
      [:concat :a StringExpressionArg :b StringExpressionArg :more (rest StringExpressionArg)]
      [:count-where :pred Filter]
      [:- :x NumericExpressionArg :y NumericExpressionArgOrInterval :more (rest NumericExpressionArgOrInterval)]
      [:asc :field FieldOrAggregationReference]
      [:cum-count :field (optional Field)]
      [:value :value s/Any :type-info (s/maybe ValueTypeInfo)]
      [:or
       :first-clause
       (s/recursive #'Filter)
       :second-clause
       (s/recursive #'Filter)
       :other-clauses
       (rest (s/recursive #'Filter))]
      [:exp :x NumericExpressionArg]
      [:time :time (s/cond-pre java.time.LocalTime java.time.OffsetTime) :unit TimeUnit]
      [:between :field OrderComparable :min OrderComparable :max OrderComparable]
      [:sum-where :field-or-expression FieldOrExpressionDef :pred Filter]
      [:not :clause (s/recursive #'Filter)]
      [:cum-sum :field-or-expression FieldOrExpressionDef]
      [:coalesce :a ExpressionArg :b ExpressionArg :more (rest ExpressionArg)]
      [:is-null :field Field]
      [:/ :x NumericExpressionArg :y NumericExpressionArg :more (rest NumericExpressionArg)]
      [:>= :field OrderComparable :value-or-field OrderComparable]
      [:not-empty :field Field]
      [:distinct :field-or-expression FieldOrExpressionDef]
      [:percentile :field-or-expression FieldOrExpressionDef :percentile NumericExpressionArg]
      [:round :x NumericExpressionArg]
      [:power :x NumericExpressionArg :y NumericExpressionArg]
      [:aggregation-options :aggregation UnnamedAggregation :options AggregationOptions]
      [:+ :x NumericExpressionArg :y NumericExpressionArgOrInterval :more (rest NumericExpressionArgOrInterval)]
      [:abs :x NumericExpressionArg]
      [:median :field-or-expression FieldOrExpressionDef]
      [:share :pred Filter]
      [:case :clauses CaseClauses :options (optional CaseOptions)]
      [:segment :segment-id (s/cond-pre helpers/IntGreaterThanZero helpers/NonBlankString)]
      [:max :field-or-expression FieldOrExpressionDef]
      [:!= :field EqualityComparable :value-or-field EqualityComparable :more-values-or-fields (rest EqualityComparable)]
      [:count :field (optional Field)]
      [:lower :s StringExpressionArg]
      [:length :s StringExpressionArg]
      [:trim :s StringExpressionArg]
      [:and
       :first-clause
       (s/recursive #'Filter)
       :second-clause
       (s/recursive #'Filter)
       :other-clauses
       (rest (s/recursive #'Filter))]
      [:avg :field-or-expression FieldOrExpressionDef]
      [:aggregation :aggregation-clause-index s/Int]
      nil
      schema=>
      ```
      Unverified
      b7ca7e5d
  23. Mar 23, 2021
  24. Mar 15, 2021
    • dpsutton's avatar
      Semantic types 2 effective type (#15022) · 6b8ddc84
      dpsutton authored
      
      * First pass using coercions
      
      * Coercions
      
      * Handle effective_type coercion_strategy in test data sets
      
      * special-type -> semantic type in sample db
      
      ```clojure
      user> (def config (metabase.db.spec/h2 {:db (str "/Users/dan/projects/clojure/metabase/resources/sample-dataset.db"
                                                       ";UNDO_LOG=0;CACHE_SIZE=131072;QUERY_CACHE_SIZE=128;COMPRESS=TRUE;"
                                                       "MULTI_THREADED=TRUE;MVCC=TRUE;DEFRAG_ALWAYS=TRUE;MAX_COMPACT_TIME=5000;"
                                                       "ANALYZE_AUTO=100")}))
      user> (jdbc/execute! config ["UPDATE _metabase_metadata
                              SET keypath = 'PEOPLE.ZIP.semantic_type'
                              WHERE keypath = 'PEOPLE.ZIP.special_type'" ])
      [1]
      user> (jdbc/execute! config ["UPDATE _metabase_metadata
                              SET keypath = 'REVIEWS.BODY.semantic_type'
                              WHERE keypath = 'REVIEWS.BODY.special_type'" ])
      [1]
      ```
      
      * Correct mismatch in validation preventing sync
      
      * fixing up alternative date tests
      
      * More passing tests
      
      * Tests for values, nested queries, fetch metadata
      
      * tests
      
      * tests passing
      
      * Fixup mongo qp for coercions
      
      locally i have some failing tests that are off by 1 errors:
      
      Fail in compile-time-interval-test
      
      [36m:mongo[0m Make sure time-intervals work the way they're supposed to. [:time-interval $date -4 :month] should give us something like Oct 01 2020 - Feb 01 2021 if today is Feb 17 2021
      
      expected: [{$match {$and [{:$expr {$gte [$date {:$dateFromString {:dateString 2020-10-01T00:00Z}}]}} {:$expr {$lt [$date {:$dateFromString {:dateString 2021-02-01T00:00Z}}]}}]}} {$group {_id {date~~~day {:$let {:vars {:parts {:$dateToParts {:date $date}}}, :in {:$dateFromParts {:year $$parts.year, :month $$parts.month, :day $$parts.day}}}}}}} {$sort {_id 1}} {$project {_id false, date~~~day $_id.date~~~day}} {$sort {date~~~day 1}} {$limit 1048576}]
      
        actual: [{"$match"
                  {"$and"
                   [{:$expr {"$gte" ["$date" {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
                    {:$expr {"$lt" ["$date" {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}
                 {"$group"
                  {"_id"
                   {"date~~~day"
                    {:$let
                     {:vars {:parts {:$dateToParts {:date "$date"}}},
                      :in {:$dateFromParts {:year "$$parts.year", :month "$$parts.month", :day "$$parts.day"}}}}}}}
                 {"$sort" {"_id" 1}}
                 {"$project" {"_id" false, "date~~~day" "$_id.date~~~day"}}
                 {"$sort" {"date~~~day" 1}}
                 {"$limit" 1048576}]
          diff: - [{"$match"
                    {"$and"
                     [{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-10-01T00:00Z"}}]}}
                      {:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-02-01T00:00Z"}}]}}]}}]
                + [{"$match"
                    {"$and"
                     [{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
                      {:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}]
      
      * ee fixes
      
      * UI to set coercion type
      
      * Don't need to populate effective-type here
      
      it actually has knock on effects:
      - does more work now as almost every field has an update to do in
      `add-extra-metadata`
      - we have databases that have state that we don't create. druid for
      example has stuff to mimic the dataset in tqpt/with-flattened-dbdef on
      checkins but we don't actually create this. And our dbdef has a field
      called "date" that is not present in the druid db, so if we attempt to
      add metadata it fails and kills the rest of the metadata that we add.
      - tests need this metadata to be present and the error causes field
      visibilities (for example) to not be set
      
      * Docstrings on shared lib
      
      * Add effective and coercion to redshift expectations
      
      * Fixup google analytics
      
      * Derecordize instead of recordize the expectation
      
      object details didn't work out well here. they added way more stuff
      from the db than what is flowing through here.
      
      ```clojure
        actual: {:field
                 {:name "DATE",
                  :parent_id nil,
                  :table_id 69,
                  :base_type :type/Date,
                  :effective_type :type/Date,
                  :coercion_strategy nil,
                  :semantic_type nil},
                 :value {:type :date/all-options, :value "past5days"}}
          diff: - {:field
                   {:description nil,
                    :database_type "VARCHAR",
                    :fingerprint_version 0,
                    :has_field_values nil,
                    :settings nil,
                    :caveats nil,
                    :fk_target_field_id nil,
                    :custom_position 0,
                    :active true,
                    :last_analyzed nil,
                    :position 1,
                    :visibility_type :normal,
                    :preview_display true,
                    :database_position 0,
                    :fingerprint nil,
                    :points_of_interest nil}}
      ```
      
      Object defaults adds quite a bit of stuff such that we'd be dissoc'ing
      more than we are currently adding in
      
      Co-authored-by: default avatarCam Saul <1455846+camsaul@users.noreply.github.com>
      Unverified
      6b8ddc84
  25. Mar 05, 2021
    • Cam Saul's avatar
      Shared CLJ/CLJS lib (#14980) · cca03d22
      Cam Saul authored
      * Shared CLJ/CLJS lib (PoC/WIP)
      
      * PoC 2.0
      
      * Fixes :wrench:
      
      * More test fixes :wrench:
      
      * Bump shadow-cljs version
      
      * Fix more stuff
      
      * Need to ^:export the exports
      
      * CI fixes :wrench:
      
      * Add eslintignore
      
      * Ignore cljs files for FE code coverage
      
      * Try prefixing CLJS -> JS import with goog:
      
      * Revert indentation change
      
      * No goog:
      
      * Add .prettierignore
      
      * Use advanced build for now for JS tests unit we can figure out how to make it work
      Unverified
      cca03d22
Loading