Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Dec 28, 2021
    • dpsutton's avatar
      Handle nested queries which have agg at top level and nested (#19437) · 50818a92
      dpsutton authored
      * Handle nested queries which have agg at top level and nested
      
      Previously when matching columns in the outer query with columns in the
      inner query we had use id, and then recently, field_ref. This is
      problematic for aggregations.
      
      Consider https://github.com/metabase/metabase/issues/19403
      
      The mbql for this query is
      
      ```clojure
      {:type :query
       :query {:aggregation [[:aggregation-options
                              [:count]
                              {:name "count"}]
                             [:aggregation-options
                              [:avg
                               [:field
                                "sum"
                                {:base-type :type/Float}]]
                              {:name "avg"}]]
               :limit 10
               :source-card-id 1960
               :source-query {:source-table 1
                              :aggregation [[:aggregation-options
                                             [:sum
                                              [:field 23 nil]]
                                             {:name "sum"}]
                                            [:aggregation-options
                                             [:max
                                              [:field 28 nil]]
                                             {:name "max"}]]
                              :breakout [[:field 26 nil]]
                              :order-by [[:asc
                                          [:field 26 nil]]]}}
       :database 1}
      ```
      
      The aggregations in the top level select will be type checked as :name
      "count" :field_ref [:aggregation 0]. The aggregations in the nested
      query will be turned into :name "sum" :field_ref [:aggregation 0]! This
      is because aggregations are numbered "on their level" and not
      globally. So when the fields on the top level look at the metadata for
      the nested query and merge it, it unifies the two [:aggregation 0]
      fields but this is INCORRECT. These aggregations are not the same, they
      just happen to be the first aggregations at each level.
      
      Its illustrative to see what a (select * from (query with aggregations))
      looks like:
      
      ```clojure
      {:database 1
       :query {:source-card-id 1960
               :source-metadata [{:description "The type of product, valid values include: Doohicky, Gadget, Gizmo and Widget"
                                  :semantic_type :type/Category
                                  :coercion_strategy nil
                                  :name "CATEGORY"
                                  :field_ref [:field 26 nil]
                                  :effective_type :type/Text
                                  :id 26
                                  :display_name "Category"
                                  :fingerprint {:global {:distinct-count 4
                                                         :nil% 0}
                                                :type {:type/Text {:percent-json 0
                                                                   :percent-url 0
                                                                   :percent-email 0
                                                                   :percent-state 0
                                                                   :average-length 6.375}}}
                                  :base_type :type/Text}
                                 {:name "sum"
                                  :display_name "Sum of Price"
                                  :base_type :type/Float
                                  :effective_type :type/Float
                                  :semantic_type nil
                                  :field_ref [:aggregation 0]}
                                 {:name "max"
                                  :display_name "Max of Rating"
                                  :base_type :type/Float
                                  :effective_type :type/Float
                                  :semantic_type :type/Score
                                  :field_ref [:aggregation 1]}]
               :fields ([:field 26 nil]
                        [:field
                         "sum"
                         {:base-type :type/Float}]
                        [:field
                         "max"
                         {:base-type :type/Float}])
               :source-query {:source-table 1
                              :aggregation [[:aggregation-options
                                             [:sum
                                              [:field 23 nil]]
                                             {:name "sum"}]
                                            [:aggregation-options
                                             [:max
                                              [:field 28 nil]]
                                             {:name "max"}]]
                              :breakout [[:field 26 nil]]
                              :order-by [[:asc
                                          [:field 26 nil]]]}}
       :type :query
       :middleware {:js-int-to-string? true
                    :add-default-userland-constraints? true}
       :info {:executed-by 1
              :context :ad-hoc
              :card-id 1960
              :nested? true
              :query-hash #object["[B" 0x10227bf4 "[B@10227bf4"]}
       :constraints {:max-results 10000
                     :max-results-bare-rows 2000}}
      ```
      
      The important bits are that it understands the nested query's metadata
      to be
      
      ```clojure
      {:name "sum"
       :display_name "Sum of Price"
       :field_ref [:aggregation 0]}
      {:name "max"
       :display_name "Max of Rating"
       :field_ref [:aggregation 1]}
      ```
      
      And the fields on the outer query to be:
      ```clojure
      ([:field
        "sum"
        {:base-type :type/Float}]
       [:field
        "max"
        {:base-type :type/Float}])
      ```
      
      So there's the mismatch: the field_ref on the outside is [:field "sum"]
      but on the inside the field_ref is [:aggregation 0]. So the best way to
      match up when "looking down" into sub queries is by id and then by name.
      
      * Some drivers return 4.0 instead of 4 so make them all ints
      
      * Handle dataset metadata in a special way
      
      rather than trying to set confusing merge rules, just special case
      metadata from datasets.
      
      Previously, was trying to merge the "preserved keys" on top of the
      normal merge order. This caused lots of issues. They were
      trivial. Native display names are very close to the column name, whereas
      mbql names go through some humanization. So when you select price
      from (select PRICE ...) its an mbql with a nested native query. The
      merge order meant that the display name went from "Price"
      previously (the mbql nice name for the outer select) to "PRICE", the
      underlying native column name. Now we don't special case the display
      name (and other fields) of regular source-metadata.
      
      Also, there were issues with mbql on top of an mbql dataset. Since it is
      all mbql, everything is pulled from the database. So if there were
      overrides in the nested mbql dataset, like description, display name,
      etc, the outer field select already had the display name, etc. from the
      database rather than allowing the edits to override from the nested
      query.
      
      Also, using a long `:source-query/dataset?` keyword so it is far easier
      to find where this is set. With things called just `:dataset` it can be
      quite hard to find where these keys are used. When using the namespaced
      keyword, greping and finding usages is trivial. And the namespace gives
      more context
      Unverified
      50818a92
  2. Dec 15, 2021
  3. Sep 14, 2021
  4. Sep 09, 2021
  5. Aug 11, 2021
    • dpsutton's avatar
      Changing type (#16776) · c60dfc5f
      dpsutton authored
      * Move sync executor to bespoke namespace
      
      * Refingerprint fields on type change
      
      * Check if can connect in refingerprint-field!
      
      * docstring update
      
      * Cleanup tests a bit
      
      * Error handling
      
      * Table field values endpoint on sync.concurrent executor
      
      * ns sort api field
      
      * ns cleanup
      Unverified
      c60dfc5f
  6. Aug 04, 2021
    • Cam Saul's avatar
      Minor test runner/config fixes (#17323) · ad69111e
      Cam Saul authored
      * Fix JUnit output not correctly stripping ANSI color codes
      
      * Remove find-tests profiling log message
      
      * 1-arg arity of metabase.test-runner/run
      
      * Fix bug with metabase.test-runner/find-tests with a single test name
      Unverified
      ad69111e
  7. Jul 30, 2021
  8. Jun 22, 2021
    • dpsutton's avatar
      Move *ignore-cached-results* to middleware (#16695) · 83f948ff
      dpsutton authored
      * Move *ignore-cached-results* to middleware
      
      Had tried to adjust the binding to work, but the binding ended before
      all of the bound functions and such were submitted to work, so when
      that code actually executed the binding was no longer in
      place. Annoying and it should be in middleware anyways
      
      * Remove now unused import (cache became keyword in middleware not binding)
      Unverified
      83f948ff
  9. Jun 02, 2021
    • dpsutton's avatar
      Authority Level on collections (formerly Typed collections) (#16191) · ad753b18
      dpsutton authored
      * Add type on collection
      
      * Search with collection type
      
      * Cleanup bounded heap accumulator
      
      * Make search conditionally bump official collection items
      
      * collection api and tests
      
      * Put collection type onto hydrated cards on dashboards
      
      * Move official collection type scoring into ee codebase
      
      * ensure ee and oss agree when not official collection
      
      * Mark Collections tree with type
      
      * Remove unnecessary `and`s when no check for enhancements
      
      * Tests for setting collection tree type
      
      * Include hydration for collection type on dashboards
      
      * Make sure created test collections are cleaned up
      
      * Cleanup tests, don't search on collection_type
      
      looks for all text fields and searches the term against that. official
      would bring back all official types. no bueno
      
      * Docstring on protocol and don't shadow comparator
      
      * update to new ee impl var passing style
      
      * Collection model ensures no type change on personal collection
      
      * Check for personal collection when updating type
      
      model checks for personal collection and rejects in the update but
      doesn't check for children of personal collection.
      
      * Update checking for unchangeable properties of a personal collection
      
      * Cleanup: type collection tree, batched hydration, combine error checks
      
      * Cleanup
      
      * move bounded-heap accumulator to utils
      
      * switch to test.check testing
      
      * Bad test just to see what our CI will report
      
      * remove purposeful failing test (was for CI)
      
      * collection.type -> collection.authority_level
      
      * Test the actual ee scoring impl, not the wrapped one
      
      locally i had enhancements enabled so the tests were passing, but in
      CI it was failing as it did not have enhancements enabled
      Unverified
      ad753b18
  10. May 17, 2021
    • Cam Saul's avatar
      Add Semantic/* and Relation/* ancestor types (#15994) · 9700fe5b
      Cam Saul authored
      * Port legacy data type migrations -> Liquibase
      
      * Fix migration IDs
      
      * Field type validation/error handling
      
      * Have semantic type fallback to nil
      
      * Fix semantic-type-migrations-test
      
      * Fix migrations
      
      * Revert accidental changes
      
      * Semantic/* & Relation/* ancestor types
      
      * Fix stray Relation/PK and Relation/FKs
      
      * Semantic/* and Relation/* ancestor types
      
      * cljs test fix
      
      * Fix :require
      
      * FE test fixes :wrench:
      
      * Test fixes :wrench:
      
      * prettier
      
      * PR  f e e d b a c k
      
      * Use medium size CircleCI image for Presto to prevent all the OOMs
      
      * Backport dir-locals tweaks from hierarchy PR
      
      * Redshift: only sync the test schema (faster CI and fix failure)
      
      * Better error handling for sync in tests
      
      * Revert accidental commit
      
      * Redshift test fixes :wrench:
      Unverified
      9700fe5b
  11. Apr 13, 2021
    • dpsutton's avatar
      Remove unixtimestamp type (#15533) · 4910a3e2
      dpsutton authored
      * Remove type/UNIX* and type/ISO8601* from frontend
      
      * Set effective-type and coercion strategy when syncing
      
      these values will most likely only be available when using tests, as
      metadata is very unlikely to have this. As far as I can tell the
      toucannery apparatus is the only bit that has this. Its quite
      artificial. I'm not sure if this is a good idea.
      
      * :type/UNIXTimestampSeconds and type/ISO8601DateTimeString out of type
      
      remove the coercions from the type hierarchy. This brought up a
      strange issue that has been present for a while: all liquidbase
      migrations run and then all data migrations run. They are not
      interleaved. This allows for the following scenario:
      
      - data migration migrates all X values to Y in version 26
      - liquibase migration migrates all Y values to Z in version 28
      
      But these are not run in version order, but all liquibase migrations
      are run and then all data migrations are run. If you were on an old
      version for a while, you could end up with Y values even though the
      liquibase migration which comes "after" the data migration turns all Y
      values into Z values.
      
      This impacts this change because a data migration in this way:
      - liquibase changesets 287, 288, 289, and 290 remove all 'type/UNIX*'
      and 'type/ISO8601*' semantic types. These were in 0.39
      - data migration `migrate-field-types` added in 0.20.0 was looking for
      special_type's of "timestamp_milliseconds" and migrating them to
      "type/UNIXTimestampMilliseconds".
      
      Since the liquibase runs before the migrate-field-types, it's possible
      for a 'type/UNIX*' semantic type to reappear. And since these were
      removed from the type system it would fail validation and blow up. In
      this case it actually can't happen since the field its attempting to
      migrate (special_type) no longer exists. But since the migrations are
      not interleaved this problem still exists.
      
      * whatever prettier
      
      * Appease the machines
      
      * Restore the unix and iso semantic types to hierarchy
      
      migration tests still use these
      `(impl/test-migrations [283 296] [migrate!] ...)`
      
      A few things in tension: the field requires valid semantic_types,
      these _were_ valid semantic_types until v39.
      
      * Remove old "coercion" semantic types
      
      * Migrate the v20 version semantic types for unix epoch
      
      * Time travelling migrations
      
      these target v20 and v29 which at the time they existed had a column
      special type not semantic type. But these run after all of the schema
      migrations, not interleaved, so they will have a semantic_type not
      special_type when they actually run even though they target v20 or v29
      data. strange world
      
      * add migration's new checksum
      Unverified
      4910a3e2
  12. Apr 12, 2021
    • Dalton's avatar
      Dashboard parameter field filter operators feature flag (#15519) · a362e2f3
      Dalton authored
      
      * Backend feature flag for new field filters
      
      * Feature flag new parameter options
      
      When the "field-filter-operators-enabled?" flag is disabled we do the following:
      1. Replace new operator options with old category and location/city, etc., options
         in the PARAMETER_OPTIONS list found in metabase/meta/Parameter.js
      2. Hide numbers section in the PARAMETER_SECTIONS list found in
         metabase/meta/Dashboard.js
      3. Return args as-is in the mapUIParameterToQueryParameter function
         found in metabase/meta/Parameter.js
      
      React/UI code handles both old options and new options so doesn't need
      to change. Old parameter types like "category" and "location/city" are
      treated like "string/=" in the UI but retain their own parameter type
      when used to send a new query.
      
      * Fix FE issues caused by meta/Parameter refactor
      
      * mock the field operator param flag to make tests pass
      
      * add/fix cypress tests
      
      * fix import in ParametersPopover
      
      * update widget tag type
      
      * Enable field filter operators for cypress tests
      
      * Question marks are questionable
      
      * Conditionally use category or string/= if field filters are enabled
      
      * rmv mocks where we don't need them
      
      * rmv mock from chained-filters test
      
      * env vars as string in project.clj, alignment
      
      Co-authored-by: default avatardan sutton <dan@dpsutton.com>
      Unverified
      a362e2f3
  13. Mar 31, 2021
  14. Mar 23, 2021
  15. Feb 25, 2021
    • Cam Saul's avatar
      Optimize relative datetime filters (#14835) · 538e5e38
      Cam Saul authored
      1. Rename optimize-datetime-filters middleware -> optimize-temporal-filters (it's more accurate, because this also
      optimizes date or time filter clauses)
      
      2. optimize-temporal-filters middleware now optimizes relative-datetime clauses (which represent a moment in time
      relative to when the query is ran, e.g. "last month") in addition to absolute-datetime clauses (which represent an
      absolute moment in time, e.g. 2021-02-15T14:40:00-08:00) . This middleware rewrites queries so we filter against
      specific temporal ranges without casting the column itself, meaning we can leverage indexes on that column. See #11837
      for more details
      
      3. Added new validate-temporal-bucketing middleware that throws an Exception if you try to do something that makes no
      sense, e.g. bucket a DATE field by :time or a TIME field by :month. This is a better situation then running the query
      and waiting for the DB to complain. (In practice, I don't think the FE client would let you generate a query like this
      in the first place)
      
      4. Fix random test failures for MySQL in task-history-cleanup-test
      Unverified
      538e5e38
  16. Jan 15, 2021
    • dpsutton's avatar
      Sync startup cleanups (#14428) · 4028dc65
      dpsutton authored
      * Log db info from db query
      
      we're already making the query might as well also get the information
      we want to log
      
      * Alignment settings in .dir-locals
      
      * Inverted logic for logging fingerprint vs refingerprint
      Unverified
      4028dc65
  17. Jan 14, 2021
  18. Jan 07, 2021
  19. Jan 05, 2021
  20. Oct 05, 2020
    • Paul Rosenzweig's avatar
      Dashboard interactivity: custom drill through and chained filters (#13354) · c4353a1f
      Paul Rosenzweig authored
      
      * dashboard interactivity: custom drill through and chained filters
      
      (cherry picked from commit f3ea3099da981c967fcffd5c7e318c88064c0a96)
      
      * add missing param
      
      * Better error message & test for cases where you don't have Field perms
      
      (cherry picked from commit 2ae8174c388a105170bb316b6266b900f8844c62)
      
      * Fix lint error
      
      (cherry picked from commit aa967359ecf9705f28dded2b5b30eb42ebec156e)
      
      * type another letter to avoid ordering issue
      
      (cherry picked from commit 5f285219c07eeff41b3d1fbfc091092a6ce60ed4)
      
      * fix failing cy tests
      
      (cherry picked from commit 8dbb9561a21f853a550d4ab1385fbdf142ea36e6)
      
      * update test
      
      * Fix merge
      
      * sort correctly
      
      * Fix namespace decl sorting
      
      * use another within block for the "add to dash" modal
      
      Co-authored-by: default avatarCam Saul <github@camsaul.com>
      Unverified
      c4353a1f
  21. Jul 20, 2020
    • Robert Roland's avatar
      Update eslint (#12951) · d86b85ce
      Robert Roland authored
      
      * Update eslint
      
      Updates eslint, babel and plugins to the latest compatible versions
      
      Drops the 'no-color-literals' parameter which doesn't exist (looks like
      it's actually part of eslint-plugin-react-native which we don't use)
      
      adding a dirlocal to make sure js2-mode doesn't confuse you with type
      errors that aren't actually errors because of flowtype and such
      
      * update generated css classes in snapshots
      
      Co-authored-by: default avatarPaul Rosenzweig <paul.a.rosenzweig@gmail.com>
      Unverified
      d86b85ce
  22. Feb 19, 2020
  23. Dec 03, 2019
  24. Oct 07, 2019
  25. Jul 29, 2019
  26. Jun 05, 2019
  27. Mar 13, 2019
  28. Dec 19, 2017
  29. Feb 01, 2017
  30. Jan 27, 2017
  31. Dec 06, 2016
  32. Nov 02, 2016
  33. Oct 19, 2016
  34. Sep 19, 2016
  35. Jul 20, 2016
  36. Jul 18, 2016
  37. Jul 13, 2016
  38. Jul 08, 2016
  39. Jul 05, 2016
Loading