Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Oct 06, 2023
  2. Oct 05, 2023
  3. Oct 04, 2023
  4. Oct 03, 2023
  5. Oct 02, 2023
  6. Sep 29, 2023
    • Case Nelson's avatar
    • Mark Bastian's avatar
      This is a small set of changes pulled from another PR whose contents (#34228) · 92faa8e2
      Mark Bastian authored
      probably won't be merged. The intent of these changes is to be
      explicit in what is passed into each of the card creation functions,
      ensure consistency of types, and eliminate redundant or non-required
      arguments.
      
      `build-dashcard` was the target of this simplification. The
      `available-values` argument is no longer necessary as that data is
      sourced from other arguments. the `satisfied-*` arguments are no
      longer seqs of satisfied filter, metric, and dimension definitions,
      but maps of names of those items to the definitions. This is only
      needed because satisfied metrics needs metrics names when adding
      metadata. However, all values were modified into a consistent shape.
      
      In other areas, if an aggregate object with several keys is passed
      to a function that uses only one element of the object, only the
      inner element is passed.
      
      In card-candidates, `:matches` is dissoced from `satisfied-dimensions`
      to remove the extra payload that isn't needed in that argument.
      This makes it clear that satisfied field values are passed into card
      creation functions via the `satisfied-bindings` and not cruft from
      earlier stage.
      
      As an aside, one potential fix that would substantially improve the
      readibility of this entire codebase would be to move the element
      names in the one key to value maps as is done for dimensions,
      metrics, filters, and cards into the elements as a named key.
      Carrying that shape forward from the yaml files requires a ton of
      first this or zipmap that to recover the names into whatever the
      current context is.
      92faa8e2
  7. Sep 27, 2023
  8. Sep 26, 2023
    • adam-james's avatar
      Alert API sends all alert information to any user. Change this to onl… (#34067) · 0920254f
      adam-james authored
      * Alert API sends all alert information to any user. Change this to only show those pulses the user has created or is
      recipient of
      
      * Adjust tests to show that only admins see all alerts. Regular users only see their alerts.
      
      * Change docstring to describe behaviour of the endpoint.
      0920254f
    • Cam Saul's avatar
      Remove deprecated `current-db-time` method (#32686) · 47a52b57
      Cam Saul authored
      
      * Remove deprecated current-db-time method
      
      * Address PR feedback
      
      * sync-timezone! should handle java.time.ZoneId and java.time.ZoneOffset
      
      * Allow time zone offsets as zone IDs
      
      * Fix tests and suppress clj-kondo on a few forms
      
      * Suppress warning on one more occurrence of db-default-timezone
      
      * Map offset Z to time zone UTC when syncing
      
      * Accept zone offsets, zone ids and strings for h2 and oracle
      
      * Mark snowflake as not supporting report timezone
      
      * Revert "Mark snowflake as not supporting report timezone"
      
      This reverts commit 70b15a60334fda707d695bc6c01621961f1c5aa8.
      
      * Fix report-timezone handling
      
      * Numeric timezone
      
      ---------
      
      Co-authored-by: default avatarmetamben <103100869+metamben@users.noreply.github.com>
      Co-authored-by: default avatarTamás Benkő <tamas@metabase.com>
      47a52b57
    • lbrdnk's avatar
    • dpsutton's avatar
      X ray stage 1 (#34026) · 400850b9
      dpsutton authored
      * reintroduce the functions
      
      * using affinities
      
      had to update the affinities map to handle multiple definitions of the
      same affinity name.
      
      ```clojure
      core=> (let [affinities (-> ["table" "GenericTable"]
                                  dashboard-templates/get-dashboard-template
                                  dash-template->affinities-map)]
               (affinities "HourOfDayCreateDate"))
      [["HourOfDayCreateDate"
        {:dimensions ["CreateTimestamp"], :metrics ["Count"], :score 50}]
       ["HourOfDayCreateDate"
        {:dimensions ["CreateTime"], :metrics ["Count"], :score 50}]]
      ```
      
      Here there were two cards defined as "HourOfDateCreateDate". One looked
      for a createtimestamp, the other for a createtime. When treating the
      affinities as unique names, the one with createtime clobbered the
      createtimestamp and then it went unsatisfied.
      
      Now we keep both definitions around. The group by is stable so the
      timestamp comes first and in match that one will be matched.
      
      * i don't like reduced for some reason
      
      * fixup affinities
      
      1. Affinities were in wrong shape:
      
      before:
      
      ```clojure
      {"card-name" [["card-name" definition]
                    ["card-name" definition]]
       ,,,
      ```
      
      new shape:
      
      ```clojure
      {"card-name" [definition
                    definition]
       ,,,
      ```
      
      ex:
      
      ```clojure
      core=> (let [affinities (-> ["table" "GenericTable"]
                                  dashboard-templates/get-dashboard-template
                                  dash-template->affinities-map)]
               (affinities "HourOfDayCreateDate"))
      ({:dimensions ["CreateTimestamp"], :metrics ["Count"], :score 50}
       {:dimensions ["CreateTime"], :metrics ["Count"], :score 50})
      ```
      
      2. Erroring on unfindable filters/metrics
      using `{:filter [:dimension ::unsatisfiable]}` didn't work correctly
      because of this:
      
      ```clojure
      ;; it doesn't look for dimensions by keyword
      core=> (dashboard-templates/collect-dimensions [:dimension ::nope])
      ()
      core=> (dashboard-templates/collect-dimensions [:dimension (str ::nope)])
      (":metabase.automagic-dashboards.core/nope")
      ```
      
      * Updating dash-template->affinities-map to dash-template->affinities.
      
      Two key changes were made:
      - The card name is now a part of the 'affinity' object (under the `:affinity-name` key).
        This flattening should also make affinities easier to deal with and more flexible
        (e.g. arbitrary groupings).
      - All base dimensions are exploded out into these affinity objects under the `:base-dims` key.
        Note that this may result in growth of the number of affinities when named items are
        repeated with different definitions. For example, card names are not unique, resulting in
        N affinities per card. Also, metrics and filters need not be unique. GenericTable, for
        example, has 6 definitions of the Last30Days filter. This will result in 6X the number of
        affinities created for each card using that filter.
      
      This change encapsulates affinities, as before you'd need to know what dimensions underlied
      any given metric or filter. ATM, we do not package each filter or metric definition in each
      affinity object, but perhaps that would be worth doing in the future for complete encapsulation.
      
      This change also allows for very simple matching logic. To match card affinities, for example,
      you just filter all affinities for which the affinity dimensions are a subset of the provided
      dimensions.
      
      * Updated tests for new affinity code. Removed accidental cruft.
      
      * Change shape of affinities
      
      The "satisfied-affins" are of this shape:
      ```clojure
      {"AverageIncomeByMonth" #{"Income" "Timestamp"},
       "AverageDiscountByMonth" #{"Income" "Discount" "Timestamp"}}
      ```
      
      And they are ordered so it can drive the `make-cards` function in the
      future. The idea is for right now we'll look up the card based on
      affinity name, and when multiple cards found, the set of dimensions they
      depend on. And that will drive the layout.
      
      But in the future, just the affinity itself will drive how we make a
      card layout. This is the firs step towards that.
      
      * Satisfied-affinities shape is ordered map to vector of sets of dims
      
      ```clojure
      {"RowcountLast30Days" [#{"CreateTimestamp"}],
       "Rowcount" [#{}]}
      ```
      
      Since each could be met in a few definitions in multiple ways
      
      * Update comment
      
      ```clojure
      core=> (let [affinities (-> ["table" "GenericTable"]
                                  dashboard-templates/get-dashboard-template
                                  dash-template->affinities)]
               (match-affinities affinities {:available-dimensions {"JoinDate" :whatever}}))
      {"Rowcount" [#{}],
       "RowcountLast30Days" [#{"JoinDate"}],
       "CountByJoinDate" [#{"JoinDate"}],
       "DayOfWeekJoinDate" [#{"JoinDate"}],
       "DayOfMonthJoinDate" [#{"JoinDate"}],
       "MonthOfYearJoinDate" [#{"JoinDate"}],
       "QuerterOfYearJoinDate" [#{"JoinDate"}]}
      ```
      
      * comment block is helpful :octopus:
      
      
      
      * Drive `make-cards` from affinities
      
      old style was make-cards looped over all cards to see which ones were
      satisfied. Now we've taken a notion of "interestingness" which we call
      affinities: which dimensions are interesting in combination with each
      other. Right now these are derived from card combinations but that will
      change going forward.
      
      So now going into the make-cards loop are interesting combinations and
      we then grab a card-template from the combination. Again, it's a double
      lookup back to cards but this lets us break that cycle and come up with
      interesting card templates just based on the groupings themselves.
      
      in the future, we're going to want an affinity to produce multiple
      card-templates so this will become a mapcat of some sorts rather than a
      map.
      
      * Removing pre-check from card-candidates and corresponding unit test as this is not an invariant -- card-candidates should always be satisfied with our affinity mapping.
      
      * comment and docstring
      
      * Changed names for clarity (affinity -> affinity-sets) and modified
      the return value of match-affinities to have values of sets of sets
      rather than vectors of sets. This makes matching simpler and easier.
      
      Added a docstring to CardTemplateProvider and started adding some tests.
      
      * Revert "Changed names for clarity (affinity -> affinity-sets) and modified"
      
      This reverts commit dd2aef1fea8e6deb5f970e51f698e4b72fa97b32.
      
      * Something about either the cherry-pick or stale state made the previous
      change of affinities (as a vector of set) to affinity-sets (a set of sets)
      cause failures. It may just be that the implementation was broken and
      the tests passed due to a stale state ¯\_(ツ)_/¯.
      
      This picks out the clarity and doc changes and reverts the set of sets.
      
      * Adding schemas for the affinity functions in metabase.automagic-dashboards.core
      
      * Created the AffinitySetProvider protocol, which, given an item, will
      provide the set of affinities (a set of set of dimensions) required to
      bind to that item. The initial implementation reifies the protocol
      over a dashboard template and provides affinity sets for cards, but
      this protocol could be extended to provide affinities for whatever
      object we desire.
      
      The initial implementation looks like so:
      
      ```clojure
      (p/defprotocol+ AffinitySetProvider
        "For some item, determine the affinity sets of that item. This is a set of sets, each underlying set being a set of
        dimensions that, if satisfied, specify affinity to the item."
        (create-affinity-sets [this item]))
      
      (mu/defn base-dimension-provider :- [:fn #(satisfies? AffinitySetProvider %)]
        "Takes a dashboard template and produces a function that takes a dashcard template and returns a seq of potential
        dimension sets that can satisfy the card."
        [{card-metrics :metrics card-filters :filters} :- ads/dashboard-template]
        (let [dim-groups (fn [items]
                           (-> (->> items
                                    (map
                                      (fn [item]
                                        [(ffirst item)
                                         (set (dashboard-templates/collect-dimensions item))]))
                                    (group-by first))
                               (update-vals (fn [v] (mapv second v)))
                               (update "this" conj #{})))
              m->dims    (dim-groups card-metrics)
              f->dims    (dim-groups card-filters)]
          (reify AffinitySetProvider
            (create-affinity-sets [_ {:keys [dimensions metrics filters]}]
              (let [dimset                (set (map ffirst dimensions))
                    underlying-dim-groups (concat (map m->dims metrics) (map f->dims filters))]
                (set
                  (map
                    (fn [lower-dims] (reduce into dimset lower-dims))
                    (apply math.combo/cartesian-product underlying-dim-groups))))))))
      ```
      
      * Adding specs for dashcard, dashcards, context (minimal) and instrumenting
      make-cards. I'm wondering if we should move card-candidates into the
      layout-producer protocol since we're using known affinities to make a thing,
      not to make a baby step to make a thing. The one gotcha in this is that
      there's a positional index inserted in there which IDK how we use other
      than maybe layout ATM.
      
      * Added more schemas and externalized the matching of affinities to
      potential dimensions (or any map conforming to the a map of dimension
      names to matches of items). This generalizes our ability to match
      of affinity groups with "things" that we want to generate.
      
      * Renaming output of all-satisfied-bindings to satisfied-bindings to prevent variable shadowing error.
      
      ---------
      
      Co-authored-by: default avatarMark Bastian <markbastian@gmail.com>
      400850b9
    • Cal Herries's avatar
      481604ae
    • Tim Macdonald's avatar
      Use Bigints for CSV uploads (#34118) · ecf9d549
      Tim Macdonald authored
      [Fixes #34080]
      ecf9d549
    • Ngoc Khuat's avatar
      Fix the flaky test 'downgrade-dashboard-tabs-test' (#34115) · 3a1b10c6
      Ngoc Khuat authored
      * Fix the flaky test 'downgrade-dashboard-tabs-test' by bumping the
      migration version we upgrade to during the test. I'm not fully sure why
      it was flaky in the first place. For some reasons the rollback step
      doesn't get triggered on MySQL even though liqubiase does acknowledge
      that the migration got ran?? eitherway bumping the version seems to take
      the flaky a way.
      3a1b10c6
    • Cal Herries's avatar
  9. Sep 25, 2023
  10. Sep 22, 2023
Loading