Skip to content
Snippets Groups Projects
Unverified Commit 400850b9 authored by dpsutton's avatar dpsutton Committed by GitHub
Browse files

X ray stage 1 (#34026)

* reintroduce the functions

* using affinities

had to update the affinities map to handle multiple definitions of the
same affinity name.

```clojure
core=> (let [affinities (-> ["table" "GenericTable"]
                            dashboard-templates/get-dashboard-template
                            dash-template->affinities-map)]
         (affinities "HourOfDayCreateDate"))
[["HourOfDayCreateDate"
  {:dimensions ["CreateTimestamp"], :metrics ["Count"], :score 50}]
 ["HourOfDayCreateDate"
  {:dimensions ["CreateTime"], :metrics ["Count"], :score 50}]]
```

Here there were two cards defined as "HourOfDateCreateDate". One looked
for a createtimestamp, the other for a createtime. When treating the
affinities as unique names, the one with createtime clobbered the
createtimestamp and then it went unsatisfied.

Now we keep both definitions around. The group by is stable so the
timestamp comes first and in match that one will be matched.

* i don't like reduced for some reason

* fixup affinities

1. Affinities were in wrong shape:

before:

```clojure
{"card-name" [["card-name" definition]
              ["card-name" definition]]
 ,,,
```

new shape:

```clojure
{"card-name" [definition
              definition]
 ,,,
```

ex:

```clojure
core=> (let [affinities (-> ["table" "GenericTable"]
                            dashboard-templates/get-dashboard-template
                            dash-template->affinities-map)]
         (affinities "HourOfDayCreateDate"))
({:dimensions ["CreateTimestamp"], :metrics ["Count"], :score 50}
 {:dimensions ["CreateTime"], :metrics ["Count"], :score 50})
```

2. Erroring on unfindable filters/metrics
using `{:filter [:dimension ::unsatisfiable]}` didn't work correctly
because of this:

```clojure
;; it doesn't look for dimensions by keyword
core=> (dashboard-templates/collect-dimensions [:dimension ::nope])
()
core=> (dashboard-templates/collect-dimensions [:dimension (str ::nope)])
(":metabase.automagic-dashboards.core/nope")
```

* Updating dash-template->affinities-map to dash-template->affinities.

Two key changes were made:
- The card name is now a part of the 'affinity' object (under the `:affinity-name` key).
  This flattening should also make affinities easier to deal with and more flexible
  (e.g. arbitrary groupings).
- All base dimensions are exploded out into these affinity objects under the `:base-dims` key.
  Note that this may result in growth of the number of affinities when named items are
  repeated with different definitions. For example, card names are not unique, resulting in
  N affinities per card. Also, metrics and filters need not be unique. GenericTable, for
  example, has 6 definitions of the Last30Days filter. This will result in 6X the number of
  affinities created for each card using that filter.

This change encapsulates affinities, as before you'd need to know what dimensions underlied
any given metric or filter. ATM, we do not package each filter or metric definition in each
affinity object, but perhaps that would be worth doing in the future for complete encapsulation.

This change also allows for very simple matching logic. To match card affinities, for example,
you just filter all affinities for which the affinity dimensions are a subset of the provided
dimensions.

* Updated tests for new affinity code. Removed accidental cruft.

* Change shape of affinities

The "satisfied-affins" are of this shape:
```clojure
{"AverageIncomeByMonth" #{"Income" "Timestamp"},
 "AverageDiscountByMonth" #{"Income" "Discount" "Timestamp"}}
```

And they are ordered so it can drive the `make-cards` function in the
future. The idea is for right now we'll look up the card based on
affinity name, and when multiple cards found, the set of dimensions they
depend on. And that will drive the layout.

But in the future, just the affinity itself will drive how we make a
card layout. This is the firs step towards that.

* Satisfied-affinities shape is ordered map to vector of sets of dims

```clojure
{"RowcountLast30Days" [#{"CreateTimestamp"}],
 "Rowcount" [#{}]}
```

Since each could be met in a few definitions in multiple ways

* Update comment

```clojure
core=> (let [affinities (-> ["table" "GenericTable"]
                            dashboard-templates/get-dashboard-template
                            dash-template->affinities)]
         (match-affinities affinities {:available-dimensions {"JoinDate" :whatever}}))
{"Rowcount" [#{}],
 "RowcountLast30Days" [#{"JoinDate"}],
 "CountByJoinDate" [#{"JoinDate"}],
 "DayOfWeekJoinDate" [#{"JoinDate"}],
 "DayOfMonthJoinDate" [#{"JoinDate"}],
 "MonthOfYearJoinDate" [#{"JoinDate"}],
 "QuerterOfYearJoinDate" [#{"JoinDate"}]}
```

* comment block is helpful :octopus:



* Drive `make-cards` from affinities

old style was make-cards looped over all cards to see which ones were
satisfied. Now we've taken a notion of "interestingness" which we call
affinities: which dimensions are interesting in combination with each
other. Right now these are derived from card combinations but that will
change going forward.

So now going into the make-cards loop are interesting combinations and
we then grab a card-template from the combination. Again, it's a double
lookup back to cards but this lets us break that cycle and come up with
interesting card templates just based on the groupings themselves.

in the future, we're going to want an affinity to produce multiple
card-templates so this will become a mapcat of some sorts rather than a
map.

* Removing pre-check from card-candidates and corresponding unit test as this is not an invariant -- card-candidates should always be satisfied with our affinity mapping.

* comment and docstring

* Changed names for clarity (affinity -> affinity-sets) and modified
the return value of match-affinities to have values of sets of sets
rather than vectors of sets. This makes matching simpler and easier.

Added a docstring to CardTemplateProvider and started adding some tests.

* Revert "Changed names for clarity (affinity -> affinity-sets) and modified"

This reverts commit dd2aef1fea8e6deb5f970e51f698e4b72fa97b32.

* Something about either the cherry-pick or stale state made the previous
change of affinities (as a vector of set) to affinity-sets (a set of sets)
cause failures. It may just be that the implementation was broken and
the tests passed due to a stale state ¯\_(ツ)_/¯.

This picks out the clarity and doc changes and reverts the set of sets.

* Adding schemas for the affinity functions in metabase.automagic-dashboards.core

* Created the AffinitySetProvider protocol, which, given an item, will
provide the set of affinities (a set of set of dimensions) required to
bind to that item. The initial implementation reifies the protocol
over a dashboard template and provides affinity sets for cards, but
this protocol could be extended to provide affinities for whatever
object we desire.

The initial implementation looks like so:

```clojure
(p/defprotocol+ AffinitySetProvider
  "For some item, determine the affinity sets of that item. This is a set of sets, each underlying set being a set of
  dimensions that, if satisfied, specify affinity to the item."
  (create-affinity-sets [this item]))

(mu/defn base-dimension-provider :- [:fn #(satisfies? AffinitySetProvider %)]
  "Takes a dashboard template and produces a function that takes a dashcard template and returns a seq of potential
  dimension sets that can satisfy the card."
  [{card-metrics :metrics card-filters :filters} :- ads/dashboard-template]
  (let [dim-groups (fn [items]
                     (-> (->> items
                              (map
                                (fn [item]
                                  [(ffirst item)
                                   (set (dashboard-templates/collect-dimensions item))]))
                              (group-by first))
                         (update-vals (fn [v] (mapv second v)))
                         (update "this" conj #{})))
        m->dims    (dim-groups card-metrics)
        f->dims    (dim-groups card-filters)]
    (reify AffinitySetProvider
      (create-affinity-sets [_ {:keys [dimensions metrics filters]}]
        (let [dimset                (set (map ffirst dimensions))
              underlying-dim-groups (concat (map m->dims metrics) (map f->dims filters))]
          (set
            (map
              (fn [lower-dims] (reduce into dimset lower-dims))
              (apply math.combo/cartesian-product underlying-dim-groups))))))))
```

* Adding specs for dashcard, dashcards, context (minimal) and instrumenting
make-cards. I'm wondering if we should move card-candidates into the
layout-producer protocol since we're using known affinities to make a thing,
not to make a baby step to make a thing. The one gotcha in this is that
there's a positional index inserted in there which IDK how we use other
than maybe layout ATM.

* Added more schemas and externalized the matching of affinities to
potential dimensions (or any map conforming to the a map of dimension
names to matches of items). This generalizes our ability to match
of affinity groups with "things" that we want to generate.

* Renaming output of all-satisfied-bindings to satisfied-bindings to prevent variable shadowing error.

---------

Co-authored-by: default avatarMark Bastian <markbastian@gmail.com>
parent 481604ae
No related branches found
No related tags found
No related merge requests found
......@@ -81,46 +81,51 @@
fields specified in the template, then build metrics, filters, and finally cards based on the bound dimensions.
"
(:require
[buddy.core.codecs :as codecs]
[cheshire.core :as json]
[clojure.math.combinatorics :as math.combo]
[clojure.string :as str]
[clojure.walk :as walk]
[clojure.zip :as zip]
#_{:clj-kondo/ignore [:deprecated-namespace]}
[java-time :as t]
[kixi.stats.core :as stats]
[kixi.stats.math :as math]
[medley.core :as m]
[metabase.automagic-dashboards.dashboard-templates :as dashboard-templates]
[metabase.automagic-dashboards.filters :as filters]
[metabase.automagic-dashboards.populate :as populate]
[metabase.automagic-dashboards.visualization-macros :as visualization]
[metabase.db.query :as mdb.query]
[metabase.driver :as driver]
[metabase.mbql.normalize :as mbql.normalize]
[metabase.mbql.predicates :as mbql.preds]
[metabase.mbql.util :as mbql.u]
[metabase.models.card :as card :refer [Card]]
[metabase.models.database :refer [Database]]
[metabase.models.field :as field :refer [Field]]
[metabase.models.interface :as mi]
[metabase.models.metric :as metric :refer [Metric]]
[metabase.models.query :refer [Query]]
[metabase.models.segment :refer [Segment]]
[metabase.models.table :refer [Table]]
[metabase.query-processor.util :as qp.util]
[metabase.related :as related]
[metabase.sync.analyze.classify :as classify]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
[metabase.util.i18n :as i18n :refer [deferred-tru trs tru trun]]
[metabase.util.log :as log]
#_{:clj-kondo/ignore [:deprecated-namespace]}
[metabase.util.schema :as su]
[ring.util.codec :as codec]
[schema.core :as s]
[toucan2.core :as t2]))
[buddy.core.codecs :as codecs]
[cheshire.core :as json]
[clojure.math.combinatorics :as math.combo]
[clojure.set :as set]
[clojure.string :as str]
[clojure.walk :as walk]
[clojure.zip :as zip]
[flatland.ordered.map :refer [ordered-map]]
#_{:clj-kondo/ignore [:deprecated-namespace]}
[java-time :as t]
[kixi.stats.core :as stats]
[kixi.stats.math :as math]
[medley.core :as m]
[metabase.automagic-dashboards.dashboard-templates :as dashboard-templates]
[metabase.automagic-dashboards.filters :as filters]
[metabase.automagic-dashboards.populate :as populate]
[metabase.automagic-dashboards.schema :as ads]
[metabase.automagic-dashboards.visualization-macros :as visualization]
[metabase.db.query :as mdb.query]
[metabase.driver :as driver]
[metabase.mbql.normalize :as mbql.normalize]
[metabase.mbql.predicates :as mbql.preds]
[metabase.mbql.util :as mbql.u]
[metabase.models.card :as card :refer [Card]]
[metabase.models.database :refer [Database]]
[metabase.models.field :as field :refer [Field]]
[metabase.models.interface :as mi]
[metabase.models.metric :as metric :refer [Metric]]
[metabase.models.query :refer [Query]]
[metabase.models.segment :refer [Segment]]
[metabase.models.table :refer [Table]]
[metabase.query-processor.util :as qp.util]
[metabase.related :as related]
[metabase.sync.analyze.classify :as classify]
[metabase.util :as u]
[metabase.util.date-2 :as u.date]
[metabase.util.i18n :as i18n :refer [deferred-tru trs tru trun]]
[metabase.util.log :as log]
[metabase.util.malli :as mu]
#_{:clj-kondo/ignore [:deprecated-namespace]}
[metabase.util.schema :as su]
[potemkin :as p]
[ring.util.codec :as codec]
[schema.core :as s]
[toucan2.core :as t2]))
(def ^:private public-endpoint "/auto/dashboard/")
......@@ -769,24 +774,6 @@
(map filters/field-reference->id)
set)))
(defn- potential-card-dimension-bindings
"Compute all potential assignments (bindings) of identified fields (assigned to dimensions) to
required dimensions in the card definition."
[{:keys [tables]}
{:keys [available-dimensions]}
{card-query :query}
{:keys [satisfied-dimensions satisfied-metrics satisfied-filters]}]
(let [dimension-locations [satisfied-dimensions satisfied-metrics satisfied-filters card-query]
used-dimensions (dashboard-templates/collect-dimensions dimension-locations)
matched-fields (map
(some-fn
#(get-in available-dimensions [% :matches])
(comp #(filter-tables % tables) dashboard-templates/->entity))
used-dimensions)]
(->> matched-fields
(apply math.combo/cartesian-product)
(map (partial zipmap used-dimensions)))))
(defn- build-dashcard
"Build a dashcard from the given context, card template, common entities, and field bindings.
......@@ -849,27 +836,25 @@
"Generate all potential cards given a card definition and bindings for
dimensions, metrics, and filters."
[{:keys [query-filter] :as context}
{:keys [available-dimensions available-metrics available-filters] :as available-values}
satisfied-bindings
{:keys [available-metrics available-filters] :as available-values}
{required-dimensions :dimensions
required-metrics :metrics
required-filters :filters
card-title :title
:as card-template}]
(if (and (every? available-dimensions (map ffirst required-dimensions))
(every? available-metrics required-metrics)
(every? available-filters required-filters))
(let [satisfied-metrics (map available-metrics required-metrics)
satisfied-filters (cond-> (map available-filters required-filters)
query-filter
(conj {:filter query-filter}))
satisfied-dimensions (map (comp (partial into [:dimension]) first) required-dimensions)
satisfied-values {:satisfied-dimensions satisfied-dimensions
:satisfied-metrics satisfied-metrics
:satisfied-filters satisfied-filters}]
(->> (potential-card-dimension-bindings context available-values card-template satisfied-values)
(filter (partial valid-bindings? context satisfied-dimensions))
(map (partial build-dashcard context available-values card-template satisfied-values))))
(log/debugf "Card %s cannot satisfy required dimensions." card-title)))
(let [satisfied-metrics (map available-metrics required-metrics)
satisfied-filters (cond-> (map available-filters required-filters)
query-filter
(conj {:filter query-filter}))
satisfied-dimensions (map (comp (partial into [:dimension]) first) required-dimensions)
satisfied-values {:satisfied-dimensions satisfied-dimensions
:satisfied-metrics satisfied-metrics
:satisfied-filters satisfied-filters}
dimension-locations [satisfied-dimensions satisfied-metrics satisfied-filters]
used-dimensions (set (dashboard-templates/collect-dimensions dimension-locations))]
(->> (satisfied-bindings (set used-dimensions))
(filter (partial valid-bindings? context satisfied-dimensions))
(map (partial build-dashcard context available-values card-template satisfied-values)))))
(defn- matching-dashboard-templates
"Return matching dashboard templates ordered by specificity.
......@@ -927,6 +912,121 @@
(assoc field :db db)))))]
(constantly source-fields)))))
(p/defprotocol+ AffinitySetProvider
"For some item, determine the affinity sets of that item. This is a set of sets, each underlying set being a set of
dimensions that, if satisfied, specify affinity to the item."
(create-affinity-sets [this item]))
(mu/defn base-dimension-provider :- [:fn #(satisfies? AffinitySetProvider %)]
"Takes a dashboard template and produces a function that takes a dashcard template and returns a seq of potential
dimension sets that can satisfy the card."
[{card-metrics :metrics card-filters :filters} :- ads/dashboard-template]
(let [dim-groups (fn [items]
(-> (->> items
(map
(fn [item]
[(ffirst item)
(set (dashboard-templates/collect-dimensions item))]))
(group-by first))
(update-vals (fn [v] (mapv second v)))
(update "this" conj #{})))
m->dims (dim-groups card-metrics)
f->dims (dim-groups card-filters)]
(reify AffinitySetProvider
(create-affinity-sets [_ {:keys [dimensions metrics filters]}]
(let [dimset (set (map ffirst dimensions))
underlying-dim-groups (concat (map m->dims metrics) (map f->dims filters))]
(set
(map
(fn [lower-dims] (reduce into dimset lower-dims))
(apply math.combo/cartesian-product underlying-dim-groups))))))))
(mu/defn dash-template->affinities :- ads/affinities
"Takes a dashboard template, pulls the affinities from its cards, adds the name of the card
as the affinity name, and adds in the set of all required base dimensions to satisfy the card.
As card, filter, and metric names need not be unique, the number of affinities computed may
be larger than the number of distinct card names and affinities are a sequence, not a map.
These can easily be grouped by :affinity-name, :base-dims, etc. to efficiently select
the appropriate affinities for a given situation.
eg:
(dash-template->affinities-map
(dashboard-templates/get-dashboard-template [\"table\" \"TransactionTable\"]))
[{:metrics [\"TotalOrders\"]
:score 100
:dimensions []
:affinity-name \"Rowcount\"
:base-dims #{}}
{:filters [\"Last30Days\"]
:metrics [\"TotalOrders\"]
:score 100
:dimensions []
:affinity-name \"RowcountLast30Days\"
:base-dims #{\"Timestamp\"}}
...
].
"
[{card-templates :cards :as dashboard-template} :- ads/dashboard-template]
;; todo: cards can specify native queries with dimension template tags. See
;; resources/automagic_dashboards/table/example.yaml
;; note that they can specify dimension dependencies and ALSO table dependencies:
;; - Native:
;; title: Native query
;; # Template interpolation works the same way as in title and description. Field
;; # names are automatically expanded into the full TableName.FieldName form.
;; query: select count(*), [[State]]
;; from [[GenericTable]] join [[UserTable]] on
;; [[UserFK]] = [[UserPK]]
;; visualization: bar
(let [provider (base-dimension-provider dashboard-template)]
(letfn [(card-deps [card]
(-> (select-keys card [:dimensions :filters :metrics :score])
(update :dimensions (partial mapv ffirst))))]
(mapcat
(fn [card-template]
(let [[card-name template] (first card-template)]
(for [base-dims (create-affinity-sets provider template)]
(assoc (card-deps template)
:affinity-name card-name
:base-dims base-dims))))
card-templates))))
(mu/defn match-affinities :- ads/affinity-matches
"Return an ordered map of affinity names to the set of dimensions they depend on."
[affinities :- ads/affinities
available-dimensions :- [:set [:string {:min 1}]]]
;; Since the affinities contain the exploded base-dims, we simply do a set filter on the affinity names as that is
;; how we currently match to existing cards.
(let [met-affinities (filter (fn [{:keys [base-dims] :as _v}]
(set/subset? base-dims available-dimensions))
affinities)]
(reduce (fn [m {:keys [affinity-name base-dims]}]
(update m affinity-name (fnil conj []) base-dims))
(ordered-map)
met-affinities)))
(comment
(dash-template->affinities
(dashboard-templates/get-dashboard-template ["table" "TransactionTable"]))
;; example call
(let [affinities (-> ["table" "GenericTable"]
dashboard-templates/get-dashboard-template
dash-template->affinities)]
(match-affinities affinities #{"JoinDate"}))
;; example call where one affinity matches on two sets of dimensions: "OrdersBySource" matches
;; on [#{"SourceSmall" "Timestamp"} #{"SourceMedium"}]
(let [affinities (-> ["table" "TransactionTable"]
dashboard-templates/get-dashboard-template
dash-template->affinities)]
(match-affinities affinities
#{"SourceSmall" "Timestamp" "SourceMedium"}))
)
(s/defn ^:private make-base-context
"Create the underlying context to which we will add metrics, dimensions, and filters.
......@@ -942,17 +1042,53 @@
:query-filter (filters/inject-refinement (:query-filter root)
(:cell-query root))}))
(defn- make-cards
(p/defprotocol+ CardTemplateProducer
"Given an affinity name and corresponding affinity sets, produce a card template whose base dimensions match
one of the affinity sets. Example:
Given an affinity-name \"AverageQuantityByMonth\" and affinities [#{\"Quantity\" \"Timestamp\"}], the producer
should produce a card template for which the base dimensions of the card are #{\"Quantity\" \"Timestamp\"}.
"
(create-template [_ affinity-name affinities]))
(mu/defn card-based-layout :- [:fn #(satisfies? CardTemplateProducer %)]
"Returns an implementation of `CardTemplateProducer`. This is a bit circular right now as we break the idea of cards
being the driver. Affinities are sets of dimensions that are interesting together. We mine the card template
definitions for these. And then when we want to make a layout, we use the set of interesting dimensions and the name
of that interestingness to find the card that originally defined it. But this gives us the seam to break this
connection. We can independently come up with a notion of interesting combinations and then independently come up
with how to put that in a dashcard."
[{template-cards :cards :as dashboard-template} :- ads/dashboard-template]
(let [by-name (update-vals (group-by ffirst template-cards) #(map (comp val first) %))
resolve-overloading (fn [affinities cards]
(let [provider (base-dimension-provider dashboard-template)]
(some
(fn [card]
(let [dimsets (create-affinity-sets provider card)]
(when (some dimsets affinities) card)))
cards)))]
(reify CardTemplateProducer
(create-template [_ affinity-name affinities]
(let [possible-cards (by-name affinity-name)]
(if (= (count possible-cards) 1)
(first possible-cards)
(resolve-overloading affinities possible-cards)))))))
(mu/defn make-cards :- ads/dashcards
"Create cards from the context using the provided template cards.
Note that card, as destructured here, is a template baked into a dashboard template and is not a db entity Card."
[context available-values {card-templates :cards}]
(some->> card-templates
(map first)
(map-indexed (fn [position [identifier card-template]]
(some->> (assoc card-template :position position)
(card-candidates context available-values)
not-empty
(hash-map (name identifier)))))
[context :- ads/context
available-values :- ads/available-values
satisfied-affinities :- ads/affinity-matches
satisfied-bindings :- [:map-of ads/dimension-set ads/dimension-maps]
layout-producer :- [:fn #(satisfies? CardTemplateProducer %)]]
(some->> satisfied-affinities
(map-indexed (fn [position [affinity-name affinities]]
(let [card-template (create-template layout-producer affinity-name affinities)]
(some->> (assoc card-template :position position)
(card-candidates context satisfied-bindings available-values)
not-empty
(hash-map (name affinity-name))))))
(apply merge-with (partial max-key (comp :score first)) {})
vals
(apply concat)))
......@@ -999,6 +1135,43 @@
:name (:name entity)
:score dashboard-templates/max-score})))
(mu/defn satisified-bindings :- ads/dimension-maps
"Take an affinity set (a set of dimensions) and a map of dimension to bound items and return all possible realized
affinity combinations for this affinity set and binding."
[affinity-set :- ads/dimension-set
available-dimensions :- ads/dimension-bindings]
(->> affinity-set
(map
(fn [affinity-dimension]
(let [{:keys [matches]} (available-dimensions affinity-dimension)]
matches)))
(apply math.combo/cartesian-product)
(mapv (fn [combos] (zipmap affinity-set combos)))))
(mu/defn all-satisfied-bindings :- [:map-of ads/dimension-set ads/dimension-maps]
"Compute all potential combinations of dimensions for each affinity set."
[distinct-affinity-sets :- [:sequential ads/dimension-set]
available-dimensions :- ads/dimension-bindings]
(let [satisfied-combos (map #(satisified-bindings % available-dimensions)
distinct-affinity-sets)]
(zipmap distinct-affinity-sets satisfied-combos)))
(comment
(let [{template-dimensions :dimensions
:as dashboard-template} (dashboard-templates/get-dashboard-template ["table" "GenericTable"])
model (t2/select-one :model/Card 2)
base-context (make-base-context (->root model))
affinities (dash-template->affinities dashboard-template)
available-dimensions (->> (bind-dimensions base-context template-dimensions)
(add-field-self-reference base-context))
satisfied-affinities (match-affinities affinities (set (keys available-dimensions)))
distinct-affinity-sets (-> satisfied-affinities vals distinct flatten)]
(update-vals
(all-satisfied-bindings distinct-affinity-sets available-dimensions)
(fn [v]
(mapv (fn [combo] (update-vals combo #(select-keys % [:name]))) v))))
)
(s/defn ^:private apply-dashboard-template
"Apply a 'dashboard template' (a card template) to the root entity to produce a dashboard
(including filters and cards).
......@@ -1012,21 +1185,30 @@
:keys [dashboard-template-name dashboard_filters]
:as dashboard-template} :- dashboard-templates/DashboardTemplate]
(log/debugf "Applying dashboard template '%s'" dashboard-template-name)
(let [dimensions (->> (bind-dimensions base-context template-dimensions)
(add-field-self-reference base-context))
(let [available-dimensions (->> (bind-dimensions base-context template-dimensions)
(add-field-self-reference base-context))
;; Satisfied metrics and filters are those for which there is a dimension that can be bound to them.
available-metrics (->> (resolve-available-dimensions dimensions template-metrics)
(add-metric-self-reference base-context)
(into {}))
available-filters (into {} (resolve-available-dimensions dimensions template-filters))
available-values {:available-dimensions dimensions
:available-metrics available-metrics
:available-filters available-filters}
cards (make-cards base-context available-values dashboard-template)]
available-metrics (->> (resolve-available-dimensions available-dimensions template-metrics)
(add-metric-self-reference base-context)
(into {}))
available-filters (into {} (resolve-available-dimensions available-dimensions template-filters))
available-values {:available-dimensions available-dimensions
:available-metrics available-metrics
:available-filters available-filters}
;; for now we construct affinities from cards
affinities (dash-template->affinities dashboard-template)
;; get the suitable matches for them
satisfied-affinities (match-affinities affinities (set (keys available-dimensions)))
distinct-affinity-sets (-> satisfied-affinities vals distinct flatten)
cards (make-cards base-context
available-values
satisfied-affinities
(all-satisfied-bindings distinct-affinity-sets available-dimensions)
(card-based-layout dashboard-template))]
(when (or (not-empty cards) (nil? template-cards))
[(assoc (make-dashboard root dashboard-template base-context available-values)
:filters (->> dashboard_filters
(mapcat (comp :matches dimensions))
(mapcat (comp :matches available-dimensions))
(remove (comp (singular-cell-dimensions root) id-or-name)))
:cards cards)
dashboard-template
......
(ns metabase.automagic-dashboards.schema
(:require [malli.core :as mc]))
;; --
(def context
"The big ball of mud data object from which we generate x-rays"
(mc/schema
[:map
[:source any?]
[:root any?]
[:tables {:optional true} any?]
[:query-filter {:optional true} any?]]))
(def dashcard
"The base unit thing we are trying to produce in x-rays"
;; TODO - Beef these specs up, esp. the any?s
(mc/schema
[:map
[:dataset_query {:optional true} any?]
[:dimensions {:optional true} [:sequential string?]]
[:group {:optional true} string?]
[:height pos-int?]
[:metrics {:optional true} any?]
[:position {:optional true} nat-int?]
[:score {:optional true} number?]
[:title {:optional true} string?]
[:visualization {:optional true} any?]
[:width pos-int?]
[:x_label {:optional true} string?]]))
(def dashcards
"A bunch of dashcards"
(mc/schema [:maybe [:sequential dashcard]]))
;;
(def dimension-value
"A specification for the basic keys in the value of a dimension template."
(mc/schema
[:map
[:field_type
[:or
[:tuple :keyword]
[:tuple :keyword :keyword]]]
[:score {:optional true} nat-int?]
[:max_cardinality {:optional true} nat-int?]
[:named {:optional true} [:string {:min 1}]]]))
(def dimension-template
"A specification for the basic keys in a dimension template."
(mc/schema
[:map-of
{:min 1 :max 1}
[:string {:min 1}]
dimension-value]))
(def metric-value
"A specification for the basic keys in the value of a metric template."
(mc/schema
[:map
[:metric [:vector some?]]
[:score {:optional true} nat-int?]
;[:name some?]
]))
(def metric-template
"A specification for the basic keys in a metric template."
(mc/schema
[:map-of
{:min 1 :max 1}
[:string {:min 1}]
metric-value]))
(def filter-value
"A specification for the basic keys in the value of a filter template."
(mc/schema
[:map
[:filter [:vector some?]]
[:score nat-int?]]))
(def filter-template
"A specification for the basic keys in a filter template."
(mc/schema
[:map-of
{:min 1 :max 1}
[:string {:min 1}]
filter-value]))
(def card-value
"A specification for the basic keys in the value of a card template."
(mc/schema
[:map
[:dimensions {:optional true} [:vector (mc/schema
[:map-of
{:min 1 :max 1}
[:string {:min 1}]
[:map
[:aggregation {:optional true} string?]]])]]
[:metrics {:optional true} [:vector string?]]
[:filters {:optional true} [:vector string?]]
[:score {:optional true} nat-int?]]))
(def card-template
"A specification for the basic keys in a card template."
(mc/schema
[:map-of
{:min 1 :max 1}
[:string {:min 1}]
card-value]))
(def dashboard-template
"A specification for the basic keys in a dashboard template."
(mc/schema
[:map
[:dimensions {:optional true} [:vector dimension-template]]
[:metrics {:optional true} [:vector metric-template]]
[:filters {:optional true} [:vector filter-template]]
[:cards {:optional true} [:vector card-template]]]))
;; Available values schema -- These are items for which fields have been successfully bound
(def available-values
"Specify the shape of things that are available after dimension to field matching for affinity matching"
(mc/schema
[:map
[:available-dimensions [:map-of [:string {:min 1}] any?]]
[:available-metrics [:map-of [:string {:min 1}] any?]]
[:available-filters [:map-of [:string {:min 1}] any?]]]))
;; Schemas for "affinity" functions as these can be particularly confusing
(def dimension-set
"A set of dimensions that belong together. This is the basic unity of affinity."
[:set string?])
(def affinity
"A collection of things that go together. In this case, we're a bit specialized on
card affinity, but the key element in the structure is `:base-dims`, which are a
set of dimensions which, when satisfied, enable this affinity object."
(mc/schema
[:map
[:dimensions {:optional true} [:vector string?]]
[:metrics {:optional true} [:vector string?]]
[:filters {:optional true} [:vector string?]]
[:score {:optional true} nat-int?]
[:affinity-name string?]
[:base-dims dimension-set]]))
(def affinities
"A sequence of affinity objects."
(mc/schema
[:sequential affinity]))
(def affinity-matches
"A map of named affinities to all dimension sets that are associated with this name."
(mc/schema
[:map-of
:string
[:vector dimension-set]]))
(def item
"A \"thing\" that we bind to, consisting, generally, of at least a name and id"
(mc/schema
[:map
[:id {:optional true} nat-int?]
[:name {:optional true} string?]]))
(def dimension-bindings
"A map of named dimensions to a map containing a sequence of matching items satisfying this dimension"
(mc/schema
[:map-of
:string
[:map [:matches [:sequential item]]]]))
(def dimension-map
"A map of dimension names to item satisfying that dimensions"
(mc/schema
[:map-of :string item]))
(def dimension-maps
"A sequence of dimension maps"
(mc/schema
[:sequential dimension-map]))
(comment
(require '[malli.generator :as mg])
(mg/sample dashboard-template)
(mg/sample affinities)
(mg/sample affinity-matches))
This diff is collapsed.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment