Skip to content
Snippets Groups Projects
Unverified Commit e2e6f138 authored by adam-james's avatar adam-james Committed by GitHub
Browse files

Add dev fn to render dashboards with static-viz renderers (#36451) 

* Add dev fn to render dashboards with static-viz renderers

Run `(dev.render-png/render-dashboard-to-html 1)` to render the dashboard with id 1 to a handy html file.

This file will render each dashcard in the dashboard 3 ways:
 - as a png, like you would see in Slack or in an email body (if it's a chart)
 - as html/svg, like you'd see in the email body (for tables).. might also be helpful to inspect the svg output from
 the graaljs interpreter
 - 10 row table representing what the .csv attachment for the dashcard would look like

Note: to get this branch working properly, there are a couple lines commented out to pass the linter... it's a work in
progress and will be cleaned up for better use.

Todo:
 - [ ] avoid with-redefs to pass the linter
 - [ ] add a preview endpoint that can be used so that you don't need to run a Clojure repl to get the same result.

* Eliminate with-redefs to allow csv 'render'

* Make a 'preview' ns to power `api/pulse/preview_dashboard/:id`

* Make dev.render-png dashboard preview fns the same as the preview ns

* Dashboard Subscription preview ns now also uses csv attachment code

This change now uses the csv attachment code to include an html table representing what we expect to see from csv exports.

* Add dev fn to render dashboards with static-viz renderers

Run `(dev.render-png/render-dashboard-to-html 1)` to render the dashboard with id 1 to a handy html file.

This file will render each dashcard in the dashboard 3 ways:
 - as a png, like you would see in Slack or in an email body (if it's a chart)
 - as html/svg, like you'd see in the email body (for tables).. might also be helpful to inspect the svg output from
 the graaljs interpreter
 - 10 row table representing what the .csv attachment for the dashcard would look like

Note: to get this branch working properly, there are a couple lines commented out to pass the linter... it's a work in
progress and will be cleaned up for better use.

Todo:
 - [ ] avoid with-redefs to pass the linter
 - [ ] add a preview endpoint that can be used so that you don't need to run a Clojure repl to get the same result.

* Eliminate with-redefs to allow csv 'render'

* Make a 'preview' ns to power `api/pulse/preview_dashboard/:id`

* Make dev.render-png dashboard preview fns the same as the preview ns

* Dashboard Subscription preview ns now also uses csv attachment code

This change now uses the csv attachment code to include an html table representing what we expect to see from csv exports.

* Collect all element styles into a single style tag to use with the server's nonce

This allows the dashboard_preview endpoint to render properly by gathering all of the element styles into a single
style tag, which can then be given the server's nonce value so that CSP doesn't strip the style out.

This is useful for the preview endpoint so that we get an accurate picture of what our tables will look like in
emails, plus it makes the presentation look just a bit more readable.

* dev render_png fns match output used on the endpoint now too.

* Hickory dependency out of dev into regular deps
parent ab5b0673
Branches
Tags
No related merge requests found
Hide whitespace changes
Inline Side-by-side
deps.edn
+ 1
− 1
View file @ e2e6f138
......@@ -130,6 +130,7 @@
org.clojars.pntblnk/clj-ldap {:mvn/version "0.0.17"} ; LDAP client
org.bouncycastle/bcpkix-jdk18on {:mvn/version "1.77"} ; Bouncy Castle crypto library -- explicit version of BC specified to resolve illegal reflective access errors
org.bouncycastle/bcprov-jdk18on {:mvn/version "1.77"}
org.clj-commons/hickory {:mvn/version "0.7.3"} ; Parse HTML into Clojure data structures
org.clojure/clojure {:mvn/version "1.11.1"}
org.clojure/core.async {:mvn/version "1.6.681"
:exclusions [org.clojure/tools.reader]}
......@@ -232,7 +233,6 @@
[org.ow2.asm/asm-all]}
lambdaisland/deep-diff2 {:mvn/version "2.10.211"} ; way better diffs
methodical/methodical {:mvn/version "0.15.1"} ; drop-in replacements for Clojure multimethods and adds several advanced features
org.clj-commons/hickory {:mvn/version "0.7.3"} ; Parse HTML into Clojure data structures
org.clojure/algo.generic {:mvn/version "0.1.3"}
pjstadig/humane-test-output {:mvn/version "0.11.0"}
reifyhealth/specmonstah {:mvn/version "2.1.0"
......
dev/src/dev/render_png.clj
+ 132
− 34
View file @ e2e6f138
......@@ -2,14 +2,19 @@
"Improve feedback loop for dealing with png rendering code. Will create images using the rendering that underpins
pulses and subscriptions and open those images without needing to send them to slack or email."
(:require
[clojure.data.csv :as csv]
[clojure.java.io :as io]
[clojure.java.shell :as sh]
[hiccup.core :as hiccup]
[metabase.email.messages :as messages]
[metabase.models :refer [Card]]
[metabase.models.card :as card]
[metabase.pulse :as pulse]
[metabase.pulse.markdown :as markdown]
[metabase.pulse.render :as render]
[metabase.pulse.render.test-util :as render.tu]
[metabase.pulse.render.image-bundle :as img]
[metabase.pulse.render.png :as png]
[metabase.pulse.render.style :as style]
[metabase.query-processor :as qp]
[metabase.test :as mt]
[toucan2.core :as t2])
......@@ -85,29 +90,121 @@
(.deleteOnExit tmp-file)
(open tmp-file)))
(comment
(render-card-to-png 1)
(let [{:keys [content]} (render-pulse-card 1)]
(open-hiccup-as-html content))
;; open viz in your browser
(-> [["A" "B"]
[1 2]
[30 20]]
(render.tu/make-viz-data :line {:goal-line {:graph.goal_label "Target"
:graph.goal_value 20}})
:viz-tree
open-hiccup-as-html)
(-> [["As" "Bs" "Cs" "Ds" "Es"]
["aa" "bb" "cc" "dd" "ee"]
["aaa" "bbb" "ccc" "ddd" "eee"]]
(render.tu/make-viz-data :table {:reordered-columns {:order [2 3 1 0 4]}
:custom-column-names {:names ["-A-" "-B-" "-C-" "-D-"]}
:hidden-columns {:hide [0 2]}})
:viz-tree
open-hiccup-as-html))
(def ^:private execute-dashboard #'pulse/execute-dashboard)
(defn render-dashboard-to-pngs
"Given a dashboard ID, renders each dashcard, including Markdown, to its own temporary png image, and opens each one."
[dashboard-id]
(let [user (t2/select-one :model/User)
dashboard (t2/select-one :model/Dashboard :id dashboard-id)
dashboard-results (execute-dashboard {:creator_id (:id user)} dashboard)]
(doseq [{:keys [card dashcard result] :as dashboard-result} dashboard-results]
(let [render (if card
(render/render-pulse-card :inline (pulse/defaulted-timezone card) card dashcard result)
{:content [:div {:style (style/style {:font-family "Lato"
:font-size "0.875em"
:font-weight "400"
:font-style "normal"
:color "#4c5773"
:-moz-osx-font-smoothing "grayscale"})}
(markdown/process-markdown (:text dashboard-result) :html)]
:attachments nil})
png-bytes (-> render (png/render-html-to-png 1000))
tmp-file (java.io.File/createTempFile "card-png" ".png")]
(with-open [w (java.io.FileOutputStream. tmp-file)]
(.write w ^bytes png-bytes))
(.deleteOnExit tmp-file)
(open tmp-file)))))
(def ^:private table-style-map
{:border "1px solid black"
:border-collapse "collapse"
:padding "5px"})
(def ^:private table-style
(style/style table-style-map))
(def ^:private csv-row-limit 10)
(defn- csv-to-html-table [csv-string]
(let [rows (csv/read-csv csv-string)]
[:table {:style table-style}
(for [row (take (inc csv-row-limit) rows)] ;; inc row-limit to include the header and the expected # of rows
[:tr {:style table-style}
(for [cell row]
[:td {:style table-style} cell])])]))
(def ^:private result-attachment #'messages/result-attachment)
(defn- render-csv-for-dashcard
[part]
(-> part
(assoc-in [:card :include_csv] true)
result-attachment
first
:content
slurp
csv-to-html-table))
(defn- render-one-dashcard
[{:keys [card dashcard result] :as dashboard-result}]
(letfn [(cellfn [content]
[:td {:style (style/style (merge table-style-map {:max-width "400px"}))}
content])]
(if card
(let [base-render (render/render-pulse-card :inline (pulse/defaulted-timezone card) card dashcard result)
html-src (-> base-render :content)
img-src (-> base-render
(png/render-html-to-png 1200)
img/render-img-data-uri)
csv-src (render-csv-for-dashcard dashboard-result)]
[:tr
(cellfn (:name card))
(cellfn [:img {:style (style/style {:max-width "400px"}) :src img-src}])
(cellfn html-src)
(cellfn csv-src)])
[:tr
(cellfn nil)
(cellfn
[:div {:style (style/style {:font-family "Lato"
:font-size "13px" #_ "0.875em"
:font-weight "400"
:font-style "normal"
:color "#4c5773"
:-moz-osx-font-smoothing "grayscale"})}
(markdown/process-markdown (:text dashboard-result) :html)])
(cellfn nil)])))
(defn render-dashboard-to-hiccup
"Given a dashboard ID, renders all of the dashcards to hiccup datastructure."
[dashboard-id]
(let [user (t2/select-one :model/User)
dashboard (t2/select-one :model/Dashboard :id dashboard-id)
dashboard-results (execute-dashboard {:creator_id (:id user)} dashboard)
render (->> (map render-one-dashcard (map #(assoc % :dashboard-id dashboard-id) dashboard-results))
(into [[:tr
[:th {:style (style/style table-style-map)} "Card Name"]
[:th {:style (style/style table-style-map)} "PNG"]
[:th {:style (style/style table-style-map)} "HTML"]
[:th {:style (style/style table-style-map)} "CSV"]]])
(into [:table {:style (style/style table-style-map)}]))]
render))
(defn render-dashboard-to-html
"Given a dashboard ID, renders all of the dashcards into an html document."
[dashboard-id]
(hiccup/html (render-dashboard-to-hiccup dashboard-id)))
(defn render-dashboard-to-html-and-open
"Given a dashboard ID, renders all of the dashcards to an html file and opens it."
[dashboard-id]
(let [html-str (render-dashboard-to-html dashboard-id)
tmp-file (File/createTempFile "card-html" ".html")]
(with-open [w (io/writer tmp-file)]
(.write w ^String html-str))
(.deleteOnExit tmp-file)
(open tmp-file)))
(comment
;; This form has 3 cards:
......@@ -119,16 +216,17 @@
;; - The plain question will not have custom formatting applied
;; - The model and derived query will have custom formatting applied
(mt/dataset sample-dataset
(mt/with-temp [Card {base-card-id :id} {:dataset_query {:database (mt/id)
:type :query
:query {:source-table (mt/id :orders)
:expressions {"Tax Rate" [:/
[:field (mt/id :orders :tax) {:base-type :type/Float}]
[:field (mt/id :orders :total) {:base-type :type/Float}]]},
:fields [[:field (mt/id :orders :tax) {:base-type :type/Float}]
[:field (mt/id :orders :total) {:base-type :type/Float}]
[:expression "Tax Rate"]]
:limit 10}}}
(mt/with-temp [Card {base-card-id :id}
{:dataset_query {:database (mt/id)
:type :query
:query {:source-table (mt/id :orders)
:expressions {"Tax Rate" [:/
[:field (mt/id :orders :tax) {:base-type :type/Float}]
[:field (mt/id :orders :total) {:base-type :type/Float}]]},
:fields [[:field (mt/id :orders :tax) {:base-type :type/Float}]
[:field (mt/id :orders :total) {:base-type :type/Float}]
[:expression "Tax Rate"]]
:limit 10}}}
Card {model-card-id :id} {:dataset true
:dataset_query {:type :query
:database (mt/id)
......
src/metabase/api/pulse.clj
+ 23
− 2
View file @ e2e6f138
......@@ -4,6 +4,7 @@
[clojure.set :refer [difference]]
[compojure.core :refer [GET POST PUT]]
[hiccup.core :refer [html]]
[hiccup.page :refer [html5]]
[metabase.api.alert :as api.alert]
[metabase.api.common :as api]
[metabase.api.common.validation :as validation]
......@@ -23,6 +24,7 @@
[metabase.plugins.classloader :as classloader]
[metabase.public-settings.premium-features :as premium-features]
[metabase.pulse]
[metabase.pulse.preview :as preview]
[metabase.pulse.render :as render]
[metabase.query-processor :as qp]
[metabase.query-processor.middleware.permissions :as qp.perms]
......@@ -268,13 +270,29 @@
(let [card (api/read-check Card id)
result (pulse-card-query-results card)]
{:status 200
:body (html
:body (html5
[:html
[:body {:style "margin: 0;"}
(binding [render/*include-title* true
render/*include-buttons* true]
(render/render-pulse-card-for-display (metabase.pulse/defaulted-timezone card) card result))]])}))
(api/defendpoint GET "/preview_dashboard/:id"
"Get HTML rendering of a Dashboard with `id`.
This endpoint relies on a custom middleware defined in `metabase.pulse.preview/style-tag-nonce-middleware` to
allow the style tag to render properly, given our Content Security Policy setup. This middleware is attached to these
routes at the bottom of this namespace using `metabase.api.common/define-routes`."
[id]
{id ms/PositiveInt}
(api/read-check :model/Dashboard id)
{:status 200
:headers {"Content-Type" "text/html"}
:body (preview/style-tag-from-inline-styles
(html5
[:body [:h2 (format "Backend Artifacts Preview for Dashboard %s" id)]
(preview/render-dashboard-to-html id)]))})
(api/defendpoint GET "/preview_card_info/:id"
"Get JSON object containing HTML rendering of a Card with `id` and other information."
[id]
......@@ -332,4 +350,7 @@
(t2/delete! PulseChannelRecipient :id pcr-id))
api/generic-204-no-content)
(api/define-routes)
(def ^:private style-nonce-middleware
(partial preview/style-tag-nonce-middleware "/api/pulse/preview_dashboard"))
(api/define-routes style-nonce-middleware)
src/metabase/pulse/preview.clj 0 → 100644
+ 170
− 0
View file @ e2e6f138
(ns metabase.pulse.preview
"Improve the feedback loop for Dashboard Subscription outputs."
(:require
[clojure.data.csv :as csv]
[clojure.string :as str]
[clojure.zip :as zip]
[hiccup.core :as hiccup]
[hickory.core :as hik]
[hickory.render :as hik.r]
[hickory.zip :as hik.z]
[metabase.email.messages :as messages]
[metabase.pulse :as pulse]
[metabase.pulse.markdown :as markdown]
[metabase.pulse.render :as render]
[metabase.pulse.render.image-bundle :as img]
[metabase.pulse.render.png :as png]
[metabase.pulse.render.style :as style]
[toucan2.core :as t2]))
(set! *warn-on-reflection* true)
(def ^:private table-style-map
{:border "1px solid black"
:border-collapse "collapse"
:padding "5px"})
(def ^:private table-style
(style/style table-style-map))
(def ^:private csv-row-limit 10)
(defn- csv-to-html-table [csv-string]
(let [rows (csv/read-csv csv-string)]
[:table {:style table-style}
(for [row (take (inc csv-row-limit) rows)] ;; inc row-limit to include the header and the expected # of rows
[:tr {:style table-style}
(for [cell row]
[:td {:style table-style} cell])])]))
(def ^:private result-attachment #'messages/result-attachment)
(defn- render-csv-for-dashcard
[part]
(-> part
(assoc-in [:card :include_csv] true)
result-attachment
first
:content
slurp
csv-to-html-table))
(defn- render-one-dashcard
[{:keys [card dashcard result] :as dashboard-result}]
(letfn [(cellfn [content]
[:td {:style (style/style (merge table-style-map {:max-width "400px"}))}
content])]
(if card
(let [base-render (render/render-pulse-card :inline (pulse/defaulted-timezone card) card dashcard result)
html-src (-> base-render :content)
img-src (-> base-render
(png/render-html-to-png 1200)
img/render-img-data-uri)
csv-src (render-csv-for-dashcard dashboard-result)]
[:tr
(cellfn (:name card))
(cellfn [:img {:style (style/style {:max-width "400px"}) :src img-src}])
(cellfn html-src)
(cellfn csv-src)])
[:tr
(cellfn nil)
(cellfn
[:div {:style (style/style {:font-family "Lato"
:font-size "13px" #_ "0.875em"
:font-weight "400"
:font-style "normal"
:color "#4c5773"
:-moz-osx-font-smoothing "grayscale"})}
(markdown/process-markdown (:text dashboard-result) :html)])
(cellfn nil)])))
(def ^:private execute-dashboard #'pulse/execute-dashboard)
(defn render-dashboard-to-hiccup
"Given a dashboard ID, renders all of the dashcards to hiccup datastructure."
[dashboard-id]
(let [user (t2/select-one :model/User)
dashboard (t2/select-one :model/Dashboard :id dashboard-id)
dashboard-results (execute-dashboard {:creator_id (:id user)} dashboard)
render (->> (map render-one-dashcard (map #(assoc % :dashboard-id dashboard-id) dashboard-results))
(into [[:tr
[:th {:style (style/style table-style-map)} "Card Name"]
[:th {:style (style/style table-style-map)} "PNG"]
[:th {:style (style/style table-style-map)} "HTML"]
[:th {:style (style/style table-style-map)} "CSV"]]])
(into [:table {:style (style/style table-style-map)}]))]
render))
(defn render-dashboard-to-html
"Given a dashboard ID, renders all of the dashcards into an html document."
[dashboard-id]
(hiccup/html (render-dashboard-to-hiccup dashboard-id)))
(defn- collect-inline-style
[style-lines {:keys [attrs] :as node}]
(let [{:keys [style]} attrs]
(if style
(let [{:keys [id] :or {id (str (gensym "inline"))}} attrs]
(swap! style-lines assoc id style)
(-> node
(update :attrs dissoc :style)
(update :attrs assoc :id id)))
node)))
(defn- css-str-fragment
[[id css-str]]
(format "#%s {%s}" id css-str))
(defn- style-node
[style-lines-map]
{:type :element
:tag :style
:attrs {:nonce "%NONCE%"}
:content [(str/join "\n" (map css-str-fragment style-lines-map))]})
(defn- move-inline-styles
[hickory-tree]
(let [zipper (hik.z/hickory-zip hickory-tree)
style-lines (atom {})
xf-tree (loop [loc zipper]
(if (zip/end? loc)
(zip/root loc)
(recur (zip/next (zip/edit loc (partial collect-inline-style style-lines))))))]
(update xf-tree :content
(fn [v]
(vec (conj (seq v) (style-node @style-lines)))))))
(defn style-tag-from-inline-styles
"Collects styles defined on element 'style' attributes and adds them to a single inline style tag.
Each element that does not already have an 'id' attribute will have one generated, and the style will be added under that id, or the element's existing id.
For example, the html string \"<p style='color: red;'>This is red text.</p>\" Will result in a CSS map-entry
that looks like: #inline12345 {color: red;}.
This approach will capture all inline styles but is naive and will result in lots of style duplications. Since this
is a simple preview endpoint not meant for heavy use outside of manual checks, this slower approach seems ok for now (as of 2023-12-18)."
[html-str]
(-> html-str
hik/parse
hik/as-hickory
move-inline-styles
hik.r/hickory-to-html))
(defn- add-style-nonce [request response]
(update response :body (fn [html-str]
(str/replace html-str #"%NONCE%" (:nonce request)))))
(defn style-tag-nonce-middleware
"Constructs a middleware handler function that adds the generated nonce to an html string.
This is only designed to be used with an endpoint that returns an html string response containing
a style tag with an attribute 'nonce=%NONCE%'. Specifcally, this was designed to be used with the
endpoint `api/pulse/preview_dashboard/:id`."
[only-this-uri handler]
(fn [request respond raise]
(let [{:keys [uri]} request]
(handler
request
(if (str/starts-with? uri only-this-uri)
(comp respond (partial add-style-nonce request))
respond)
raise))))
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment