Skip to content
Snippets Groups Projects
  • dpsutton's avatar
    ea0d4329
    Fix for toucan2 laziness in hydration (#28201) · ea0d4329
    dpsutton authored
    https://github.com/camsaul/toucan2/pull/126
    
    Editing the data model loads a _lot_ of data. It hits the following form
    from the api:
    
    ```clojure
    (hydrate [:tables [:fields [:target :has_field_values]
                       :has_field_values] :segments :metrics])
    ```
    
    On my test instances, with 1500 tables each with 14 fields: 15mb and 359
    tables each with 14 fields: 3.69mb. Not great.
    
    If you want to recreate, create an empty db, sync it, then
    
    ```clojure
      (time
       (let [conn (sql-jdbc.conn/db->pooled-connection-spec 3)]
         (dotimes [n 359]
           (jdbc/execute! conn [(format "create table if not exists foo_%d (id int, col_1 text, col_2 text, col_3 text, col_4 text, col_5 text, col_6 text, col_7 text, col_8 text, col_9 text, col_10 text, col_11 text, col_12 text, col_13 text, col_14 text)" n)]))))
    
      (sync/sync-database! (db/select-one 'Database :id 3) {:scan :schema})
    
      (binding [api/*current-user-permissions-set* (delay #{"/"})]
        (count (#'metabase.api.database/db-metadata 3 true true)))
    ```
    
    And the source of the bug was a classic recur with a lazy concat.
    
    ```diff
    diff --git a/src/toucan2/tools/hydrate.clj b/src/toucan2/tools/hydrate.clj
    index e1d0459..10c9c49 100644
    --- a/src/toucan2/tools/hydrate.clj
    +++ b/src/toucan2/tools/hydrate.clj
    @@ -456,7 +456,7 @@
         (if (empty? hydrated-instances)
           (concat acc (map :instance annotated-instances))
           (let [[not-hydrated [_needed-hydration & more]] (split-with (complement :needs-hydration?) annotated-instances)]
    -        (recur (concat acc (map :instance not-hydrated) [(first hydrated-instances)])
    +        (recur (vec (concat acc (map :instance not-hydrated) [(first hydrated-instances)]))
                    more
                    (rest hydrated-instances))))))
    
    ```
    Fix for toucan2 laziness in hydration (#28201)
    dpsutton authored
    https://github.com/camsaul/toucan2/pull/126
    
    Editing the data model loads a _lot_ of data. It hits the following form
    from the api:
    
    ```clojure
    (hydrate [:tables [:fields [:target :has_field_values]
                       :has_field_values] :segments :metrics])
    ```
    
    On my test instances, with 1500 tables each with 14 fields: 15mb and 359
    tables each with 14 fields: 3.69mb. Not great.
    
    If you want to recreate, create an empty db, sync it, then
    
    ```clojure
      (time
       (let [conn (sql-jdbc.conn/db->pooled-connection-spec 3)]
         (dotimes [n 359]
           (jdbc/execute! conn [(format "create table if not exists foo_%d (id int, col_1 text, col_2 text, col_3 text, col_4 text, col_5 text, col_6 text, col_7 text, col_8 text, col_9 text, col_10 text, col_11 text, col_12 text, col_13 text, col_14 text)" n)]))))
    
      (sync/sync-database! (db/select-one 'Database :id 3) {:scan :schema})
    
      (binding [api/*current-user-permissions-set* (delay #{"/"})]
        (count (#'metabase.api.database/db-metadata 3 true true)))
    ```
    
    And the source of the bug was a classic recur with a lazy concat.
    
    ```diff
    diff --git a/src/toucan2/tools/hydrate.clj b/src/toucan2/tools/hydrate.clj
    index e1d0459..10c9c49 100644
    --- a/src/toucan2/tools/hydrate.clj
    +++ b/src/toucan2/tools/hydrate.clj
    @@ -456,7 +456,7 @@
         (if (empty? hydrated-instances)
           (concat acc (map :instance annotated-instances))
           (let [[not-hydrated [_needed-hydration & more]] (split-with (complement :needs-hydration?) annotated-instances)]
    -        (recur (concat acc (map :instance not-hydrated) [(first hydrated-instances)])
    +        (recur (vec (concat acc (map :instance not-hydrated) [(first hydrated-instances)]))
                    more
                    (rest hydrated-instances))))))
    
    ```
Code owners
Assign users and groups as approvers for specific file changes. Learn more.