Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Mar 31, 2023
    • dpsutton's avatar
      Don't log in bigquery results hotpath (#29727) · dbc4a37d
      dpsutton authored
      * Don't log in bigquery results hotpath
      
      Right now we log in the parser of bigquery results for unrecognized
      types. But the problem is that we log for each _value_ and not each
      column. This is causing an _enormous_ amount of logs and performance
      penalty.
      
      See
      - https://github.com/metabase/metabase/issues/29118 (performance)
      - https://github.com/metabase/metabase/issues/28868 (filling disk space)
      
      This log was added between 45.1 and 45.3
      
      ```diff
      ❯ git diff v0.45.1..v0.45.3 modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/**
      diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
      index a0d8081c30..f367199b55 100644
      --- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
      +++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj
      @@ -22,7 +22,7 @@
                   [metabase.util :as u]
                   [metabase.util.date-2 :as u.date]
                   [metabase.util.honeysql-extensions :as hx]
      -            [metabase.util.i18n :refer [tru]]
      +            [metabase.util.i18n :refer [trs tru]]
                   [pretty.core :refer [PrettyPrintable]]
                   [schema.core :as s])
         (:import [com.google.cloud.bigquery Field$Mode FieldValue]
      @@ -88,7 +88,8 @@
           (parse-fn v)))
      
       (defmethod parse-result-of-type :default
      -  [_ column-mode _ v]
      +  [column-type column-mode _ v]
      +  (log/warn (trs "Warning: missing type mapping for parsing BigQuery results of type {0}." column-type))
         (parse-value column-mode v identity))
      ```
      
      The result is that selecting 50,000 rows for download in excel:
      
      | version                      | time       |
      |------------------------------|------------|
      | 0.45.1                       | 28 seconds |
      | 0.45.3                       | 52 seconds |
      | 0.45.3 with logging disabled | 30 seconds |
      
      (disable logging by adding `<Logger
      name="metabase.driver.bigquery-cloud-sdk.query-processor"
      level="ERROR"/>` and `-Dlog4j2.configurationFile=log4j2.xml` to jar
      startup)
      
      For the query (3 string columns, 5 rows):
      
      ```sql
      SELECT game_id, first_name, last_name
      FROM `bigquery-public-data.ncaa_basketball.mbb_players_games_sr`
      LIMIT 5
      ```
      
      BEFORE:
      
      ```
      ```
      2023-03-31 17:17:52,146 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING.
      2023-03-31 17:17:52,155 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 795.2 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (2 idle, 0 queued) (192 total active threads) Queries in flight: 0 (0 queued)
      ```
      
      Note this is 15 logs (3 columns x 5 rows)
      
      AFTER:
      
      ```
      2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column game_id of type STRING.
      2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column first_name of type STRING.
      2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column last_name of type STRING.
      2023-03-31 17:19:15,757 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 973.5 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (3 idle, 0 queued) (193 total active threads) Queries in flight: 0 (0 queued)
      ```
      
      * unused require to appease our overlords
      dbc4a37d
    • Luis Paolini's avatar
      Update Clojure CLI on CI and Dockerfile (#29723) · 6c62db77
      Luis Paolini authored
      * Update Clojure CLI
      
      Most important change: using stable tools.deps
      
      https://clojure.org/releases/tools
      6c62db77
    • Jeff Bruemmer's avatar
      docs - add link to cloud docs (#29733) · 14d7a90c
      Jeff Bruemmer authored
      14d7a90c
    • Denis Berezin's avatar
    • Braden Shepherdson's avatar
      [MLv2] Test `metabase.lib.convert` by round-tripping every `mbql-query` (#29487) · 61d382f1
      Braden Shepherdson authored
      This is a great way to make sure the conversion in both directions
      is robust; every feature of MBQL is exercised somewhere in the BE tests.
      
      In a few cases the `mbql-query` is called from a context where a
      `(is ...)` test won't work - those places can use a `mbql-query-no-test`
      instead. There were actually only two of these.
      
      This exposed several issues in `->legacy-MBQL` which have now been fixed.
      61d382f1
    • Nemanja Glumac's avatar
      Revert "Toggle JSON unfolding by field (#28742)" (#29720) · b92aaea1
      Nemanja Glumac authored
      This reverts commit 25a1b259.
      b92aaea1
    • Nemanja Glumac's avatar
      4d1db365
    • Anton Kulyk's avatar
      Clean up MLv2 usage on the FE (#29683) · de27e29e
      Anton Kulyk authored
      * Add more MLv2 wrappers
      
      * Don't use MLv2 directly
      
      * Update tests
      
      * Add dedicated `OrderByClause` type
      
      * Rename `query` to `fromLegacyQuery`
      
      * Remove too precise `orderableColumns` test
      
      Agreed that should be better covered on the CLJS side
      de27e29e
    • Nemanja Glumac's avatar
    • Cal Herries's avatar
      Toggle JSON unfolding by field (#28742) · 25a1b259
      Cal Herries authored
      
      * Use type/JSON instead of type/SerializedJSON
      
      * Tidy migration
      
      * Update migration
      
      * Fix rollback migration for h2
      
      * whitespace
      
      * Add test for migration
      
      * Fix test
      
      * Add rollback
      
      * whitespace
      
      * Test JSONB type as well as JSON
      
      * Don't fingerprint JSON columns
      
      * Remove comment, that might be wrong in the future
      
      * Update test
      
      * Add tests for fingerprinting
      
      * Use base-type JSON for fingerprinting base query
      
      * Add test for visibility-type=details-only
      
      * undo .
      
      * Change migration id
      
      * Fix migration test
      
      * Merge master
      
      * Exclude mariadb from tests
      
      * Make is-mariadb? public
      
      * Migration for adding nfc_enabled
      
      * Add nfc_enabled to field settings
      
      * Update describe-nested-field-columns to only unfold fields that have not been disabled
      
      * Remove spy
      
      * Tidy
      
      * Fix
      
      * Clear nested fields immediately if folding is disabled
      
      * Clear nested fields on nfc_enabled change
      
      * Trim trailing whitespace
      
      * Tidy
      
      * Fix
      
      * Add enable-json-unfolding-test
      
      * Move to field api test
      
      * nfc_enabled -> json_unfolding
      
      * Tidy test
      
      * Make json-unfolding in database details just the default for new settings
      
      * Restore original visibility-type logic
      
      * Fix
      
      * Sync field json_unfolding according to db json_folding
      
      * Rename to json-unfolding-default
      
      * Add test for the case when the json-unfolding is false for the database
      
      * Implement default json unfolding for first sync
      
      * Update comment
      
      * Update comment
      
      * Restore PUT field
      
      * Migration for populating json_unfolding for mysql and postgres
      
      * Remove migration and use default on the frontend instead
      
      * Update json-unfolding database setting copy
      
      * Move Unfold JSON setting under semantic type setting
      
      * Separate sentences with spaces
      
      * Capitalize
      
      * Restore handleChangeSemanticType
      
      * Use base_type TYPE.JSON instead
      
      * Change order of migrations
      
      * Add h2 migration
      
      * Add test for migration
      
      * Update test description
      
      * Remove validCheckSum
      
      Co-authored-by: default avatarNgoc Khuat <qn.khuat@gmail.com>
      
      * Use ?? instead of ||
      
      * Fix tests
      
      * Remove outdated serdes stuff
      
      * Remove unnecessary and
      
      * Fix mysql migration
      
      * Remove unused require
      
      * Fix test
      
      * Add false default value for json_unfolding
      
      * Fix json-unfolding nil case
      
      * Other suggestions
      
      * whitespace
      
      * Tidy describe-nested-field-columns
      
      * Update comment
      
      * Remove unused clear-nested-fields!
      
      * Remove unused return value
      
      * Fix H2 migration to use base_type not database_type
      
      * Always set json-unfolding during sync
      
      * Fix test
      
      * Fix test
      
      * Fix test
      
      * Add comment explaining nested-field-column support for MySQL
      
      * Fix tests
      
      * Fix tests
      
      * Fix test
      
      * Fix test
      
      * Default json_unfolding to false for new fields
      
      * Fix merge
      
      * Add json_unfolding to mock tables
      
      * Don't capitalize prepositions
      
      * Update setting description
      
      * whitespace
      
      * whitespace
      
      * Fix fetch_metadata
      
      * Fix fetch_metadata
      
      * Fix tests
      
      * Fix test
      
      * Fix clj-kondo
      
      * Remove postgres database-supports test
      
      * Fix postgres test
      
      * Fix postgres test
      
      * Fix mysql migration
      
      * Fix clj-kondo
      
      * Don't test mariadb
      
      * Fix mysql test
      
      * Fix mysql json-unfolding nil case
      
      * Add comments to test
      
      * Add upterm step to mariadb test
      
      * Move upterm step before tests
      
      * Fix postgresql migration
      
      * move upterm step to test-driver action
      
      * Remove upterm step from drivers.yml
      
      * Comment out everything else in test-driver
      
      * Fix mariadb migration
      
      * Remove upterm action
      
      * Whitespace
      
      * Only handle JSONObjects with JSON_VALUE, not JSONArray
      
      * Fix fields.sync_metadata/update-field-metadata-if-needed!
      
      * Add comment explaining `json-unfolding-default`
      
      * Add comment for JSON_VALUE test
      
      * Update json-unfolding-default-true-test to use fresh db
      
      * Fix test descriptions
      
      * Don't capitalize setting display name
      
      * Fix update json_unfolding
      
      * Fix unfolding json for only one JSON column, not all
      
      * Add e2e test for JSON unfolding setting
      
      * Fix test
      
      * whitespace
      
      * Fix test
      
      * Fix json-unfolding for nested field columns
      
      * Fix mysql and postgres tests for nested field columns
      
      * Coerce nil json-unfolding to false for sync_metadata
      
      * Add test for nil json-unfolding for sync_metadata
      
      * Fix test
      
      * Don't update json_unfolding from intial sync
      
      * Add json_unfolding to update-field-test
      
      * Fix tests
      
      ---------
      
      Co-authored-by: default avatarNgoc Khuat <qn.khuat@gmail.com>
      25a1b259
  2. Mar 30, 2023
  3. Mar 29, 2023
Loading