This project is mirrored from https://github.com/metabase/metabase.
Pull mirroring updated .
- Apr 03, 2023
-
-
Noah Moss authored
-
Denis Berezin authored
-
metabase-bot[bot] authored
Co-authored-by:
Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
-
- Mar 31, 2023
-
-
metabase-bot[bot] authored
* Don't log in bigquery results hotpath Right now we log in the parser of bigquery results for unrecognized types. But the problem is that we log for each _value_ and not each column. This is causing an _enormous_ amount of logs and performance penalty. See - https://github.com/metabase/metabase/issues/29118 (performance) - https://github.com/metabase/metabase/issues/28868 (filling disk space) This log was added between 45.1 and 45.3 ```diff ❯ git diff v0.45.1..v0.45.3 modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/** diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj index a0d8081c30..f367199b55 100644 --- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj +++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj @@ -22,7 +22,7 @@ [metabase.util :as u] [metabase.util.date-2 :as u.date] [metabase.util.honeysql-extensions :as hx] - [metabase.util.i18n :refer [tru]] + [metabase.util.i18n :refer [trs tru]] [pretty.core :refer [PrettyPrintable]] [schema.core :as s]) (:import [com.google.cloud.bigquery Field$Mode FieldValue] @@ -88,7 +88,8 @@ (parse-fn v))) (defmethod parse-result-of-type :default - [_ column-mode _ v] + [column-type column-mode _ v] + (log/warn (trs "Warning: missing type mapping for parsing BigQuery results of type {0}." column-type)) (parse-value column-mode v identity)) ``` The result is that selecting 50,000 rows for download in excel: | version | time | |------------------------------|------------| | 0.45.1 | 28 seconds | | 0.45.3 | 52 seconds | | 0.45.3 with logging disabled | 30 seconds | (disable logging by adding `<Logger name="metabase.driver.bigquery-cloud-sdk.query-processor" level="ERROR"/>` and `-Dlog4j2.configurationFile=log4j2.xml` to jar startup) For the query (3 string columns, 5 rows): ```sql SELECT game_id, first_name, last_name FROM `bigquery-public-data.ncaa_basketball.mbb_players_games_sr` LIMIT 5 ``` BEFORE: ``` ``` 2023-03-31 17:17:52,146 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,155 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 795.2 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (2 idle, 0 queued) (192 total active threads) Queries in flight: 0 (0 queued) ``` Note this is 15 logs (3 columns x 5 rows) AFTER: ``` 2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column game_id of type STRING. 2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column first_name of type STRING. 2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column last_name of type STRING. 2023-03-31 17:19:15,757 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 973.5 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (3 idle, 0 queued) (193 total active threads) Queries in flight: 0 (0 queued) ``` * unused require to appease our overlords Co-authored-by:
dpsutton <dan@dpsutton.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
- Mar 30, 2023
-
-
metabase-bot[bot] authored
From https://github.com/metabase/metabase/issues/29545 Co-authored-by:
Luis Paolini <paoliniluis@gmail.com>
-
- Mar 29, 2023
-
-
metabase-bot[bot] authored
* fix typo * fix links --------- Co-authored-by:
Noah Moss <noahbmoss@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
- Mar 28, 2023
-
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Roman Abdulmanov <roman@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Luiz Arakaki <luiz.arakaki@metabase.com>
-
Nemanja Glumac authored
* Updating to cypress 12 * updating deps * Updating Tests. Be Green Plz * removing custom test id from select button * Select Button Tuning Co-authored-by:
Nick Fitzpatrick <nick@metabase.com>
-
metabase-bot[bot] authored
Add a reproduction for dashboard sticky filter incorrectly positioned when moving between dashboards (#29497) (#29596) * Add a repro for #26230 * Move the prep phase to the `beforeEach` * Use `visitDashboard` helper * Wait for the second dashboard to load * You don't have to use `within` here * Add a comment * Add main repro assertion * Greatly simplify creation of the markdown card * Update the test title --------- Co-authored-by:
Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev> Co-authored-by:
Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
-
metabase-bot[bot] authored
Co-authored-by:
Luiz Arakaki <luiz.arakaki@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Ryan Laurie <30528226+iethree@users.noreply.github.com>
-
- Mar 27, 2023
-
-
Denis Berezin authored
-
metabase-bot[bot] authored
* Configure foreign key support for Mongo to allow implicit joins Co-authored-by:
metamben <103100869+metamben@users.noreply.github.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
Jeff Bruemmer authored
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
* Revert "Fix zoom in drilldown for FKs (#28022)" (#29551) --------- Co-authored-by:
Alexander Polyankin <alexander.polyankin@metabase.com> Co-authored-by:
Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
-
- Mar 25, 2023
-
-
metabase-bot[bot] authored
* keep standalone values when collapsing column * adjusting unit test * PR Feedback * adding space * removing only * Adjusting behavior and tests * adjusting data_grid unit tests Co-authored-by:
Nick Fitzpatrick <nick@metabase.com>
-
- Mar 24, 2023
-
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
Aleksandr Lesnenko authored
-
metabase-bot[bot] authored
Co-authored-by:
Mark Bastian <markbastian@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Denis Berezin <denis.berezin@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
* Separate date from datetime binning options The BE sends over binning indexes for each field and a top level map for how to interpret these keys. ```javascript ❯ http get http://localhost:3000/api/table/10658/query_metadata Cookie:$SESSION -pb { "active": true, "caveats": null, "created_at": "2023-03-21T22:20:42.363169Z", "db": { "engine": "postgres", "features": [ "full-join", ... ], "id": 499, ... }, "db_id": 499, "dimension_options": { "0": { "mbql": [ "field", null, { "temporal-unit": "day" } ], "name": "Day", "type": "type/Date"}, "1": { "mbql": [ "field", null, { "temporal-unit": "week" } ], "name": "Week", "type": "type/Date" }, "10": { "mbql": ["field", null, { "temporal-unit": "quarter-of-year"}], "name": "Quarter of Year", "type": "type/Date" }, "11": { "mbql": [ "field", null, { "temporal-unit": "minute" } ], "name": "Minute", "type": "type/DateTime"}, "12": { "mbql": [ "field", null, { "temporal-unit": "hour"}], "name": "Hour", "type": "type/DateTime"}, ... // more options. They have a type, an id, and an mbql clause }, "display_name": "Different Times", "entity_type": null, "fields": [{"name": "id",}, {"name": "dt", "dimension_options": [ "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25"] "database_type": "timestamp", "effective_type": "type/DateTime", }, {"name": "t", "dimension_options": [ "26", "27", "28" ], "effective_type": "type/Time", "database_type": "time" }, {"name": "d", "database_type": "date", "dimension_options": [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10" ], "effective_type": "type/Date", } ], "name": "different_times", ... } ``` This PR adds some new types just for dates so that you don't get offered by group a date by hour or minute. This adds new indexes to this list, which makes us wary that these indexes have leaked out and cannot be changed. But that is not the case. This is essentially compression on the response. When the FE saves a query, it does not include the index but uses the mbql query to insert the proper temporal-unit: ```clojure ;; a "date" field choosing the quarter option {:type :query, :query {:source-table 5, :breakout [[:field 49 {:temporal-unit :quarter}]], :aggregation [[:count]]}, :database 1} ;; the same query but binning by week {:database 1, :query {:source-table 5, :breakout [[:field 49 {:temporal-unit :week}]], :aggregation [[:count]]}, :type :query} ``` So the index is relative to just the editing session of the query, and the indexes do not have to be stable when we add new ones. * athena and mongo (others?) don't have a date type the test expects a date field to have date binning options. previous, we made no distinction for binning options for date vs datetime fields. Now we do and have to manage expectations in the tests. * mongo returns type/Instant Co-authored-by:
dpsutton <dan@dpsutton.com>
-
- Mar 23, 2023
-
-
metabase-bot[bot] authored
* add permissions and test * address comments * fix whitespace * remove ehitespace * revert vscode changes Co-authored-by:
Jerry Huang <34140255+qwef@users.noreply.github.com>
-
metabase-bot[bot] authored
Co-authored-by:
Alexander Polyankin <alexander.polyankin@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Nick Fitzpatrick <nick@metabase.com>
-
metabase-bot[bot] authored
* Improvement of recent_views endpoint query (#29259) * WIP improvement of recent_views endpoint query * Adjust query to actually include models (cards with :dataset true) This was not part of the recent_views before, so I think it's actually an improvement to the feature as well as improvement to the characteristics of the query itself. * Linter fixes * One more take on recent_views query. Last 30 days, all req's in 1 query This query combines view_log and query_execution Keep query_executions when they are card with context 'question' Keep any card views when they are 'datasets' (aka models) Remove duplicates with GROUP BY model, model_id Sort by timestamp desceneding Remove any views that are bookmarked by using NOT IN * An improved query that handles all logic in one db query * Adjust limit in case archived cards/dashboards are in the list * Make fn private for linter * Add one additional constraint to prevent pinned items from counting as viewed * Address review feedback * Accidentally slipped this in... not needed on this PR * Add toucan2 require --------- Co-authored-by:
adam-james <21064735+adam-james-v@users.noreply.github.com> Co-authored-by:
Adam James <adam.vermeer2@gmail.com>
-
Anton Kulyk authored
* Break down `ActionForm` component (#29290) * Move `ActionFormFieldWidget` to action components * Move `inputTypeHasOptions` to shared utils * Move `FormFieldEditor` to `FormCreator` * Clean up `ActionForm` props * Clean up `FormCreator` props * Move most of `ActionForm` utils to shared utils * Move `reorderFields` to `FormCreator` * Extract `FormCreator` resps from `ActionForm` * Use default export for `ActionForm` * Consolidate action form logic (#29326) * Make `ActionForm` accept an `action` prop * Make field settings inside `ActionForm` if missing * Clean date-time values inside `ActionForm` * WIP * Clean and filter values on submit in `ActionForm` * Sort `ActionVizForm` props * Sort `Action` props * Use more generic `WritebackAction` type * Move more logic into the `ActionForm` * Add `useActionForm` hook * Fix parameter type in tests * Fix submitting a form when all parameters mapped * Move utils to `useActionForm` * Fix submit button label in public action e2e tests * Filter changed values only for implicit update actions * Rename `cleanInitialValues` to `cleanedInitialValues` * Fix `shouldShowConfirmation` can be a string * Nice boring code * Fix types in tests * Small rename * Move `useActionForm` utils closer to source (#29328) * Fix `ActionForm` ignored submit errors * Fix default values display in the form editor * Fix names
-