This project is mirrored from https://github.com/metabase/metabase.
Pull mirroring updated .
- Apr 03, 2023
-
-
dpsutton authored
* Create `cache_info` table in our persisted schemas ```sql -- postgres/redshift test-data=# select * from metabase_cache_424a9_379.cache_info ; key | value ------------------+-------------------------------------- settings-version | 1 created-at | 2023-03-28 instance-uuid | 407e4ba8-2bab-470f-aeb5-9fc63fd18c4e instance-name | Metabase Test (4 rows) ``` ```sql --mysql mysql> select * from metabase_cache_424a9_435.cache_info; +------------------+--------------------------------------+ | key | value | +------------------+--------------------------------------+ | settings-version | 1 | | created-at | 2023-03-28 | | instance-uuid | 407e4ba8-2bab-470f-aeb5-9fc63fd18c4e | | instance-name | Metabase Test | +------------------+--------------------------------------+ ``` our key values in v1: ```clojure (defn kv-table-values "Version 1 of the values to go in the key/value table `cache_info` table." [] [{:key "settings-version" :value "1"} {:key "created-at" ;; "2023-03-28" :value (.format (LocalDate/now) DateTimeFormatter/ISO_LOCAL_DATE)} {:key "instance-uuid" :value (public-settings/site-uuid)} {:key "instance-name" :value (public-settings/site-name)}]) ``` This will enable us to delete cached schemas in shared databases (cloud versions like redshift) without risking wiping out a cached schema during an active test run. The code to delete the schemas will be something like, ```clojure (def spec (sql-jdbc.conn/connection-details->spec :redshift {:host "xxx.us-east-1.reshift.amazonaws.com" :db "db" :port 5439 :user "user" :password "password"})) (let [days-prior 1 ;; probably 2 to handle crossing day time. Can also adjust to 6 hours, etc threshold (.minus (java.time.LocalDate/now) days-prior java.time.temporal.ChronoUnit/DAYS)] (doseq [schema (->> ["select nspname from pg_namespace where nspname like 'metabase_cache%'"] (jdbc/query spec) (map :nspname))] (let [[{created :value}] (jdbc/query spec (sql/format {:select [:value] :from [(keyword schema "cache_info")] :where [:= :key "created-at"]} {:dialect :ansi}))] (when created (let [creation-day (java.time.LocalDate/parse created)] (when (.isBefore creation-day threshold) (jdbc/execute! spec [(format "drop schema %s cascade" schema)]))))))) ``` or if we want to do it in a single query: ```clojure schemas=> (let [days-prior 1 threshold (.minus (java.time.LocalDate/now) days-prior java.time.temporal.ChronoUnit/DAYS)] (let [schemas (->> ["select nspname from pg_namespace where nspname like 'metabase_cache%'"] (jdbc/query spec) (map :nspname))] (jdbc/with-db-connection [conn spec] (jdbc/execute! conn [(format "set search_path= %s;" (str/join ", " schemas))]) (let [sql (sql/format {:select [:schema :created-at] :from {:union-all (for [schema schemas] {:select [[[:inline schema] :schema] [{:select [:value] :from [(keyword schema "cache_info")] :where [:= :key [:inline "created-at"]]} :created-at]]})}} {:dialect :ansi}) old? (fn [{create-str :created-at}] (let [created (java.time.LocalDate/parse create-str)] (.isBefore created threshold))) schemas-to-delete (->> (jdbc/query spec sql) (filter old?) (map :schema))] schemas-to-delete)))) ("metabase_cache_424a9_503") ;; when there are any, string join them and delete them as above ``` * Use Instants so we can compare by hours instead of just days * require toucan as db
-
Anton Kostenko authored
-
metabase-bot[bot] authored
Co-authored-by:
Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
-
Noah Moss authored
-
Denis Berezin authored
-
metabase-bot[bot] authored
Co-authored-by:
Nemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
-
- Mar 31, 2023
-
-
metabase-bot[bot] authored
* Don't log in bigquery results hotpath Right now we log in the parser of bigquery results for unrecognized types. But the problem is that we log for each _value_ and not each column. This is causing an _enormous_ amount of logs and performance penalty. See - https://github.com/metabase/metabase/issues/29118 (performance) - https://github.com/metabase/metabase/issues/28868 (filling disk space) This log was added between 45.1 and 45.3 ```diff ❯ git diff v0.45.1..v0.45.3 modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/** diff --git a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj index a0d8081c30..f367199b55 100644 --- a/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj +++ b/modules/drivers/bigquery-cloud-sdk/src/metabase/driver/bigquery_cloud_sdk/query_processor.clj @@ -22,7 +22,7 @@ [metabase.util :as u] [metabase.util.date-2 :as u.date] [metabase.util.honeysql-extensions :as hx] - [metabase.util.i18n :refer [tru]] + [metabase.util.i18n :refer [trs tru]] [pretty.core :refer [PrettyPrintable]] [schema.core :as s]) (:import [com.google.cloud.bigquery Field$Mode FieldValue] @@ -88,7 +88,8 @@ (parse-fn v))) (defmethod parse-result-of-type :default - [_ column-mode _ v] + [column-type column-mode _ v] + (log/warn (trs "Warning: missing type mapping for parsing BigQuery results of type {0}." column-type)) (parse-value column-mode v identity)) ``` The result is that selecting 50,000 rows for download in excel: | version | time | |------------------------------|------------| | 0.45.1 | 28 seconds | | 0.45.3 | 52 seconds | | 0.45.3 with logging disabled | 30 seconds | (disable logging by adding `<Logger name="metabase.driver.bigquery-cloud-sdk.query-processor" level="ERROR"/>` and `-Dlog4j2.configurationFile=log4j2.xml` to jar startup) For the query (3 string columns, 5 rows): ```sql SELECT game_id, first_name, last_name FROM `bigquery-public-data.ncaa_basketball.mbb_players_games_sr` LIMIT 5 ``` BEFORE: ``` ``` 2023-03-31 17:17:52,146 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,147 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,149 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,150 WARN bigquery-cloud-sdk.query-processor :: Warning: missing type mapping for parsing BigQuery results of type STRING. 2023-03-31 17:17:52,155 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 795.2 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (2 idle, 0 queued) (192 total active threads) Queries in flight: 0 (0 queued) ``` Note this is 15 logs (3 columns x 5 rows) AFTER: ``` 2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column game_id of type STRING. 2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column first_name of type STRING. 2023-03-31 17:19:15,694 WARN driver.bigquery-cloud-sdk :: Warning: missing type mapping for parsing BigQuery results column last_name of type STRING. 2023-03-31 17:19:15,757 DEBUG middleware.log :: POST /api/dataset 202 [ASYNC: completed] 973.5 ms (6 DB calls) App DB connections: 0/10 Jetty threads: 4/50 (3 idle, 0 queued) (193 total active threads) Queries in flight: 0 (0 queued) ``` * unused require to appease our overlords Co-authored-by:
dpsutton <dan@dpsutton.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
- Mar 30, 2023
-
-
metabase-bot[bot] authored
From https://github.com/metabase/metabase/issues/29545 Co-authored-by:
Luis Paolini <paoliniluis@gmail.com>
-
- Mar 29, 2023
-
-
metabase-bot[bot] authored
* fix typo * fix links --------- Co-authored-by:
Noah Moss <noahbmoss@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
- Mar 28, 2023
-
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Roman Abdulmanov <roman@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Luiz Arakaki <luiz.arakaki@metabase.com>
-
Nemanja Glumac authored
* Updating to cypress 12 * updating deps * Updating Tests. Be Green Plz * removing custom test id from select button * Select Button Tuning Co-authored-by:
Nick Fitzpatrick <nick@metabase.com>
-
metabase-bot[bot] authored
Add a reproduction for dashboard sticky filter incorrectly positioned when moving between dashboards (#29497) (#29596) * Add a repro for #26230 * Move the prep phase to the `beforeEach` * Use `visitDashboard` helper * Wait for the second dashboard to load * You don't have to use `within` here * Add a comment * Add main repro assertion * Greatly simplify creation of the markdown card * Update the test title --------- Co-authored-by:
Mahatthana (Kelvin) Nomsawadi <me@bboykelvin.dev> Co-authored-by:
Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
-
metabase-bot[bot] authored
Co-authored-by:
Luiz Arakaki <luiz.arakaki@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Ryan Laurie <30528226+iethree@users.noreply.github.com>
-
- Mar 27, 2023
-
-
Denis Berezin authored
-
metabase-bot[bot] authored
* Configure foreign key support for Mongo to allow implicit joins Co-authored-by:
metamben <103100869+metamben@users.noreply.github.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
Jeff Bruemmer authored
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
* Revert "Fix zoom in drilldown for FKs (#28022)" (#29551) --------- Co-authored-by:
Alexander Polyankin <alexander.polyankin@metabase.com> Co-authored-by:
Nemanja <31325167+nemanjaglumac@users.noreply.github.com>
-
- Mar 25, 2023
-
-
metabase-bot[bot] authored
* keep standalone values when collapsing column * adjusting unit test * PR Feedback * adding space * removing only * Adjusting behavior and tests * adjusting data_grid unit tests Co-authored-by:
Nick Fitzpatrick <nick@metabase.com>
-
- Mar 24, 2023
-
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
Aleksandr Lesnenko authored
-
metabase-bot[bot] authored
Co-authored-by:
Mark Bastian <markbastian@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Denis Berezin <denis.berezin@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Jeff Bruemmer <jeff.bruemmer@gmail.com>
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-
metabase-bot[bot] authored
* Separate date from datetime binning options The BE sends over binning indexes for each field and a top level map for how to interpret these keys. ```javascript ❯ http get http://localhost:3000/api/table/10658/query_metadata Cookie:$SESSION -pb { "active": true, "caveats": null, "created_at": "2023-03-21T22:20:42.363169Z", "db": { "engine": "postgres", "features": [ "full-join", ... ], "id": 499, ... }, "db_id": 499, "dimension_options": { "0": { "mbql": [ "field", null, { "temporal-unit": "day" } ], "name": "Day", "type": "type/Date"}, "1": { "mbql": [ "field", null, { "temporal-unit": "week" } ], "name": "Week", "type": "type/Date" }, "10": { "mbql": ["field", null, { "temporal-unit": "quarter-of-year"}], "name": "Quarter of Year", "type": "type/Date" }, "11": { "mbql": [ "field", null, { "temporal-unit": "minute" } ], "name": "Minute", "type": "type/DateTime"}, "12": { "mbql": [ "field", null, { "temporal-unit": "hour"}], "name": "Hour", "type": "type/DateTime"}, ... // more options. They have a type, an id, and an mbql clause }, "display_name": "Different Times", "entity_type": null, "fields": [{"name": "id",}, {"name": "dt", "dimension_options": [ "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25"] "database_type": "timestamp", "effective_type": "type/DateTime", }, {"name": "t", "dimension_options": [ "26", "27", "28" ], "effective_type": "type/Time", "database_type": "time" }, {"name": "d", "database_type": "date", "dimension_options": [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10" ], "effective_type": "type/Date", } ], "name": "different_times", ... } ``` This PR adds some new types just for dates so that you don't get offered by group a date by hour or minute. This adds new indexes to this list, which makes us wary that these indexes have leaked out and cannot be changed. But that is not the case. This is essentially compression on the response. When the FE saves a query, it does not include the index but uses the mbql query to insert the proper temporal-unit: ```clojure ;; a "date" field choosing the quarter option {:type :query, :query {:source-table 5, :breakout [[:field 49 {:temporal-unit :quarter}]], :aggregation [[:count]]}, :database 1} ;; the same query but binning by week {:database 1, :query {:source-table 5, :breakout [[:field 49 {:temporal-unit :week}]], :aggregation [[:count]]}, :type :query} ``` So the index is relative to just the editing session of the query, and the indexes do not have to be stable when we add new ones. * athena and mongo (others?) don't have a date type the test expects a date field to have date binning options. previous, we made no distinction for binning options for date vs datetime fields. Now we do and have to manage expectations in the tests. * mongo returns type/Instant Co-authored-by:
dpsutton <dan@dpsutton.com>
-
- Mar 23, 2023
-
-
metabase-bot[bot] authored
* add permissions and test * address comments * fix whitespace * remove ehitespace * revert vscode changes Co-authored-by:
Jerry Huang <34140255+qwef@users.noreply.github.com>
-
metabase-bot[bot] authored
Co-authored-by:
Alexander Polyankin <alexander.polyankin@metabase.com>
-
metabase-bot[bot] authored
Co-authored-by:
Natalie <nat@metabase.com>
-