Skip to content
Snippets Groups Projects
user avatar
Mark Bastian authored
* Renamed columns are displayed correctly in downloads (#18572)

When custom viz settings or metadata is applied to questions and models, the downloaded artifact labels do not always mirror what is displayed on the tabular view in Metabase. This happens for several reasons:

- The `:visualization_settings` were not applied to CSV and JSON downloads
- `update-card-viz-settings` in `metabase.query-processor.middleware.visualization-settings` did not correctly merge keys, resulting in dropped column names in Excel file
- The logic for reconciling column metadata in a result set and the keys in `:visualization_settings` of a query result is not straightforward

This PR is an attempt to move this consistency in the right direction without fixing all the challenges of field matching as discussed in [this thread](https://metaboat.slack.com/archives/CKZEMT1MJ/p1703091539541279) and [this issue](Duplicated columns in a source query cannot be distinguished #36185) as this sounds like a significant effort.

The following changes were made:

1. Update `metabase.query-processor.middleware.visualization-settings/update-card-viz-settings` to merge field settings, if present, into `column-viz-settings` without introducing new keys, especially ambiguous keys as follows:

```clojure
(defn- update-card-viz-settings
  "For each field, fetch its settings from the QP store, convert the settings into the normalized form
  for visualization settings, and then merge in the card-level column settings."
  [column-viz-settings field-ids]
  ;; Retrieve field-level settings
  (let [field-id->settings (reduce
                             (fn [m field-id]
                               (let [field-settings      (:settings (lib.metadata/field (qp.store/metadata-provider) field-id))
                                     norm-field-settings (normalize-field-settings field-id field-settings)]
                                 (assoc m field-id norm-field-settings)))
                             {}
                             field-ids)]
    ;; For each column viz setting, if there is a match on the field settings, merge it in,
    ;; with the column viz settings being the default in the event of conflicts.
    (reduce-kv
      (fn [coll {field-id ::mb.viz/field-id :as k} column-viz-setting]
        (assoc coll k (merge (get field-id->settings field-id {}) column-viz-setting)))
      {}
      column-viz-settings)))
```

The primary difference is rather than merging in new keys as `{::mb.viz/field-id field-id} -> field-settings` this code will merge settings found by field id with any `column-viz-setting` containing that field id as part of its key. The merge prefers `column-viz-settings`.

This should be an improvement over what already exists.

2. Move `column-titles` from `metabase.query-processor.streaming.xlsx` to `metabase.query-processor.streaming.common` and use this common function for CSV and JSON export names.

Note that the lifted `column-titles` has some erroneous logic:

- It relies on column `:name` or `:id` as unique identifiers -- they are not always unique.
- It expects the `col-settings` key format to be exactly `{::mb.viz/field-id id}` or `{::mb.viz/column-name name}` -- Sometimes these keys have additional keys. One change made to this function is normalizing `col-settings` to conform to the expected format by removing extra keys. While this isn't a perfect solution, it conforms with the intent of what already exists and is an improvement in the majority of cases.

The _right_ thing to do is _probably_ to use `metabase.lib.equality/find-matching-column` to match the column and settings keys, _but_ the `col-settings` keys are in a weird format that isn't really conducive to doing this. I'd rather we make an incremental step forward and try doing this in another effort.

This second change should not make anything that wasn't already problematic worse, while making the general case better.

* Updated `update-card-viz-settings` for unique fields case

When `update-card-viz-settings` is called, sometimes fields have ids that aren't part of the `column-viz-settings` map. For cases where they are part of the viz settings map, they are merged in, potentially more than once. For cases where the field ids aren't found in the column viz settings but do have settings, we add these in as new entries.

* Test for JSON keys

* Fixed broken unit test that had the wrong data shape.
f8b533c9
History

Metabase

Metabase is the easy, open-source way for everyone in your company to ask questions and learn from data.

Metabase Product Screenshot

Latest Release codecov Docker Pulls

Features

Take a tour of Metabase.

Supported databases

Installation

Metabase can be run just about anywhere. Check out our Installation Guides.

Contributing

Quick Setup: Dev environment

In order to spin up a development environment, you need to start the front end and the backend as follows:

Frontend quick setup

The following command will install the Javascript dependencies:

$ yarn install

To build and run without watching changes:

$ yarn build

To build and run with hot-reload:

$ yarn build-hot 

Backend quick setup

In order to run the backend, you'll need to build the drivers first, and then start the backend:

$ ./bin/build-drivers.sh
$ clojure -M:run

For a more detailed setup of a dev environment for Metabase, check out our Developers Guide.

Internationalization

We want Metabase to be available in as many languages as possible. See which translations are available and help contribute to internationalization using our project over at POEditor. You can also check out our policies on translations.

Extending Metabase

Hit our Query API from Javascript to integrate analytics. Metabase enables your application to:

  • Build moderation interfaces.
  • Export subsets of your users to third party marketing automation software.
  • Provide a custom customer lookup application for the people in your company.

Check out our guide, Working with the Metabase API.

Security Disclosure

See SECURITY.md for details.

License

This repository contains the source code for both the Open Source edition of Metabase, released under the AGPL, as well as the commercial editions of Metabase, which are released under the Metabase Commercial Software License.

See LICENSE.txt for details.

Unless otherwise noted, all files © 2023 Metabase, Inc.

Metabase Experts

If you’d like more technical resources to set up your data stack with Metabase, connect with a Metabase Expert.