Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Sep 21, 2022
  2. Sep 20, 2022
  3. Sep 19, 2022
    • dpsutton's avatar
      Don't use persisted model tables for segmented users (#25347) · 32403f0d
      dpsutton authored
      * Don't use persisted model tables for segmented users
      
      This actually isn't a bug, but due to very subtle and arbitrary reasons.
      
      For background about why we need to ensure this never happens, we cannot
      use persisted models when sandboxing is at play. A simple example is as
      follows: make a model on a products table that does not select the
      category. Have a sandbox on category such that someone can only see
      products of category "Gizmo". the model lacks the category column but we
      insert a where clause that still works. When the model is persisted,
      there is no category column in the underlying table so sandboxing cannot
      possibly work: the data necessary to filter is no longer associated with
      the rest of the data in the model.
      
      The fix for this is quite simple: in
      `metabase.query-processor.middleware.fetch-source-query` we only splice
      in the persisted query if the user is not a segmented user (product name
      for sandboxing).
      
      ```clojure
      (and persisted-info/*allow-persisted-substitution*
           (not (segmented-user?))  ;; <----- new check
           (:active card)
           (:definition card)
           (:query_hash card)
           (= (:query_hash card) (persisted-info/query-hash (:dataset_query card)))
           (= (:definition card) (persisted-info/metadata->definition (:result_metadata card)
                                                                      (:table_name card)))
           (= (:state card) "persisted"))
      ```
      
      Technical details about why this bug did not manifest
      
      When swapping out a card__<id> to a source query, if its a model we will
      see if it is persisted, and if so, we will use the native sql to select
      from the persisted table. It does this by adding the native sql at a key
      called `:persisted-info/native` and a middleware
      `#'qp.persistence/substitute-persisted-query` walks the query replacing
      the query with the native:
      
      ```clojure
      ;; metabase.query-processor.middleware.persistence
      
          (mbql.u/replace query
            (x :guard (every-pred map? :persisted-info/native))
            {:native (:persisted-info/native x)})
      ```
      
      There is also a middleware that walks through the query looking for
      tables with gtaps on them and replacing them. By change, the sandboxing
      middleware runs immediately before the substitute-persisted middleware!
      
      ```clojure
         ;; literally the previous middleware
         (resolve 'ee.sandbox.rows/apply-sandboxing)
         #'qp.persistence/substitute-persisted-query
      ```
      
      If you swap the order of these two sandboxing is broken. As is, it
      "works" but not by design, just by happenstance. The sandboxing
      middleware just did not know that the `:persisted-info/native` key meant
      that a native query was to be substituted. In the reverse order, the
      native query is already substituted and there is no change for the
      sandboxing to occur.
      
      The obvious fix is to ensure that we never even attempt to use the
      persisted tables and that is what this PR does.
      
      * Eastwood doesn't like shadowing like this
      
      * Rearrange check order for tests
      
      `segmented-user?` throws if there is no bound user. A test in
      `fetch-source-query-test` was failing because there was no user bound,
      but it wasn't attempting to swap out a persisted table, it just didn't
      expect to need a user.
      
      Moving it lower lets it short circuit on other bits that are bound to
      fail (definition, query_hash, etc) requiring persistence before we check
      for a bound user
      Unverified
      32403f0d
    • Noah Moss's avatar
      LDAP setup improvements (#25421) · b18d53e8
      Noah Moss authored
      
      * first pass at LDAP setup improvements
      
      * fix lint errors
      
      * fix LDAP api tests
      
      * WIP test for new setting setter
      
      * fix setting test
      
      * set ldap-enabled to true in ldap server macro
      
      * try to fix java11 tests
      
      * Update src/metabase/integrations/ldap.clj
      
      Co-authored-by: default avatarmetamben <103100869+metamben@users.noreply.github.com>
      
      * add transaction
      
      * remove ldap-ever-enabled? setting and revert some of the logic that is no longer necessary
      
      * set ldap-enabled via the ldap api and add tests
      
      * fix tests and lint
      
      * fix error on settings save
      
      * fix cypress test
      
      * actually fix cypress
      
      Co-authored-by: default avatarmetamben <103100869+metamben@users.noreply.github.com>
      Unverified
      b18d53e8
    • Alexander Polyankin's avatar
    • dpsutton's avatar
      initial prometheus sketch (#25149) · 00530c98
      dpsutton authored
      * initial prometheus sketch
      
      need env variable to start: MB_PROMETHEUS_SERVER_PORT=9191
      
      sample of output:
      
      ```
      '# HELP jvm_threads_current Current thread count of a JVM
      '# TYPE jvm_threads_current gauge
      jvm_threads_current 81.0
      '# HELP jvm_threads_daemon Daemon thread count of a JVM
      '# TYPE jvm_threads_daemon gauge
      jvm_threads_daemon 36.0
      '# HELP jvm_threads_peak Peak thread count of a JVM
      '# TYPE jvm_threads_peak gauge
      jvm_threads_peak 81.0
      '# HELP jvm_threads_started_total Started thread count of a JVM
      '# TYPE jvm_threads_started_total counter
      jvm_threads_started_total 104.0
      '# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
      '# TYPE jvm_threads_deadlocked gauge
      jvm_threads_deadlocked 0.0
      ```
      
      request:
      
      ```
      ❯ http localhost:9191/metrics
      HTTP/1.1 200 OK
      Content-Length: 7329
      Content-Type: text/plain; version=0.0.4; charset=utf-8
      Date: Wed, 31 Aug 2022 16:13:38 GMT
      Server: Jetty(9.4.48.v20220622)
      
      '# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
      '# TYPE jvm_gc_collection_seconds summary
      jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 41.0
      jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 0.586
      jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
      jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
      '# HELP jvm_threads_current Current thread count of a JVM
      '# TYPE jvm_threads_current gauge
      ```
      
      * Log on unparseable prometheus port
      
      * Clean up prometheus and save test
      
      * typehint
      
      * Jetty collector
      
      * Reset system to nil when shutting down
      
      * c3p0 stats
      
      * Clean up, document, and add tests
      
      * Error message for failure to bind to port
      
      Starting up with prometheus port set to the main webserver port to get
      the error:
      
      ```
      MB_JETTY_PORT=3006 MB_DB_CONNECTION_URI="postgres://..." \
      MB_PROMETHEUS_SERVER_PORT=3006 java -jar locally-built.jar
      ```
      yields the following error:
      
      ```shell
      2022-09-09 10:08:52,000 INFO metabase.core :: Setting up prometheus metrics
      2022-09-09 10:08:52,002 INFO metabase.prometheus :: Starting prometheus metrics collector
      2022-09-09 10:08:52,016 INFO metabase.prometheus :: Starting prometheus metrics web-server on port 3,006
      2022-09-09 10:08:52,036 ERROR metabase.core :: Metabase Initialization FAILED
      clojure.lang.ExceptionInfo: Failed to initialized Prometheus on port 3,006 {:port 3006}
      [stacktrace ...]
      Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:3006
      [stacktrace ...]
      Caused by: java.net.BindException: Address already in use
      ```
      
      * Test for error message
      
      * Move prometheus to analytics folder
      
      * Str port so log does not add commas to number
      
      eg:
      > Starting prometheus metrics web-server on port 9,191
      
      * make some test functions private
      
      * docstring on defsetting
      
      * align lets
      
      * ns docstring changes
      
      include the env variable for ease of understanding
      
      * sort ns
      
      * remove some reflection warnings
      
      * reorder FE lines due to some new linter
      
      * Cleanup: i18n descriptions, typo, List/of
      
      * bit more concise c3p0 collection
      
      * no longer need the helper `->array`
      
      * clean up doseq for c3p0 measurements
      Unverified
      00530c98
    • Mahatthana (Kelvin) Nomsawadi's avatar
      Make table column width reasonable (#25436) · d6bb49fd
      Mahatthana (Kelvin) Nomsawadi authored
      * Make table column width reasonable
      
      Truncate column name that is longer than 16 characters
      
      * Add a test to check header truncation of tables
      
      Also adjusted the shape of the header fn's return data a bit -> the `[:th ... ]` vectors aren't wrapped in a seq anymore.
      
      * Fix wrong max column character length :face_palm:
      
      
      
      * Update static viz table style
      
      * Address PR comment, simplifying the code
      
      Co-authored-by: default avatarAdam James <adam.vermeer2@gmail.com>
      Unverified
      d6bb49fd
  4. Sep 16, 2022
  5. Sep 15, 2022
Loading