Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Dec 16, 2022
    • metamben's avatar
      Report DBMS versions as a JSON string (#27267) · 2eb0f03a
      metamben authored
      Without this conversion the data ends up as an EDN string in the
      usage_stats table and that's not easily accessible in PostgreSQL.
      Unverified
      2eb0f03a
    • Cal Herries's avatar
      Refactor datetime-diff implementations: sqlserver, mysql, bigquery, redshift,... · 615c5a4e
      Cal Herries authored
      Refactor datetime-diff implementations: sqlserver, mysql, bigquery, redshift, snowflake, vertica, postgres (#27262)
      
      * Refactor bigquery datetime-diff to use multimethod
      
      * Refactor snowflake datetime-diff to use multimethod
      
      * Refactor SQL server datetime-diff to use multimethod
      
      * Refactor mysql datetime-diff to use multimethod
      
      * Refactor redshift datetime-diff to use multimethod
      
      * Simplify vertica datetime-diff
      
      * Tidy postgres datetime-diff
      
      * ; -> ;; in snowflake comment
      
      * Add to bigquery comment
      Unverified
      615c5a4e
  2. Dec 15, 2022
  3. Dec 14, 2022
    • Natalie's avatar
    • Jeff Bruemmer's avatar
    • Noah Moss's avatar
      BE endpoint to clear group memberships (#26555) · 7828c1e6
      Noah Moss authored
      * BE endpoint to clear group membership
      
      * update ee group manager tests with clear-memberships
      Unverified
      7828c1e6
    • Mahatthana (Kelvin) Nomsawadi's avatar
      Fix can't change locked parameter names in full-app embedding (#27179) · 51fc3624
      Mahatthana (Kelvin) Nomsawadi authored
      * Fix can't change locked parameter names in full-app embedding
      
      * Remove unused param
      
      * Add tests
      Unverified
      51fc3624
    • Noah Moss's avatar
      Hide auth config pages from setting managers (#27187) · 90320851
      Noah Moss authored
      * make auth settings adminOnly on FE
      
      * WIP backend changes
      
      * Update relevant settings
      
      * address lint warning
      
      * new tests, and a bunch of reorganization
      
      * remove broken EE settings test, and fix validation in google & ldap endpoints
      
      * fix cypress tests
      
      * add clarifying table to setting docs
      Unverified
      90320851
    • Nemanja Glumac's avatar
    • Gustavo Saiani's avatar
    • Nick Fitzpatrick's avatar
      Transition InputBlurChange to use core Input component (#27122) · 31999007
      Nick Fitzpatrick authored
      * Convert to TypeScript, handle ColumnItem and MetadataTable
      
      * more progress
      
      * handled numeric inputs
      
      * Transition InputBlurChange to core Input component
      
      * Removing input class from legacy date picker just incase
      
      * Small style adjustments in Admin and Viz Settings
      
      * working on tests
      
      * Tests passing
      
      * Adjusting styling
      
      * PR Cleanup
      
      * add input size=large (#27217)
      
      * Suppress console log when using `with-log-messages-for-level` (#26468)
      
      * disable additivity when using "with-log-level"
      
      * only set when parent is root
      
      * wording
      
      * remove a test that is no longer needed
      
      * fix indents
      
      * one missing indent fix
      
      * [CI] Add `concurrency` to the `drivers` workflow (#27223)
      
      * Fix datetime-diff helper text (#27224)
      
      * Improve datetimeDiff type error message (#27225)
      
      * Update test
      
      * Update implementations
      
      * Relieve db pressure on api/health check (#27192)
      
      * Relieve db pressure on api/health check
      
      https://github.com/metabase/metabase/issues/26266
      
      Servers under heavy load can be slow to respond to the api/health
      check. This can lead to k8s killing healthy instances happily humming
      along serving requests.
      
      One idea floated was to use QoSFilters
      https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/servlets/QoSFilter.html
      to prioritize those requests in front of others. But I suspect this
      might not be our bottleneck.
      
      Our health endpoint was updated to see if it could acquire an endpoint
      when we were dealing with connection pool issues. We were reporting the
      instance was healthy once it has finished the init process, but would
      report healthy if 60/15 app-db connections were used and no actual
      queries could complete.
      
      The remedy was adding
      `(sql-jdbc.conn/can-connect-with-spec? {:datasource (mdb.connection/data-source)})`
      to the endpoint. But now to get information about the health of the
      system we have to wait in the queue to get a datasource.
      
      The hope is that this change which monitors for recent db
      checkins (query success) and checkouts (query begun) can be a proxy for
      db activity without having to wait for a connection and hit the db ourselves.
      
      Some simple and crude benchmarking:
      - use `siege` to hit `api/database/<app-db>/sync_schema`
      - in a separate tab, use `siege` to hit `api/health`
      
      Three trials with unconditional db access and conditional db
      access (look for recent activity set by the new `ConnectionCustomizer`).
      
      One siege client is synching the app-db's schema with 80 clients each
      sending 60 requests. the other has 1 client sending 60 requests to api/health.
      
      Run             |  Elapsed Time | max tx  | tx rate
       before change  |    7.16s      |  0.79s  |  8.38 tx/s
       before change  |   23.91s      |  1.44s  |  2.51 tx/s
       before change  |   13.00s      |  0.50s  |  4.62 tx/s
      ----------------------------------------------------
       after change   |    4.46s      |  0.27s  |  13.45 tx/s
       after change   |    5.81s      |  0.61s  |  10.33 tx/s
       after change   |    4.54s      |  0.44s  |  13.22 tx/s
      
      Full(er) results below:
      
      ```
      Unconditional db access
      =======================
      
      siege -c80 -r 40 "http://localhost:3000/api/database/2/sync_schema POST" -H "Cookie: $SESSION"
      
      siege -c 1 -r 60 "http://localhost:3000/api/health
      
      "
      
      Elapsed time:		        7.16 secs
      Response time:		        0.12 secs
      Transaction rate:	        8.38 trans/sec
      Longest transaction:	        0.79
      Shortest transaction:	        0.01
      
      Elapsed time:		       23.91 secs
      Response time:		        0.40 secs
      Transaction rate:	        2.51 trans/sec
      Longest transaction:	        1.44
      Shortest transaction:	        0.02
      
      Elapsed time:		       13.00 secs
      Response time:		        0.22 secs
      Transaction rate:	        4.62 trans/sec
      Longest transaction:	        0.50
      Shortest transaction:	        0.06
      
      Conditional db access
      ==============================================================
      
      Elapsed time:		        4.46 secs
      Response time:		        0.07 secs
      Transaction rate:	       13.45 trans/sec
      Longest transaction:	        0.27
      Shortest transaction:	        0.01
      
      Elapsed time:		        5.81 secs
      Response time:		        0.10 secs
      Transaction rate:	       10.33 trans/sec
      Longest transaction:	        0.61
      Shortest transaction:	        0.00
      
      Elapsed time:		        4.54 secs
      Response time:		        0.08 secs
      Transaction rate:	       13.22 trans/sec
      Longest transaction:	        0.44
      Shortest transaction:	        0.01
      ```
      
      * Remove reflection in `.put` call (not the reflections trategy)
      
      also remove the call to `classloader/the-classloader` as it did nothing
      
      * Comment and settle on a single method
      
      * tests
      
      * select from db twice
      
      had a failure in CI. give it time to do its thing with another db call
      
      * block to wait for timestamp update?
      
      * unflake the tests
      
      tasks and events from outside the thread can hit the db. the
      ConnectionCustomizer is also run from c3p0 controlled threads so we
      can't easily isolate everything to our thread
      
      Was running
      
      ```clojure
      (comment
        (dotimes [n 5]
          (dotimes [_ 100]
            (recent-activity-test)
            (CheckinTracker-test))
          (println (* (inc n) 100)))
          )
      ```
      
      to run the tests 500 times and would keep getting flakes at a rate
      ~1/100 to 1/500. Just frustration for the future.
      
      * typehint
      
      * Switch it up a bit
      
      Tests were flaking in h2 and I don't know why. I'm switching to just
      updating recent activity on most methods.
      
      * final touches
      
      Co-authored-by: default avatarAleksandr Lesnenko <alxnddr@users.noreply.github.com>
      Co-authored-by: default avatarNgoc Khuat <qn.khuat@gmail.com>
      Co-authored-by: default avatarNemanja Glumac <31325167+nemanjaglumac@users.noreply.github.com>
      Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
      Co-authored-by: default avatardpsutton <dan@dpsutton.com>
      Unverified
      31999007
    • dpsutton's avatar
      Relieve db pressure on api/health check (#27192) · 0975f5d9
      dpsutton authored
      * Relieve db pressure on api/health check
      
      https://github.com/metabase/metabase/issues/26266
      
      Servers under heavy load can be slow to respond to the api/health
      check. This can lead to k8s killing healthy instances happily humming
      along serving requests.
      
      One idea floated was to use QoSFilters
      https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/servlets/QoSFilter.html
      to prioritize those requests in front of others. But I suspect this
      might not be our bottleneck.
      
      Our health endpoint was updated to see if it could acquire an endpoint
      when we were dealing with connection pool issues. We were reporting the
      instance was healthy once it has finished the init process, but would
      report healthy if 60/15 app-db connections were used and no actual
      queries could complete.
      
      The remedy was adding
      `(sql-jdbc.conn/can-connect-with-spec? {:datasource (mdb.connection/data-source)})`
      to the endpoint. But now to get information about the health of the
      system we have to wait in the queue to get a datasource.
      
      The hope is that this change which monitors for recent db
      checkins (query success) and checkouts (query begun) can be a proxy for
      db activity without having to wait for a connection and hit the db ourselves.
      
      Some simple and crude benchmarking:
      - use `siege` to hit `api/database/<app-db>/sync_schema`
      - in a separate tab, use `siege` to hit `api/health`
      
      Three trials with unconditional db access and conditional db
      access (look for recent activity set by the new `ConnectionCustomizer`).
      
      One siege client is synching the app-db's schema with 80 clients each
      sending 60 requests. the other has 1 client sending 60 requests to api/health.
      
      Run             |  Elapsed Time | max tx  | tx rate
       before change  |    7.16s      |  0.79s  |  8.38 tx/s
       before change  |   23.91s      |  1.44s  |  2.51 tx/s
       before change  |   13.00s      |  0.50s  |  4.62 tx/s
      ----------------------------------------------------
       after change   |    4.46s      |  0.27s  |  13.45 tx/s
       after change   |    5.81s      |  0.61s  |  10.33 tx/s
       after change   |    4.54s      |  0.44s  |  13.22 tx/s
      
      Full(er) results below:
      
      ```
      Unconditional db access
      =======================
      
      siege -c80 -r 40 "http://localhost:3000/api/database/2/sync_schema POST" -H "Cookie: $SESSION"
      
      siege -c 1 -r 60 "http://localhost:3000/api/health"
      
      Elapsed time:		        7.16 secs
      Response time:		        0.12 secs
      Transaction rate:	        8.38 trans/sec
      Longest transaction:	        0.79
      Shortest transaction:	        0.01
      
      Elapsed time:		       23.91 secs
      Response time:		        0.40 secs
      Transaction rate:	        2.51 trans/sec
      Longest transaction:	        1.44
      Shortest transaction:	        0.02
      
      Elapsed time:		       13.00 secs
      Response time:		        0.22 secs
      Transaction rate:	        4.62 trans/sec
      Longest transaction:	        0.50
      Shortest transaction:	        0.06
      
      Conditional db access
      ==============================================================
      
      Elapsed time:		        4.46 secs
      Response time:		        0.07 secs
      Transaction rate:	       13.45 trans/sec
      Longest transaction:	        0.27
      Shortest transaction:	        0.01
      
      Elapsed time:		        5.81 secs
      Response time:		        0.10 secs
      Transaction rate:	       10.33 trans/sec
      Longest transaction:	        0.61
      Shortest transaction:	        0.00
      
      Elapsed time:		        4.54 secs
      Response time:		        0.08 secs
      Transaction rate:	       13.22 trans/sec
      Longest transaction:	        0.44
      Shortest transaction:	        0.01
      ```
      
      * Remove reflection in `.put` call (not the reflections trategy)
      
      also remove the call to `classloader/the-classloader` as it did nothing
      
      * Comment and settle on a single method
      
      * tests
      
      * select from db twice
      
      had a failure in CI. give it time to do its thing with another db call
      
      * block to wait for timestamp update?
      
      * unflake the tests
      
      tasks and events from outside the thread can hit the db. the
      ConnectionCustomizer is also run from c3p0 controlled threads so we
      can't easily isolate everything to our thread
      
      Was running
      
      ```clojure
      (comment
        (dotimes [n 5]
          (dotimes [_ 100]
            (recent-activity-test)
            (CheckinTracker-test))
          (println (* (inc n) 100)))
          )
      ```
      
      to run the tests 500 times and would keep getting flakes at a rate
      ~1/100 to 1/500. Just frustration for the future.
      
      * typehint
      
      * Switch it up a bit
      
      Tests were flaking in h2 and I don't know why. I'm switching to just
      updating recent activity on most methods.
      Unverified
      0975f5d9
    • Cal Herries's avatar
      Improve datetimeDiff type error message (#27225) · 8e385d49
      Cal Herries authored
      * Update test
      
      * Update implementations
      Unverified
      8e385d49
    • Cal Herries's avatar
      Fix datetime-diff helper text (#27224) · e3bdada4
      Cal Herries authored
      Unverified
      e3bdada4
    • Nemanja Glumac's avatar
    • Ngoc Khuat's avatar
      Suppress console log when using `with-log-messages-for-level` (#26468) · ed1d503f
      Ngoc Khuat authored
      * disable additivity when using "with-log-level"
      
      * only set when parent is root
      
      * wording
      
      * remove a test that is no longer needed
      
      * fix indents
      
      * one missing indent fix
      Unverified
      ed1d503f
  4. Dec 13, 2022
Loading