Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Aug 06, 2023
    • Cam Saul's avatar
      QP: use Malli (#32657) · 18c1b198
      Cam Saul authored
      * QP: use Malli
      
      * Fix schema names
      
      * Fix missing namespaces
      
      * Fix busted stuff
      
      * Simplify the new schema
      
      * Test fix :wrench:
      
      * Test fixes :wrench:
      
      * MongoDB fixes :wrench:
      
      * FAST mu/defn
      
      * Malli registry
      
      * Misc fixes
      
      * Some cleanup
      
      * Misc cleanup and test fixes :wrench:
      
      * mu/disable-enforcement
      
      * Test fix
      
      * Fix Cljs version of `disable-enforcement`
      
      * Misc improvements
      
      * Update tests
      
      * Fix Kondo warnings
      
      * Fix Kondo warnings
      
      * Fix busted Malli schema
      
      * PR feedback
      
      * Fix busted schema
      
      * Kondo fixx
      
      * Test fixes :wrench:
      
      * Use Malli version of `describe` now that PR was merged
      
      * Work around https://github.com/metosin/malli/issues/924
      
      * Test fixes :wrench:
      
      * Improvements
      
      * Test fix
      Unverified
      18c1b198
  2. Aug 02, 2023
  3. Jun 17, 2023
  4. May 31, 2023
  5. May 15, 2023
  6. May 12, 2023
    • dpsutton's avatar
      Hides malli's instrumentation messages (#30737) · 97108afc
      dpsutton authored
      Before:
      ```
      ❯ clj -M:"$ALIASES"
      WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.algo.generic.math-functions, being replaced by: #'clojure.algo.generic.math-functions/abs
      operator.clj:172 recur arg for primitive local: sum is not matching primitive, had: Object, needed: long
      Auto-boxing loop arg: sum
      Clojure 1.12.0-alpha2
      user=> (do (dev) (start!))
      Warning: environ value /Users/dan/.sdkman/candidates/java/current for key :java-home has been overwritten with /Users/dan/.sdkman/candidates/java/17.0.1-zulu/zulu-17.jdk/Contents/Home
      2023-05-12 14:26:37,091 INFO metabase.util :: Maximum memory available to JVM: 8.0 GB
      2023-05-12 14:26:47,248 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. :unlock:
       For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
      ..instrumented #'metabase.util.malli/with-api-error-message
      ..instrumented #'metabase.util.honey-sql-2/identifier
      ..instrumented #'metabase.util.honey-sql-2/normalize-type-info
      ..instrumented #'metabase.util.honey-sql-2/with-database-type-info
      ..instrumented #'metabase.util.honey-sql-2/cast
      ..instrumented #'metabase.util.honey-sql-2/quoted-cast
      ```
      
      After:
      
      ```
      ❯ clj -M:"$ALIASES"
      WARNING: abs already refers to: #'clojure.core/abs in namespace: clojure.algo.generic.math-functions, being replaced by: #'clojure.algo.generic.math-functions/abs
      operator.clj:172 recur arg for primitive local: sum is not matching primitive, had: Object, needed: long
      Auto-boxing loop arg: sum
      Clojure 1.12.0-alpha2
      user=> (do (dev) (start!))
      Warning: environ value /Users/dan/.sdkman/candidates/java/current for key :java-home has been overwritten with /Users/dan/.sdkman/candidates/java/17.0.1-zulu/zulu-17.jdk/Contents/Home
      2023-05-12 15:11:21,678 INFO metabase.util :: Maximum memory available to JVM: 8.0 GB
      2023-05-12 15:11:30,806 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. :unlock:
       For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
      util.cljc:346 recur arg for primitive local: cumulative_byte_count is not matching primitive, had: Object, needed: long
      Auto-boxing loop arg: cumulative-byte-count
      WARNING: infinite? already refers to: #'kixi.stats.math/infinite? in namespace: kixi.stats.core, being replaced by: #'clojure.core/infinite?
      2023-05-12 15:11:55,615 INFO driver.impl :: Registered abstract driver :sql  :truck:
      ```
      Unverified
      97108afc
  7. May 05, 2023
    • Cal Herries's avatar
      Tidy driver/database-supports? defmethods and remove usage of driver/supports? (#29516) · ea25b693
      Cal Herries authored
      * Tidy database-supports? defmethods
      
      * Fix oracle build
      
      * Fix driver
      
      * Fix typo
      
      * prefer driver/database-supports? instead of driver/supports?
      
      * driver/supports? -> driver/database-supports? in tests
      
      * Fix foreign-key support for sqlite in tests
      
      * supports? -> database-supports?
      
      * `supports?` -> `database-supports?`
      
      * supports? -> database-supports?
      
      * Update supports? docstring
      
      * Fix tests
      
      * Fix test
      
      * Remove unused require
      
      * Add mt/with-everything-store in tests
      
      * fix test
      
      * Fix tests
      
      * Add note in driver changelog
      Unverified
      ea25b693
  8. Apr 19, 2023
    • Cam Saul's avatar
      MLv2: Implement support for `:value` clause (#30163) · 2f817f52
      Cam Saul authored
      * Don't load metabase.query-processor-test.test-mlv2 until we load QP
      
      * Add schema and tests for `:value`
      
      * Don't skip tests for #29895
      
      * Conversion logic for `:value`
      
      * pMBQL normalization should handle `:effective-type`
      
      * Options should validate `:base-type`, `:effective-type`, etc. if present
      
      * Use `:effective-type` for type checking if present
      Unverified
      2f817f52
  9. Apr 12, 2023
    • dpsutton's avatar
      Move require of test.test-mlv2 out of main code (#30037) · 3812b7c2
      dpsutton authored
      in query_processor.clj we were requiring
      
      ```clojure
      (when-not config/is-prod?
        (classloader/require 'metabase.query-processor-test.test-mlv2))
      ```
      
      And this was blowing up as it was not present in the aliases used for
      cypress and the FE crew.
      
      Now user.clj requires it in the same way that it requires
      `[metabase.test-runner.assert-exprs]` so we can run tests with
      `partial=` and their ilk. And it is required by the test entry namespace
      so CI and test runners find it as well.
      
      The `resolve` command in query_processor.clj now works in dev and CI
      
      ```clojure
      query-processor=> (resolve 'metabase.query-processor-test.test-mlv2/post-processing-middleware)
      ,#'metabase.query-processor-test.test-mlv2/post-processing-middleware
      ```
      (comma to prevent commit message from eliding as a comment)
      
      so presumably tests that require the middleware tricks from this test
      namespace will continue to work.
      Unverified
      3812b7c2
  10. Apr 11, 2023
  11. Mar 23, 2023
  12. Mar 21, 2023
  13. Mar 20, 2023
    • Ngoc Khuat's avatar
      db/select-* to toucan 2 (#29319) · 952af410
      Ngoc Khuat authored
      * db/select-one-field -> t2/select-one-fn
      
      * `db/select-field` -> `t2/select-fn-set`
      
      * db/select-ids -> t2/select-pk-set
      
      * db/select-field->field -> t2/select-fn->fn
      
      * db/select-field->id -> t2/select-fn->pk
      
      * `db/select-id->field` -> `t2/select-pk->field`
      
      * add discourage var
      Unverified
      952af410
  14. Mar 17, 2023
  15. Mar 08, 2023
  16. Feb 10, 2023
  17. Feb 07, 2023
  18. Feb 02, 2023
    • Bryan Maass's avatar
      insert next-gen permission paths alongside currently active permission paths +... · c46337ad
      Bryan Maass authored
      insert next-gen permission paths alongside currently active permission paths + classification (#27911)
      
      * implement move, which returns v2 paths
      
      - TODO: insert these into the db
      
      (move v1-path) => [v2 paths]
      
      * cleanup + add some schemas
      
      * generative tests 4 permission path classification
      
      * whitespace lint
      
      * detect data, query, and paths for v2
      
      * calling move on v2 paths is a no-op
      
      * differentiate between v1 and v2 permissions
      
      quickchecking for move, classify-path, and classify-data-path
      
      * fix tests + add idempotency test
      
      * add tests for classification of permission paths
      
      - rename move to ->v2-path
      - move some fxns around
      - ascii art in test
      
      * making the legos line up
      
      - need to insert both v1 and v2 versions of paths (of course)
      - valid-path? has to allow v2 paths to be inserted
      
      * replace mu/with-api-error-message
      
      * linter + code quality fixes
      
      * privatize rx->kind
      
      * remove some changes that should be 4 anotherbranch
      
      * revert ns
      
      * delete v2 permissions in
      
      - they aren't handled by the perm graph parser, so they don't get propagated into "old graph", so the diff between old and new indicates that they need to be rewritten.
      
      * only delete v2 paths for the current group_id
      
      - reorder declarations in models.permissions
      
      * remove extra line break
      
      * Update src/metabase/models/permissions.clj
      
      fix typo
      
      Co-authored-by: default avatarmetamben <103100869+metamben@users.noreply.github.com>
      
      ---------
      
      Co-authored-by: default avatarmetamben <103100869+metamben@users.noreply.github.com>
      Unverified
      c46337ad
  19. Feb 01, 2023
  20. Jan 26, 2023
  21. Jan 04, 2023
    • metamben's avatar
      Cherry pick be data apps minus apps and pages (#27413) · 3b8f0d88
      metamben authored
      
      * Implicit action pre-fetch values (#25940)
      
      * [Apps] Implicit action pre-fetch values
      
      WIP To return a parameter-id: value map in order to prefill ui.
      
      * Use jsonstring encoded parameters
      
      * Change parameters exposed by implicit actions
      
      Previously, we would look at a model's source-table to determine which
      parameters were valid for implicit actions. So if an implicit action is
      on a User model that hides the user.password column, you could still
      update or insert the password.
      
      With this commit, if the model doesn't expose a source table column, it
      is not a valid parameter. In the example user.password could not be
      inserted/updated or fetched for prefilling.
      
      * Fixing tests
      
      * Show data app page ID params in editing mode (#25943)
      
      * Data apps layout updates (#25930)
      
      * play with grid sizes to make apps feel appier [ci skip]
      
      * move app nav to bar inside app
      
      * only hide other header buttons outside of edit mode
      
      * tweak overflow issue
      
      * fix header width on data apps
      
      * add control to see app contents
      
      * set data apps layout constants contextually [ci skip]
      
      * remove hardcoded value [ci skip]
      
      * extract conextual header styles [ci skip]
      
      * set max-width on paramaters for data apps [ci skip]
      
      * move data apps nav deeper, only show if not editing [ci skip]
      
      * add spacing to contents trigger, rm old code [ci skip]
      
      * rm table thing for now [ci skip]
      
      * Fixes for data app layout updates (#25962)
      
      * Reorder import
      
      * Fix type
      
      * Fix missing translation, move out URL
      
      * Remove not used import
      
      * Rework `getSelectedItems` utility for data apps
      
      * Fix selected data app page highlighting
      
      * Don't reload app navbar on page change
      
      * Change nav item links look
      
      * Remove `DataAppPageSidebarLink`
      
      * Turn nav into HTML list
      
      * Use styled components
      
      * Fix edit button covered by visualizations
      
      * Fix opening homepage
      
      * Remove redundant prop
      
      Co-authored-by: default avatarAnton Kulyk <kuliks.anton@gmail.com>
      
      * First take at parameterized data app page titles (#25938)
      
      * Bring `DataAppContext` back
      
      * Pass `onFocus` and `onBlur` to `EditableText`
      
      * Export `EditableTextProps`
      
      * Add basic `DataAppPageTitle` component
      
      * Use `DataAppPageTitle`
      
      * Fix component name
      
      * Add primitive autocompletion to page title input
      
      * Add `title_template` to `DataAppNavItem` type
      
      * Tweak value management in `DataAppPageTitle`
      
      * Handle `null` values in titles more gracefully
      
      * Don't show suggestions without detail cards
      
      * Don't add whitespace when pasting a token
      
      * Don't update app's collection all the time
      
      * Add data app related selectors to dashboard selectors
      
      * Add redux state to keep title template changes
      
      * Update page title template on page save
      
      * Wire up `DataAppPageTitle` with dashboard code
      
      * Fix unit tests
      
      * Reset state on exiting editing mode
      
      * Add back button to template data picker
      
      * Fix test that didn't include pk on insert
      
      * Use the actual pk value
      
      Co-authored-by: default avatarAnton Kulyk <kuliks.anton@gmail.com>
      Co-authored-by: default avatarKyle Doherty <5248953+kdoh@users.noreply.github.com>
      
      * Revert "Remove data app permission code (#26016)"
      
      This reverts commit b8c92cf1.
      
      * Remove unused imports
      
      * Don't require the table_id on result_metadata elements (#26073)
      
      In some cases (which ones still has to be clarified) the result metadata doesn't
      contain the table_id and so no parameters are provided. In fact the table ID is
      probably not needed anyway because fields are already table specific.
      
      * Support filtering cards by used model (#26088)
      
      This change adds a new filter option for the GET /card endpoint.
      
      * Support filtering apps using a given model (#26124)
      
      * Handle type errors more cleanly (#26120)
      
      * [Apps] Handle type errors more cleanly
      
      Adds a number of tests for different db datatypes, these can be expanded
      on as we solve.
      
      Catches most exceptions during execution and logs then throws a generic
      message that leaks less of the details.
      
      * Make sure we order-by expected results
      
      * Fix tests, allow purposeful exception through
      
      * Accept model-ids for scaffolding apps (#26139)
      
      * [Apps] Accept model-ids for scaffolding apps
      
      WIP - accept model-ids but there's still some issues, might be able to
      start building UI from this point though.
      
      * Maybe working model
      
      * Work with card__n table-ids passed in. Fix problems with detail page
      
      * Dynamically add actions that exist on models
      
      * Update tests for existing custom actions
      
      * Can't def tru
      
      * Address review comments
      
      * Add required to action parameters (#26296)
      
      Looks at `database_required` field values for implicit actions, and
      `required` template-tags for custom query actions. Bubble those values
      up onto `action.parameters`. Finally, look at `requires_pk` to see if pk
      fields should be marked as required regardless.
      
      * Cleanup scaffolded back button (#26317)
      
      * Forward action execution error messages to the frontend (#26469)
      
      This is a crude temporary solution necessary to be able to debug problems related to executing actions.
      Although error parsing has been implemented for Postgres (only) with the goal to provide information for
      the FE like `{"errors":{"id":"violates foreign key constraint note_subject_id_fkey"}}`. This code doesn't
      care about the error message itself, it just forwards the error message from the DB to the FE.
      
      * Actions for public pages (#26554)
      
      * [Apps] Testing out public dashboards
      
      * Add WIP endpoints for public execution
      
      * Add throttle check to public execution route
      
      * Bump attempts-threshold to 1000
      
      * Bump threshold to 5000
      
      * Add test
      
      * Fix linter
      
      * Review changes
      
      * Fix lint
      
      * Run publicly shared actions with full permissions (#26610)
      
      * Support actions with MySQL and MariaDB (#26573)
      
      Support actions with MySQL/MariaDB
      
      * Check permissions for implicit actions (#26652)
      
      * Check permissions when executing implicit actions
      
      * Fix some action tests and reduce permissions boilerplate
      
      * Revert "[Apps] Disable data-apps prior to release (#25982)"
      
      This reverts commit 51147027.
      
      * WIP: Unify action representation (#26717) (#27349)
      
      * [Apps] WIP initial db migrations for giving implicit actions an action_id and removing card from query_action
      
      * Make crud work for http actions
      
      * Make implicit crud work
      
      * Rename implicit_action.namespace to kind. Rewrite merged-model-actions to actions-with-implicit-parameters
      
      * Fix query action creation
      
      * Make actions.test-util/with-action create a holding model
      
      * Fix unified-action-create-test
      
      * Fix get-action-test
      
      * Add tests for creating implicit and http dashcard actions
      
      * Enable query action in dashcard-action-create-update-test
      
      * Fix metabase.api.dashboard-test/dashcard-query-action-execution-test
      
      * Fix metabase.api.dashboard-test/dashcard-http-action-execution-test
      
      * Fix metabase.api.dashboard-test/dashcard-action-execution-auth-test
      
      * Fix metabase.api.dashboard-test/dashcard-action-execution-granular-auth-test
      
      * Make implicit action execution work
      
      * Scaffolding implicit actions from table
      
      * Fix metabase.api.action-test
      
      * Make scaffolding work with existing models. Update with-action macro to allow multiple actions
      
      * Fix metabase.api.public-test
      
      * Remove unused names from metabase.models.action
      
      * Remove some unused imports
      
      * Remove an unused import
      
      * Remove model_action and fix a bunch of tests
      
      * Fix dashcard-implicit-action-only-expose-and-allow-model-fields
      
      * Fix action type inheritance
      
      * Fill in remarks and add description to action
      
      * Add forward migration for existing actions and rollbacks that remove existing actions
      
      * Prevent overwriting parameters of custom actions
      
      * Support setting description on actions
      
      * Update migration to work with mysql - rollback is still not quite working in mysql due to database_id fk
      
      * Rename with-action macro to with-actions and document it
      
      * Remove report_card.is_write
      
      * Fix some more tests
      
      * Fix rollback across all appdb drivers, add not null constraints, use action.name to hold implicit slugs
      
      * Pull name and description from is_write card
      
      * Add name to scaffolded implicit actions
      
      * Remove ModelAction
      
      * Remove is_write from comments too
      
      * Different databases reference capture groups in regexp_replace differently. Also drop model_action table
      
      * Update migrations to pass linter.
      
      Introduce ${capturegroup.ref} to handle the different ways to reference
      capture groups within a regexp_replace. This allows collapsing large
      data migrations into a single `sql` element that the linter requires.
      
      * Add kind to action-mapper so we can sort as we wish
      
      * Add name to with-temp Action
      
      * Adjust FE to query actions migration (#26764)
      
      * Remove `is_write` card notion
      
      * Remove action `slug` from action types
      
      * Fix typo in `estimateCardSize` name
      
      * Remove hanging card's `is_write` prop
      
      * Remove not used writeback actions
      
      * Update endpoints
      
      * Remove not used prop
      
      * Remove action `slug` usage
      
      * Update actions entity
      
      * Tweak action creator to handle new actions shape
      
      * Fix saving action dashcards
      
      * Fix actions API usage
      
      How did that work before? :thinking:
      
      
      
      * Pass `database_id` when creating query actions
      
      * Remove custom list API implementation for actions
      
      * Clean up types
      
      Co-authored-by: default avatarTamás Benkő <tamas@metabase.com>
      
      * Change regexp_replace to substring to support mysql - this is ok because we have consistent json serialization.
      
      * Check permissions on the model in the action API
      
      * Remove unused required namespace
      
      * Throw an exception when demoting a model with actions to question
      
      * Allow saved parameters on implicit actions to add to implicit parameters
      
      Co-authored-by: default avatarCase Nelson <case@metabase.com>
      Co-authored-by: default avatarAnton Kulyk <kuliks.anton@gmail.com>
      
      Co-authored-by: default avatarmetamben <103100869+metamben@users.noreply.github.com>
      Co-authored-by: default avatarCase Nelson <case@metabase.com>
      
      * empty commit to kick CI
      
      * bump CI
      
      * Fix down migrations
      
       change v45.00-002 sets up a fk fk_query_action_ref_action_id. h2 v2 for
       some strange reason requires us to remove the constraint before we can
       remove the primary key. But then we have to restore the state that
       v45.00-002 creates manualy
      
      * Allow for version for `(migrate! :down 45)`
      
      * Remove data apps parts not planned for 0.46
      
      Removed app container, app scaffolding and pages (dashboards with is_app_page)
      
      * Address linter issues
      
      * Make some cosmetic changes to reduce the diff
      
      * Make new permission API endpoints use defendpoint-schema
      
      * Fix bad automerge
      
      * Skip Mongo 4.2 instead of 4.0
      
      Co-authored-by: default avatarCase Nelson <case@metabase.com>
      Co-authored-by: default avatarAnton Kulyk <kuliks.anton@gmail.com>
      Co-authored-by: default avatarKyle Doherty <5248953+kdoh@users.noreply.github.com>
      Co-authored-by: default avatardan sutton <dan@dpsutton.com>
      Unverified
      3b8f0d88
  22. Jan 03, 2023
  23. Dec 20, 2022
  24. Dec 09, 2022
  25. Nov 18, 2022
    • adam-james's avatar
      Group Axes for Multi-series static viz (#26145) · 03373a33
      adam-james authored
      * First pass at grouping axes
      
      This PR uses the results_metadata key to look at fingerprints for numerical axes to try determine if each series on
      the Y axis can be sanely represented on the same axis.
      
      This is done by calculating an overlap (some value between 0 and 1) and grouping all axes on the LEFT if they pass the threshold, which is
      some value between 0 and 1. The overlap is always calculated when there is SOME overlap between the ranges of each
      axis, and is calculated as:
      
      ```clojure
      (/ (- (max maximums) (min minimums))
         (- (min maxiumums) (max minimums)))
      ```
      
      This is done to try catch situations where one column's range is entirely inside the other, but is much smaller (- max
      min); such a case would have a small percent overlap by the above calculation, and implies that it might be better to
      split the axes.
      
      * Address feedback.
      
      * Fix shape of data in tests of 2 private fns
      
      * Add test for split axes
      
      * Fixed error in test util
      
      * render-utils does with-redef, disallows parallel tests. Makes sense
      Unverified
      03373a33
  26. Oct 21, 2022
    • adam-james's avatar
      Correctly Apply Column Formatting to Static Visualizations (#25493) · d1c61272
      adam-james authored
      * Column settings correct keys. Show Axes labels logic defaults 'true'
      
      These are two small changes related to static viz column formatting.
      
      The show axes labels logic change fixes a faulty assumption from an earlier PR where it was assumed that if
      :labels_enabled key was NOT present, then it was 'false'. The default however, is 'true', so the check had to change
      to reflect the frontend.
      
      Column settings have to be formatted so that the js code running in the backend looks in the right places. For
      example, `:number-format` must become `:number_format`. I move the function that does that change up so that it can be
      rolled up into `settings-from-column` to reduce the number of call sites for `update-col-for-js`.
      
      * Adjust number format fn to consider :decimal_digits of currencies
      
      * Simpler date-separator fix
      
      * Prevent non-temporal labels from being passed to datetime formatter
      
      * Table Temporal format columns are improved
      
      The column headers will now be properly looked up, the key format was incorrect before, causing the `:column_title`
      lookup to return nil, which ignores when a user changes the column name.
      
      Table columns with temporal formatting settings come in with `:unit` `:default`, so that case was added to the
      datetime formatter. Additionally, a helper function was added to pull the correct column viz settings to pass to the datetime.
      
      * Change some table rendering to look for '_' keys instead of '-'
      
      And also allow the field-id to take the form `[:field-id 10 nil]` OR `[:field-id 10]`
      
      * Added test utils for static-viz rendering. Docstrings for details.
      
      A big set of functions to enable cleaner static-viz rendering tests.
      
      Next push will be thorough table tests, and some other static viz cleanup
      
      * Add some viz-scenarios to help with table testing
      
      * Lint
      
      * Wrong map here
      
      * Currently unused bindings
      
      * Writing tests with render-test utils now, and catching new bugs
      
      * Use column's field ref to maybe get the column settings
      
      * Adjustments so that column-settings are properly grabbed
      
      * Number formatter no longer assumes decimal places
      
      The frontend will only display decimal places when strictly necessary OR when the column formatting dictates the
      number of decimal places to show.
      
      The number formatter on the backend made different assumptions about decimal places, so precision may be lost OR
      needlessly displayed.
      
      The formatter now checks the actual value passed in and adjusts the format string according to the decimal places
      present in the value, not just via column-settings, though the column settings will still take priority.
      
      * Shame on me. inline def is not needed here :/
      
      * Adjust number format tests to assert that now we match FE behavior
      
      * Really trying to get formatting buttoned up.
      
      * Matching frontend's default format behavior for numbers
      
      * Some tests failed because they assumed two decimal places always
      
      * Values less than one still round after 2 sig-figs
      
      * Column formatting on tables follows expected default behavior
      
      * Getting public-settings included in number and datetime formatters
      
      * Test for custom formatting public settings
      
      * Removed unused require.
      
      * address some PR feedback
      
      * Try to clarify the logic in the number formatter
      
      * Add some clarifying comments
      
      * Use the existing viz tools to help create field-refs properly
      
      * Cleaned up column-name impl. to use normed field-refs. Much nicer!
      
      * Remove some unnecessary tests, and add a negative scientific num
      
      * Simplify table columns test
      
      * Simplify and add more format settings to column-formatting test
      
      * Unused require
      
      * Another one
      
      * Fix format of column refs in test
      
      * Remove unused thread
      Unverified
      d1c61272
    • Braden Shepherdson's avatar
      Serdes v2: Refactor entire serdes flow to use real Toucan, not low level (#25981) · a9f44010
      Braden Shepherdson authored
      Previously, some automatic behavior was causing problems in serdes.
      (The worst example is generating an `entity_id` on insert while
      deserializing an entity we don't own - if deserialized again it would be
      duplicated.)
      
      A whole cascade of design choices fell out of this problem:
      - `insert!` would generate `entity_id`s, so use `simple-insert!`
      - But `simple-insert!` doesn't convert eg. MBQL `:definition` maps back
        into JSON strings
      - We got the maps as Clojure data and not JSON strings because `select`
        and `simple-select` both run `post-select` and parse the JSON.
      - So we ended up with a raw query on the select side and `simple-insert!`
        on the storage side.
      
      This change unwinds that whole stack, and instead uses a dynamic
      var to suppress the few pieces of `pre-insert` and `pre-update` logic
      that causes problems.
      
      The end result is much cleaner, and much more consistent with the rest
      of Metabase's backend logic.
      Unverified
      a9f44010
  27. Sep 15, 2022
  28. Sep 14, 2022
  29. Sep 13, 2022
  30. Sep 07, 2022
  31. Aug 05, 2022
    • dpsutton's avatar
      Fix in-memory logger (#24616) · df165ba1
      dpsutton authored
      * Fix in-memory logger
      
      Our Admin > Troubleshooting > Logs page broke, just showing a spinner
      and never showing the logs.
      
      Don't quite understand why this fixes it. Javadocs are
      https://logging.apache.org/log4j/2.x/log4j-api/apidocs/org/apache/logging/log4j/LogManager.html#getContext-boolean-
      
      ```clojure
      logger=> (log/warn "test")
      nil
      logger=> (count @messages*)
      0
      ;; no in-memory logs so page is empty
      ;; change `(LogManager/getContext true)` in the momoized ns-logger fn
      ;; and then observe:
      logger=> (log/warn "test")
      nil
      logger=> (count @messages*)
      4
      ```
      
      Some explorations that might shine some light:
      
      ```clojure
      logger=> (into {} (.getAppenders (.getLogger (LogManager/getContext false) (str *ns*))))
      {}
      logger=> (into {} (.getAppenders (.getLogger (LogManager/getContext true) (str *ns*))))
      {"metabase-appender" #object[metabase.logger.proxy$org.apache.logging.log4j.core.appender.AbstractAppender$ff19274a
                                   "0x4d680247"
                                   "metabase-appender"]}
      ```
      
      So something is not hooked up quite right.
      
      * Add tests for metabase.logger
      
      * Ensure `api/util/logs` returns logs
      
      * tests
      
      * Check for presence of `MetabaseLoggerFactory` rather than whole str
      
      When in a namespace with a record, `Foo resolves to ns.Foo. But outside
      it resolves to ns/Foo. When running tests from the command line *ns* is
      user so it gets more complicated.
      
      * Kinda playing whackamole™ with `(LogManager/getContext true)`
      
      * Remove custom memoizing logger
      
      History:
      
      39.2 we set `Multi-Release: true` in our manifest file and query speed
      drops like a stone. Jeff was able to track this to our logger calls in
      tight loops.
      
      We revert the multi-release and keep seeing the warning on startup
      
      > WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
      
      Benchmarking on 39.2
      (bench (log/trace "hi")) -> 15383 ns
      
      So we freaked out and set up a memoizing logger factory
      
      (bench (log/trace "hi")) -> 141 ns
      
      What a win.
      
      But we never noticed that the default *logger-factory* being picked up
      was slf4j ( `(log.impl/name log/*logger-factory*)` -> "org.slf4j" ). On
      39.2 if you set the factory to the log4j2 version you get back to a
      great number: `(bench (log/trace "hi"))` -> 25 ns
      
      And thus ensuring that our logger is the default log4j2 version is even
      faster than our memoizing logger-factory.
      
      Memoizing factory: 141 ns
      slf4j factory: 2269 ns
      log4j2 factory: 31 ns
      
      What does `(LogManager/getContext false)` mean versus using `true`? We
      only need and want a single context. But log4j2 by default uses a
      context selector called `ClassLoaderContextSelector`. We could put all
      of this behind us if we used a context selector type
      `BasicContextSelector` but this is surprisingly hard to do: you have to
      set a system property. And since all of this stuff gets initialized in
      static initializers, we don't have an opportunity to do this
      programmatically. The way around this is to require people to pass this
      system property on startup which is not acceptable.
      
      So getContext true checks for a threadlocal context in a specific static
      variable and falls back to a Default context. getContext false looks at
      classloaders and ends up at a different context. BUT: the log.impl
      version uses a closure over getContext false instead of getting it each
      time. And I suspect that when it does this there is only one so it is
      the default and continues to use this one. In our LoggerFactory
      implementation we were looking up the context each time. This still
      seems to work and everything is playing nice in our application
      classloader but its totally possible that our drivers are not hitting
      this. I'll have to investigate this.
      
      Verification:
      - build the uberjar locally (`bin/build`)
      - copy to some temp directory and also copy criterium.jar
      
      ```shell
      MB_JETTY_PORT=4000 java "$(socket-repl 4001)" -cp locally-built.jar:criterium.jar metabase.core
      ```
      
      ```clojure
      /tmp/locally-built via :coffee: v17.30 on :cloud:  metabase-query
      ❯ nc localhost 4001
      user=> (doto 'metabase.logger require in-ns)
      metabase.logger
      metabase.logger=> (require '[criterium.core :refer [bench]])
      nil
      metabase.logger=> (bench (log/trace "hi"))
      Evaluation count : 1686535500 in 60 samples of 28108925 calls.
                   Execution time mean : 22.487972 ns
          Execution time std-deviation : 0.101004 ns
         Execution time lower quantile : 22.326806 ns ( 2.5%)
         Execution time upper quantile : 22.648368 ns (97.5%)
                         Overhead used : 6.924761 ns
      nil
      metabase.logger=> (count (messages))
      358
      metabase.logger=>
      ```
      
      Verifies that we are on the order of 22 nanoseconds and the in-memory
      logger has messages in it.
      
      * Appease our namespace linters
      
      * I'll unplug you ns linter
      
      * Better tests and ns docstring
      
      * Bootstrap to set system properties
      
      New entrypoint for the application: metabase.bootstrap
      
      sets two properties for logging (context selector, log4j2 factory) and
      ensures those properties are set before any logging code is loaded.
      
      * docstrings and clean ns
      
      * metabase.logger ns docstring cleanup
      
      * docstring
      
      * rename a test now that there's no memoization
      
      * add logger properties to :dev profile
      
      * Revert "add logger properties to :dev profile"
      
      This reverts commit 4f09fa3b631f882a3c5edcab4508769ffb20d4fa.
      
      * deps
      Unverified
      df165ba1
  32. May 26, 2022
  33. May 13, 2022
    • dpsutton's avatar
      Record startup times (#22707) · ae032bb0
      dpsutton authored
      * Record startup times
      
      updated `dev/start!`. `mb.core/init!` itself sets up the db, loads
      plugins, and sets the init status so those were superfluous.
      
      * clean up dev namespace
      
      * testing doesn't call init so set a value
      Unverified
      ae032bb0
  34. Apr 19, 2022
    • Braden Shepherdson's avatar
      Make namespace aliasing consistent everywhere; enforce with clj-kondo (#21738) · 19beda53
      Braden Shepherdson authored
      * Make namespace aliasing consistent everywhere; enforce with clj-kondo
      
      See the table of aliases in .clj-kondo/config.edn
      
      Notable patterns:
      - `[metabase.api.foo :as api.foo]`
      - `[metabase.models.foo :as foo]`
      - `[metabase.query-processor.foo :as qp.foo]`
      - `[metabase.server.middleware.foo :as mw.foo]`
      - `[metabase.util.foo :as u.foo]`
      - `[clj-http.client :as http]` and `[metabase.http-client :as client]`
      
      Fixes #19930.
      Unverified
      19beda53
  35. Feb 15, 2022
Loading