Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. May 10, 2022
  2. May 09, 2022
  3. May 08, 2022
  4. May 06, 2022
  5. May 05, 2022
  6. May 04, 2022
  7. May 03, 2022
  8. May 02, 2022
    • dpsutton's avatar
    • dpsutton's avatar
      Persisted models schema (#21109) · c504a12e
      dpsutton authored
      * dir locals for api/let-404
      
      * Driver supports persisted model
      
      * PersistedInfo model
      
      far easier to develop this model with the following sql:
      
      ```sql
      create table persisted_info (
         id serial primary key not null
        ,db_id int references metabase_database(id) not null
        ,card_id int references report_card(id) not null
        ,question_slug text not null
        ,query_hash text not null
        ,table_name text not null
        ,active bool not null
        ,state text not null
        ,UNIQUE (db_id, card_id)
      )
      
      ```
      and i'll make the migration later. Way easier to just dorp table, \i
      persist.sql and keep developing without worrying about the migration
      having changed so it can't rollback, SHAs, etc
      
      * Persisting api (not making/deleting tables yet)
      
      http POST "localhost:3000/api/card/4075/persist" Cookie:$COOKIE -pb
      http DELETE "localhost:3000/api/card/4075/persist" Cookie:$COOKIE -pb
      
      useful from commandline (this is httpie)
      
      * Pull format-name into ddl.i
      
      * Postgres ddl
      
      * Hook up endpoints
      
      * move schema-name into interface
      
      * better jdbc connection management
      
      * Hotswap peristed tables into qp
      
      * clj-kondo fixes
      
      * docstrings
      
      * bad alias in test infra
      
      * goodbye testing format-name function
      
      left over. everything uses ddl.i/format-name and this rump was left
      
      * keep columns in persisted info
      
      columns that are in the persisted query. I thought about a tuple of
      [col-name type] instead of just the col-name. I didn't do this this type
      because I want to ensure that we compute the db-type in ONLY ONE WAY
      ever and i wasn't ready to commit to that yet. I'm not sure this is
      necessary in the future so it remains out now.
      
      Context: we hot-swap the persisted table in for the original
      query. Match up on query hash remaining the same. It continues to use
      the metadata from the original query and just `select cols from table`
      
      * Add migration for persisted_info table
      
      also removes the db_id. Don't know why i was thinking that was
      necessary. also means we don't need another unique constraint on (db_id,
      card_id) since we can just mark the card_id as unique. no idea what i
      was thinking.
      
      * fix ns in a sad manner :(
      
      far better to just have no alias to indicate it is required for side
      effects.
      
      * Dont hardcode a card-id :(:(:( my B
      
      * copy the PersistedInfo
      
      * ns cleanup, wrong alias, reflection warning
      
      * Check that state of persisted_info is persisted
      
      * api to enable persistence on a db
      
      i'm not wild about POST /api/database/:id/persist and POST
      /api/database/:id/unpersist but carrying on. left a note about it.
      
      So now you can enable persistence on a db, enable persistence on a model
      by posting to api/card/:id/persist and everything works.
      
      What does not work yet is the unpersisting or re-persisting of models
      when using the db toggle.
      
      * Add refresh_begin and refresh_end to persisted_info
      
      This information helps us with two bits:
      - when we need to chunk refreshing models, this lets us order by
      staleness so we can refresh a few models and pick up later
      - if we desire, we can look at the previous elapsed time of refreshes
      and try to gauge amount of work we want. This gives us a bit of
      look-ahead. We can of course track our progress as we go but there's no
      way to know if the next refresh might take an hour. This gives us a bit
      of insight.
      
      * Refresh tables every 8 hours ("0 0 0/8 * * ? *")
      
      Tables are refreshed every 8 hours. There is one single job doing this
      named "metabase.task.PersistenceRefresh.job" but it has 0 triggers by
      default. Each database with persisted models will add a trigger to this
      to refresh those models every 8 hours.
      
      When you unpersist a model, it will immediately remove the table and
      then delete the persisted_info record.
      
      When you mark a database as persist false, it will immediately mark all
      persisted_info rows as inactive and deleteable, and unschedule its
      trigger. A background thread will then start removing the tables.
      
      * Schedule refreshing on startup, watching for already scheduled
      
      does not allow for schedule changes but that's a future endeavor
      
      * appease our linter overlords
      
      * Dynamic var to inbhit persistence when refreshing
      
      also, it checked the state against "active" instead of "persisted" which
      is really freaky. how has this worked in the past if thats the case?
      
      * api docstrings on card persist
      
      * docstring
      
      * Don't sync the persisted schemas
      
      * Fix bad sql when no deleteable rows
      
      getting error with bad sql when there were no ids
      
      * TaskHistory for refreshing
      
      * Add created_at to persist_info table
      
      helpful if this ever ends up in the audit section
      
      * works on redshift
      
      hooked up the hierarchy and redshift is close enought that it just works
      
      * Remove persist_info record after deleting "deleteable"
      
      * Better way to check that something exists
      
      * POST /api/<card-id>/refresh
      
      api to refresh a model's persisted record
      
      * return a 204 from refreshing
      
      * Add buttons to persist/unpersist a database and a model for PoC (#21344)
      
      * Redshift and postgres report true for persist-models
      
      there are separate notions of persistence is possible vs persistence is
      enabled. Seems like we're just gonna check details for enabled and rely
      on the driver multimethod for whether it is possible.
      
      * feature for enabled, hydrate card with persisted
      
      two features: :persist-models for which dbs support it, and
      :persist-models-enabled for when that option is enabled.
      
      POST to api/<card-id>/unpersist
      
      hydrate persisted on cards so FE can display persist/unpersist for
      models
      
      * adjust migration number
      
      * remove deferred-tru :shrug:
      
      
      
      * conditionally hydrate persisted on models only
      
      * Look in right spot for persist-models-enabled
      
      * Move persist enabled into options not details
      
      changing details recomposes the pool, which is especially bad now that
      we have refresh tasks going on reusing the same connection
      
      * outdated comment
      
      * Clean up source queries from persisted models
      
      their metadata might have had [:field 19 nil] field_refs and we should
      substitute just [:field "the-name" {:base-type :type/Whatever-type}
      since it will be a select from a native query.
      
      Otherwise you get the following:
      
      ```
      2022-03-31 15:52:11,579 INFO api.dataset :: Source query for this query is Card 4,088
      2022-03-31 15:52:11,595 WARN middleware.fix-bad-references :: Bad :field clause [:field 4070 nil] for field "category.catid" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,596 WARN middleware.fix-bad-references :: Bad :field clause [:field 4068 nil] for field "category.catgroup" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,596 WARN middleware.fix-bad-references :: Bad :field clause [:field 4071 nil] for field "category.catname" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,596 WARN middleware.fix-bad-references :: Bad :field clause [:field 4069 nil] for field "category.catdesc" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,611 WARN middleware.fix-bad-references :: Bad :field clause [:field 4070 nil] for field "category.catid" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,611 WARN middleware.fix-bad-references :: Bad :field clause [:field 4068 nil] for field "category.catgroup" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,611 WARN middleware.fix-bad-references :: Bad :field clause [:field 4071 nil] for field "category.catname" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,611 WARN middleware.fix-bad-references :: Bad :field clause [:field 4069 nil] for field "category.catdesc" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,622 WARN middleware.fix-bad-references :: Bad :field clause [:field 4070 nil] for field "category.catid" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,622 WARN middleware.fix-bad-references :: Bad :field clause [:field 4068 nil] for field "category.catgroup" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,622 WARN middleware.fix-bad-references :: Bad :field clause [:field 4071 nil] for field "category.catname" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      2022-03-31 15:52:11,623 WARN middleware.fix-bad-references :: Bad :field clause [:field 4069 nil] for field "category.catdesc" at [:fields]: clause should have a :join-alias. Unable to infer an appropriate join. Query may not work as expected.
      ```
      I think its complaining that that table is not joined in the query and
      giving up.
      
      While doing this i see we are hitting the database a lot:
      
      ```
      2022-03-31 22:52:18,838 INFO api.dataset :: Source query for this query is Card 4,111
      2022-03-31 22:52:18,887 INFO middleware.fetch-source-query :: Substituting cached query for card 4,111 from metabase_cache_1e483_229.model_4111_redshift_c
      2022-03-31 22:52:18,918 INFO middleware.fetch-source-query :: Substituting cached query for card 4,111 from metabase_cache_1e483_229.model_4111_redshift_c
      2022-03-31 22:52:18,930 INFO middleware.fetch-source-query :: Substituting cached query for card 4,111 from metabase_cache_1e483_229.model_4111_redshift_c
      ```
      
      I tried to track down why we are doing this so much but couldn't get
      there.
      
      I think I need to ensure that we are using the query store annoyingly :(
      
      * Handle native queries
      
      didn't nest the vector in the `or` clause correctly. that was truthy
      only when the mbql-query local was truthy. Can't put the vector `[false
      mbql-query]` there and rely on that behavior
      
      * handle datetimetz in persisting
      
      * Errors saved into persisted_info
      
      * Reorder migrations to put v43.00-047 before 048
      
      * correct arity mismatch in tests
      
      * comment in refresh task
      
      * GET localhost:3000/api/persist
      
      Returns persisting information:
      - most information from the `persist_info` table. Excludes a few
      columns (query_hash, question_slug, created_at)
      - adds database name and card name
      - adds next fire time from quartz scheduling
      
      ```shell
      ❯ http GET "localhost:3000/api/persist" Cookie:$COOKIE -pb
      [
          {
              "active": false,
              "card_name": "hooking reviews to events",
              "columns": [
                  "issue__number",
                  "actor__login",
                  "user__login",
                  "submitted_at",
                  "state"
              ],
              "database_id": 19,
              "database_name": "pg-testing",
              "error": "No method in multimethod 'field-base-type->sql-type' for dispatch value: [:postgres :type/DateTimeWithLocalTZ]",
              "id": 4,
              "next-fire-time": "2022-04-06T08:00:00.000Z",
              "refresh_begin": "2022-04-05T20:16:54.654283Z",
              "refresh_end": "2022-04-05T20:16:54.687377Z",
              "schema_name": "metabase_cache_1e483_19",
              "state": "error",
              "table_name": "model_4077_hooking_re"
          },
          {
              "active": true,
              "card_name": "redshift Categories",
              "columns": [
                  "catid",
                  "catgroup",
                  "catname",
                  "catdesc"
              ],
              "database_id": 229,
              "database_name": "redshift",
              "error": null,
              "id": 3,
              "next-fire-time": "2022-04-06T08:00:00.000Z",
              "refresh_begin": "2022-04-06T00:00:01.242505Z",
              "refresh_end": "2022-04-06T00:00:01.825512Z",
              "schema_name": "metabase_cache_1e483_229",
              "state": "persisted",
              "table_name": "model_4088_redshift_c"
          }
      ]
      
      ```
      
      * include card_id in /api/persist
      
      * drop table if exists
      
      * Handle rescheduling refresh intervals
      
      There is a single global value for the refresh interval. The API
      requires it to be 1<=value<=23. There is no validation if someone
      changes the value in the db or with an env variable. Setting this to a
      nonsensical value could cause enormous load on the db so they shouldn't
      do that.
      
      On startup, unschedule all tasks and then reschedule them to make sure
      that they have the latest value.
      
      One thing to note: there is a single global value but i'm making a task
      for each database. Seems like an obvious future enhancement so I don't
      want to deal with migrations. Figure this gives us the current spec
      behavior to have a trigger for each db with the same value and lets us
      get more interesting using the `:options` on the database in the
      future.
      
      * Mark as admin not internal
      
      lets it show up in `api/setting/` . I'm torn on how special this value
      is. Is it the setting code's requirement to invoke the reschedule
      refresh triggers or should that be on the setting itself.
      
      It feels "special" and can do a lot of work from such just setting an
      integer. There's a special endpoint to set it which is aware, and thus
      would be a bit of an error to set this setting through the more
      traditional setting endpoint
      
      * Allow for "once a day" refresh interval
      
      * Global setting to enable/disable
      
      post api/persist/enable
      post api/persist/disable
      
      enable allows for other scheduling operations (enabling on a db, and
      then on a model).
      
      Disable will
      - update each enabled database and disable in options
      - update each persisted_info record and set it inactive and state
      deleteable
      - unschedule triggers to refresh
      - schedule task to unpersist each model (deleting table and associated
      pesisted_info row)
      
      * offset and limits on persisted info list
      
      ```shell
      http get "localhost:3000/api/persist?limit=1&offset=1" Cookie:$COOKIE -pb
      {
          "data": [
              {
                  "active": true,
                  "card_id": 4114,
                  "card_name": "Categories from redshift",
                  "columns": [
                      "catid",
                      "catgroup",
                      "catname",
                      "catdesc"
                  ],
                  "database_id": 229,
                  "database_name": "redshift",
                  "error": null,
                  "id": 12,
                  "next-fire-time": "2022-04-08T00:00:00.000Z",
                  "refresh_begin": "2022-04-07T22:12:49.209997Z",
                  "refresh_end": "2022-04-07T22:12:49.720232Z",
                  "schema_name": "metabase_cache_1e483_229",
                  "state": "persisted",
                  "table_name": "model_4114_categories"
              }
          ],
          "limit": 1,
          "offset": 1,
          "total": 2
      }
      ```
      
      * Include collection id, name, and authority level
      
      * Include creator on persisted-info records
      
      * Add settings to manage model persistence globally (#21546)
      
      * Common machinery for running steps
      
      * Add model cache refreshes monitoring page (#21551)
      
      * don't do shenanigans
      
      * Refresh persisted and error persisted_info rows
      
      * Remarks on migration column
      
      * Lint nits (sorted-ns and docstrings)
      
      * Clean up unused function, docstring
      
      * Use `onChanged` prop to call extra endpoints (#21593)
      
      * Tests for persist-refresh
      
      * Reorder requires
      
      * Use quartz for individual refreshing for safety
      
      switch to using one-off jobs to refresh individual tables. Required
      adding some job context so we know which type to run.
      
      Also, cleaned up the interface between ddl.interface and the
      implementations. The common behaviors of advancing persisted-info state,
      setting active, duration, etc are in a public `persist!` function which
      then calls to the multimethod `persist!*` function for just the
      individual action on the cached table.
      
      Still more work to be done:
      - do we want creating and deleting to be put into this type of system?
      Quite possible
      - we still don't know if a query is running against the cached table
      that can prevent dropping the table. Perhaps using some delay to give
      time for any running query to finish. I don't think we can easily solve
      this in general because another instance in the cluster could be
      querying against it and we don't have any quick pub/sub type of
      information sharing. DB writes would be quite heavy.
      - clean up the ddl.i/unpersist method in the same way we did refresh and
      persist. Not quite clear what to do about errors, return values, etc.
      
      * Update tests with more job-info in context
      
      * Fix URL type conflicts
      
      * Whoops get rid of our Thread/sleep test :)
      
      * Some tests for the new job-data, clean up task history saving
      
      * Fix database model persistence button states (#21636)
      
      * Use plain database instance on form
      
      * Fix DB model persistence toggle button state
      
      * Add common `getSetting` selector
      
      * Don't show caching button when turned off globally
      
      * Fix text issue
      
      * Move button from "Danger zone"
      
      * Fix unit test
      
      * Skip default setting update request for model persistence settings (#21669)
      
      * Add a way to skip default setting update request
      
      * Skip default setting update for persistence
      
      * Add changes for front end persistence
      
      - Order by refresh_begin descending
      - Add endpoint /persist/:persisted-info-id for fetching a single entry.
      
      * Move PersistInfo creation into interface function
      
      * Hide model cache monitoring page when caching is turned off (#21729)
      
      * Add persistence setting keys to `SettingName` type
      
      * Conditionally hide "Tools" from admin navigation
      
      * Conditionally hide caching Tools tab
      
      * Add route guard for Tools
      
      * Handle missing settings during init
      
      * Add route for fetching persistence by card-id
      
      * Wrangling persisted-info states
      
      Make quartz jobs handle any changes to database.
      Routes mark persisted-info state and potentially trigger jobs.
      Job read persisted-info state.
      
      Jobs
      
      - Prune
      -- deletes PersistedInfo `deleteable`
      -- deletes cache table
      
      - Refresh
      -- ignores `deletable`
      -- update PersistedInfo `refreshing`
      -- drop/create/populate cache table
      
      Routes
      
      card/x/persist
      - creates the PersistedInfo `creating`
      - trigger individual refresh
      
      card/x/unpersist
      - marks the PersistedInfo `deletable`
      
      database/x/unpersist
      - marks the PersistedInfos `deletable`
      - stops refresh job
      
      database/x/persist
      - starts refresh job
      
      /persist/enable
      - starts prune job
      
      /persist/disable
      - stops prune job
      - stops refresh jobs
      - trigger prune once
      
      * Save the definition on persist info
      
      This removes the columns and query_hash columns in favor of definition.
      
      This means, that if the persisted understanding of the model is
      different than the actual model during fetch source query we won't
      substitute.
      
      This makes sure we keep columns and datatypes in line.
      
      * Remove columns from api call
      
      * Add a cache section to model details sidebar (#21771)
      
      * Extract `ModelCacheRefreshJob` type
      
      * Add model cache section to sidebar
      
      * Use `ModelCacheRefreshStatus` type name
      
      * Add endpoint to fetch persistence info by model ID
      
      * Use new endpoint at QB
      
      * Use `CardId` from `metabase-types/api`
      
      * Remove console.log
      
      * Fix `getPersistedModelInfoByModelId` selector
      
      * Use `t` instead of `jt`
      
      * Provide seam for prune testing
      
      - Fix spelling of deletable
      
      * Include query hash on persisted_info
      
      we thought we could get away with just checking the definition but that
      is schema shaped. So if you changed a where clause we should invalidate
      but the definition would be the same (same table name, columns with
      types).
      
      * Put random hash in PersistedInfo test defaults
      
      * Fixing linters
      
      * Use new endpoint for model cache refresh modal (#21742)
      
      * Use new endpoint for cache status modal
      
      * Update refresh timestamps on refresh
      
      * Move migration to 44
      
      * Dispatch on initialized driver
      
      * Side effects get bangs!
      
      * batch hydrate :persisted on cards
      
      * bang on `models.persisted-info/make-ready!`
      
      * Clean up a doc string
      
      * Random fixes: docstrings, make private, etc
      
      * Bangs on side effects
      
      * Rename global setting to `persisted-models-enabled`
      
      felt awkward (enabled-persisted-models) and renamed to make it a bit
      more natural. If you are developing you need to set the new value to
      true and then your state will stay the same
      
      * Rename parameter for site-uuid-str for clarity
      
      * Lint cleanups
      
      interesting that the compojure one is needed for clj-kondo. But i guess
      it makes sense since there is a raw `GET` in `defendpoint`.
      
      * Docstring help
      
      * Unify type :type/DateTimeWithTZ and :type/DateTimeWithLocalTZ
      
      both are "TIMESTAMP WITH TIME ZONE". I had got an error and saw that the
      type was timestamptz so i used that. They are synonyms although it might
      require an extension.
      
      * Make our old ns linter happy
      
      Co-authored-by: default avatarAlexander Polyankin <alexander.polyankin@metabase.com>
      Co-authored-by: default avatarAnton Kulyk <kuliks.anton@gmail.com>
      Co-authored-by: default avatarCase Nelson <case@metabase.com>
      Unverified
      c504a12e
    • Braden Shepherdson's avatar
      One Clojure lint to rule them all (#22213) · 390d10e5
      Braden Shepherdson authored
      Background:
      
      We had two lint configs: lint-config.edn which was enfored by Github Actions, and
      .clj-kondo/config.edn, the default location used by editors.
      
      
      Problem:
      
      Patchy enforcement of the (larger) config in .clj-kondo/config.edn since not
      everyone uses an editor.
      
      Some new lint, like canonical namespace aliases (#21738, #19930) is not enforced
      automatically and has begun to bit-rot.
      
      Solution:
      
      Combine both configs, fix any outstanding lint, and update our tooling to use
      the unified config.
      
      Caveats:
      
      Anyone who has their local editor configured to use lint-config.edn will have
      their linting broken by this change.
      Unverified
      390d10e5
  9. Apr 29, 2022
    • Noah Moss's avatar
    • Noah Moss's avatar
      bump snowplow-java-tracker (#22267) · 357e0edc
      Noah Moss authored
      * bump snowplow
      
      * bufferSize -> batchSize
      
      * pin jackson-databind to 2.13.2.2
      Unverified
      357e0edc
    • dpsutton's avatar
      Handle locationlization errors from java.util.Locale (#22226) · 64b5c40c
      dpsutton authored
      
      * Handle locationlization errors from java.util.Locale
      
      Indonesian has code "id". But
      ```
      ;; java 11
      (str (java.util.Locale/forLanguageTag "id")) => "in"
      
      ;; java 17
      (str (java.util.Locale/forLanguageTag "id")) => "id"
      ```
      
      And this was bad. Because the frontend sends us the language to use:
      "id" for indonesian. We write that down. And then when we need to
      construct the index.html page with the correct translations, instead of
      using "id"-- the value we were given-- we construct a Locale from "id"
      and then get the str value of that which is "in". We don't have a in.edn
      file, there's no frontend in.json file, and moment.js has no in
      localization. And this all happened because we kept this value we all
      agreed upon in a java util class and then asked it for that value back.
      
      * Localization names have hyphens, files underscores
      
      * Added docstring
      
      * docstrings
      
      * Assert language id and POE header value in tests
      
      * localized json files don't exist until a build step
      
      Co-authored-by: default avatarAdam James <adam.vermeer2@gmail.com>
      Unverified
      64b5c40c
    • Noah Moss's avatar
      Fix XLSX column names for remapped cols (#22212) · 74c3fec5
      Noah Moss authored
      * fix column name for remapped fields
      
      * unskip repro
      
      * remove csv from cypress repro
      
      * fix cypress test
      
      * also check that :remapped_from is set on col metadata
      
      * revert import syntax
      Unverified
      74c3fec5
  10. Apr 28, 2022
  11. Apr 27, 2022
    • Noah Moss's avatar
      Migrate LDAP code to using `defenterprise` (#21687) · 2167b788
      Noah Moss authored
      * scaffolding, docstring & arg parsing
      
      * tweaks
      
      * registry and basic oss->ee dispatch
      
      * stubbed defenterprise-ee
      
      * full defenterprise-ee impl
      
      * fix build and tweak docstring
      
      * remove debug code
      
      * basic tests
      
      * use defonce
      
      * support for :any feature
      
      * schema support
      
      * fix schema test
      
      * switch to using format
      
      * support for schema on return val
      
      * tweak macro
      
      * move schema validation to submacros
      
      * do ee fn resolution at runtime
      
      * do conditionals at macro-expansion time
      
      * adjust semantics & refactor
      
      * fix test
      
      * add memoization for ee resolution
      
      * fix tests
      
      * emit register-mapping! call to avoid eval
      
      * call starts-with on symbol
      
      * remove schema support
      
      * use spec for arg parsing, generate fns, and remove :error fallback option
      
      * clean ns
      
      * change schema alias to schema
      
      * add :arglists meta and small refactor
      
      * refactor
      
      * move EE logic to fn
      
      * validate that the correct args are passed in EE/OSS namespaces, and make :feature required
      
      * remove logic for handling nil :feature
      
      * fix lint errors
      
      * fix ee macros
      
      * defenterprise-schema macro & tests
      
      * clean ns
      
      * propogate metadata
      
      * don't memoize ee resolution in dev to make debugging easier
      
      * dont do ee fn resolution during macroexpansion
      
      * migrate LDAP functions to defenterprise
      
      * lint
      
      * require ee ns separately so that EE settings are available
      
      * revert change to function mocked in tests
      
      * add back code lost in merge
      
      * clean ns
      
      * clean ns again
      
      * minor refactor and cleanup
      
      * major refactor
      
      * fix defenterprise-schema syntax
      
      * fix import
      
      * major refactor
      
      * update docstrings
      
      * more docstring tweaks
      
      * try to fix lint error on ldap PR
      
      * add lint-as mappings for defenterprise and defenterprise-schema to clj-kondo config
      
      * clj-kondo/ignore use of spec alias
      
      * use destructuring in arg lists since it's now supported
      
      * add docstring to def emitted by macro, and some formatting tweaks
      Unverified
      2167b788
    • Case Nelson's avatar
      Validate datasets are found when checking bigquery (#22144) · f4e49dbd
      Case Nelson authored
      * Validate datasets are found when checking bigquery
      
      Fixes #19709
      
      * Address PR feedback
      
      Made a general mechanism to pass expected messages to users in
      api/database via ex-info. This allows us to suppress logging for
      "unexceptional" exceptions that one can expect to hit while setting up
      drivers.
      
      * Only validate when filters are set
      
      Also removed the dataset list from the exception as it's not surfaced to
      users.
      Unverified
      f4e49dbd
    • Noah Moss's avatar
      `defenterprise` macro (#21672) · 8f9b5957
      Noah Moss authored
      * scaffolding, docstring & arg parsing
      
      * tweaks
      
      * registry and basic oss->ee dispatch
      
      * stubbed defenterprise-ee
      
      * full defenterprise-ee impl
      
      * fix build and tweak docstring
      
      * remove debug code
      
      * basic tests
      
      * use defonce
      
      * support for :any feature
      
      * schema support
      
      * fix schema test
      
      * switch to using format
      
      * support for schema on return val
      
      * tweak macro
      
      * move schema validation to submacros
      
      * do ee fn resolution at runtime
      
      * do conditionals at macro-expansion time
      
      * adjust semantics & refactor
      
      * fix test
      
      * add memoization for ee resolution
      
      * fix tests
      
      * emit register-mapping! call to avoid eval
      
      * call starts-with on symbol
      
      * remove schema support
      
      * use spec for arg parsing, generate fns, and remove :error fallback option
      
      * clean ns
      
      * change schema alias to schema
      
      * add :arglists meta and small refactor
      
      * refactor
      
      * move EE logic to fn
      
      * validate that the correct args are passed in EE/OSS namespaces, and make :feature required
      
      * remove logic for handling nil :feature
      
      * fix lint errors
      
      * fix ee macros
      
      * defenterprise-schema macro & tests
      
      * clean ns
      
      * propogate metadata
      
      * don't memoize ee resolution in dev to make debugging easier
      
      * dont do ee fn resolution during macroexpansion
      
      * minor refactor and cleanup
      
      * major refactor
      
      * update docstrings
      
      * more docstring tweaks
      
      * try to fix lint error on ldap PR
      
      * use dan's suggestion
      Unverified
      8f9b5957
    • Bryan Maass's avatar
    • Case Nelson's avatar
  12. Apr 26, 2022
    • Nick Fitzpatrick's avatar
      Gh 21549 snippet permissions (#21927) · f83386d3
      Nick Fitzpatrick authored
      
      * Remove right click option when unable to write to collections
      
      * Permissions for snippets root requires :namespace key on the collection
      
      * pr feedback
      
      Co-authored-by: default avatarCase Nelson <case@metabase.com>
      Unverified
      f83386d3
    • dpsutton's avatar
      Enable :unsorted-required-namespaces and lint cleanups (#22087) · b519dd8a
      dpsutton authored
      ```
      ❯ clj-kondo --lint src:shared/src
      src/metabase/api/common/validation.clj:2:33: warning: Inconsistent alias. Expected str instead of string.
      src/metabase/api/database.clj:664:5: warning: Redundant let expression.
      src/metabase/api/table.clj:18:45: warning: Inconsistent alias. Expected field-values instead of sync.field-values.
      src/metabase/cmd.clj:123:5: warning: unused binding docs
      src/metabase/db/data_migrations.clj:15:52: warning: Inconsistent alias. Expected perms-group instead of group.
      src/metabase/db/data_migrations.clj:55:15: warning: #'metabase.db.data-migrations/data-migrations is deprecated
      src/metabase/db/data_migrations.clj:62:24: warning: #'metabase.db.data-migrations/data-migrations is deprecated
      src/metabase/db/data_migrations.clj:63:7: warning: #'metabase.db.data-migrations/run-migration-if-needed! is deprecated
      src/metabase/db/data_migrations.clj:154:1: warning: #'metabase.db.data-migrations/defmigration is deprecated
      src/metabase/db/data_migrations.clj:201:1: warning: #'metabase.db.data-migrations/defmigration is deprecated
      src/metabase/db/setup.clj:59:5: warning: Missing catch or finally in try
      src/metabase/driver/postgres.clj:18:63: warning: Inconsistent alias. Expected sql-jdbc.describe-table instead of sql-jdbc.sync.describe-table.
      src/metabase/driver/postgres.clj:304:9: warning: unused binding nfc-path
      src/metabase/driver/postgres.clj:323:60: warning: unused binding fields-fields
      src/metabase/driver/sql_jdbc/sync/describe_table.clj:253:39: error: Expected: set or nil, received: seq.
      src/metabase/driver/sql_jdbc/sync/describe_table.clj:254:39: error: Expected: set or nil, received: seq.
      src/metabase/driver/sql_jdbc/sync/describe_table.clj:333:11: warning: unused binding map-inner
      src/metabase/models/secret.clj:248:23: warning: use :else as the catch-all test expression in cond
      src/metabase/pulse/render/body.clj:216:64: warning: unused binding cols
      src/metabase/task/refresh_slack_channel_user_cache.clj:52:25: error: clojurewerkz.quartzite.schedule.simple/with-interval-in-seconds is called with 1 arg but expects 2
      linting took 5988ms, errors: 3, warnings: 17
      ```
      
      ```
      ❯ clj-kondo --lint src:shared/src
      src/metabase/task/refresh_slack_channel_user_cache.clj:52:25: error: clojurewerkz.quartzite.schedule.simple/with-interval-in-seconds is called with 1 arg but expects 2
      linting took 5270ms, errors: 1, warnings: 0
      ```
      
      Want to call out a few things:
      
      The following all related to the new paths into json feature. Wanted
      howon to verify that nothing got dropped with these unused bindings.
      
      ```
      src/metabase/driver/postgres.clj:304:9: warning: unused binding nfc-path
      src/metabase/driver/postgres.clj:323:60: warning: unused binding fields-fields
      src/metabase/driver/sql_jdbc/sync/describe_table.clj:253:39: error: Expected: set or nil, received: seq.
      src/metabase/driver/sql_jdbc/sync/describe_table.clj:254:39: error: Expected: set or nil, received: seq.
      src/metabase/driver/sql_jdbc/sync/describe_table.clj:333:11: warning: unused binding map-inner
      ```
      Unverified
      b519dd8a
    • Bryan Maass's avatar
      Better logs for rotate encryption key + better command error messages (#21901) · 58a03477
      Bryan Maass authored
      * explain what is going on during encryption key rotation
      * better argument error handling for commands
      Unverified
      58a03477
    • metamben's avatar
      Deserialize dashboard Starting From/Exclude datetime filters (#22035) · 59f97880
      metamben authored
      Deserialize dashboard Starting From/Exclude datetime filters
      Unverified
      59f97880
  13. Apr 25, 2022
  14. Apr 23, 2022
  15. Apr 22, 2022
    • Cam Saul's avatar
      Handle March 31st + 3 months (June 31st?) for Oracle (#21841) · b9cedccc
      Cam Saul authored
      * Handle March 31st + 3 months for Oracle (#10072)
      
      * Optimize out some casting in Oracle
      
      * rx util support varargs inside `opt`
      
      * hx/ math operators like + and - should propagate type information
      
      * Some dox tweaks
      
      * Fix SQLite busted behavior
      
      * Avoid unneeded casting in Vertica when adding temporal intervals
      
      * Lint error fixes
      
      * BigQuery fix for #21969
      
      * Add testing context for tests for #21968 and #21971
      Unverified
      b9cedccc
Loading