Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Jan 07, 2022
  2. Dec 28, 2021
    • Cam Saul's avatar
      Database-local Settings (#19399) · b9bee5dc
      Cam Saul authored
      * Rename setting/get and setting/all; GeoJSON via env var tests
      
      * Fix typo (thanks @noahmoss)
      
      * Support Database-local Settings in metabase.models.setting itself
      
      * Rework Settings code so it can handle possibly-already-deserialized values
      
      * Database-local Settings
      
      * Remove empty part of docstring
      
      * Appease linters
      
      * Update dox again
      
      * Use text.type for the new column
      
      * Test fixes :wrench:
      
      * Test fix :wrench:
      
      * Test fix :wrench:
      
      * All negative integer values when setting :integer Setting with a string
      Unverified
      b9bee5dc
  3. Dec 20, 2021
    • Jeff Evans's avatar
      Fix Liquibase checksum failure (#19429) · a4b1af29
      Jeff Evans authored
      Set validCheckSum values for id 313
      Unverified
      a4b1af29
    • Jeff Evans's avatar
      Fix file upload of secret values (#19398) · b8b1b28a
      Jeff Evans authored
      * Fix file upload of secret values
      
      Fix logic in `db-details-client->server` to properly consider the `-options` suffixed connection property from the client version, to determine the treatment to be applied (i.e. base64 decode)
      
      Change logic in `db-details-client->server` to `assoc` nil rather than `dissoc` unused keywords (so they override the saved db details on merge)
      
      Update `driver.u/database->driver*` so that the engine loaded from the database instance is *always* a keyword
      
      Update `metabase.test.data.oracle/connection-details` to pass the property vals through the client->server translation layer
      
      Change `creator_id` in `secret` model to be nullable (since, when creating test DBs, the API current user is deliberately set to nil)
      
      Switch the logic in Oracle SSL test to iterate on the file-path and uploaded variants to the test itself to get around weird CircleCI/environment issues
      
      Use `rds_root_ca_truststore.jks` for the test instead since we don't need the full cacerts for this
      Unverified
      b8b1b28a
  4. Dec 06, 2021
  5. Nov 23, 2021
  6. Nov 12, 2021
  7. Nov 11, 2021
    • Jeff Evans's avatar
      Change query_cache.results column to correct blob type (#18848) · 6a40ee38
      Jeff Evans authored
      * Change query_cache.results column to correct blob type
      
      Add Liquibase changeSet to modifyDataType of query_cache.results to ${blob.type} (an idempotent operation on most DBs)
      
      Add schema migrations test to confirm, which simulates the broken app DB state, then runs the migration, then confirms the fixed type
      Unverified
      6a40ee38
    • dpsutton's avatar
      Nest qbnewb under /modal to allow for other modals (#18926) · 11d9bd55
      dpsutton authored
      * Nest qbnewb under /modal to allow for other modals
      
      move the api from api/user/:id/qbnewb -> api/user/:id/modal/qbnewb to
      allow for the upcoming new dismissable modal
      
      * Add additional datasetnewb route
      
      staying with the same naming scheme as qbnewb. the db field is
      is_datasetnewb, the route just takes datasetnewb.
      
      I'm not wild about the default is true, set to false, and the naming
      scheme. But will be easier to migrate and easier for FE to use if we
      continue the defaults/logic etc.
      
      * Remark on db change in migration file
      Unverified
      11d9bd55
  8. Nov 04, 2021
    • Jeff Evans's avatar
      Change all active TEXT columns in MySQL app DB to LONGTEXT (#18749) · b4610877
      Jeff Evans authored
      
      * Change all active TEXT columns in MySQL app DB to LONGTEXT
      
      Add new text.type Liquibase property to dynamically select the correct text type to use, per DB (which is LONGTEXT for MySQL), very similar to the existing blob.type one
      
      Adding new migrations that update all existing app DB TEXT columns to LONGTEXT columns, only for mariadb/mysql
      
      Update migrations linter to ensure no new text types get added, via a new predicate that searches for text types being added
      
      Do the same logic for "blob" now (should be "${blob.type}"), update rule and test
      
      Add test for migrations, that checks the new types, for all types in scope for this PR, to ensure that they have all been changed to their expected DB-specific type (either LONGTEXT for MySQL or TEXT/CLOB for others)
      
      Updating a couple migrations to remove special casing that was originally done only for MySQL to simply make them universally "${text.type}", in order to unify behavior and reduce surprises going forward
      
      Co-authored-by: default avatarCam Saul <github@camsaul.com>
      Unverified
      b4610877
  9. Nov 03, 2021
    • Cam Saul's avatar
      Add NOT NULL constraint to Card.database_id; attempt to set database_id if possible (#18472) · 83de3e86
      Cam Saul authored
      * Drop long-unused Table.entity_name column
      
      * Add NOT NULL constraint to Card.database_id
      
      * Test fix :wrench:
      
      * Test fix :wrench:
      
      * Fix migration
      
      * Add note about H2 shell to deps.edn
      
      * Add new SQL migration to attempt to set database_id when unset
      
      * Remove data migration that is now done in Liquibase land
      
      * Oops, '$.database', not '$.database_id'
      
      * Parse ints as signed rather than unsigned just to be safe (they *might* be -1337 if they're really broken)
      
      * Don't run for H2
      
      * Update comment
      
      * Fix migration indentation
      
      * Clean namespace
      
      * Use new migration number.
      
      * Use the new migration numbers
      
      * Adopt new migration numbering scheme
      
      * Fix comments
      
      * Fix MySQL + MariaDB insanity
      
      * Fix ID range validation
      
      * Actually 382 is the last legacy ID
      
      * Improved validation and tests
      
      * Adopt the new-new migration ID format.
      
      * Test fixes :wrench:
      
      * Fix merge
      
      * Simplify precondition
      Unverified
      83de3e86
    • Cam Saul's avatar
      Add index to ModerationReview moderated_item_type + moderated_item_id (#18799) · 7d614623
      Cam Saul authored
      * Revert changes from Jeff's PR
      
      * Add index to ModerationReview moderated_item_type + moderated_item_id
      
      * Add to 0.41.2 instead
      
      * Require explicit index name for createIndex
      
      * Move 41.2 migrations to after the 41.0 migrations
      
      * Adopt new migration numbering scheme
      
      * Fix comments
      
      * Fix MySQL + MariaDB insanity
      
      * Fix ID range validation
      
      * Actually 382 is the last legacy ID
      
      * Improved validation and tests
      
      * Adopt the new-new migration ID format.
      
      * Test fixes :wrench:
      Unverified
      7d614623
    • Cam Saul's avatar
      Adopt new migration numbering scheme (#18821) · 2fee04b7
      Cam Saul authored
      * Adopt new migration numbering scheme
      
      * Fix comments
      
      * Fix MySQL + MariaDB insanity
      
      * Fix ID range validation
      
      * Actually 382 is the last legacy ID
      
      * Improved validation and tests
      
      * Adopt the new-new migration ID format.
      
      * Test fixes :wrench:
      Unverified
      2fee04b7
  10. Nov 02, 2021
  11. Oct 26, 2021
    • dpsutton's avatar
      Dataset schema (#18629) · 977a1765
      dpsutton authored
      * Add dataset flag to Card
      
      * Set dataset viz type to table
      
      * Include datasets in collection children
      
      * Ensure can write as dataset, include more tests
      Unverified
      977a1765
  12. Oct 21, 2021
    • Jeff Evans's avatar
      Secrets :closed_lock_with_key: PR 2 - Add secret model (#17649) · 0d7786a1
      Jeff Evans authored
      Add secret model
      
      Add Liquibase migration for secret table
      
      Implement bare bones model namespace for secret
      
      Add simple test to ensure secret values are stored and retrieved under an encryption key (or not)
      
      Add :creator_id to secret model to capture the user who created this secret instance/version
      
      Support updating secret values as part of encryption key rotation
      
      Add test assertions for secret values in the `rotate-encryption-key!-test`
      Unverified
      0d7786a1
  13. Oct 20, 2021
  14. Oct 19, 2021
    • Dennis Schridde's avatar
      Fix precondition of change set 97 (#16095) · 2d88ae48
      Dennis Schridde authored
      * Fix precondition of change set 97
      
      Without the `type` and with the space Liquibase is unable to parse this
      precondition.
      
      During `lein test` it outputs:
      ```
      [clojure-agent-send-off-pool-0] DEBUG liquibase.changelog - Running Changeset:migrations/000_migrations.yaml::97::senior
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - Changeset migrations/000_migrations.yaml::97::senior
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - Added 0.32.0
      [clojure-agent-send-off-pool-0] INFO  liquibase.changelog - Marking ChangeSet: migrations/000_migrations.yaml::97::senior ran despite precondition failure due to onFail='MARK_RAN':
                liquibase.yaml : DBMS Precondition failed: expected null, got h2
      
      [clojure-agent-send-off-pool-0] DEBUG liquibase.changelog - Skipping ChangeSet: migrations/000_migrations.yaml::97::senior
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - Executing with the 'jdbc' executor
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - 1 row(s) affected
      ```
      
      After this change the output changes to:
      ```
      [clojure-agent-send-off-pool-0] DEBUG liquibase.changelog - Running Changeset:migrations/000_migrations.yaml::97::senior
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - Changeset migrations/000_migrations.yaml::97::senior
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - Added 0.32.0
      [clojure-agent-send-off-pool-0] INFO  liquibase.changelog - Marking ChangeSet: migrations/000_migrations.yaml::97::senior ran despite precondition failure due to onFail='MARK_RAN':
                liquibase.yaml : DBMS Precondition failed: expected mysql,mariadb, got h2
      
      [clojure-agent-send-off-pool-0] DEBUG liquibase.changelog - Skipping ChangeSet: migrations/000_migrations.yaml::97::senior
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - Executing with the 'jdbc' executor
      [clojure-agent-send-off-pool-0] DEBUG liquibase.executor - 1 row(s) affected
      ```
      
      For documentation of the syntax cf.
       https://docs.liquibase.com/concepts/advanced/preconditions.html
      
      
      
      * Extend migration linter to check dbms preconditions
      
      * Also validate the `type` field of the `dbms` precondition
      
      Co-authored-by: default avatardpsutton <dan@dpsutton.com>
      Unverified
      2d88ae48
  15. Sep 28, 2021
    • Howon Lee's avatar
      Audit cache controls 2 (#18046) · d7898af8
      Howon Lee authored
      Cache controls all landed but is lacking in the audit capacity wanted for in the notion doc. This PR adds that audit capability and by the by changes the ViewLog model in order to be able to deliver on the question of whether ViewLogs were cache hits or not.
      Unverified
      d7898af8
  16. Sep 09, 2021
  17. Aug 24, 2021
  18. Aug 03, 2021
    • Dalton's avatar
      Add the ability to verify/unverify questions (#17030) · b7fa2f34
      Dalton authored
      * rmv old bucm icons and remove verified fill color
      
      * add moderation action section to sidebar
      
      * add moderation review icon to the saved question header button
      
      * hide moderation section when is not a moderator
      
      * add UI for ModerationReviewBanner
      
      * Backend for moderation-review
      
      - create table moderation_review. Same as before but also has a
        "most_recent" boolean flag for the most recent moderation for easy
        lookup
      - POST /moderation-review/ . Status can be "verified" or nil
      - must be an admin to post
      - No PUT or edit route yet. Not sure if this is even
        necessary. _MAYBE_ to edit the text, but certainly not for the
        status, ids, etc. If there's to be history, let's build some history
      - Ensure we never have more than 10 reviews. Adding a new review will
        delete the older ones, mark all old ones as not `most_recent`, and
        add the newest one as `most_recent true`
      - Ensure the card actually exists before creating the mod review
      - Since admin only at this time, don't need to check moderate
        permission or view permission
      - When hydrating ensure reviews are ordered by id desc. Should mimic
        the created_at desc
      
      * fix moderation review banner tooltip offset
      
      * disable verification button when already verified
      
      * rmv iconOnly prop because it seems to do nothing
      
      * update getLatestModerationReview to rely on most_recent boolean
      
      * Return 400 on invalid status to post /moderation-review
      
      the schema was using keywords on the left hand side rather than the
      symbols. Required a change to the docstring generator, when it made a
      docstring for enums, it would call (sort (:vs enum)) and need to
      string em.
      
      * Add ModerationReview model to models.clj and copy infra
      
      * hydrate moderation reviews on cards
      
      * clean up + wire up to BE + ensure mod buttons don't show for normal users
      
      * rmv unused moderation redux logic from QuestionDetailsSidebarPanel
      
      * finish writing unit tests for FE
      
      * ensure getIconForReview returns an object
      
      * enable/disable verify button tooltip when unverified/verified
      
      * add e2e tests
      
      * fix tests
      
      * styling tweaks
      
      * more styling on moderationReviewBanner
      
      * add function for abbreviated timestamp
      
      * increase fontsize of timestamp back to 12
      
      * fix tooltip offset
      
      * ensure custom locale is separate from 'en' and not used for other languages
      
      * Deletion moderation reviews when deleting cards
      
      i had actually thought this was a much larger problem. But it turns
      out we almost never delete cards (thanks comment!). And so we won't
      really generate a lot of garbage.
      
      I was worried that since we aren't using actual foreign keys but just
      `moderated_item_type "card"` and `moderated_item_id 2` we would have
      deleted cards with these moderation reviews but that is not the case
      as the cards aren't deleted.
      
      * hide verify disabled button when a question is verified
      
      * update test to use queryByTestId
      
      * Hydrate moderation reviews on cards on ordered cards
      
      * Handle mysql's lack of offset functionality
      
      mysql cannot handle just a `offset` clause, it also needs a limit
      
      clause grammar from
      https://dev.mysql.com/doc/refman/8.0/en/select.html:
      
      [LIMIT {[offset,] row_count | row_count OFFSET offset}]
      
      select id, name from metabase_field offset 5;         -- errors
      select id, name from metabase_field limit 2 offset 5; -- works
      
      Since our numbers are so small here there is no worry and just do the
      offset in memory rather than jump through hoops for different dbs.
      
      * Batch hydrate moderation reviews
      
      * Don't let /api/user/:userId failure conceal moderation banner
      
      * fix moderation cy tests
      
      * work around possible bug in toucan hydration
      
      dashboards hydrate ordered cards
      (hydrate [:ordered_cards [:card :moderation_reviews] :series])
      
      Ordered_cards are dashboard_cards which have an optional card_id. But
      toucan hydration doesn't filter out the nils as they go down. It seems
      toucan returns a nil card, and then when hydrating the
      moderation_review, passes the collection of all "cards" including the
      nil ones into the hydration function for moderation_reviews
      
      This feels like a bug to me
      
      * Cleanup moderation warnings
      
      * Docstring in moderation review
      
      * include hoisted moderated_status on cards in collections api
      
      * Expect unverified in test :wrench:
      
      
      
      Co-authored-by: default avatardan sutton <dan@dpsutton.com>
      Co-authored-by: default avatarMaz Ameli <maz@metabase.com>
      Co-authored-by: default avataralxnddr <alxnddr@gmail.com>
      Unverified
      b7fa2f34
  19. Jun 08, 2021
    • Howon Lee's avatar
      Friendly names bugfix 2 (#16482) · c00ffb45
      Howon Lee authored
      Friendly names bugfix 1 was a noop because of a typo in the WHERE clause of the DB migration. This one has been manually tested to go and sync on all 4 supported MB db's.
      Unverified
      c00ffb45
  20. Jun 07, 2021
  21. Jun 03, 2021
  22. Jun 02, 2021
    • dpsutton's avatar
      Authority Level on collections (formerly Typed collections) (#16191) · ad753b18
      dpsutton authored
      * Add type on collection
      
      * Search with collection type
      
      * Cleanup bounded heap accumulator
      
      * Make search conditionally bump official collection items
      
      * collection api and tests
      
      * Put collection type onto hydrated cards on dashboards
      
      * Move official collection type scoring into ee codebase
      
      * ensure ee and oss agree when not official collection
      
      * Mark Collections tree with type
      
      * Remove unnecessary `and`s when no check for enhancements
      
      * Tests for setting collection tree type
      
      * Include hydration for collection type on dashboards
      
      * Make sure created test collections are cleaned up
      
      * Cleanup tests, don't search on collection_type
      
      looks for all text fields and searches the term against that. official
      would bring back all official types. no bueno
      
      * Docstring on protocol and don't shadow comparator
      
      * update to new ee impl var passing style
      
      * Collection model ensures no type change on personal collection
      
      * Check for personal collection when updating type
      
      model checks for personal collection and rejects in the update but
      doesn't check for children of personal collection.
      
      * Update checking for unchangeable properties of a personal collection
      
      * Cleanup: type collection tree, batched hydration, combine error checks
      
      * Cleanup
      
      * move bounded-heap accumulator to utils
      
      * switch to test.check testing
      
      * Bad test just to see what our CI will report
      
      * remove purposeful failing test (was for CI)
      
      * collection.type -> collection.authority_level
      
      * Test the actual ee scoring impl, not the wrapped one
      
      locally i had enhancements enabled so the tests were passing, but in
      CI it was failing as it did not have enhancements enabled
      Unverified
      ad753b18
  23. May 10, 2021
  24. Apr 22, 2021
    • dpsutton's avatar
      Rename collection revision (#15616) · a88351d7
      dpsutton authored
      * Table collection_revision -> collection_permission_graph_revision
      
      in anticipation of getting collection revisions in, need the table
      name as the previous table named this was tracking changes to the
      permission graph
      
      * rename collection permission graph revision alias
      
      * Remove unnecessary or for precondition
      
      * Rename sequence in sql
      
      liquibase has a renameSequence changeset but it doesn't work for h2
      obviously. However, it doesn't allow this constraint to be honored
      with a precondition. Just having this changeset on h2 blows up
      Unverified
      a88351d7
  25. Apr 13, 2021
    • dpsutton's avatar
      Remove unixtimestamp type (#15533) · 4910a3e2
      dpsutton authored
      * Remove type/UNIX* and type/ISO8601* from frontend
      
      * Set effective-type and coercion strategy when syncing
      
      these values will most likely only be available when using tests, as
      metadata is very unlikely to have this. As far as I can tell the
      toucannery apparatus is the only bit that has this. Its quite
      artificial. I'm not sure if this is a good idea.
      
      * :type/UNIXTimestampSeconds and type/ISO8601DateTimeString out of type
      
      remove the coercions from the type hierarchy. This brought up a
      strange issue that has been present for a while: all liquidbase
      migrations run and then all data migrations run. They are not
      interleaved. This allows for the following scenario:
      
      - data migration migrates all X values to Y in version 26
      - liquibase migration migrates all Y values to Z in version 28
      
      But these are not run in version order, but all liquibase migrations
      are run and then all data migrations are run. If you were on an old
      version for a while, you could end up with Y values even though the
      liquibase migration which comes "after" the data migration turns all Y
      values into Z values.
      
      This impacts this change because a data migration in this way:
      - liquibase changesets 287, 288, 289, and 290 remove all 'type/UNIX*'
      and 'type/ISO8601*' semantic types. These were in 0.39
      - data migration `migrate-field-types` added in 0.20.0 was looking for
      special_type's of "timestamp_milliseconds" and migrating them to
      "type/UNIXTimestampMilliseconds".
      
      Since the liquibase runs before the migrate-field-types, it's possible
      for a 'type/UNIX*' semantic type to reappear. And since these were
      removed from the type system it would fail validation and blow up. In
      this case it actually can't happen since the field its attempting to
      migrate (special_type) no longer exists. But since the migrations are
      not interleaved this problem still exists.
      
      * whatever prettier
      
      * Appease the machines
      
      * Restore the unix and iso semantic types to hierarchy
      
      migration tests still use these
      `(impl/test-migrations [283 296] [migrate!] ...)`
      
      A few things in tension: the field requires valid semantic_types,
      these _were_ valid semantic_types until v39.
      
      * Remove old "coercion" semantic types
      
      * Migrate the v20 version semantic types for unix epoch
      
      * Time travelling migrations
      
      these target v20 and v29 which at the time they existed had a column
      special type not semantic type. But these run after all of the schema
      migrations, not interleaved, so they will have a semantic_type not
      special_type when they actually run even though they target v20 or v29
      data. strange world
      
      * add migration's new checksum
      Unverified
      4910a3e2
  26. Apr 09, 2021
  27. Apr 06, 2021
    • Tim Macdonald's avatar
      Dashboard Subscription Filtering (#15404) · ce41deaf
      Tim Macdonald authored
      
      * Backend for dashboard sub filters, including EE implementation
      
      * Fix pulse/update-notification! to respect parameters
      
      * Add some tests
      
      * add FE EE parameter section to dashboard subscriptions
      
      * fix improper filter counting for label
      
      * hide parameters section when dashboard has none
      
      * fix thrown errorin pulses list sidebar
      
      * fix linting error
      
      * add cypress tests
      
      * rmv describe.only from test
      
      * refactor FE code a little
      
      * Merge pulse/dashboard parameters correctly for dashboard subs
      
      * fix issues with getting list of active pulse parameters
      
      * update formatting of multi value parameters
      
      * fix cypress test
      
      * tweak pulse details styling
      
      * move ParametersSection to enterprise tree
      
      Co-authored-by: default avatarDalton Johnson <daltojohnso@users.noreply.github.com>
      Unverified
      ce41deaf
  28. Mar 19, 2021
    • Cam Saul's avatar
      Record login history (#15129) · 3a5eface
      Cam Saul authored
      * Record login history [WIP]
      
      * WIP
      
      * Add API endpoint and ugly React component
      
      * Fix fn name
      
      * Lookin good
      
      * Test fixes :wrench:
      
      * Prettier
      
      * Test fix
      
      * Test fixes :wrench:
      
      * Better login history info
      
      * Less ugly login history & tests
      
      * Update migration numbers
      
      * Deduplicate https-request? util function from metabase.server.middleware .session & .util
      
      * Refactor -- new Ring request utils namespace
      
      * Test fixes :wrench:
      
      * Reorganize stuff a bit; API endpoint tests
      
      * Tests for login/logout recording history
      
      * Test fix :wrench:
      
      * Test fixes :wrench:
      
      * Test fix :wrench:
      
      * Copy FE test fixture DB so it stops getting changed every time we run tests
      
      * Test fixes :wrench:
      
      
      
      * layout and code style tweaks
      
      * Fix ns sorting
      
      Co-authored-by: default avatarKyle Doherty <5248953+kdoh@users.noreply.github.com>
      Unverified
      3a5eface
    • dpsutton's avatar
      Fix typo in semantic migration (#15218) · 3099c35a
      dpsutton authored
      * Fix typo in semantic migration
      
      * Update semantic migrations for mysql compatibility
      
      mysql updates as it goes, not in a row level transaction. So
      
      ```sql
      update a = null,
             b = (case a ...)
      where a is not null
      ```
      
      mysql will use the new null value of a rather than the pre-udpate
      value of a whereas pg and h2 will do row level transaction so you are
      considering the pre-updated value of a.
      
      Solution is to update these migrations to leave semantic type alone
      and then one final set semantic_type = null where coercion_strategy is
      not null. This is sensical since these two properties used to both be
      in semantic_type, so if we extracted a coercion it could not have had
      a meaningful semantic_type before.
      Unverified
      3099c35a
  29. Mar 15, 2021
    • dpsutton's avatar
      Semantic types 2 effective type (#15022) · 6b8ddc84
      dpsutton authored
      
      * First pass using coercions
      
      * Coercions
      
      * Handle effective_type coercion_strategy in test data sets
      
      * special-type -> semantic type in sample db
      
      ```clojure
      user> (def config (metabase.db.spec/h2 {:db (str "/Users/dan/projects/clojure/metabase/resources/sample-dataset.db"
                                                       ";UNDO_LOG=0;CACHE_SIZE=131072;QUERY_CACHE_SIZE=128;COMPRESS=TRUE;"
                                                       "MULTI_THREADED=TRUE;MVCC=TRUE;DEFRAG_ALWAYS=TRUE;MAX_COMPACT_TIME=5000;"
                                                       "ANALYZE_AUTO=100")}))
      user> (jdbc/execute! config ["UPDATE _metabase_metadata
                              SET keypath = 'PEOPLE.ZIP.semantic_type'
                              WHERE keypath = 'PEOPLE.ZIP.special_type'" ])
      [1]
      user> (jdbc/execute! config ["UPDATE _metabase_metadata
                              SET keypath = 'REVIEWS.BODY.semantic_type'
                              WHERE keypath = 'REVIEWS.BODY.special_type'" ])
      [1]
      ```
      
      * Correct mismatch in validation preventing sync
      
      * fixing up alternative date tests
      
      * More passing tests
      
      * Tests for values, nested queries, fetch metadata
      
      * tests
      
      * tests passing
      
      * Fixup mongo qp for coercions
      
      locally i have some failing tests that are off by 1 errors:
      
      Fail in compile-time-interval-test
      
      [36m:mongo[0m Make sure time-intervals work the way they're supposed to. [:time-interval $date -4 :month] should give us something like Oct 01 2020 - Feb 01 2021 if today is Feb 17 2021
      
      expected: [{$match {$and [{:$expr {$gte [$date {:$dateFromString {:dateString 2020-10-01T00:00Z}}]}} {:$expr {$lt [$date {:$dateFromString {:dateString 2021-02-01T00:00Z}}]}}]}} {$group {_id {date~~~day {:$let {:vars {:parts {:$dateToParts {:date $date}}}, :in {:$dateFromParts {:year $$parts.year, :month $$parts.month, :day $$parts.day}}}}}}} {$sort {_id 1}} {$project {_id false, date~~~day $_id.date~~~day}} {$sort {date~~~day 1}} {$limit 1048576}]
      
        actual: [{"$match"
                  {"$and"
                   [{:$expr {"$gte" ["$date" {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
                    {:$expr {"$lt" ["$date" {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}
                 {"$group"
                  {"_id"
                   {"date~~~day"
                    {:$let
                     {:vars {:parts {:$dateToParts {:date "$date"}}},
                      :in {:$dateFromParts {:year "$$parts.year", :month "$$parts.month", :day "$$parts.day"}}}}}}}
                 {"$sort" {"_id" 1}}
                 {"$project" {"_id" false, "date~~~day" "$_id.date~~~day"}}
                 {"$sort" {"date~~~day" 1}}
                 {"$limit" 1048576}]
          diff: - [{"$match"
                    {"$and"
                     [{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-10-01T00:00Z"}}]}}
                      {:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-02-01T00:00Z"}}]}}]}}]
                + [{"$match"
                    {"$and"
                     [{:$expr {"$gte" [nil {:$dateFromString {:dateString "2020-11-01T00:00Z"}}]}}
                      {:$expr {"$lt" [nil {:$dateFromString {:dateString "2021-03-01T00:00Z"}}]}}]}}]
      
      * ee fixes
      
      * UI to set coercion type
      
      * Don't need to populate effective-type here
      
      it actually has knock on effects:
      - does more work now as almost every field has an update to do in
      `add-extra-metadata`
      - we have databases that have state that we don't create. druid for
      example has stuff to mimic the dataset in tqpt/with-flattened-dbdef on
      checkins but we don't actually create this. And our dbdef has a field
      called "date" that is not present in the druid db, so if we attempt to
      add metadata it fails and kills the rest of the metadata that we add.
      - tests need this metadata to be present and the error causes field
      visibilities (for example) to not be set
      
      * Docstrings on shared lib
      
      * Add effective and coercion to redshift expectations
      
      * Fixup google analytics
      
      * Derecordize instead of recordize the expectation
      
      object details didn't work out well here. they added way more stuff
      from the db than what is flowing through here.
      
      ```clojure
        actual: {:field
                 {:name "DATE",
                  :parent_id nil,
                  :table_id 69,
                  :base_type :type/Date,
                  :effective_type :type/Date,
                  :coercion_strategy nil,
                  :semantic_type nil},
                 :value {:type :date/all-options, :value "past5days"}}
          diff: - {:field
                   {:description nil,
                    :database_type "VARCHAR",
                    :fingerprint_version 0,
                    :has_field_values nil,
                    :settings nil,
                    :caveats nil,
                    :fk_target_field_id nil,
                    :custom_position 0,
                    :active true,
                    :last_analyzed nil,
                    :position 1,
                    :visibility_type :normal,
                    :preview_display true,
                    :database_position 0,
                    :fingerprint nil,
                    :points_of_interest nil}}
      ```
      
      Object defaults adds quite a bit of stuff such that we'd be dissoc'ing
      more than we are currently adding in
      
      Co-authored-by: default avatarCam Saul <1455846+camsaul@users.noreply.github.com>
      Unverified
      6b8ddc84
  30. Feb 25, 2021
    • Cam Saul's avatar
      Optimize relative datetime filters (#14835) · 538e5e38
      Cam Saul authored
      1. Rename optimize-datetime-filters middleware -> optimize-temporal-filters (it's more accurate, because this also
      optimizes date or time filter clauses)
      
      2. optimize-temporal-filters middleware now optimizes relative-datetime clauses (which represent a moment in time
      relative to when the query is ran, e.g. "last month") in addition to absolute-datetime clauses (which represent an
      absolute moment in time, e.g. 2021-02-15T14:40:00-08:00) . This middleware rewrites queries so we filter against
      specific temporal ranges without casting the column itself, meaning we can leverage indexes on that column. See #11837
      for more details
      
      3. Added new validate-temporal-bucketing middleware that throws an Exception if you try to do something that makes no
      sense, e.g. bucket a DATE field by :time or a TIME field by :month. This is a better situation then running the query
      and waiting for the DB to complain. (In practice, I don't think the FE client would let you generate a query like this
      in the first place)
      
      4. Fix random test failures for MySQL in task-history-cleanup-test
      Unverified
      538e5e38
  31. Feb 15, 2021
    • dpsutton's avatar
      Semantic types pt 1 (special-type -> semantic-type rename) (#14780) · 80a88883
      dpsutton authored
      * Semantic types migration
      
      * Fixup for basic querying
      
      * Remove the relation type migration.
      
      it makes the diff far too hard at the moment. need to be able to just
      move everything from special_type -> semantic_type and then correct
      the few that look at it to care about the effective type.
      
      the PK/FK stuff gets really invasive and needs to be in its own much
      smaller changeset
      
      * Just rename special_type -> semantic_type for first change
      
      * Special type -> semantic type
      
      * special-type -> semantic-type
      
      * SpecialType -> SemanticType
      
      * special type -> semantic type
      
      mostly in documentation, docstrings, etc
      
      * Fix tests which relied on order of sets
      
      database position was annotated by map-index'ing over the
      set. changing `:special-type` to `:semantic-type` changed the order of
      the seq produced from it.
      
      * special -> semantic in schema_metadata
      
      * Tim is awesome: Undo overeager special->semantic in docstrings
      
      * Un-rename semantic_type in data_migrations
      
      These migrations are run unless a migrations table marks them as
      already having run. If they haven't run, then the db is older and the
      column is special_type most likely. So we let them run as
      `:special_type` and add some error handling to the migration runner
      that is _opt in_.
      
      ```clojure
      (defmigration ^{:author "camsaul", :added "0.20.0", :catch? true}
        migrate-field-types
        ...
        )
      
      (try
        (@migration-var)
        (catch Exception e
          (if catch? ;; catch? from metadata
            (log/warn (format "Data migration %s failed: %s" migration-name (.getMessage e)))
            (throw e))))
      ```
      
      * Fix merged master changes
      
      just accepted their changes and fixed up rather than fix conflicts.
      Unverified
      80a88883
  32. Jan 08, 2021
  33. Jan 05, 2021
  34. Dec 16, 2020
Loading