This project is mirrored from https://github.com/metabase/metabase.
Pull mirroring updated .
- Feb 17, 2022
-
-
adam-james authored
* Migrations for the new 'Annotate Useful Events' project Users will be able to create 'events' in timelines, which are accessible via collections. Two migrations are introduced to facilitate the new feature: timeline table, and event table. The following columns are presumed necessary based on the design document: ** TimeLine - id - name - description - collection_id - archived - creator_id - created_at - updated_by ?? - updated_at *** Event - id - timeline_id - name - description markdown (max length 255 for some reason) - date - time is optional - timezone (optional) - icon (set of 6) - archived status - created by - created at - created through: api or UI - modified by ?? - modified at * Changes to events schema - add icon onto timeline - make icon on event nullable - just move to a single timestamp on event, no boolean or anything - rename event table to `timeline_events` since "event" is so generic * dir-locals indentation * Timeline api and model - patched up the migration to make collection_id nullable since that is the root collection - followed the layout of api/metric and did the generic model stuff * Select keys with keywords, not destructured nils :( also, can't return the boolean from the update, but select it again * Enable the automatic updated_at * Boolean for "time matters" and string timezone column on events * clean up migration, rename modified_at -> updated_at for benefit of `:timestamped? true` bits * basic timeline event model namespace * Timeline Events initial API Just beginning the basics for the API for timeline events. - need to find a way to check permissions - have to still check if the endpoint is returning stuff properly * Singularize timeline_event tablename * Timeline events api hooked up * Singularize timeline event api namespace * unused comment * indent spec maps on routes * Make name optional on timeline PUT * Update collection api for timelines endpoints - add /root/timelines endpoint for when collection id is null - add /:id/timelines endpoint Created a hydration function to hydrate the timeline events when fetching timelines. * TimelineEvent permissions - crate a permissions objects set via the timeline's permissions * Move to using new permissions setup previously was explicitly checking permissions of the underlying timeline from the event, but now we just do the standard read/write check on the event, and the permissions set knows it is based on the permission set of the underlying timeline (which is based on the permissions set of the underlying collection) * Items api includes timelines * Strip of icon from other rows * Indices on timeline and timeline_event timeline access by collection_id timeline_event by both timeline_id and (timeline_id, timestamp) for when looking for events within a certain range. We will always have been narrowed by timeline ids (per the collection containing the question, or by manually enabling certain timelines) so we don't need a huge index on the timestamp itself. * Skeleton of range-based query * Initial timeline API tests Began with some simple Auth tests and basic GET tests, using a Collection with an id as well as the special root collection. * Fix docstring on api timeline namespace * Put timeline's events at `:events` not `timeline-events` * Add api/card/:id/timelines At the moment api/card/:id/timelines and api/collection/:id/timelines do the same thing: they return all of the timelines in the collection or in the collection belonging to the card. In the future this may drift so they share an implementation at the moment. * Put creator on timeline on api endpoints for timeline by id and timelines by card and collection * Hydrate creator on events Bit tricky stuff here. The FE wants the events at "events" not "timeline-events" i guess due to the hyphen that js can never quite be sure isn't subtraction. To use the magic of nested hydration `(hydrate timelines [:events :creator])`, the events have to be put at the declared hydration key. I had wanted the hydration `:timeline-events` for clarity in the code but want `:events` to be the key. But when you do this, the hydration thinks it didn't do any work and cannot add the creator. So the layout of the datastructure is tied to the name of the hydration key. Makes sense but I wish I had more control since `:events` is so generic. * Add include param check to allow timeline GET to hydrate w/ events. The basic check is in place for include=events in the following ways: GET /api/timeline/:id?include=events GET /api/collection/root/timelines?include=events If include param is omitted, timelines are hydrated only with the creator. Events are always hydrated with creator. * fix hyphen typo in api doc * Change schema for include=events to s/enum to enforce proper param Had used just a s/Str validation, which allows the include parameter value to be anything, which isn't great. So, switched it to an enum. Can always change this later to be a set of valid enums, similar to the `model` parameter on collection items. * Fixed card/:id/timelines failing w/ wrong arity The `include` parameter is passed to `timeline/timelines-for-collection` to control hydration of the events for that timeline. Since the card API currently calls this function too, it requires that parameter be passed along. As well, I abstracted the `include-events-schema` and have it in metabase.api.timeline. Not married to it, but since the schema was being used across timeline, collection, and card API, it seems like a good idea to define it in one place only. Not sure if it should be metabase.api.timeline or metabase.models.timeline * used proper migration numbers (claimed in migrations slack channel) * Add archived=true functionality There's one subtle issue remaining, and that is that when archived=true, :creator is not hydrated on the events. I'll look into fixing that. In the meantime, the following changes are made: - :events returs empty list `[]` instead of `null` when there are no events in a timeline - archived events can be fetched via `/api/timeline/:id?include=events&archived=true` - similarly, archived events can be fetched with: - `/api/collection/root|:id/timelines?include=events&archived=true` - `/api/card/:id/timelines?include=events&archived=true` Just note the caveat for the time being (no creator hydrated on events when fetching archived) Fix pending. * Altered the hydration so creator is always part of a hydrated event Adjusted the hydration of timelines as follows: - hydration always grabs all events - at the endpoint's implementation function, the timeline(s) are altered by filtering on archived true or false Not sure if this is the best approach, but for now it should work * Create GET endpoint for /api/timeline and allow archived=true Added a missed GET endpoint for fetching all the timelines (optionally fetch archived) * reformat def with docstring * Timeline api updated to properly filter archived/un-archived Use archived=true pattern in the query params for the timeline endpoint to allow FE to get what they need. Work in progress on the API tests too. * Timeline-events API tests Timeline Events API endpoint tests, at least the very basics. * Timeline Hydration w/ Events tests Added a test for Events hydration in the timeline API test namespace. * TimelineEvent Model test NS May be a bit unnecessary to test the hydrate function, but the namespace is there in case we need to add more tests over time. Also adjusted out a comment and added a library to timeline_test ns * Added metabase.models.timeline-test NS Once again, this may be a bit unnecessary as a test ns for now, but could be the right place to add tests if the feature grows in the future. * Clean up handling of archived - we only want to show archived events when viewing a timeline by id and specifically asking for archived events. Presumably this is some administrative screen. We don't want to allow all these options when viewing a viz as its just overload and they are archived for a reason. If FE really want them they can go by id of the timeline. Note it will return all events, not just the archived ones. - use namespaced keywords in the options map. Getting timelines necessarily requires getting their events sometimes. And to avoid confusion about the archived state, we have options like `:timeline/events?`, `:timeline/archived?`, `:events/start`, `:events/end`, and `:events/all?` so they are all in one map and not nested or ambiguous. * Clean up some tests noticable differences: timelines with archived=true return all events as if it is "include archived" rather than only show archived. * PUT /api/timeline/:id now archives all events when TL is archived We can archive/un-archive timelines, and all timeline events associated with that timeline will follow suit. This does lose state when, for example, there is a mix of archived/un-archived events, as there is no notion of 'archived-by' to check against. But, per discussion in the pod-discovery slack channel, this is ok for now. It is also how we handle other archiving scenarios anyway, with items on collections being archived when a collection is archived. * cleanup tests * Include Timeline and TimelineEvent in models * Add tt/WithTempDefaults impls for Timeline/TimelineEvent lets us remove lots of the `(merge defaults {...})` that was in the code. Defaults are ```clojure Timeline (fn [_] {:name "Timeline of bird squawks" :creator_id (rasta-id)}) TimelineEvent (fn [_] {:name "default timeline event" :timestamp (t/zoned-date-time) :timezone "US/Pacific" :time_matters true :creator_id (rasta-id)}) ``` * Add timeline and timeline event to copy infra * Timeline Test checking that archiving a Timeline archives its events A Timeline can be archived and its events should also be archived. A Timeline can also be unarchived, which should also unarchive the events. * Docstring clarity on apis * Remove commented out form * Reorder migrations * Clean ns clj-kondo ignores those but our linter does not * Correct casing on api schema for include * Ensure cleanup of timeline from test * DELETE for timeline and timeline-event * Poison collection items timeline query when is_pinned timelines have no notion of pinning and were coming back in both queries for pinned and not pinned. This adds a poison clause 1=2 when searching for pinned timelines since none are but they don't have a column saying that. * Clean up old comment and useless tests comment still said to poison the query when getting pinned state and that function had been added. Tests were asserting count = 2 and also the set of names had two things in them. Close enough since there are no duplicate names * Use TemporalString schema on timeline event routes Co-authored-by:
dan sutton <dan@dpsutton.com>
-
- Dec 28, 2021
-
-
dpsutton authored
* Handle nested queries which have agg at top level and nested Previously when matching columns in the outer query with columns in the inner query we had use id, and then recently, field_ref. This is problematic for aggregations. Consider https://github.com/metabase/metabase/issues/19403 The mbql for this query is ```clojure {:type :query :query {:aggregation [[:aggregation-options [:count] {:name "count"}] [:aggregation-options [:avg [:field "sum" {:base-type :type/Float}]] {:name "avg"}]] :limit 10 :source-card-id 1960 :source-query {:source-table 1 :aggregation [[:aggregation-options [:sum [:field 23 nil]] {:name "sum"}] [:aggregation-options [:max [:field 28 nil]] {:name "max"}]] :breakout [[:field 26 nil]] :order-by [[:asc [:field 26 nil]]]}} :database 1} ``` The aggregations in the top level select will be type checked as :name "count" :field_ref [:aggregation 0]. The aggregations in the nested query will be turned into :name "sum" :field_ref [:aggregation 0]! This is because aggregations are numbered "on their level" and not globally. So when the fields on the top level look at the metadata for the nested query and merge it, it unifies the two [:aggregation 0] fields but this is INCORRECT. These aggregations are not the same, they just happen to be the first aggregations at each level. Its illustrative to see what a (select * from (query with aggregations)) looks like: ```clojure {:database 1 :query {:source-card-id 1960 :source-metadata [{:description "The type of product, valid values include: Doohicky, Gadget, Gizmo and Widget" :semantic_type :type/Category :coercion_strategy nil :name "CATEGORY" :field_ref [:field 26 nil] :effective_type :type/Text :id 26 :display_name "Category" :fingerprint {:global {:distinct-count 4 :nil% 0} :type {:type/Text {:percent-json 0 :percent-url 0 :percent-email 0 :percent-state 0 :average-length 6.375}}} :base_type :type/Text} {:name "sum" :display_name "Sum of Price" :base_type :type/Float :effective_type :type/Float :semantic_type nil :field_ref [:aggregation 0]} {:name "max" :display_name "Max of Rating" :base_type :type/Float :effective_type :type/Float :semantic_type :type/Score :field_ref [:aggregation 1]}] :fields ([:field 26 nil] [:field "sum" {:base-type :type/Float}] [:field "max" {:base-type :type/Float}]) :source-query {:source-table 1 :aggregation [[:aggregation-options [:sum [:field 23 nil]] {:name "sum"}] [:aggregation-options [:max [:field 28 nil]] {:name "max"}]] :breakout [[:field 26 nil]] :order-by [[:asc [:field 26 nil]]]}} :type :query :middleware {:js-int-to-string? true :add-default-userland-constraints? true} :info {:executed-by 1 :context :ad-hoc :card-id 1960 :nested? true :query-hash #object["[B" 0x10227bf4 "[B@10227bf4"]} :constraints {:max-results 10000 :max-results-bare-rows 2000}} ``` The important bits are that it understands the nested query's metadata to be ```clojure {:name "sum" :display_name "Sum of Price" :field_ref [:aggregation 0]} {:name "max" :display_name "Max of Rating" :field_ref [:aggregation 1]} ``` And the fields on the outer query to be: ```clojure ([:field "sum" {:base-type :type/Float}] [:field "max" {:base-type :type/Float}]) ``` So there's the mismatch: the field_ref on the outside is [:field "sum"] but on the inside the field_ref is [:aggregation 0]. So the best way to match up when "looking down" into sub queries is by id and then by name. * Some drivers return 4.0 instead of 4 so make them all ints * Handle dataset metadata in a special way rather than trying to set confusing merge rules, just special case metadata from datasets. Previously, was trying to merge the "preserved keys" on top of the normal merge order. This caused lots of issues. They were trivial. Native display names are very close to the column name, whereas mbql names go through some humanization. So when you select price from (select PRICE ...) its an mbql with a nested native query. The merge order meant that the display name went from "Price" previously (the mbql nice name for the outer select) to "PRICE", the underlying native column name. Now we don't special case the display name (and other fields) of regular source-metadata. Also, there were issues with mbql on top of an mbql dataset. Since it is all mbql, everything is pulled from the database. So if there were overrides in the nested mbql dataset, like description, display name, etc, the outer field select already had the display name, etc. from the database rather than allowing the edits to override from the nested query. Also, using a long `:source-query/dataset?` keyword so it is far easier to find where this is set. With things called just `:dataset` it can be quite hard to find where these keys are used. When using the namespaced keyword, greping and finding usages is trivial. And the namespace gives more context
-
- Dec 15, 2021
-
-
Cam Saul authored
* Fix logging utils * Test fixes
-
- Sep 14, 2021
-
-
Cam Saul authored
* Add new Settings (placeholders) for the new 0.41.0+ premium features * Code cleanup * Add new features to the `:premium-features` Setting (for FE consumption) * Fix busted ns declarations
-
- Sep 09, 2021
-
-
Cam Saul authored
-
- Aug 11, 2021
-
-
dpsutton authored
* Move sync executor to bespoke namespace * Refingerprint fields on type change * Check if can connect in refingerprint-field! * docstring update * Cleanup tests a bit * Error handling * Table field values endpoint on sync.concurrent executor * ns sort api field * ns cleanup
-
- Aug 04, 2021
-
-
Cam Saul authored
* Fix JUnit output not correctly stripping ANSI color codes * Remove find-tests profiling log message * 1-arg arity of metabase.test-runner/run * Fix bug with metabase.test-runner/find-tests with a single test name
-
- Jul 30, 2021
-
-
Cam Saul authored
-
- Jun 22, 2021
-
-
dpsutton authored
* Move *ignore-cached-results* to middleware Had tried to adjust the binding to work, but the binding ended before all of the bound functions and such were submitted to work, so when that code actually executed the binding was no longer in place. Annoying and it should be in middleware anyways * Remove now unused import (cache became keyword in middleware not binding)
-
- Jun 02, 2021
-
-
dpsutton authored
* Add type on collection * Search with collection type * Cleanup bounded heap accumulator * Make search conditionally bump official collection items * collection api and tests * Put collection type onto hydrated cards on dashboards * Move official collection type scoring into ee codebase * ensure ee and oss agree when not official collection * Mark Collections tree with type * Remove unnecessary `and`s when no check for enhancements * Tests for setting collection tree type * Include hydration for collection type on dashboards * Make sure created test collections are cleaned up * Cleanup tests, don't search on collection_type looks for all text fields and searches the term against that. official would bring back all official types. no bueno * Docstring on protocol and don't shadow comparator * update to new ee impl var passing style * Collection model ensures no type change on personal collection * Check for personal collection when updating type model checks for personal collection and rejects in the update but doesn't check for children of personal collection. * Update checking for unchangeable properties of a personal collection * Cleanup: type collection tree, batched hydration, combine error checks * Cleanup * move bounded-heap accumulator to utils * switch to test.check testing * Bad test just to see what our CI will report * remove purposeful failing test (was for CI) * collection.type -> collection.authority_level * Test the actual ee scoring impl, not the wrapped one locally i had enhancements enabled so the tests were passing, but in CI it was failing as it did not have enhancements enabled
-
- May 17, 2021
-
-
Cam Saul authored
* Port legacy data type migrations -> Liquibase * Fix migration IDs * Field type validation/error handling * Have semantic type fallback to nil * Fix semantic-type-migrations-test * Fix migrations * Revert accidental changes * Semantic/* & Relation/* ancestor types * Fix stray Relation/PK and Relation/FKs * Semantic/* and Relation/* ancestor types * cljs test fix * Fix :require * FE test fixes
* Test fixes * prettier * PR f e e d b a c k * Use medium size CircleCI image for Presto to prevent all the OOMs * Backport dir-locals tweaks from hierarchy PR * Redshift: only sync the test schema (faster CI and fix failure) * Better error handling for sync in tests * Revert accidental commit * Redshift test fixes
-
- Apr 13, 2021
-
-
dpsutton authored
* Remove type/UNIX* and type/ISO8601* from frontend * Set effective-type and coercion strategy when syncing these values will most likely only be available when using tests, as metadata is very unlikely to have this. As far as I can tell the toucannery apparatus is the only bit that has this. Its quite artificial. I'm not sure if this is a good idea. * :type/UNIXTimestampSeconds and type/ISO8601DateTimeString out of type remove the coercions from the type hierarchy. This brought up a strange issue that has been present for a while: all liquidbase migrations run and then all data migrations run. They are not interleaved. This allows for the following scenario: - data migration migrates all X values to Y in version 26 - liquibase migration migrates all Y values to Z in version 28 But these are not run in version order, but all liquibase migrations are run and then all data migrations are run. If you were on an old version for a while, you could end up with Y values even though the liquibase migration which comes "after" the data migration turns all Y values into Z values. This impacts this change because a data migration in this way: - liquibase changesets 287, 288, 289, and 290 remove all 'type/UNIX*' and 'type/ISO8601*' semantic types. These were in 0.39 - data migration `migrate-field-types` added in 0.20.0 was looking for special_type's of "timestamp_milliseconds" and migrating them to "type/UNIXTimestampMilliseconds". Since the liquibase runs before the migrate-field-types, it's possible for a 'type/UNIX*' semantic type to reappear. And since these were removed from the type system it would fail validation and blow up. In this case it actually can't happen since the field its attempting to migrate (special_type) no longer exists. But since the migrations are not interleaved this problem still exists. * whatever prettier * Appease the machines * Restore the unix and iso semantic types to hierarchy migration tests still use these `(impl/test-migrations [283 296] [migrate!] ...)` A few things in tension: the field requires valid semantic_types, these _were_ valid semantic_types until v39. * Remove old "coercion" semantic types * Migrate the v20 version semantic types for unix epoch * Time travelling migrations these target v20 and v29 which at the time they existed had a column special type not semantic type. But these run after all of the schema migrations, not interleaved, so they will have a semantic_type not special_type when they actually run even though they target v20 or v29 data. strange world * add migration's new checksum
-
- Apr 12, 2021
-
-
Dalton authored
* Backend feature flag for new field filters * Feature flag new parameter options When the "field-filter-operators-enabled?" flag is disabled we do the following: 1. Replace new operator options with old category and location/city, etc., options in the PARAMETER_OPTIONS list found in metabase/meta/Parameter.js 2. Hide numbers section in the PARAMETER_SECTIONS list found in metabase/meta/Dashboard.js 3. Return args as-is in the mapUIParameterToQueryParameter function found in metabase/meta/Parameter.js React/UI code handles both old options and new options so doesn't need to change. Old parameter types like "category" and "location/city" are treated like "string/=" in the UI but retain their own parameter type when used to send a new query. * Fix FE issues caused by meta/Parameter refactor * mock the field operator param flag to make tests pass * add/fix cypress tests * fix import in ParametersPopover * update widget tag type * Enable field filter operators for cypress tests * Question marks are questionable * Conditionally use category or string/= if field filters are enabled * rmv mocks where we don't need them * rmv mock from chained-filters test * env vars as string in project.clj, alignment Co-authored-by:
dan sutton <dan@dpsutton.com>
-
- Mar 31, 2021
-
-
Tim Macdonald authored
* Set lein as the preferred build tool for Cider * Add search-typeahead-enabled setting
-
- Mar 23, 2021
-
-
Cam Saul authored
Port the metabase.mbql utility namespaces to ./shared/ so they can be used on both the frontend and backend.
-
- Feb 25, 2021
-
-
Cam Saul authored
1. Rename optimize-datetime-filters middleware -> optimize-temporal-filters (it's more accurate, because this also optimizes date or time filter clauses) 2. optimize-temporal-filters middleware now optimizes relative-datetime clauses (which represent a moment in time relative to when the query is ran, e.g. "last month") in addition to absolute-datetime clauses (which represent an absolute moment in time, e.g. 2021-02-15T14:40:00-08:00) . This middleware rewrites queries so we filter against specific temporal ranges without casting the column itself, meaning we can leverage indexes on that column. See #11837 for more details 3. Added new validate-temporal-bucketing middleware that throws an Exception if you try to do something that makes no sense, e.g. bucket a DATE field by :time or a TIME field by :month. This is a better situation then running the query and waiting for the DB to complain. (In practice, I don't think the FE client would let you generate a query like this in the first place) 4. Fix random test failures for MySQL in task-history-cleanup-test
-
- Jan 15, 2021
-
-
dpsutton authored
* Log db info from db query we're already making the query might as well also get the information we want to log * Alignment settings in .dir-locals * Inverted logic for logging fingerprint vs refingerprint
-
- Jan 14, 2021
-
-
Cam Saul authored
-
- Jan 07, 2021
- Jan 05, 2021
-
-
Cam Saul authored
* Fix #14114 * Test fix
-
- Oct 05, 2020
-
-
Paul Rosenzweig authored
* dashboard interactivity: custom drill through and chained filters (cherry picked from commit f3ea3099da981c967fcffd5c7e318c88064c0a96) * add missing param * Better error message & test for cases where you don't have Field perms (cherry picked from commit 2ae8174c388a105170bb316b6266b900f8844c62) * Fix lint error (cherry picked from commit aa967359ecf9705f28dded2b5b30eb42ebec156e) * type another letter to avoid ordering issue (cherry picked from commit 5f285219c07eeff41b3d1fbfc091092a6ce60ed4) * fix failing cy tests (cherry picked from commit 8dbb9561a21f853a550d4ab1385fbdf142ea36e6) * update test * Fix merge * sort correctly * Fix namespace decl sorting * use another within block for the "add to dash" modal Co-authored-by:
Cam Saul <github@camsaul.com>
-
- Jul 20, 2020
-
-
Robert Roland authored
* Update eslint Updates eslint, babel and plugins to the latest compatible versions Drops the 'no-color-literals' parameter which doesn't exist (looks like it's actually part of eslint-plugin-react-native which we don't use) adding a dirlocal to make sure js2-mode doesn't confuse you with type errors that aren't actually errors because of flowtype and such * update generated css classes in snapshots Co-authored-by:
Paul Rosenzweig <paul.a.rosenzweig@gmail.com>
-
- Feb 19, 2020
-
-
Cam Saul authored
-
- Dec 03, 2019
-
-
Cam Saul authored
-
- Oct 07, 2019
-
-
Daniel Higginbotham authored
add an endpoint so that users who don't own a pulse can unsubscribe themselves from the pulse
-
- Jul 29, 2019
-
-
Cam Saul authored
* Add Database auto_run_queries setting; code cleanup * More code cleanup & test fixes * Test fixes ?
-
- Jun 05, 2019
-
-
Cam Saul authored
* Metabase© Joins 2.0™ improvements
[ci drivers]
-
- Mar 13, 2019
-
-
Cam Saul authored
-
- Dec 19, 2017
-
-
Cam Saul authored
-
- Feb 01, 2017
-
-
Cam Saül authored
-
- Jan 27, 2017
-
-
Cam Saül authored
-
- Dec 06, 2016
-
-
Cam Saül authored
-
- Nov 02, 2016
-
-
Cam Saül authored
-
- Oct 19, 2016
-
-
Cam Saül authored
-
- Sep 19, 2016
-
-
Cam Saül authored
-
- Jul 20, 2016
-
-
Cam Saül authored
-
- Jul 18, 2016
-
-
Cam Saül authored
-
- Jul 13, 2016
-
-
Cam Saül authored
-
- Jul 08, 2016
-
-
Cam Saül authored
-