Jeff Evans
authored
Serialization refactoring and fixes ********************************************************* * Support loading linked cards in different collections * ********************************************************* Architectural changes: Lay foundation for more generic retry mechanism Establish a keyword to represent a failed name->id lookup that can happen during load, and map that to information about which part of the card resolution failed (for logging purposes) Similar to dashboards, add a separate fn for performing a load of cards Invoking the load-cards fn from the load multimethod implementation when some cards failed name resolution Test enhancements: Add new multimethod to perform more specific assertions on the loaded entities For cards - check that the native SQL of the loaded card matches that of the original (pre-dump) instance For dashboards - check that the linked card for each dashboard card series matches Bug fixes: Apply the mbql-id->fully-qualified-name fn on serialize to filter, breakout, and aggregation clauses Updating "My Test" card to include filter clause so that fix is exercised Instead of upserting users with a "changeme" password, simply do not attempt to change the password Adding :engine to the identity condition of Database (so unrelated test-data instances don't get trashed on load) Cleanup: Remove unused specs namespace **************************************** * Support dashboard card click actions * **************************************** Architectural changes: Add visualization settings shared model namespace with utility functions Adding tests for viz settings Changes to serialization: Update visualization_settings in dump to convert column_settings keys from field ID to fully qualified name, and update the click action target with the fully qualified name Also converting targetId in top level click action to fully qualified name Doing the reverse in load, using the same utility code Changes to serialization tests: Updating the existing dashcard in the "world" roundtrip test to include the click action Adding assertion for the click action in the assert-loaded-entity implementation for Dashboard ****************************************************************** * Stop nesting cards and check for unresolved query dependencies * ****************************************************************** In names.clj, stop qualifying a card's name by its dataset query's source card ID (this is what caused the "nested" directory structure) In load.clj, start checking that either the source query [:dataset_query :query] is resolved properly as a card ID or table ID, and if neither, then propagate an ::unresolved-names entry up into the card (causing it to be reloaded on second pass) Fix logging of unresolved name info Test changes: Add two new cards to the "world", one that lives in the root collection but depends on a card within a collection for its query, and one opposite (i.e. lives in a collection but depends on a root card for its query) Adding new check that collection names match after loading ************************ * Logging improvements * ************************ Include causes when logging errors to aid with diagnosis Add BEGIN and END messages for both dump and load commands Add dynamic var to suppress printing of name lookup error on first pass load ************************************************* * Handle collection namespaces and retry pulses * ************************************************* Architectural changes: Considering a collection namespace to be part of its fully qualified name (putting in a subdirectory called :namespace when writing to disk) Accounting for that change when dumping and loading, and upsert (i.e. namespace is part of unique columns) Bumping serialization format version to 2 because of this Changes to load: Add load-pulses fn that works similarly to others, to facilitate reloading Add similar logic as other entities to reload a pulse if the :card_id is not found Model changes: Add hack fn to compare namespace values for the "namespace does not change" assertion with appropriate comment Test changes: Added test for pulse with reload *********************************************** * Handle dimension entry within template-tags * *********************************************** Add :field to the recognized clauses under mbql-entity-reference? (since that is how field refs are now represented in MBQL) Add temp-field helper fn in test_util.clj to reduce some of the "world" verbosity Adding new test card to world that uses template-tags with a field reference under dimension ************************ * Handle textbox cards * ************************ Add support for passing textbox (virtual) cards through visualization_settings.cljc Updating load test to add a textbox type card to the dashboard and ensuring it is persisted *************************************** * Handle dimension parameter mappings * *************************************** In visualization_settings.cljc: Adding handling for parameterMapping Adding new roundtrip test for parameter mappings Some refactoring to split up larger fns In serialize.clj, convert all parts of the map that contain the dimension key to use fully qualified names for the respective fields In load.clj, do the opposite (convert back to new IDs) Updating test to add parameter mapping to an existing click action, and adding new assertions for that to ensure it is preserved Fixing serialize test now that the "ids" can themselves be maps (i.e within dimension vector) In visualization_settings_test.cljc Adding new roundtrip test for problem dashboard *********************************** * Handle field literals correctly * *********************************** On load, only attempt to translate field name to ID if it's actually a fully qualified field name Adding new fn to names.clj to provide that logic Adding new test card that uses such a field reference ************************************** * Accept only-db-ids option for dump * ************************************** Accept options map for the dump command. Currently, the only supported option is `:only-db-ids`, which is expected to be a set of DB IDs to dump. If set, then only these databases (and their respective tables, fields, and segments) will be dumped to the dump-dir. If not set, then the previous behavior takes effect (all DBs). Update the load_test.clj to set this to be only the new temporary DB to avoid tramping on existing `sample-dataset` and other instances ***************************************************** * Don't include personal collections of other users * ***************************************************** Use the `select-collections` fn to filter only collections that are public, or owned bythe given user (by way of email), plus all collections with those as an ancestor Updating test to assert a non-owned personal collection is not persisted (including nested), but all nested personal collections are Adding new wrapped macro to also bind collection/*allow-deleting-personal-collections* true around the with-temp* call so that personal collections can legitimately be deleted (since they're temporary); the name of this macro was chosen to preserve the same amount of spacing within `with-world` ********************************* * Support native query snippets * ********************************* Adding support for NativeQuerySnippet: - in names - in dump (putting into subdirectory of collection, as cards) - in upsert (identity condition of :name :collection_id) - in load (no retry fn necessary since they should not have any dependencies themselves) Adding a snippet and a card referencing it to roundtrip test (including a deeply nested snippet) Fixing up handling of snippet namespaced collections and adding relevant name tests ********************************* * Handle source-table for joins * ********************************* Adding `mbql-fully-qualified-names->ids*` for recursive calls, which does not normalize its args, keeping mbql-fully-qualified-names->ids` as a wrapper to that Adding clause to `mbql-fully-qualified-names->ids*` to resolve `:source-table` when it's a fully qualified table name Adding util fn to names.clj to support that scenario Updating "My Card" test card to include a join Skipping two more drivers from `dump-load-entities-test` - SQLite and Spark because they don't support FKs *************************** * Cleanup and refactoring * *************************** Responding to PR feedback Updating upsert_test.clj to check that the identity-condition keys actually exist for the models Removing legacy MBQL syntax in a couple places Adding ordering by `:id` to serialization load command for predictability Lots of fixes and refactorings in visualization_settings.cljc: - added lots of specs and fdefs - added test fixture that instruments all the speced fns and required it from both visualization_settings.cljc and load_test.clj - renamed a bunch of fns for more consistency - added and fixed some docstrings - fix implementation of `parse-json-string` for cljs ************************************************* * Preserve temporal-unit in source-field clause * ************************************************* Update serialize and match macros to keep the rest of the match, when modifying the source field ID Adding a bunch of new tables in order to perform a "bucketing on implicit join" type query Updating test_util.clj to initialize the query processor with all the new temp tables so that queries actually work ************************** * Misc cleanup and fixes * ************************** Fixing the implementation of `fully-qualified-name->context` to work properly when the entity name itself is also a model name *************************** * Handle joining to cards * *************************** Recognize :source-table pointing to a fully qualified card name in `mbql-fully-qualified-names->ids*`, and adding the unresolved name capture to that lookup Adding new util fns to capture the unresolved named entries from anywhere within the MBQL query tree: `paths-to-key-in` and `gather-all-unresolved-names` Updating `resolve-card-dataset-query` to call this new `gather-all-unresolved-names` to pull all unresolved names from the MBQL query tree into card itself Renaming `fully-qualified-name->id-rec` to `resolve-source-query` for clarity, and removing now unneeded clause To the test case, adding a card that joins to another card
Name | Last commit | Last update |
---|