adam-james
authored
* PoC Pivot table exporting. Works so far! This just hacks a couple things in. I added the :query via format_rows, which should instead have its own proper middleware Pivot export should probably not just be in the middleware namespace but in qp.pivot. Should not break the streaming results writer for non-pivots... obviously this breaks ordinary results, so that needs to be handled properly. Better documentation about how the logic actually works. Pivot tables are hard to remember/easy to forget, so some very clear diagrams/examples would probably be good. * adjust csv export to export a pivot table when pivot options exist * post processing of pivot table rows for card/dashcard downloads * WIP on getting pivot exports to work with unsaved questions * WIP getting pivot exports working on unsaved questions The dataset POST /pivot endpoint is used to get the pivot data to the frontend, but also to get the pivot query. The query that is returned by the POST comes from the json_query, which is populated by the userland-query stuff with the last run query. The pivot qp runs mulitple queries and each query runs through the regular userland query qp, so we'll 1. see lots of executions recorded and 2. end up with the wrong query in the POST return. It'll look mostly correct, but due to how the pivot qp works won't necessarily have all of the fields. That incorrect pivot query is passed to the /:export-format endpoint and we don't end up seeing correct results. Since that json_query gets fed back to the backend when doing an export, we do need it to be correct, like the query that was sent to /pivot in the first place. Since this is a WIP, I've just passed the query as a string to :original-query in :info, and if that exists, use that to fill the :json_query key. Then, also add a :was-pivot flag so that the dataset download endpoint can look at it and use the pivot qp instead of the regular qp. Even if we have the correct 'top level' query, we need to run it through the pivot qp to get the full, correct pivoted results. * Pivot Export Test Here I've added a pivot export test that has various assertions trying to capture the correct output from the pivot csv export. I've done my best to explain the tests with comments, but have also left a (comment ) form with a function `explore-pivot` to help *show* rather than tell what the expected pivot output should be. I've created this new namespace to first capture this pivot export test with the intent to add tests that can improve our confidence that ALL exports/downloads in ALL places behave as we expect in the future. It just seemed nice to start fresh here with the pivot export test first. * Don't need to make this public. * Add lots of comments to hopefully make this a bit easier to follow * Added an explore-raw-pivot fn in comment form * Update test/metabase/api/downloads_exports_test.clj Co-authored-by:Cam Saul <1455846+camsaul@users.noreply.github.com> * begin addressing feedback - finish an unfinished docstring - merge info instead of assoc - use a native query to build the pivot data instead of using a custom dataset, to keep test speed ok * Create a new ns for the pivot post processing utils Separate the post processing fns into their own namespace. This means I can easily add a test namespace for the post processing specifically as well. * Add pivot postprocessing test ns, not complete, but decent start * fix mistake in endpoint. Only add original query to first pivot q * Figured out how to keep the original query without storing as a string * Perhaps this'll fix things properly? --------- Co-authored-by:
Cam Saul <1455846+camsaul@users.noreply.github.com>
Code owners
Assign users and groups as approvers for specific file changes. Learn more.