-
- Downloads
Lean on DB queries for describe-table for Mongo (#46598) (#48572)
This PR reimplements driver/describe-table for MongoDB. Before we would query a sample of documents from a collection and analyse them in Clojure. Instead, we now now execute a query that does a similar aggregation, but most of the calculation is done in the Mongo database. Based on a few tests the performance is slightly slower when the collection contains small or deeply nested documents but much faster for large ones. But the main difference is in memory usage. This uses very little memory in the Metabase instance because all of the aggregation is done in the database. Nested fields are a naturally recursive problem but here we unroll potential recursions to a `max-depth` number of queries that look for nesting at each depth level. * ~ use DB to describe the table * ~ optimize root query * ~ nested-level-query works and gets objects too * + root query gets objects too * + driver/describe-table :mongo works * ~ remove old implementation * Various fixes for faster sync Upgraded driver to 5.2.0 Updated data load to insert many rather than 1 row at a time. Dropped max-depth to 7, see comment. --------- Co-authored-by:Cal Herries <39073188+calherries@users.noreply.github.com> Co-authored-by:
Case Nelson <case@metabase.com>
Showing
- modules/drivers/mongo/deps.edn 1 addition, 1 deletionmodules/drivers/mongo/deps.edn
- modules/drivers/mongo/src/metabase/driver/mongo.clj 252 additions, 118 deletionsmodules/drivers/mongo/src/metabase/driver/mongo.clj
- modules/drivers/mongo/src/metabase/driver/mongo/util.clj 5 additions, 0 deletionsmodules/drivers/mongo/src/metabase/driver/mongo/util.clj
- modules/drivers/mongo/test/metabase/driver/missing-fields.json 31 additions, 0 deletions...es/drivers/mongo/test/metabase/driver/missing-fields.json
- modules/drivers/mongo/test/metabase/driver/mongo/sharded_cluster_test.clj 4 additions, 1 deletion...mongo/test/metabase/driver/mongo/sharded_cluster_test.clj
- modules/drivers/mongo/test/metabase/driver/mongo_test.clj 137 additions, 34 deletionsmodules/drivers/mongo/test/metabase/driver/mongo_test.clj
- modules/drivers/mongo/test/metabase/test/data/mongo.clj 14 additions, 11 deletionsmodules/drivers/mongo/test/metabase/test/data/mongo.clj
Loading
Please register or sign in to comment