Skip to content
Snippets Groups Projects
Unverified Commit 0102793f authored by metabase-bot[bot]'s avatar metabase-bot[bot] Committed by GitHub
Browse files

Lean on DB queries for describe-table for Mongo (#46598) (#48572)


This PR reimplements driver/describe-table for MongoDB. Before we would query a sample of documents from a collection and analyse them in Clojure. Instead, we now now execute a query that does a similar aggregation, but most of the calculation is done in the Mongo database.

Based on a few tests the performance is slightly slower when the collection contains small or deeply nested documents but much faster for large ones. But the main difference is in memory usage. This uses very little memory in the Metabase instance because all of the aggregation is done in the database.


Nested fields are a naturally recursive problem but here we unroll potential recursions to a `max-depth` number of queries that look for nesting at each depth level.

* ~ use DB to describe the table

* ~ optimize root query

* ~ nested-level-query works and gets objects too

* + root query gets objects too

* + driver/describe-table :mongo works

* ~ remove old implementation

* Various fixes for faster sync

Upgraded driver to 5.2.0
Updated data load to insert many rather than 1 row at a time.
Dropped max-depth to 7, see comment.

---------

Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
Co-authored-by: default avatarCase Nelson <case@metabase.com>
parent ce5274a5
Branches
Tags v0.12.0
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment