Skip to content
Snippets Groups Projects
  • Tim Macdonald's avatar
    a06e640e
    Upload CSVs (#30256) · a06e640e
    Tim Macdonald authored
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * Add CSV uploads feature and have postgres and H2 support it (#29636)
    
    * Add csv uploads feature and have postgres support it
    
    * Add h2 support
    
    * CSV: schema detection (#29650)
    
    * CSV / Admin Upload Settings (#29709)
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * Add CSV uploads feature and have postgres and H2 support it (#29636)
    
    * Add csv uploads feature and have postgres support it
    
    * Add h2 support
    
    * Add defsettings
    
    * Add table prefix setting
    
    * add data upload settings
    
    * Add newline
    
    * update settings mock
    
    * Whether or not uploads are enabled
    
    ---------
    
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * Create table and insert rows from CSV uploads for postgres (#29677)
    
    * WIP create table and insert rows
    
    * Add newline
    
    * Use multimethods
    
    * WIP
    
    * Fix postgres
    
    * Fix postgres impl
    
    * Use detect-schema and add test
    
    * Fix load-from-csv arglists
    
    * Undo whitespace change
    
    * Use empty db instead
    
    * Add table and field tests
    
    * Remove prn
    
    * Uniquify names
    
    * newline
    
    * Add test for float
    
    * Add a test for the table contents
    
    * Tidy load-from-csv
    
    * Drop the table if loading the csv fails
    
    * Drop the table only if loading the csv fails
    
    * whitespace
    
    * Address SQL injection vulnerability
    
    * Use the slugified file-name as the table-name
    
    * Fix kondo
    
    * Using absolute path instead of relative
    
    * Fix kondo
    
    * Detect types for empty CSV files appropriately (#29754)
    
    * Create models from CSV uploads (#29966)
    
    * Wire up CSV uploads, sync the new table
    
    Still need to create a model
    
    * Check database support for CSV uploads better
    
    * Another test
    
    * Create model after sync
    
    * Fix request with formData
    
    * Fix tests
    
    * Fix tests
    
    * Revert "Fix request with formData"
    
    This reverts commit 66f146a41cf42feb34dbb03ed8566fa4220d0642.
    
    * Add whitespace
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Fix postgres support for uploads (#29922)
    
    * Check write perms for collection (#29976)
    
    * Rename metabase.csv -> metabase.upload (#29975)
    
    * Rename metabase.csv -> metabase.upload
    
    * Add docstring
    
    * Rename alias for release.common.upload
    
    ---------
    
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * WIP - 29598 csv status toast (#29757)
    
    * temp
    
    * generalize StatusListing types
    
    * Putting a pin in this for now
    
    * poorly done typing
    
    * generalizing upload function names
    
    * point to real API
    
    * Fixing Storybook
    
    * Permissions for Status Listing
    
    * use formdata and fix types
    
    ---------
    
    Co-authored-by: default avatarRyan Laurie <iethree@gmail.com>
    
    * Don't support redshift (#29981)
    
    * Handle uploads to the root collection (#29985)
    
    * Allow the uploads schema to be nil (for MySQL, etc.) (#29987)
    
    * CSV upload: fix stack overflow in detect-schema (#29983)
    
    * Fix stack overflow error on detect schema
    
    * map + vec -> mapv
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@metabase.com>
    
    * reload collection data after upload (#30009)
    
    * MySQL support for uploads (#29829)
    
    * Add MySQL support for uploads
    
    * Rename arg to upload-type
    
    * Use new ns name
    
    * Allow no table-name prefix to be specified and fix tests
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Fix setting descriptions (#30023)
    
    * Batch CSV insertion for better performance (#30028)
    
    * CSV upload e2e tests (#30005)
    
    * add CSV upload e2e tests
    
    * update tests
    
    * H2 support for uploads (#29828)
    
    * Add uploads support for H2
    
    * Rename arg
    
    * whitespace
    
    * Use new metabase.upload ns name
    
    * Fix tests for H2
    
    * Make table and field matching case insensitive
    
    * Fix kondo error
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Clear CSV upload state after a while (#30014)
    
    * clear upload status after 5s
    
    * make async behavior clearer
    
    * CSV Upload Dates and Datetimes Types (#30076)
    
    * Simple dates
    
    * Simple datetimes
    
    * Fix coalesce for date and datetime
    
    * Add mysql and h2 types
    
    * Fix test
    
    * Misc cleanup
    
    * Fix test
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Fix Kondo (rebase gone wrong)
    
    * MySQL test fix
    
    * Fix order of Metabot setting
    
    * adding unit tests to upload actions and reducers (#30161)
    
    * adding unit tests to upload actions and reducers
    
    * use fetch-mock
    
    * Adding component unit test
    
    * Check CSV Upload permissions on the frontend (#30094)
    
    * CSV / Admin Upload Settings (#29709)
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * Add CSV uploads feature and have postgres and H2 support it (#29636)
    
    * Add csv uploads feature and have postgres support it
    
    * Add h2 support
    
    * Add defsettings
    
    * Add table prefix setting
    
    * add data upload settings
    
    * Add newline
    
    * update settings mock
    
    * Whether or not uploads are enabled
    
    ---------
    
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * properly check if user can upload to a collection
    
    * do some db checking
    
    * don't check admin status for upload notification
    
    * misc cleanup
    
    * fix rebase
    
    * add loggedin check to statusListing
    
    also combines container + view component into a single file
    
    * update unit tests
    
    * update tests
    
    ---------
    
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * Csv/misc bugfixes (#30219)
    
    * Support empty header names
    
    * Handle long CSV names more appropriately
    
    Handle duplicate column names in CSVs
    
    * Test fix
    
    * Partition CSV inserts a little more intelligently
    
    * Test fix
    
    * Remove 'duplicate' from duplicated column names
    
    * More intelligent sampling
    
    * Add comment re. Postgres table-name-length-limit
    
    * Quote column names, clean up tests
    
    When unquoted, we couldn't use reserved words (true, false, group, etc.)
    as column names
    
    * Fix test
    
    Now that we quote things, H2 is case-sensitive
    
    * Use dialect-appropriate quoting
    
    Also clean up some card-filtering tests so they work with a dirty DB
    
    * Fix deduplication
    
    * Remove use of setting/get
    
    * Remove unneeded migration
    
    ---------
    
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * Fix test (#30383)
    
    ---------
    
    Co-authored-by: default avatarRyan Laurie <iethree@gmail.com>
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarRyan Laurie <30528226+iethree@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    Co-authored-by: default avatarNick Fitzpatrick <nick@metabase.com>
    Upload CSVs (#30256)
    Tim Macdonald authored
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * Add CSV uploads feature and have postgres and H2 support it (#29636)
    
    * Add csv uploads feature and have postgres support it
    
    * Add h2 support
    
    * CSV: schema detection (#29650)
    
    * CSV / Admin Upload Settings (#29709)
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * Add CSV uploads feature and have postgres and H2 support it (#29636)
    
    * Add csv uploads feature and have postgres support it
    
    * Add h2 support
    
    * Add defsettings
    
    * Add table prefix setting
    
    * add data upload settings
    
    * Add newline
    
    * update settings mock
    
    * Whether or not uploads are enabled
    
    ---------
    
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * Create table and insert rows from CSV uploads for postgres (#29677)
    
    * WIP create table and insert rows
    
    * Add newline
    
    * Use multimethods
    
    * WIP
    
    * Fix postgres
    
    * Fix postgres impl
    
    * Use detect-schema and add test
    
    * Fix load-from-csv arglists
    
    * Undo whitespace change
    
    * Use empty db instead
    
    * Add table and field tests
    
    * Remove prn
    
    * Uniquify names
    
    * newline
    
    * Add test for float
    
    * Add a test for the table contents
    
    * Tidy load-from-csv
    
    * Drop the table if loading the csv fails
    
    * Drop the table only if loading the csv fails
    
    * whitespace
    
    * Address SQL injection vulnerability
    
    * Use the slugified file-name as the table-name
    
    * Fix kondo
    
    * Using absolute path instead of relative
    
    * Fix kondo
    
    * Detect types for empty CSV files appropriately (#29754)
    
    * Create models from CSV uploads (#29966)
    
    * Wire up CSV uploads, sync the new table
    
    Still need to create a model
    
    * Check database support for CSV uploads better
    
    * Another test
    
    * Create model after sync
    
    * Fix request with formData
    
    * Fix tests
    
    * Fix tests
    
    * Revert "Fix request with formData"
    
    This reverts commit 66f146a41cf42feb34dbb03ed8566fa4220d0642.
    
    * Add whitespace
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Fix postgres support for uploads (#29922)
    
    * Check write perms for collection (#29976)
    
    * Rename metabase.csv -> metabase.upload (#29975)
    
    * Rename metabase.csv -> metabase.upload
    
    * Add docstring
    
    * Rename alias for release.common.upload
    
    ---------
    
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * WIP - 29598 csv status toast (#29757)
    
    * temp
    
    * generalize StatusListing types
    
    * Putting a pin in this for now
    
    * poorly done typing
    
    * generalizing upload function names
    
    * point to real API
    
    * Fixing Storybook
    
    * Permissions for Status Listing
    
    * use formdata and fix types
    
    ---------
    
    Co-authored-by: default avatarRyan Laurie <iethree@gmail.com>
    
    * Don't support redshift (#29981)
    
    * Handle uploads to the root collection (#29985)
    
    * Allow the uploads schema to be nil (for MySQL, etc.) (#29987)
    
    * CSV upload: fix stack overflow in detect-schema (#29983)
    
    * Fix stack overflow error on detect schema
    
    * map + vec -> mapv
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@metabase.com>
    
    * reload collection data after upload (#30009)
    
    * MySQL support for uploads (#29829)
    
    * Add MySQL support for uploads
    
    * Rename arg to upload-type
    
    * Use new ns name
    
    * Allow no table-name prefix to be specified and fix tests
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Fix setting descriptions (#30023)
    
    * Batch CSV insertion for better performance (#30028)
    
    * CSV upload e2e tests (#30005)
    
    * add CSV upload e2e tests
    
    * update tests
    
    * H2 support for uploads (#29828)
    
    * Add uploads support for H2
    
    * Rename arg
    
    * whitespace
    
    * Use new metabase.upload ns name
    
    * Fix tests for H2
    
    * Make table and field matching case insensitive
    
    * Fix kondo error
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Clear CSV upload state after a while (#30014)
    
    * clear upload status after 5s
    
    * make async behavior clearer
    
    * CSV Upload Dates and Datetimes Types (#30076)
    
    * Simple dates
    
    * Simple datetimes
    
    * Fix coalesce for date and datetime
    
    * Add mysql and h2 types
    
    * Fix test
    
    * Misc cleanup
    
    * Fix test
    
    ---------
    
    Co-authored-by: default avatarTim Macdonald <tim@tsmacdonald.com>
    
    * Fix Kondo (rebase gone wrong)
    
    * MySQL test fix
    
    * Fix order of Metabot setting
    
    * adding unit tests to upload actions and reducers (#30161)
    
    * adding unit tests to upload actions and reducers
    
    * use fetch-mock
    
    * Adding component unit test
    
    * Check CSV Upload permissions on the frontend (#30094)
    
    * CSV / Admin Upload Settings (#29709)
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * allow formData uploads
    
    * basic CSV uploads from collections
    
    * add dropzone support for collections
    
    * Add CSV uploads feature and have postgres and H2 support it (#29636)
    
    * Add csv uploads feature and have postgres support it
    
    * Add h2 support
    
    * Add defsettings
    
    * Add table prefix setting
    
    * add data upload settings
    
    * Add newline
    
    * update settings mock
    
    * Whether or not uploads are enabled
    
    ---------
    
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * properly check if user can upload to a collection
    
    * do some db checking
    
    * don't check admin status for upload notification
    
    * misc cleanup
    
    * fix rebase
    
    * add loggedin check to statusListing
    
    also combines container + view component into a single file
    
    * update unit tests
    
    * update tests
    
    ---------
    
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * Csv/misc bugfixes (#30219)
    
    * Support empty header names
    
    * Handle long CSV names more appropriately
    
    Handle duplicate column names in CSVs
    
    * Test fix
    
    * Partition CSV inserts a little more intelligently
    
    * Test fix
    
    * Remove 'duplicate' from duplicated column names
    
    * More intelligent sampling
    
    * Add comment re. Postgres table-name-length-limit
    
    * Quote column names, clean up tests
    
    When unquoted, we couldn't use reserved words (true, false, group, etc.)
    as column names
    
    * Fix test
    
    Now that we quote things, H2 is case-sensitive
    
    * Use dialect-appropriate quoting
    
    Also clean up some card-filtering tests so they work with a dirty DB
    
    * Fix deduplication
    
    * Remove use of setting/get
    
    * Remove unneeded migration
    
    ---------
    
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    
    * Fix test (#30383)
    
    ---------
    
    Co-authored-by: default avatarRyan Laurie <iethree@gmail.com>
    Co-authored-by: default avatarCal Herries <39073188+calherries@users.noreply.github.com>
    Co-authored-by: default avatarRyan Laurie <30528226+iethree@users.noreply.github.com>
    Co-authored-by: default avatarCallum Herries <hi@callumherries.com>
    Co-authored-by: default avatarNick Fitzpatrick <nick@metabase.com>
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
This project manages its dependencies using yarn. Learn more