-
dpsutton authored
* dir locals for api/let-404 * Driver supports persisted model * PersistedInfo model far easier to develop this model with the following sql: ```sql create table persisted_info ( id serial primary key not null ,db_id int references metabase_database(id) not null ,card_id int references report_card(id) not null ,question_slug text not null ,query_hash text not null ,table_name text not null ,active bool not null ,state text not null ,UNIQUE (db_id, card_id) ) ``` and i'll make the migration later. Way easier to just dorp table, \i persist.sql and keep developing without worrying about the migration having changed so it can't rollback, SHAs, etc * Persisting api (not making/deleting tables yet) http POST "localhost:3000/api/card/4075/persist" Cookie:$COOKIE -pb http DELETE "localhost:3000/api/card/4075/persist" Cookie:$COOKIE -pb useful from commandline (this is httpie) * Pull format-name into ddl.i * Postgres ddl * Hook up endpoints * move schema-name into interface * better jdbc connection management * Hotswap peristed tables into qp * clj-kondo fixes * docstrings * bad alias in test infra * goodbye testing format-name function left over. everything uses ddl.i/format-name and this rump was left * keep columns in persisted info columns that are in the persisted query. I thought about a tuple of [col-name type] instead of just the col-name. I didn't do this this type because I want to ensure that we compute the db-type in ONLY ONE WAY ever and i wasn't ready to commit to that yet. I'm not sure this is necessary in the future so it remains out now. Context: we hot-swap the persisted table in for the original query. Match up on query hash remaining the same. It continues to use the metadata from the original query and just `select cols from table` * Add migration for persisted_info table also removes the db_id. Don't know why i was thinking that was necessary. also means we don't need another unique constraint on (db_id, card_id) since we can just mark the card_id as unique. no idea what i was thinking. * fix ns in a sad manner :( far better to just have no alias to indicate it is required for side effects. * Dont hardcode a card-id :(:(:( my B * copy the PersistedInfo * ns cleanup, wrong alias, reflection warning * Check that state of persisted_info is persisted * api to enable persistence on a db i'm not wild about POST /api/database/:id/persist and POST /api/database/:id/unpersist but carrying on. left a note about it. So now you can enable persistence on a db, enable persistence on a model by posting to api/card/:id/persist and everything works. What does not work yet is the unpersisting or re-persisting of models when using the db toggle. * Add refresh_begin and refresh_end to persisted_info This information helps us with two bits: - when we need to chunk refreshing models, this lets us order by staleness so we can refresh a few models and pick up later - if we desire, we can look at the previous elapsed time of refreshes and try to gauge amount of work we want. This gives us a bit of look-ahead. We can of course track our progress as we go but there's no way to know if the next refresh might take an hour. This gives us a bit of insight. * Refresh tables every 8 hours ("0 0 0/8 * * ? *") Tables are refreshed every 8 hours. There is one single job doing this named "metabase.task.PersistenceRefresh.job" but it has 0 triggers by default. Each database with persisted models will add a trigger to this to refresh those models every 8 hours. When you unpersist a model, it will immediately remove the table and then delete the persisted_info record. When you mark a database as persist false, it will immediately mark all persisted_info rows as inactive and deleteable, and unschedule its trigger. A background thread will then start removing the tables. * Schedule refreshing on startup, watching for already scheduled does not allow for schedule changes but that's a future endeavor * appease our linter overlords * Dynamic var to inbhit persistence when refreshing also, it checked the state against "active" instead of "persisted" which is really freaky. how has this worked in the past if thats the case? * api docstrings on card persist * docstring * Don't sync the persisted schemas * Fix bad sql when no deleteable rows getting error with bad sql when there were no ids * TaskHistory for refreshing * Add created_at to persist_info table helpful if this ever ends up in the audit section * works on redshift hooked up the hierarchy and redshift is close enought that it just works * Remove persist_info record after deleting "deleteable" * Better way to check that something exists * POST /api/<card-id>/refresh api to refresh a model's persisted record * return a 204 from refreshing * Add buttons to persist/unpersist a database and a model for PoC (#21344) * Redshift and postgres report true for persist-models there are separate notions of persistence is possible vs persistence is enabled. Seems like we're just gonna check details for enabled and rely on the driver multimethod for whether it is possible. * feature for enabled, hydrate card with persisted two features: :persist-models for which dbs support it, and :persist-models-enabled for when that option is enabled. POST to api/<card-id>/unpersist hydrate persisted on cards so FE can display persist/unpersist for models * adjust migration number * remove deferred-tru
Alexander Polyankin <alexander.polyankin@metabase.com> Co-authored-by:
Anton Kulyk <kuliks.anton@gmail.com> Co-authored-by:
Case Nelson <case@metabase.com>
dpsutton authored* dir locals for api/let-404 * Driver supports persisted model * PersistedInfo model far easier to develop this model with the following sql: ```sql create table persisted_info ( id serial primary key not null ,db_id int references metabase_database(id) not null ,card_id int references report_card(id) not null ,question_slug text not null ,query_hash text not null ,table_name text not null ,active bool not null ,state text not null ,UNIQUE (db_id, card_id) ) ``` and i'll make the migration later. Way easier to just dorp table, \i persist.sql and keep developing without worrying about the migration having changed so it can't rollback, SHAs, etc * Persisting api (not making/deleting tables yet) http POST "localhost:3000/api/card/4075/persist" Cookie:$COOKIE -pb http DELETE "localhost:3000/api/card/4075/persist" Cookie:$COOKIE -pb useful from commandline (this is httpie) * Pull format-name into ddl.i * Postgres ddl * Hook up endpoints * move schema-name into interface * better jdbc connection management * Hotswap peristed tables into qp * clj-kondo fixes * docstrings * bad alias in test infra * goodbye testing format-name function left over. everything uses ddl.i/format-name and this rump was left * keep columns in persisted info columns that are in the persisted query. I thought about a tuple of [col-name type] instead of just the col-name. I didn't do this this type because I want to ensure that we compute the db-type in ONLY ONE WAY ever and i wasn't ready to commit to that yet. I'm not sure this is necessary in the future so it remains out now. Context: we hot-swap the persisted table in for the original query. Match up on query hash remaining the same. It continues to use the metadata from the original query and just `select cols from table` * Add migration for persisted_info table also removes the db_id. Don't know why i was thinking that was necessary. also means we don't need another unique constraint on (db_id, card_id) since we can just mark the card_id as unique. no idea what i was thinking. * fix ns in a sad manner :( far better to just have no alias to indicate it is required for side effects. * Dont hardcode a card-id :(:(:( my B * copy the PersistedInfo * ns cleanup, wrong alias, reflection warning * Check that state of persisted_info is persisted * api to enable persistence on a db i'm not wild about POST /api/database/:id/persist and POST /api/database/:id/unpersist but carrying on. left a note about it. So now you can enable persistence on a db, enable persistence on a model by posting to api/card/:id/persist and everything works. What does not work yet is the unpersisting or re-persisting of models when using the db toggle. * Add refresh_begin and refresh_end to persisted_info This information helps us with two bits: - when we need to chunk refreshing models, this lets us order by staleness so we can refresh a few models and pick up later - if we desire, we can look at the previous elapsed time of refreshes and try to gauge amount of work we want. This gives us a bit of look-ahead. We can of course track our progress as we go but there's no way to know if the next refresh might take an hour. This gives us a bit of insight. * Refresh tables every 8 hours ("0 0 0/8 * * ? *") Tables are refreshed every 8 hours. There is one single job doing this named "metabase.task.PersistenceRefresh.job" but it has 0 triggers by default. Each database with persisted models will add a trigger to this to refresh those models every 8 hours. When you unpersist a model, it will immediately remove the table and then delete the persisted_info record. When you mark a database as persist false, it will immediately mark all persisted_info rows as inactive and deleteable, and unschedule its trigger. A background thread will then start removing the tables. * Schedule refreshing on startup, watching for already scheduled does not allow for schedule changes but that's a future endeavor * appease our linter overlords * Dynamic var to inbhit persistence when refreshing also, it checked the state against "active" instead of "persisted" which is really freaky. how has this worked in the past if thats the case? * api docstrings on card persist * docstring * Don't sync the persisted schemas * Fix bad sql when no deleteable rows getting error with bad sql when there were no ids * TaskHistory for refreshing * Add created_at to persist_info table helpful if this ever ends up in the audit section * works on redshift hooked up the hierarchy and redshift is close enought that it just works * Remove persist_info record after deleting "deleteable" * Better way to check that something exists * POST /api/<card-id>/refresh api to refresh a model's persisted record * return a 204 from refreshing * Add buttons to persist/unpersist a database and a model for PoC (#21344) * Redshift and postgres report true for persist-models there are separate notions of persistence is possible vs persistence is enabled. Seems like we're just gonna check details for enabled and rely on the driver multimethod for whether it is possible. * feature for enabled, hydrate card with persisted two features: :persist-models for which dbs support it, and :persist-models-enabled for when that option is enabled. POST to api/<card-id>/unpersist hydrate persisted on cards so FE can display persist/unpersist for models * adjust migration number * remove deferred-tru
Code owners
Assign users and groups as approvers for specific file changes. Learn more.