Skip to content
Snippets Groups Projects
This project is mirrored from https://github.com/metabase/metabase. Pull mirroring updated .
  1. Mar 12, 2021
    • Robert Roland's avatar
      Clean up sync scheduling (#15043) · 6668b4ee
      Robert Roland authored
      It's possible for the scheduler to get in a weird state if the sync
      fails while it executes.
      
      This change makes it *only* recreate a job/task if the schedule has
      changed or if it is missing. Previously, clearing the state at every
      start had bad effects if the JVM had terminated during the sync.
      
      Adds a vector of Exception classes that signal a "fatal" exception
      during sync for a specific database. If these exceptions occur, the sync
      for that database stops and will pick up next time.
      
      This will have to be expanded per driver, but I don't see a way around
      that, as each driver will have its own, unique way of failing.
      
      metabase/metabase#14817
      Unverified
      6668b4ee
  2. Oct 23, 2020
  3. Apr 02, 2019
  4. Feb 04, 2019
  5. Jul 09, 2018
  6. Jan 02, 2018
    • Ryan Senior's avatar
      Change database sync and field values job scheduling [ci drivers] · b94938e4
      Ryan Senior authored
      Previously we configured each database to have separate trigger + job
      instances for sync+analyze and field values scanning. We were using
      the default threadcount and "do nothing" upon misfires. This resulted
      in a race condition like below.
      
      By default, sync and field-values scans are scheduled at 50 minutes
      past the hour (two separate jobs). As an example assume we have 10
      databases configured. This would result in 20 jobs firing at 50
      minutes past the hour. The order of execution of these jobs is not
      defined. By default, we have 4 threads to execute these jobs. If the 4
      threads are busy when a trigger occurs, it will wait up to one
      minute (the default). If it can't get a thread in 1 minute, it will
      misfire. The policy we had setup will discard the job when it
      misfires. The end result of this is a race condition where we could
      potentially not run sync on many databases and also be running 4 scans
      or syncs at the same time, potentially causing memory issues.
      
      This commit switches to a single job for sync and a single job for
      field value scans. Every database will have a trigger for sync and a
      trigger for field values, but only 1 instance of sync and field values
      scanning will run at a time. In the event of a misfire, it will
      execute it as soon as it's able and only once.
      b94938e4
  7. Jun 15, 2016
Loading