Troubleshooting the app database (#17669)

docs/operations-guide/migrating-from-h2.md

@@ -17,7 +17,7 @@ docker cp metabase:/metabase.db/metabase.db.mv.db ./
 
 The above command would copy the database file to the directory you ran the command from. With your database file outside of the container, all you need to do is follow the "How to migrate" steps below.
 
-## How to migrate 
+## How to migrate
 
 Metabase provides a custom migration command for upgrading H2 application database files by copying their data to a new database. Here's what you'll do:
 
@@ -51,4 +51,4 @@ Metabase expects that you'll run the command against a brand-new (empty) databas
 
 ### Troubleshooting
 
-If you get an error, check out [Error when loading application database from H2](../troubleshooting-guide/loading-from-h2.md).
+If you get an error, check out [this troubleshooting guide](../troubleshooting-guide/loading-from-h2.md).

docs/operations-guide/running-metabase-on-heroku.md

@@ -37,7 +37,7 @@ Now that you’ve installed Metabase, it’s time to [set it up and connect it t
 ### Troubleshooting
 
 * If your Metabase instance is getting stuck part way through the initialization process and only every shows roughly 30% completion on the loading progress.
-    * The most likely culprit here is a stale database migrations lock that was not cleared.  This can happen if for some reason Heroku kills your Metabase dyno at the wrong time during startup.  __To fix it:__ you can either clear the lock using the built-in [release-locks](../troubleshooting-guide/application-database.md) command line function, or if needed you can login to your Metabase application database directly and delete the row in the `DATABASECHANGELOGLOCK` table.  Then just restart Metabase.
+    * The most likely culprit here is a stale database migrations lock that was not cleared.  This can happen if for some reason Heroku kills your Metabase dyno at the wrong time during startup.  __To fix it:__ you can either clear the lock using the built-in [release-locks](../troubleshooting-guide/loading-from-h2.md) command line function, or if needed you can login to your Metabase application database directly and delete the row in the `DATABASECHANGELOGLOCK` table.  Then just restart Metabase.
 
 
 ## Deploying New Versions of Metabase
@@ -51,7 +51,7 @@ Here's each step:
 * Clone the metabase-deploy repo to your local machine:
 
 ```bash
-git clone https://github.com/metabase/metabase-deploy.git  
+git clone https://github.com/metabase/metabase-deploy.git
 cd metabase-deploy
 ```
 
@@ -97,7 +97,7 @@ git push master
 
 ### Database Syncs
 
-You may want to ensure that your staging database is synced with production before you deploy a new version. Luckily with Heroku you can restore a backup from one app to another. 
+You may want to ensure that your staging database is synced with production before you deploy a new version. Luckily with Heroku you can restore a backup from one app to another.
 
 For example, assuming your production app is named `awesome-metabase-prod`, this command will create a backup:
 
@@ -117,12 +117,12 @@ Once this is done, restart your staging app and begin testing.
 
 ### Pinning Metabase versions
 
-For whatever reason, should you want to pin Metabase to a specific version, you can append the version number to the buildpack URL (as long as that tag exists in the [`metabase-buildpack`](https://github.com/metabase/metabase-buildpack) repository). 
+For whatever reason, should you want to pin Metabase to a specific version, you can append the version number to the buildpack URL (as long as that tag exists in the [`metabase-buildpack`](https://github.com/metabase/metabase-buildpack) repository).
 
 If you haven't cloned the `metabase-deploy` repository, this can be done with the Heroku CLI:
 
 ```bash
-heroku buildpacks:set --index 2 https://github.com/metabase/metabase-buildpack#0.34.1 \ 
+heroku buildpacks:set --index 2 https://github.com/metabase/metabase-buildpack#0.34.1 \
   --app <YOUR-APP-NAME>
 ```
 

docs/troubleshooting-guide/application-database.md

-# Trouble with the Metabase Application Database
-
-<div class='doc-toc' markdown=1>
-- [Metabase fails to start due to database locks](#database-locks)
-- [Metabase H2 application database gets corrupted](#h2-app-db-corrupted)
-- [Metabase fails to connect to H2 Database on Windows 10](#h2-connect-windows-10)
-</div>
-
-Metabase stores information about users, questions, and so on in a database of its own that we call the "application database". The default application database that ships with Metabase is an [H2 database][what-is-h2], which is not recommended for production. To use a production-ready database for you Metabase application database, see [Migrating from H2][migrating]. 
-
-<h2 id="database-locks">Metabase fails to start due to database locks</h2>
-
-Sometimes Metabase will fail to start up because a database lock did not clear properly. 
-
-**How to detect this:** The error message will look something like:
-
-```
-liquibase.exception.DatabaseException: liquibase.exception.LockException: Could not acquire change log lock.
-```
-
-**How to fix this:** When this happens, open a shell on the server where Metabase is installed and run:
-
-```
-java -jar metabase.jar migrate release-locks
-```
-
-This command will manually clear the locks. When it's done, restart your Metabase instance.
-
-<h2 id="h2-app-db-corrupted">Metabase H2 application database gets corrupted</h2>
-
-By default, Metabase uses [H2][what-is-h2] for its application database. Because H2 is an on-disk database, it's sensitive to filesystem errors, such as a drive being corrupted or a file not being flushed properly. In these situations, you'll see errors on startup. 
-
-**How to detect this:** Error messages will vary, but one example is:
-
-```
-myUser@myIp:~$ java -cp metabase.jar org.h2.tools.RunScript -script whatever.sql -url jdbc:h2:~/metabase.db
-Exception in thread "main" org.h2.jdbc.JdbcSQLException: Row not found when trying to delete from index """"".I37: ( /* key:7864 */ X'5256470012572027c82fc5d2bfb855264ab45f8fec4cf48b0620ccad281d2fe4', 165)" [90112-194]
-    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
-    [etc]
-```
-
-**How to fix this:** To attempt to recover a corrupted H2 file with a recent version of Metabase, try the commands shown below:
-
-```
-java -cp metabase.jar org.h2.tools.Recover
-mv metabase.db.mv.db metabase.old.db
-touch metabase.db.mv.db
-java -cp target/uberjar/metabase.jar org.h2.tools.RunScript -script metabase.db.h2.sql -url jdbc:h2:`pwd`/metabase.db
-```
-
-If you're using a legacy Metabase H2 application database (where the database file is named 'metabase.db.h2.db') use the command below instead:
-
-```
-java -cp metabase.jar org.h2.tools.Recover
-mv metabase.db.h2.db metabase.old.db
-touch metabase.db.h2.db
-java -cp target/uberjar/metabase.jar org.h2.tools.RunScript -script metabase.db.h2.sql -url jdbc:h2:`pwd`/metabase.db;MV_STORE=FALSE
-```
-
-Not all H2 errors are recoverable (which is why if you're using H2, _please_ have a backup strategy for the application database file).
-
-<h2 id="h2-connect-windows-10">Metabase fails to connect to H2 Database on Windows 10</h2>
-
-In some situations on Windows 10, the Metabase JAR needs to have permissions to create local files for the application database. 
-
-**How to detect this:** If the Metabase JAR lacks permissions, you might see an error message like this when running the JAR:
-
-```
-Exception in thread "main" java.lang.AssertionError: Assert failed: Unable to connect to Metabase DB.
-```
-
-**How to fix this:** You can unblock the file by right-clicking on it, clicking "Properties," and then clicking "Unblock." 
-
-[what-is-h2]: ../faq/setup/what-is-h2.html
-[migrating]: ../operations-guide/migrating-from-h2.html

docs/troubleshooting-guide/index.md

@@ -30,9 +30,7 @@ Problems, their causes, how to detect them, and how to fix them.
 
 - [Metabase isn't sending email][not-sending-email].
 
-- [Using the default H2 application database][appdb].
-
-- [Loading an exported H2 application database][loadh2].
+- [Using or migrating from an H2 application database][appdb].
 
 - [The dates and times in my questions and charts are wrong][incorrect-times].
 
@@ -92,7 +90,6 @@ Metabase adds new features and squashes bugs with each release. [Upgrading to th
 [learn]: https://www.metabase.com/learn
 [linked-filters]: ./linked-filters.html
 [login]: ./cant-log-in.html
-[loadh2]: ./loading-from-h2.html
 [not-sending-email]: ./cant-send-email.html
 [proxies]: ./proxies.html
 [releases]: https://github.com/metabase/metabase/releases

docs/troubleshooting-guide/loading-from-h2.md

-# Loading exported application database fails
+---
+redirect_from:
+  - ./loading-from-h2.html
+---
 
-If you've been using the default H2 application database that ships with Metabase, and want to [migrate from the default H2 application database][migrate] to a production database like [PostgreSQL][postgres] or MySQL/MariaDB, you'll need to use the `load-from-h2` command, which will fail if the database filename is incorrect. 
+# Using or migrating from an H2 application database
 
-**How to detect this:** When running the `load-from-h2` command, you'll see an error that looks something like:
+You have installed Metabase, but:
+
+- You're trying to migrate the application database from H2 to another database and something has gone wrong,
+- You're trying to downgrade rather than upgrade,
+- Metabase logs a `liquibase` error message when you try to run it,
+- Metabase logs another error message that mentions `H2` or `h2` while it is running, or
+- You're on Windows 10 and get a warning about file permissions.
+
+## Are you currently using H2 as your application database?
+
+**Root cause:** Metabase stores information about users, questions, and so on in a database of its own called the "application database", or "app database" for short. By default Metabase uses [H2][what-is-h2] for the app database, but we don't recommended it for production---because it's an on-disk database, it's sensitive to filesystem errors, such as a drive being corrupted or a file not being flushed properly.
+
+**Steps to take:**
+
+1. To check what you're using as the app database, go to **Admin Panel**, open the **Troubleshooting** tab, scroll down to "Diagnostic Info", and look for the `application-database` key in the JSON it displays.
+2. See [Migrating from H2][migrate] for instructions on how to migrate to a more robust app database.
+
+## Are you trying to migrate the application database from H2 to something else?
+
+**Root cause:** You are trying to [migrate][migrate] the app database from H2 to a production database such as PostgreSQL or MySQL/MariaDB using the `load-from-h2` command, but this has failed because the database filename is incorrect with an error message like:
+
+```
+Command failed with exception: Unsupported database file version or invalid file header in file <YOUR FILENAME>
+```
+
+**Steps to take:**
+
+1.  Create a copy of the exported H2 database (see [Backing up Metabase Application Data][backup]). _Do not proceed until you have done this_ in case something goes wrong.
+
+2.  Check that the H2 database file you exported is named `metabase.db.mv.db`.
+
+3.  H2 automatically adds `.mv.db` extension to the database path you specify on the command line, so make sure the path to the DB file you pass to the command does _not_ include the `.mv.db` extension. For example, if you've exported an application database, and you want to load the data from that H2 database into a PostgreSQL database using `load-from-h2`, your command will look something like:
+
+    ```
+    export MB_DB_TYPE=postgres
+    export MB_DB_DBNAME=metabase
+    export MB_DB_PORT=5432
+    export MB_DB_USER=<username>
+    export MB_DB_PASS=<password>
+    export MB_DB_HOST=localhost
+    java -jar metabase.jar load-from-h2 /path/to/metabase.db # do not include .mv.db
+    ```
+
+If you're using [Metabase Enterprise Edition][enterprise], you can use [serialization][serialization-docs] to snapshot your application database. Serialization is useful when you want to [preload questions and dashboards][serialization-learn] in a new Metabase instance.
+
+## Are you trying to downgrade?
+
+**Root cause:** Metabase does not support downgrading (i.e., reverting to an early version of the application).
+
+**Steps to take:**
+
+1.  Shut down Metabase.
+2.  Restore the backup copy of the app database you made before trying to upgrade or downgrade.
+3.  Restore the JAR file or container of the older version you want to revert to.
+4.  Restart Metabase.
+
+## Is the app database locked?
+
+**Root cause:** Sometimes Metabase fails to start up because an app database lock did not clear properly during a previous run. The error message looks something like:
 
 ```
-Command failed with exception: Unsupported database file version or invalid file header in file <YOUR FILENAME> 
+liquibase.exception.DatabaseException: liquibase.exception.LockException: Could not acquire change log lock.
 ```
 
-**How to fix this:** First, make sure to create a copy of the exported H2 database (see [Backing up Metabase Application Data][backup]).
+**Steps to take:**
 
-Next, check that the exported, application database file (the H2 database you exported) is named `metabase.db.mv.db`. H2 automatically adds a `.mv.db` extension to the database path you specify, so make sure the path to the DB file you pass to the command does not include the `.mv.db` extension.
+1.  Open a shell on the server where Metabase is installed and manually clear the locks by running:
 
-For example, if you've exported an application database, and you want to load the data from that H2 database into a PostgreSQL database using `load-from-h2`, your command will look something like:
+    ```
+    java -jar metabase.jar migrate release-locks
+    ```
+
+2.  Once this command completes, restart your Metabase instance normally (_without_ the `migrate release-locks` flag).
+
+## Is the app database corrupted?
+
+**Root cause:** H2 is less reliable than production-quality database management systems, and sometimes the database itself becomes corrupted. This can result in loss of data in the app database, but can _not_ damage data in the databases that Metabase is connected.
+
+**Steps to take:** Error messages can vary depending on how the app database was corrupted, but in most cases the log message will mention `h2`. A typical command and message are:
+
+```
+myUser@myIp:~$ java -cp metabase.jar org.h2.tools.RunScript -script whatever.sql -url jdbc:h2:~/metabase.db
+Exception in thread "main" org.h2.jdbc.JdbcSQLException: Row not found when trying to delete from index """"".I37: ( /* key:7864 */ X'5256470012572027c82fc5d2bfb855264ab45f8fec4cf48b0620ccad281d2fe4', 165)" [90112-194]
+    at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
+    [etc]
+```
+
+**How to fix this:** not all H2 errors are recoverable (which is why if you're using H2, _please_ have a backup strategy for the application database file).
+
+If you are running a recent version and using H2, the app database is stored in `metabase.db.mv.db`. - Open a shell on the server where the Metabase instance is running and attempt to recover the corrupted H2 file by running the following four commands:
+
+```
+java -cp metabase.jar org.h2.tools.Recover
+
+mv metabase.db.mv.db metabase-old.db.mv.db
+
+touch metabase.db.mv.db
+
+java -cp target/uberjar/metabase.jar org.h2.tools.RunScript -script metabase.db.h2.sql -url jdbc:h2:`pwd`/metabase.db
+```
+
+## Are you running Metabase with H2 on Windows 10?
+
+**Root cause:** In some situations on Windows 10, the Metabase JAR needs to have permissions to create local files for the application database. When running the JAR, you'll see an error message like this:
 
 ```
-export MB_DB_TYPE=postgres
-export MB_DB_DBNAME=metabase
-export MB_DB_PORT=5432
-export MB_DB_USER=<username>
-export MB_DB_PASS=<password>
-export MB_DB_HOST=localhost
-java -jar metabase.jar load-from-h2 /path/to/metabase.db # do not include .mv.db
+Exception in thread "main" java.lang.AssertionError: Assert failed: Unable to connect to Metabase DB.
 ```
 
-If you're using Metabase Enterprise Edition, you should check out the [Serialization][serialization-docs] feature to snapshot your application database. Serialization is useful when you want to [preload questions and dashboards][serialization-learn] in a new Metabase instance.
+**Steps to take:**
+
+1.  Right-click on the Metabase JAR file (_not_ the app database file).
+2.  Select "Properties".
+3.  Select "Unblock."
+
+## Is the application database taking too long to load?
+
+**Root cause:** You're using H2 as your app database, and the app database is so large that it can't be loaded in less than 5 seconds (which is the default timeout value). You'll see the message "Timeout" appear in the console when you try to start Metabase.
+
+**Steps to take:**
+
+1.  Use a production-quality database such as PostgreSQL for the app database (preferred).
+2.  Go to the **Admin Panel** and increase the timeout setting for the app database.
+3.  Move Metabase to a faster server (in particular, a server with faster disks).
 
-[backup]: ../operations-guide/backing-up-metabase-application-data.html
-[migrate]: ../operations-guide/migrating-from-h2.html
-[postgres]: https://www.postgresql.org/
-[serialization-docs]: ../enterprise-guide/serialization.html
-[serialization-learn]: https://www.metabase.com/learn/administration/serialization.html
+[backup]: ../operations-guide/backing-up-metabase-application-data.md
+[enterprise]: /enterprise/
+[migrate]: ../operations-guide/migrating-from-h2.md
+[serialization-docs]: ../enterprise-guide/serialization.md
+[serialization-learn]: /learn/administration/serialization.html
+[what-is-h2]: ../faq/setup/what-is-h2.md