docs - notes on memory usage (#46156)

* notes on memory usage

* typo

* Update observability-with-prometheus.md

* Update running.md

* update link

---------

Co-authored-by: Luis Paolini <paoliniluis@gmail.com>

docs/installation-and-operation/monitoring-metabase.md

@@ -7,36 +7,21 @@ redirect_from:
 
 # Monitoring Your Metabase
 
+Diagnosing performance-related issues can be a challenge. Luckily, the JVM ships with tools that can help diagnose many common issues. Enabling JMX and using a tool like VisualVM can help diagnose issues related to running out of memory, a stalled Metabase, or slow response times.
 
-Diagnosing performance related issues can be a challenge. Luckily the JVM ships with tools that can help diagnose many common issues. Enabling JMX and using a tool like VisualVM can help diagnose issues related to running out of memory, a stalled Metabase instance, and slow response times.
+This guide assumes that you have the VisualVM tool installed locally. VisualVM is included with OpenJDK and the Oracle JDK. You can find VisualJVM in the `bin` directory of the JDK installation. Some Linux distributions separate VisualVM from the JDK, in which case it's a separate `visualvm` package.
 
-This guide assumes that you have the VisualVM tool installed
-locally. VisualVM is included with OpenJDK and the Oracle JDK and is
-found in the `bin` directory of the JDK install. Some Linux
-distributions separate VisualVM from the JDK, in which case it's a
-separate `visualvm` package.
+## Connecting to a local Metabase
 
-## Connecting to a Local Metabase Instance
-
-If you have VisualVM installed on your Metabase server and are able to
-run VisualVM there, this is the easiest path as there is no need to
-setup remote communication with your metabase instance. In this
-scenario, start Metabase like you would normally and separately start
-VisualVM. Metabase will be listed under are running your Metabase
-instance
+If you have VisualVM installed on your Metabase server and are able to run VisualVM there, this is the easiest path as there is no need to setup remote communication with your metabase instance. In this scenario, start Metabase like you would normally and separately start VisualVM.
 
 ![localprocess](images/LocalProcessVisualVM.png)
 
-## Connecting to a Remote Metabase Instance
+## Connecting to a remote Metabase
 
-Monitoring a remote Metabase instance (or a local instance running in
-a docker container) is probably more common, but requires a more
-setup. First we need to specify some system properties that lets the
-JVM know that we want to allow remote monitoring. Assuming we are
-running Metabase using `java -jar metabase.jar`, we'd need change the
-invocation to the below command, which includes the properties:
+Monitoring a remote Metabase instance (or a local instance running in a docker container) is probably more common, but requires a more setup. First we need to specify some system properties that lets the JVM know that we want to allow remote monitoring. Assuming we are running Metabase using `java -jar metabase.jar`, we'd need change the invocation to the below command, which includes the properties:
 
-```
+```sh
 java --add-to-start=jmx,jmx-remote \
      -Dcom.sun.management.jmxremote \
      -Dcom.sun.management.jmxremote.port=1099 \
@@ -50,34 +35,22 @@ java --add-to-start=jmx,jmx-remote \
 
 Port `1099` is a typical RMI/JMX port, but it can be any accessible port.
 
-_Note:_ The above command opens up your application to monitoring by
-anyone and should only be used when on a trusted network for a short
-period of time. Securing this connection is possible, see [the Oracle
-docs](https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html)
-for more information.
+The above command opens up your application to monitoring by anyone and should only be used when on a trusted network for a short period of time. Securing this connection is possible, see [the Oracle
+docs](https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html).
 
-Users running Metabase in a `docker` container will need to specify
-the system properties detailed above and will also need to ensure the
-port is open. Docker allows specifying environment variables via a
-separate file that can be passed into the `docker run` invocation. You
-can create a file called `metabase-vars.env` with `JAVA_OPTS`
-specified:
+If you're running Metabase in a `docker` container, you'll need to specify the system properties detailed above and will also need to ensure the port is open. Docker allows specifying environment variables via a separate file that can be passed into the `docker run` invocation. You can create a file called `metabase-vars.env` with `JAVA_OPTS` specified:
 
 ```
 JAVA_OPTS=-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=<Metabase Hostname>
 ```
 
-_Note:_ This file expects to have each environment variable on it's own line with no line breaks
+This file expects to have each environment variable on its own line with no line breaks:
 
-```
+```sh
 docker run --env-file=metabase-vars.env -d -p 3000:3000 -p 1099:1099 -h <Metabase Hostname> --name metabase metabase/metabase
 ```
 
-The addition `-p 1099:1099` opens the JMX port for monitoring and the
-`--env-file=metabase-vars.env` passes in the extra JMX related
-environment variables. With the instance started, VisualVM needs to
-know how to connect to the running instance. First add a new remote
-instance:
+The addition `-p 1099:1099` opens the JMX port for monitoring and the `--env-file=metabase-vars.env` passes in the extra JMX related environment variables. With the instance started, VisualVM needs to know how to connect to the running instance. First add a new remote instance:
 
 ![addremotehost](images/AddRemoteHost.png)
 
@@ -85,13 +58,11 @@ Then use the hostname you specified above:
 
 ![sethostname](images/SetRemoteHostName.png)
 
-_Note:_ Your local machine needs to be able to refer to the hostname
-you specified which might require a hosts entry
+Your local machine must to be able to refer to the hostname you specified (which might require a hosts entry):
 
 ![addjmxhost](images/ClickAddJMXHost.png)
 
-The port specified for the JMX host needs to match the system property
-and the exposed port (if using Docker):
+If you're using Docker, the port specified for the JMX host needs to match the system property and the exposed port:
 
 ![jmxport](images/EnterJMXPort.png)
 
@@ -99,30 +70,23 @@ Next open the new remote JMX process:
 
 ![jmxinstance](images/OpenRemoteInstance.png)
 
-## Runtime Information
+## Runtime information
 
-Connecting to a running Metabase instance with VisualVM will make lots
-of runtime information available. This guide won't go over all of
-possibilities of the tool, but will highlight a few important pieces.
+Connecting to a running Metabase with VisualVM makes a lot of runtime information available. Here are some highlights:
 
-When running into memory related issues, typically the first question
-we want to know is what is consuming extra memory? A heap dump will
-take a snapshot of everything in memory at that specific point of
-time. That memory snapshot can be analyzed later by tools like the
-[Eclipse Memory Analyzer Tool](https://www.eclipse.org/mat/). Create a
-heap dump from the "Monitor" tab:
+### Heap dump
+
+When running into memory-related issues, typically the first question to ask is: what is consuming extra memory? A heap dump will take a snapshot of everything in memory at that specific point of time. That memory snapshot can be analyzed later by tools like the [Eclipse Memory Analyzer Tool](https://www.eclipse.org/mat/). Create a heap dump from the "Monitor" tab:
 
 ![heapdump](images/HeapDump.png)
 
-Another useful picture of a running Metabase system is a Thread
-Dump. In cases when Metabase appears hung or extremely slow, a thread
-dump will indicate what each thread is executing (or blocked on) for
-that specific point in time. Collect a thread dump via the "Threads"
-tab:
+### Thread dump
 
-![threaddump](images/ThreadDump.png)
+Another useful picture of a running Metabase system is a Thread Dump. In cases when Metabase appears stalled or extremely slow, a thread dump will indicate what each thread is executing (or blocked on) for that specific point in time Collect a thread dump via the "Threads" tab:
 
+![threaddump](images/ThreadDump.png)
 
 ## Further reading
 
+- [Running Metabase](../troubleshooting-guide/running.md)
 - [Observability with Prometheus](./observability-with-prometheus.md)

docs/installation-and-operation/observability-with-prometheus.md

@@ -10,23 +10,23 @@ You can export metrics in [Prometheus](https://prometheus.io/) format from your
 
 To give you an idea of how Metabase and Prometheus would work in your production environment, we'll walk through how to set up Metabase and Prometheus locally.
 
-### Start up Metabase with `MB_PROMETHEUS_SERVER_PORT`
+## Start up Metabase with `MB_PROMETHEUS_SERVER_PORT`
 
 Download the latest [Metabase JAR](https://www.metabase.com/start/oss/), and run Metabase using an environment variable to specify the Prometheus server port:
 
-```
+```sh
 MB_PROMETHEUS_SERVER_PORT=9191 java -jar metabase.jar
 ```
 
 The `MB_PROMETHEUS_SERVER_PORT=9191` specifies which port (`9191`) Metabase will use to send data to Prometheus. To clarify the ports that will be involved here:
 
 - Port `3000` is the port Metabase uses to serve the Metabase app. You can set another port with `MB_JETTY_PORT` (e.g., `MB_JETTY_PORT=3001`).
-  - Port `9191` (or whichever port you specified with the `MB_PROMETHEUS_SERVER_PORT` environment variable) is the port Prometheus uses to scrape metrics from Metabase.
+- Port `9191` (or whichever port you specified with the `MB_PROMETHEUS_SERVER_PORT` environment variable) is the port Prometheus uses to scrape metrics from Metabase.
 - Port `9090` is the port Prometheus uses to serve the Prometheus application.
 
 When you start Metabase, the Metabase logs will tell you that Metabase is starting the `prometheus metrics collector` and `prometheus metrics web-server`.
 
-```
+```sh
 (truncated logs)
 2022-09-01 17:47:38,808 INFO metabase.util :: Database setup took 3.4 s
 2022-09-01 17:47:38,826 INFO metabase.core :: Setting up prometheus metrics
@@ -37,15 +37,15 @@ When you start Metabase, the Metabase logs will tell you that Metabase is starti
 
 You can view your locally running Metabase at `http://localhost:3000`.
 
-### Download and configure Prometheus
+## Download and configure Prometheus
 
 [Download Prometheus](https://prometheus.io/download), and extract the files.
 
 Change into the Prometheus directory, add the following YAML file to configure your Prometheus:
 
-#### Prometheus configuration file example
+## Prometheus configuration file example
 
-```
+```yaml
 global:
   scrape_interval:     15s # By default, scrape targets every 15 seconds.
 
@@ -67,11 +67,13 @@ scrape_configs:
       - targets: ['localhost:9191']
 ```
 
-### Running Prometheus Locally
+You need to change the "target" to where Metabase is, for this particular example, Metabase resides in the same host where Prometheus is running ("localhost").
+
+## Running Prometheus Locally
 
 In a new terminal process in the Prometheus directory, run:
 
-```
+```sh
 ./prometheus --config.file=prometheus.yml
 ```
 
@@ -83,7 +85,7 @@ Then check `http://localhost:9090`. You should see the Prometheus app, and be ab
 
 Here is some sample output from Metabase:
 
-```
+```yaml
 '# HELP jvm_threads_current Current thread count of a JVM
 '# TYPE jvm_threads_current gauge
 jvm_threads_current 81.0
@@ -155,8 +157,13 @@ Metrics exported by Metabase include:
 - `process_max_fds`
 - `process_open_fds`
 - `process_start_time_seconds`
-
+- `process_virtual_memory_bytes`
+- `metabase_email_messages_total`
+- `metabase_email_messages_created`
+- `metabase_email_message_errors_total`
+- `metabase_email_message_errors_created`
 
 ## Further reading
 
+- [Running Metabase](../troubleshooting-guide/running.md)
 - [Monitoring Metabase](./monitoring-metabase.md)

docs/permissions/introduction.md

@@ -71,7 +71,7 @@ Whenever you change permissions for a group, make sure you:
 [dashboard-subscriptions]: ../dashboards/subscriptions.md
 [data-permissions]: ./data.md
 [data-sandboxing]: ./data-sandboxes.md
-[permissions]: https://www.metabase.com/learn/permissions
+[permissions]: https://www.metabase.com/learn/metabase-basics/administration/permissions
 [sandbox-columns]: https://www.metabase.com/learn/permissions/data-sandboxing-column-permissions
 [sandbox-rows]: https://www.metabase.com/learn/permissions/data-sandboxing-row-permissions
 [slack-integration]: ../configuring-metabase/slack.md

docs/troubleshooting-guide/running.md

@@ -10,17 +10,23 @@ Metabase runs on the Java Virtual Machine (JVM), and depending on how it's confi
 
 Make sure you're using a Java version of 11 or higher.
 
-## WARNING: sun.reflect.Reflection.getCallerClass is not supported
+# Metabase's memory usage
 
-Don't worry about it.
+Metabase ships as a JAR file that runs on the Java Virtual Machine (JVM).
 
-```
-WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
-```
+It's important to distinguish _Metabase's_ memory usage from the _JVM's_ memory usage.
 
-If you see the above error, ignore it. Your Metabase is perfectly healthy and performing as it should.
+The JVM will consume a constant amount of memory. By default, the JVM will use about one fourth of a machine's RAM (though you can [change how much RAM you want the JVM to use](#allocating-more-memory-to-the-jvm)).
+
+JVM applications (like Metabase) will consume and release the RAM allocated to the JVM. The JVM, however, won't release unused RAM to the machine; the JVM's memory use will be constant.
+
+So on a machine with 8 GB of RAM, by default the JVM will use 2 GB of RAM. Metabase will use some or all of these 2 GBs of JVM-allocated RAM, depending on Metabase's activity. But from the machine's perspective, the JVM will always be using that allocated 2GB of RAM, even when Metabase is only using a fraction of that allocated RAM.
 
-## Metabase fails to start due to Heap Space OutOfMemoryErrors
+## Diagnosing memory issues
+
+Given the above explanation of how the JVM handles memory, if you're having performance issues with Metabase that you don't think are due to your data warehouse, you'll want to check for these red flags:
+
+## Metabase crashes due to Java heap space `OutOfMemoryError`
 
 The JVM can normally figure out how much RAM is available on the system and automatically set a sensible upper bound for heap memory usage. On certain shared hosting environments, however, this doesn't always work as desired. The usual symptom of this is an error message like:
 
@@ -28,17 +34,34 @@ The JVM can normally figure out how much RAM is available on the system and auto
 java.lang.OutOfMemoryError: Java heap space
 ```
 
-If you are seeing this, you need to set a JVM option to tell Java know explicitly how much memory it should use for the heap. For example, your Java runtime might use the `-X` flag to do this:
+If you're seeing this "Out of memory" (OOM) error, you'll need to [allocate more memory to the JVM](#allocating-more-memory-to-the-jvm).
 
-```
+### When viewing memory usage over time as a line chart, you see a sawtooth pattern
+
+You can use tools to view how Metabase uses the memory available to it over time. Check out:
+
+- [Observability with Prometheus](../installation-and-operation/observability-with-prometheus.md)
+- [Monitoring your Metabase](../installation-and-operation/monitoring-metabase.md)
+
+The specific Prometheus metric you need to check is jvm_memory_bytes_used{area="heap"}
+
+A red flag to look out for: the sawtooth pattern. Metabase will quickly consume a lot of memory, which will trigger garbage collection, which frees up memory, which Metabase quickly consumes again. This up-down-up-down pattern of memory usage is the signature of frequent garbage collection cycles. The garbage collection will tie up CPU cycles, which can slow down your application.
+
+If you're seeing this, you'll need to [increase the amount of memory allocated to the JVM](#allocating-more-memory-to-the-jvm).
+
+## Allocating more memory to the JVM
+
+You can set a JVM option to allocate more memory to the JVM's heap. For example, your Java runtime might use the `-X` flag to do this:
+
+```sh
 java -Xmx2g -jar metabase.jar
 ```
 
-Adjust the memory allocation upward until Metabase seems happy, but make sure to keep the number lower than the total amount of RAM available on your machine, because Metabase won't be the only process running. Leaving 1--2 GB of RAM for other processes is generally enough, so you might set `-Xmx` to `1g` on a machine with 2 GB of RAM, `2g` on one with 4 GB of RAM, and so on. You may need to experiment with this settings to find one that makes Metabase and everything else play nicely together.
+Adjust the memory allocation upward until Metabase seems happy, but make sure to keep the number lower than the total amount of RAM available on your machine, because Metabase won't be the only process running. Leaving 1 to 2 GB of RAM for other processes on the machine is generally enough, so you might set `-Xmx` to `1g` on a machine with 2 GB of RAM, `2g` on one with 4 GB of RAM, and so on. You may need to experiment with this settings to find one that makes Metabase and everything else play nicely together (and this experimentation may require upgrading to a machine with more memory).
 
 You can also use the environment variable `JAVA_OPTS` to set JVM args instead of passing them directly to `java`. This is particularly useful when running the Docker image:
 
-```
+```sh
 docker run -d -p 3000:3000 -e "JAVA_OPTS=-Xmx2g" metabase/metabase
 ```
 
@@ -50,10 +73,20 @@ If the Metabase instance starts and runs for a significant amount of time before
 java -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/a/directory -jar metabase-jar
 ```
 
-The `-XX:HeapDumpPath` flag specifies where to put the dump---the current directory is the default. When an `OutOfMemoryError` occurs, it will dump an `hprof` file to the directory specified. These can be very large (i.e., the size of the `-Xmx` argument) so ensure your disk has enough space. These `hprof` files can be read with many different tools, such as `jhat` (which is included with the JDK) or the [Eclipse Memory Analyzer Tool][eclipse-memory-analyzer].
+The `-XX:HeapDumpPath` flag specifies where to put the dump---the current directory is the default. When an `OutOfMemoryError` occurs, the JVM will dump an `hprof` file to the directory specified. These `hprof` files can be large (the size of the `-Xmx` argument) so make sure your disk has enough space. These `hprof` files can be read with many different tools, such as `jhat` (which is included with the JDK) or the [Eclipse Memory Analyzer Tool][eclipse-memory-analyzer].
 
 ## Metabase cannot read or write from a file or folder (IOError)
 
 If you see an error regarding file permissions, like Metabase being unable to read a SQLite database or a custom GeoJSON map file, check out the section "Metabase can't read to/from a file or directory" in our [Docker troubleshooting guide](./docker.md).
 
+## WARNING: sun.reflect.Reflection.getCallerClass is not supported
+
+Don't worry about it.
+
+```
+WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
+```
+
+If you see the above error, ignore it. Your Metabase is perfectly healthy and performing as it should.
+
 [eclipse-memory-analyzer]: https://www.eclipse.org/mat/