Merge branch 'master' of github.com:metabase/metabase into query-lib

README.md

@@ -8,8 +8,6 @@ Metabase is the easy, open source way for everyone in your company to ask questi
 [![Circle CI](https://circleci.com/gh/metabase/metabase.svg?style=svg&circle-token=3ccf0aa841028af027f2ac9e8df17ce603e90ef9)](https://circleci.com/gh/metabase/metabase)
 [![Leiningen Dependencies Status](https://jarkeeper.com/metabase/metabase/status.svg)](https://jarkeeper.com/metabase/metabase)
 [![NPM Dependencies Status](https://david-dm.org/metabase/metabase.svg)](https://david-dm.org/metabase/metabase)
-[![Issue Stats](http://issuestats.com/github/metabase/metabase/badge/pr)](http://issuestats.com/github/metabase/metabase)
-[![Issue Stats](http://issuestats.com/github/metabase/metabase/badge/issue)](http://issuestats.com/github/metabase/metabase)
 
 # Features
 - 5 minute [setup](http://www.metabase.com/docs/latest/setting-up-metabase) (We're not kidding)

docs/README.md

 ## In-depth Guides
 
+#### [Troubleshooting Guide](troubleshooting-guide/index.md)
+Have a problem and need help? Start with our troubleshooting guide.
+
 #### [Users Guide](users-guide/start.md)
 This is the go-to guide on asking questions and sharing answers using Metabase. You'll learn in depth about how questions are expressed, how to chart answers, as well as how to share questions and create dashboards.
 

docs/troubleshooting-guide/bugs.md

+If you come across something that looks like a bug, we suggest collecting the following information to help us reproduce the issue.
+
+1. Server logs
+2. Javascript console logs
+3. Can it be reproduced on the sample dataset?
+4. Your Metabase version
+5. Where Metabase is running (Docker image, AWS Elastic Beanstalk, etc)
+6. What browser version
+
+## Helpful tidbits
+
+### Accessing the Metabase server logs
+While you can always look for the logs Metabase leaves on your server file system (or however you collect logs), if you are logged into Metabase with an admin account, you can also access them from the drop down menu in the upper right hand corner.
+
+![Logs Screenshot](images/ServerLogs.png)
+
+### Checking for Javascript errors in Chrome
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Chrome by following the instructions at 
+https://developers.google.com/web/tools/chrome-devtools/console/
+
+
+### Checking for Javascript errors in Firefox
+
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Mozilla by following the instructions at 
+https://developer.mozilla.org/en-US/docs/Tools/Web_Console
+
+### Checking for Javascript errors in Safari
+
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Safari by following the instructions at 
+
+https://developer.apple.com/library/content/documentation/AppleApplications/Conceptual/Safari_Developer_Guide/Introduction/Introduction.html
+
+### Checking for Javascript errors in Internet Explorer
+
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Internet Explorer by following the instructions at 
+
+https://msdn.microsoft.com/en-us/library/gg589530(v=vs.85).aspx
+
+For many versions this can also be accessed by pressing the F12 key on your keyboard. 
\ No newline at end of file

docs/troubleshooting-guide/datawarehouse.md

+## Troubleshooting Process
+1. Verify that the data warehouse server is running
+2. Try connecting to the data warehouse using another client from a machine you know should have access
+3. Try connecting to the data warehouse from another client from the machine you're running Metabase on
+4. Add the connection in Metabase
+5. Examine the logs to verify that the sync process started and that no errors were thrown
+6. Run a native "SELECT 1" query to verify the connection to the data warehouse
+7. If the sync process has completed, attempt to do a "Raw data" query to verify that you are able to use the database 
+
+## Specific Problems:
+
+### The Data Warehouse Server is not running:
+
+#### How to detect this -
+As silly as this sounds, occasionally database servers go down. 
+
+If you are using a hosted database service, go to it's console and verify that its status is Green. If you have direct access to a command line interface, log in and make sure that it is up and running and accepting queries.
+
+#### How to fix this -
+It's out of the scope of this troubleshooting guide to get your data warehouse server back up. Check with whoever set it up for you!
+
+
+### The Data Warehouse Server is not accepting connections from your IP:
+
+#### How to detect this -
+
+If you are able to access the server from a bastion host, or another machine, use `nc` on linux (or your operating systems equivalent) to verify that you can connect to the host on a given port.
+
+
+The port a data warehouse server software is attached to varies, but an example for a default PostgreSQL configuration (which listens on port 5432) would be 
+
+`nc -v your-db-host 5432`
+
+#### How to fix this -
+It's out of the scope of this troubleshooting guide to change your network configuration. Talk to whoever is responsible for the network your data warehouse is running on. 
+
+
+### Incorrect credentials
+
+#### How to detect this -
+If you've verified that you can connect to the host and port on the data warehouse, the next step is to check your credentials.
+
+Again, connecting to a data warehouse depends on your database server software, but for PostgreSQL, the below uses a command line interface (`psql`) to connect to your data warehouse. 
+`psql -h HOSTNAME -p PORT -d DATABASENAME -U DATABASEUSER`
+
+If your credentials are incorrect, you should see an error message letting you know if the database name or the user/password are incorrect.
+
+#### How to fix this -
+If the database name or the user/pass combination are incorrect, ask the person running your data warehouse for correct credentials.
\ No newline at end of file

docs/troubleshooting-guide/docker.md

+
+While Docker simplifies a lot of aspects of running Metabase, there are a number of potential pitfalls to be keep in mind.
+
+If you are having issues with Metabase under Docker, we recommend going through the troubleshooting process below. Then look below for the details about the specific issue you've found. 
+
+## Troubleshooting Process
+1. Check that the container is running
+2. Check that the server is running inside the container
+3. Check whether Metabase is using the correct application database
+4. Check that you can connect to the Docker host on the Metabase port
+5. Check that you can connect to the container from the Docker host
+6. Check that you can connect to the server from within the container
+
+
+## Specific Problems:
+
+### Metabase container exits without starting the server
+
+Run `docker ps` to see if the metabase container is currently running. If it is move on to the next step. 
+
+If `docker ps` does not show the running container, then list the stopped containers by running
+
+`docker ps -a | grep metabase/metabase`
+
+And look for the container that exited most recently. Note the container id.
+Look at that containers logs with
+
+`Docker logs CONTAINER_ID`
+
+
+### Metabase Container is running but the Server is not
+#### How to detect this -
+Run `docker ps` to make sure the container is running
+ 
+The server should be logging to the docker container logs. Check this by running 
+
+`docker logs CONTAINER_NAME`
+
+You should see a line like at the beginning
+```
+05-10 18:11:32 INFO metabase.util :: Loading Metabase...
+```
+
+and eventually
+```
+05-10 18:12:30 INFO metabase.core :: Metabase Initialization COMPLETE
+```
+
+If you see log lines from Metabase like the below
+
+If you see the below lines:
+```
+05-15 19:07:11 INFO metabase.core :: Metabase Shutting Down ...
+05-15 19:07:11 INFO metabase.core :: Metabase Shutdown COMPLETE
+```
+
+#### How to fix this -
+Check this for errors about connecting to the application database.
+Watch the logs to see if metabase is still being started:
+
+`Docker logs -f CONTAINER_ID`
+
+Will let you see the logs as they are printed.
+
+If the container is being killed before it finished starting it could be a health check timeout in the orchestration service used to start the container, such as docker cloud, or elastic beanstalk. 
+
+If the container is not being killed from the outside, and is failing to start anyway, this problem is probably not specific to Docker. If you are using a Metabase supplied image, you should open a github issue at github.com/metabase/metabase/issues/new
+
+
+### Not connecting to a remote application database
+#### How to detect this 
+If this is a new Metabase instance, then the database you specified via the environment variables will be empty.
+If this is an existing Metabase instance with incorrect environment parameters, the server will create a new H2 embedded database to use for application data and you’ll lines similar to the below.
+
+```
+05-10 18:11:40 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
+05-10 18:11:40 INFO metabase.db :: Verifying h2 Database Connection ...
+
+05-10 18:11:40 INFO metabase.db :: Verify Database Connection ...  ✅
+```
+
+#### How to fix this - 
+Double check you are passing environments to docker in the correct way. 
+You can list the environment variables for a container with this command:
+
+`docker inspect some-postgres -f '{{ .Config.Env }}'`
+
+
+### The Metabase server isn’t able to connect to a MySQL or PostgreSQL 
+#### How to detect this - 
+The logs for the docker container return an error message after the “Verifying Database Connection” line.
+
+#### How to fix this - 
+Try to connect with “mysql” or “psql” commands with the connection string parameters you are passing in via the environment variables http://www.metabase.com/docs/latest/operations-guide/start.html#configuring-the-metabase-application-database .
+
+If you can’t connect to the database, the problem is due to either the credentials or connectivity. Verify that the credentials are correct. If you are able to log in with those credentials from another machine then try to make the same connection from the host running the docker container.
+
+One easy way to run this is to use docker to start a container that has the appropriate client for your database. For postgres this would look like:
+
+`docker run --name postgres-client --rm -ti --entrypoint /bin/bash postgres`
+
+Then from within that container try connecting to the database host using the client command in the container such as `psql`
+If you are able to connect from another container on the same host, then try making that connection from within the metabase docker container itself. 
+
+`docker exec -ti container-name bash`
+
+And try to connect to the database host using the `nc` command and check if the connection can be opened.
+
+`nc -v your-db-host 5432`
+
+This will make it clear if this is a network or authentication problem.
+
+### The Metabase application database is not being persisted
+
+#### How to detect this - 
+This occurs if you get the Setup screen every time you start the application. The most common root cause is not giving the docker container a persistent filesystem mount to put the application database in. 
+
+#### How to fix this - 
+Make sure you are giving the container a persistent volume as per http://www.metabase.com/docs/latest/operations-guide/running-metabase-on-docker.html#mounting-a-mapped-file-storage-volume
+
+### The internal port isn’t being remapped correctly
+
+#### How to detect this - 
+Run `docker ps` and look at the port mapping
+Run `curl http://localhost:port-number-here/api/health`. This should return a response with a json reponse like
+```
+{"status":"ok"}
+```
+
+#### How to fix this - 
+Make sure you include a `-p 3000:3000` or similar remapping in the `docker run` command you execute to start the Metabase container image.
+
+
+## Helpful tidbits
+
+### How to get to the shell in the Metabase container
+
+`docker  exec -ti CONTAINER_NAME bash`
+
+### How to get the logs for the Metabase container
+
+`docker  logs -f CONTAINER_NAME`
+

docs/troubleshooting-guide/email.md

+
+## Troubleshooting Process
+1. 
+
+## Specific Problems:
+
+### Specific Problem:
+
+###  Metabase can't send email via Office365
+
+We see users report issues with sending email via Office365. We recommend using a different email delivery service if you can. 
+https://github.com/metabase/metabase/issues/4272
\ No newline at end of file

docs/troubleshooting-guide/index.md

+## What are you having trouble with?
+
+### [Logging in?](loggingin.md)
+
+### [Running Metabase on Docker?](docker.md)
+
+
+### [Connecting to databases and data warehouses with Metabase?](datawarehouse.md)
+
+### [Incorrect results due to Timezones](timezones.md)
+
+### [I think I found a bug?](bugs.md)
\ No newline at end of file

docs/troubleshooting-guide/installing.md

+## Troubleshooting Process
+1. 
+
+## Specific Problems:
+
+### Specific Problem:
+xxx
+#### How to detect this -
+xxx
+#### How to fix this -
+xxx
\ No newline at end of file

docs/troubleshooting-guide/loggingin.md

+## Troubleshooting Process
+1. Try to log in with a local account
+2. Try to log in with a Google Auth SSO account
+3. Example Javascript and Server logs if you are not able to log in.
+
+## Specific Problems:
+
+
+### Invalid Google Auth Token:
+Sometimes your token from Google will expire. 
+
+#### How to detect this -
+Open up the Javascript console. Try to log in with Google Auth, see if there are any error messages in the Javascript console indicating an invalid account.
+
+Also open up your server logs, and see if there are any errors related to authentication. If there are, try re-creating the token.
+
+#### How to fix this -
+Remove the old token from the Google Auth SSO tab in the Admin Panel, and create a new one. If the root cause was an invalid auth token, this should fix the problem.
+
+
+
+## Helpful tidbits
+
+### Accessing the Metabase server logs
+While you can always look for the logs Metabase leaves on your server file system (or however you collect logs), if you are logged into Metabase with an admin account, you can also access them from the drop down menu in the upper right hand corner.
+
+![Logs Screenshot](images/ServerLogs.png)
+
+### Checking for Javascript errors in Chrome
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Chrome by following the instructions at 
+https://developers.google.com/web/tools/chrome-devtools/console/
+
+
+### Checking for Javascript errors in Firefox
+
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Mozilla by following the instructions at 
+https://developer.mozilla.org/en-US/docs/Tools/Web_Console
+
+### Checking for Javascript errors in Safari
+
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Safari by following the instructions at 
+
+https://developer.apple.com/library/content/documentation/AppleApplications/Conceptual/Safari_Developer_Guide/Introduction/Introduction.html
+
+### Checking for Javascript errors in Internet Explorer
+
+Metabase will print debugging information and errors to your browsers Javascript Console. 
+
+You can open the javascript console in Internet Explorer by following the instructions at 
+
+https://msdn.microsoft.com/en-us/library/gg589530(v=vs.85).aspx
+
+For many versions this can also be accessed by pressing the F12 key on your keyboard. 
\ No newline at end of file

docs/troubleshooting-guide/running.md

+
+## Troubleshooting Process
+1. 
+
+## Specific Problems:
+
+
+### Specific Problem:
+xxx
+#### How to detect this -
+xxx
+#### How to fix this -
+xxx
\ No newline at end of file

docs/troubleshooting-guide/timezones.md

+## Overview
+Oftentimes the source of "wrong" numbers in charts or reports is due to an underlying timezone issue. They are extremely common, both in Metabase, and in many other analytics tools and services. 
+
+
+## Troubleshooting Process
+
+When you suspect a timezone issue, you should collect a bit of information about your overall system.
+
+1. What is the timezone of the data you think is being displayed improperly? (Eg, in the database itself)
+2. Are you using an explicit timezone setting on each timestamp or are the timestamps being stored without a timestamp (eg, `Dec 1, 2019 00:00:00Z00` is an explicitly timestamped value vs `Dec 1, 2019` where the timezone is implied)
+2. What timezone is the database server set to?
+3. What timezone is the server that is running Metabase set to?
+4. What is your Reporting Timezone setting?
+5. What is your browser timezone setting?
+
+With this information collected, you can dig into the actual "mistakes" you are seeing. Most often these occur in an aggregation. It is useful simplifying the aggregation as much as you can while still seeing the "mistake". For example if you are looking at a "Net Negative Churn by Quarter" report that is based on an underlying table consisting of orders, see if the "count of orders by Quarter" has similarly incorrect behaviour. If so, troubleshoot the second, simpler question.
+
+Once you have simplified a question as much as possible, you can try to understand exactly what timezone conversion is causing the underlying problem. In the below, we assume that you are looking at a timeseries with daily values. If your error is happening with weeks, or other time granularities, perform the same sequence of steps but translating "day" to your specific granularity. 
+
+1. Pick a specific day where you know the number is incorrect. 
+2. Click on the data point in a chart or a cell in a result table and select "View these X". 
+3. Open two other tabs in your browser with date filters changed such that one tab has the rows in the underlying table from the previous day, and the other table has the rows in the underlying table from the next day.
+4. Check that the date field being used to group the result in the underlying display is correct. If it is different from what you have stored in the database, or what you have in another tool, then the timestamp is being transformed across the board into something incorrect. This is often the case when you are using a date or time lacking an explicit timezone.
+5. If the underlying timestamps are correct (this is often the case if you are using dates or times with explicit timezones), it is likely that the individual times are being grouped into days in a different timezone than the one you want.
+6. To find out which timezone they are being transformed into, tweak the times on the date filters on the question you are looking at by moving the start time and start date backwards by an hour until you either get the correct number or you have gone back by 12 hours.
+7. If that doesn't work, try moving the start and end times forward by an hour until you either get the correct number of you've gone forward by 12 hours. 
+8. Additionally, if any of your timezones include India, you will need to do this by half hour incremenets as well. 
+9. If by this point you have reached a correct number, that means your timezone was converted by the number of hours forward or backwards you manually set the filter. If not, then the problem might not be a direct timezone issue. 
+
+Now that you have the timezone adjustment, look at the lit of timezones in the first set of steps and think about where this could have occured. 
+
+For example, lets say have a PST server timeszone, and a GMT reporting timezone. If you had to manually go back 9 hours to get correct numbers, that suggests that the conversion is not happening for some reason -- this suggests you are using timestamps without a timezone. 
+
+You can see a number of common problems below, if none of them apply, please open a bug report at www.github.com/metabase/metabase/issues/new with the above information (timezones, and the results of the second troubleshooting process) as well as your Metabase, OS and web browser versions. 
+
+## Specific Problems:
+
+### SQL queries are not respecting the Reporting Time Zone setting
+#### How to detect this -
+If you are not able to click on a cell in a result table or a chart.
+
+#### How to fix this -
+We do not currently apply a reporting timezone to the results of SQL queries, and you should manually set one. 
+
+### Dates without an explicit timezone are being converted to another day 
+#### How to detect this -
+This occurs when you are grouping by a date (vs a time) that does not have a timezone attached to it. Look at every time field your question uses in the Data Model Reference, and see if any of them is simply a "Date" field. 
+
+#### How to fix this -
+You will need to make sure the server timezone reflects the reporting timezone, as when a query is run on Metabase, the server applies the timezone it is set to, to that date. 
+
+
+### Mixing explicit and implicit timezones
+#### How to detect this -
+This often happens if you compare or perform arithmatic on two dates where one has an explicit timezone and one does not. 
+
+This typically involves a question that uses multiple fields (eg, when you filter on one timestamp and group by another). Check the timezones of each of the dates or times you are using in your question. 
+
+#### How to fix this -
+You will need to explicitly cast the timezone that does not have an explicit timezone. This will need to be done either in a SQL query or by transforming the data in your database to ensure both timestamps have timezones.

frontend/src/metabase/css/core/inputs.css

@@ -48,3 +48,9 @@
 .no-focus:focus {
   outline: 0;
 }
+
+/* prevent safari from forcing type="search" styles - issue #5225 */
+.input[type="search"] {
+  -webkit-appearance: none;
+}
+

frontend/src/metabase/questions/components/CollectionButtons.jsx

@@ -10,12 +10,12 @@ const COLLECTION_ICON_SIZE = 64;
 const COLLECTION_BOX_CLASSES = "relative block p4 hover-parent hover--visibility cursor-pointer text-centered transition-background";
 
 const CollectionButtons = ({ collections, isAdmin, push }) =>
-    <ol className="flex flex-wrap">
+    <ol className="Grid Grid--gutters Grid--fit small-Grid--1of3 md-Grid--1of4 large-Grid--guttersLg">
         { collections
             .map(collection => <CollectionButton {...collection} push={push} isAdmin={isAdmin} />)
             .concat(isAdmin ? [<NewCollectionButton push={push} />] : [])
             .map((element, index) =>
-                <li key={index} className="mr4 mb4">
+                <li key={index} className="Grid-cell">
                     {element}
                 </li>
             )
@@ -40,8 +40,6 @@ class CollectionButton extends Component {
                 <div
                     className={cx(COLLECTION_BOX_CLASSES, 'text-white-hover')}
                     style={{
-                        width: 290,
-                        height: 180,
                         borderRadius: 10,
                         backgroundColor: this.state.hovered ? color : '#fafafa'
                     }}
@@ -71,8 +69,6 @@ const NewCollectionButton = ({ push }) =>
     <div
         className={cx(COLLECTION_BOX_CLASSES, 'bg-brand-hover', 'text-brand', 'text-white-hover', 'bg-grey-0')}
         style={{
-            width: 290,
-            height: 180,
             borderRadius: 10
         }}
         onClick={() => push(`/collections/create`)}

src/metabase/api/database.clj

@@ -246,6 +246,7 @@
   (sample-data/add-sample-dataset!)
   (Database :is_sample true))
 
+
 ;;; ------------------------------------------------------------ PUT /api/database/:id ------------------------------------------------------------
 
 (api/defendpoint PUT "/:id"

src/metabase/api/public.clj

@@ -88,11 +88,11 @@
   (card-with-uuid uuid))
 
 
-
 (defn run-query-for-card-with-id
   "Run the query belonging to Card with CARD-ID with PARAMETERS and other query options (e.g. `:constraints`)."
   [card-id parameters & options]
   (u/prog1 (-> (let [parameters (if (string? parameters) (json/parse-string parameters keyword) parameters)]
+                 ;; run this query with full superuser perms
                  (binding [api/*current-user-permissions-set*     (atom #{"/"})
                            qp/*allow-queries-with-no-executor-id* true]
                    (apply card-api/run-query-for-card card-id, :parameters parameters, :context :public-question, options)))

src/metabase/api/table.clj

@@ -2,6 +2,7 @@
   "/api/table endpoints."
   (:require [clojure.tools.logging :as log]
             [compojure.core :refer [GET PUT]]
+            [medley.core :as m]
             [metabase
              [sync-database :as sync-database]
              [util :as u]]
@@ -87,6 +88,7 @@
   {include_sensitive_fields (s/maybe su/BooleanString)}
   (-> (api/read-check Table id)
       (hydrate :db [:fields :target] :field_values :segments :metrics)
+      (m/dissoc-in [:db :details])
       (update-in [:fields] (if (Boolean/parseBoolean include_sensitive_fields)
                              ;; If someone passes include_sensitive_fields return hydrated :fields as-is
                              identity

src/metabase/driver.clj

@@ -323,6 +323,17 @@
       (log/warn (format "Don't know how to map class '%s' to a Field base_type, falling back to :type/*." klass))
       :type/*))
 
+(defn values->base-type
+  "Given a sequence of VALUES, return the most common base type."
+  [values]
+  (->> values
+       (filter (complement nil?))                   ; filter out `nil` values
+       (take 1000)                                  ; take up to 1000 values
+       (group-by (comp class->base-type class))     ; now group by their base-type
+       (sort-by (comp (partial * -1) count second)) ; sort the map into pairs of [base-type count] with highest count as first pair
+       ffirst))                                     ; take the base-type from the first pair
+
+
 ;; ## Driver Lookup
 
 (defn engine->driver

src/metabase/driver/generic_sql.clj

@@ -220,6 +220,7 @@
 
 (defn honeysql-form->sql+args
   "Convert HONEYSQL-FORM to a vector of SQL string and params, like you'd pass to JDBC."
+  {:style/indent 1}
   [driver honeysql-form]
   {:pre [(map? honeysql-form)]}
   (let [[sql & args] (try (binding [hformat/*subquery?* false]

src/metabase/driver/generic_sql/query_processor.clj

@@ -243,7 +243,7 @@
       (h/limit items)
       (h/offset (* items (dec page)))))
 
-(defn- apply-source-table [_ honeysql-form {{table-name :name, schema :schema} :source-table}]
+(defn- apply-source-table [honeysql-form {{table-name :name, schema :schema} :source-table}]
   {:pre [table-name]}
   (h/from honeysql-form (hx/qualify-and-escape-dots schema table-name)))
 
@@ -252,7 +252,7 @@
   ;;    will get swapped around and  we'll be left with old version of the function that nobody implements
   ;; 2) This is a vector rather than a map because the order the clauses get handled is important for some drivers.
   ;;    For example, Oracle needs to wrap the entire query in order to apply its version of limit (`WHERE ROWNUM`).
-  [:source-table apply-source-table
+  [:source-table (u/drop-first-arg apply-source-table)
    :aggregation  #'sql/apply-aggregation
    :breakout     #'sql/apply-breakout
    :fields       #'sql/apply-fields