Skip to content
Snippets Groups Projects
Commit 2d1857d3 authored by Arthur Ulfeldt's avatar Arthur Ulfeldt Committed by GitHub
Browse files

Merge pull request #4005 from metabase/ssh-tunnel-poc 

SSH tunnels for most drivers
parents d4cb0bdf 2230751a
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 534 additions and 285 deletions
docs/administration-guide/01-managing-databases.md
+ 64
− 0
View file @ 2d1857d3
......@@ -88,6 +88,70 @@ You can also delete a database from the database list: hover over the row with t
**Caution: Deleting a database is irreversible! All saved questions and dashboard cards based on the database will be deleted as well!**
### SSH Tunneling In Metabase
---
Metabase has the ability to connect to some databases by first establishing a connection to a server in between Metabase and a data warehouse, then connect to the data warehouse using that connection as a bridge. This makes connecting to some data warehouses possible in situations that would otherwise prevent the use of Metabase.
#### When To Use This Feature
There are two basic cases for using an SSH tunnel rather than connecting directly:
* A direct connection is impossible
* A direct connection is forbidden due to a security policy
Sometimes when a data warehouse is inside an enterprise environment, direct connections are blocked by security devices such as firewalls and intrusion prevention systems. To work around this many enterprises offer a VPN, a bastion host, or both. VPNs are the more convenient and reliable option though bastion hosts are used frequently, especially with cloud providers such as Amazon Web Services where VPC (Virtual Private Clouds) don't allow direct connections. Bastion hosts offer the option to first connect to a computer on the edge of the protected network, then from that computer establish a second connection to the data warehouse on the internal network and essentially patch these two connestions together. Using the SSH tunneling feature, Metabase is able to automate this process in many cases. If a VPN is available that should be used in preference to SSH tunneling.
#### How To Use This Feature
When connecting though a bastion host:
* Answer yes to the "Use an SSH-tunnel for database connections" parameter
* Enter the hostname for the data warehouse as it is seen from inside the network in the `Host` parameter.
* Enter the data warehouse port as seen from inside the network into the `Port` parameter.
* Enter the extenal name of the bastion host as seen from the outside of the network (or wherever you are) into the `SSH tunnel host` parameter.
* Enter the ssh port as seen from outside the network into the `SSH tunnel port` parameter. This is usually 22, regardless of which data warehouse you are connecting to.
* Enter the username and password you use to login to the bastion host into the `SSH tunnel username` and `SSH tunnel password` parameters.
If you are unable to connect test your ssh credentials by connecting to the SSH server/Bastion Host using ssh directly:
ssh <SSH tunnel username>@<SSH tunnel host> -p <SSH tunnel port>
Another common case where direct connections are not possible is when connecting to a data warehouse that is only accessible locally and does not allow remote connections. In this case you will be opening an SSH connection to the data warehouse, then from there connecting back to the same computer.
* Answer yes to the "Use an SSH-tunnel for database connections" parameter
* Enter `localhost` in the `Host` parameter. This is the name the server
* Enter the same value in the `Port` parameter that you would use if you where sitting directly at the data warehouse host system.
* Enter the extenal name of the data warehouse, as seen from the outside of the network (or wherever you are) into the `SSH tunnel host` parameter.
* Enter the ssh port as seen from outside the network into the `SSH tunnel port` parameter. This is usually 22, regardless of which data warehouse you are connecting to.
* Enter the username and password you use to login to the bastion host into the `SSH tunnel username` and `SSH tunnel password` parameters.
If you have problems connecting verify the ssh host port and password by connecing manually using ssh or PuTTY on older windows systems.
#### Disadvantages to Indirect Connections
While using an ssh tunnel makes it possible to use a data warehouse that is otherwise not accessible it is almost always preferable to use a direct connection when possible:
There are several inherent limitations to connecting through a tunnel:
* If the enclosing SSH connection is closed because you put your computer to sleep or change networks, all established connections will be closed as well. This can cause delays resuming connections after suspending your laptop
* It's almost always slower. The connection has to go through an additional computer.
* Opening new connections takes longer. SSH connections are slower to establish then direct connections.
* Multiple operations over the same SSH tunnel can block each other. This can increase latency in some cases.
* The number of connections through a bastion host is often limited by organizational policy.
* Some organizations have IT security policies forbidding using SSH tunnels to bypass security perimeters.
#### What if The Built in SSH Tunnels Don't Fit My Needs?
This feature exists as a convenient wrapper around SSH and automates the common cases of connecting through a tunnel. It also makes connecting possible from systems that don't have or allow shell access. Metabase uses a built in SSH client that does not depend on the installed system's ssh client. This allows connecting from systems where it's not possible to run SSH manually, it also means that Metabase cannot take advantage of authentication services provided by the system such as Windows Domain Authentication or Kerberos Authentication.
If you need to connect using a method not enabled by Metabase, you can often accomplish this by running ssh directly:
ssh -Nf -L input-port:internal-server-name:port-on-server username@bastion-host.domain.com
This allows you to use the full array of features included in ssh. If you find yourself doing this often, please let us know so we can see about making your process more convenient through Metabase.
---
## Next: enabling features that send email
......
frontend/src/metabase/components/DatabaseDetailsForm.jsx
+ 29
− 2
View file @ 2d1857d3
......@@ -23,6 +23,8 @@ const CREDENTIALS_URL_PREFIXES = {
googleanalytics: 'https://console.developers.google.com/apis/credentials/oauthclient?project=',
};
const isTunnelField = (field) => /^tunnel-/.test(field.name);
/**
* This is a form for capturing database details for a given `engine` supplied via props.
* The intention is to encapsulate the entire <form> with standard MB form styling and allow a callback
......@@ -61,7 +63,10 @@ export default class DatabaseDetailsForm extends Component {
// go over individual fields
for (let field of engines[engine]['details-fields']) {
if (field.required && isEmpty(details[field.name])) {
// tunnel fields aren't required if tunnel isn't enabled
if (!details["tunnel-enabled"] && isTunnelField(field)) {
continue;
} else if (field.required && isEmpty(details[field.name])) {
valid = false;
break;
}
......@@ -146,7 +151,29 @@ export default class DatabaseDetailsForm extends Component {
let { engine } = this.props;
window.ENGINE = engine;
if (field.name === "is_full_sync") {
if (field.name === "tunnel-enabled") {
let on = (this.state.details["tunnel-enabled"] == undefined) ? false : this.state.details["tunnel-enabled"];
return (
<FormField key={field.name} fieldName={field.name}>
<div className="flex align-center Form-offset">
<div className="Grid-cell--top">
<Toggle value={on} onChange={(val) => this.onChange("tunnel-enabled", val)}/>
</div>
<div className="px2">
<h3>Use an SSH-tunnel for database connections</h3>
<div style={{maxWidth: "40rem"}} className="pt1">
Some database installations can only be accessed by connecting through an SSH bastion host.
This option also provides an extra layer of security when a VPN is not available.
Enabling this is usually slower than a dirrect connection.
</div>
</div>
</div>
</FormField>
)
} else if (isTunnelField(field) && !this.state.details["tunnel-enabled"]) {
// don't show tunnel fields if tunnel isn't enabled
return null;
} else if (field.name === "is_full_sync") {
let on = (this.state.details.is_full_sync == undefined) ? true : this.state.details.is_full_sync;
return (
<FormField key={field.name} fieldName={field.name}>
......
project.clj
+ 1
− 0
View file @ 2d1857d3
......@@ -46,6 +46,7 @@
"v3-rev139-1.22.0"]
[com.google.apis/google-api-services-bigquery ; Google BigQuery Java Client Library
"v2-rev342-1.22.0"]
[com.jcraft/jsch "0.1.54"] ; SSH client for tunnels
[com.h2database/h2 "1.4.194"] ; embedded SQL database
[com.mattbertolini/liquibase-slf4j "2.0.0"] ; Java Migrations lib
[com.mchange/c3p0 "0.9.5.2"] ; connection pooling library
......
src/metabase/driver.clj
+ 3
− 1
View file @ 2d1857d3
......@@ -32,7 +32,9 @@
(def ^:const connection-error-messages
"Generic error messages that drivers should return in their implementation of `humanize-connection-error-message`."
{:cannot-connect-check-host-and-port "Hmm, we couldn't connect to the database. Make sure your host and port settings are correct."
{:cannot-connect-check-host-and-port "Hmm, we couldn't connect to the database. Make sure your host and port settings are correct"
:ssh-tunnel-auth-fail "We couldn't connect to the ssh tunnel host. Check the username, password"
:ssh-tunnel-connection-fail "We couldn't connect to the ssh tunnel host. Check the hostname and port"
:database-name-incorrect "Looks like the database name is incorrect."
:invalid-hostname "It looks like your host is invalid. Please double-check it and try again."
:password-incorrect "Looks like your password is incorrect."
......
src/metabase/driver/druid.clj
+ 37
− 34
View file @ 2d1857d3
......@@ -10,7 +10,8 @@
[metabase.models
[field :as field]
[table :as table]]
[metabase.sync-database.analyze :as analyze]))
[metabase.sync-database.analyze :as analyze]
[metabase.util.ssh :as ssh]))
;;; ### Request helper fns
......@@ -30,8 +31,8 @@
(do-request http/get \"http://my-json-api.net\")"
[request-fn url & {:as options}]
{:pre [(fn? request-fn) (string? url)]}
(let [options (cond-> (merge {:content-type "application/json"} options)
(:body options) (update :body json/generate-string))
(let [options (cond-> (merge {:content-type "application/json"} options)
(:body options) (update :body json/generate-string))
{:keys [status body]} (request-fn url options)]
(when (not= status 200)
(throw (Exception. (format "Error [%d]: %s" status body))))
......@@ -53,16 +54,17 @@
(defn- do-query [details query]
{:pre [(map? query)]}
(try (vec (POST (details->url details "/druid/v2"), :body query))
(catch Throwable e
;; try to extract the error
(let [message (or (u/ignore-exceptions
(:error (json/parse-string (:body (:object (ex-data e))) keyword)))
(.getMessage e))]
(ssh/with-ssh-tunnel [details-with-tunnel details]
(try (vec (POST (details->url details-with-tunnel "/druid/v2"), :body query))
(catch Throwable e
;; try to extract the error
(let [message (or (u/ignore-exceptions
(:error (json/parse-string (:body (:object (ex-data e))) keyword)))
(.getMessage e))]
(log/error (u/format-color 'red "Error running query:\n%s" message))
;; Re-throw a new exception with `message` set to the extracted message
(throw (Exception. message e))))))
(log/error (u/format-color 'red "Error running query:\n%s" message))
;; Re-throw a new exception with `message` set to the extracted message
(throw (Exception. message e)))))))
;;; ### Sync
......@@ -76,24 +78,24 @@
:type/Text)})
(defn- describe-table [database table]
(let [details (:details database)
{:keys [dimensions metrics]} (GET (details->url details "/druid/v2/datasources/" (:name table) "?interval=1900-01-01/2100-01-01"))]
{:schema nil
:name (:name table)
:fields (set (concat
;; every Druid table is an event stream w/ a timestamp field
[{:name "timestamp"
:base-type :type/DateTime
:pk? true}]
(map (partial describe-table-field :dimension) dimensions)
(map (partial describe-table-field :metric) metrics)))}))
(ssh/with-ssh-tunnel [details-with-tunnel (:details database)]
(let [{:keys [dimensions metrics]} (GET (details->url details-with-tunnel "/druid/v2/datasources/" (:name table) "?interval=1900-01-01/2100-01-01"))]
{:schema nil
:name (:name table)
:fields (set (concat
;; every Druid table is an event stream w/ a timestamp field
[{:name "timestamp"
:base-type :type/DateTime
:pk? true}]
(map (partial describe-table-field :dimension) dimensions)
(map (partial describe-table-field :metric) metrics)))})))
(defn- describe-database [database]
{:pre [(map? (:details database))]}
(let [details (:details database)
druid-datasources (GET (details->url details "/druid/v2/datasources"))]
{:tables (set (for [table-name druid-datasources]
{:schema nil, :name table-name}))}))
(ssh/with-ssh-tunnel [details-with-tunnel (:details database)]
(let [druid-datasources (GET (details->url details-with-tunnel "/druid/v2/datasources"))]
{:tables (set (for [table-name druid-datasources]
{:schema nil, :name table-name}))})))
;;; ### field-values-lazy-seq
......@@ -163,13 +165,14 @@
:analyze-table analyze-table
:describe-database (u/drop-first-arg describe-database)
:describe-table (u/drop-first-arg describe-table)
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "http://localhost"}
{:name "port"
:display-name "Broker node port"
:type :integer
:default 8082}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "http://localhost"}
{:name "port"
:display-name "Broker node port"
:type :integer
:default 8082}]))
:execute-query (fn [_ query] (qp/execute-query do-query query))
:features (constantly #{:basic-aggregations :set-timezone :expression-aggregations})
:field-values-lazy-seq (u/drop-first-arg field-values-lazy-seq)
......
src/metabase/driver/generic_sql.clj
+ 15
− 9
View file @ 2d1857d3
......@@ -16,7 +16,9 @@
[field :as field]
[table :as table]]
[metabase.sync-database.analyze :as analyze]
[metabase.util.honeysql-extensions :as hx])
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]])
(:import [clojure.lang Keyword PersistentVector]
com.mchange.v2.c3p0.ComboPooledDataSource
[java.sql DatabaseMetaData ResultSet]
......@@ -141,13 +143,15 @@
"Create a new C3P0 `ComboPooledDataSource` for connecting to the given DATABASE."
[{:keys [id engine details]}]
(log/debug (u/format-color 'magenta "Creating new connection pool for database %d ..." id))
(let [spec (connection-details->spec (driver/engine->driver engine) details)]
(db/connection-pool (assoc spec
:minimum-pool-size 1
;; prevent broken connections closed by dbs by testing them every 3 mins
:idle-connection-test-period (* 3 60)
;; prevent overly large pools by condensing them when connections are idle for 15m+
:excess-timeout (* 15 60)))))
(let [details-with-tunnel (ssh/include-ssh-tunnel details) ;; If the tunnel is disabled this returned unchanged
spec (connection-details->spec (driver/engine->driver engine) details-with-tunnel)]
(assoc (db/connection-pool (assoc spec
:minimum-pool-size 1
;; prevent broken connections closed by dbs by testing them every 3 mins
:idle-connection-test-period (* 3 60)
;; prevent overly large pools by condensing them when connections are idle for 15m+
:excess-timeout (* 15 60)))
:ssh-tunnel (:tunnel-connection details-with-tunnel))))
(defn- notify-database-updated
"We are being informed that a DATABASE has been updated, so lets shut down the connection pool (if it exists) under
......@@ -158,7 +162,9 @@
;; remove the cached reference to the pool so we don't try to use it anymore
(swap! database-id->connection-pool dissoc id)
;; now actively shut down the pool so that any open connections are closed
(.close ^ComboPooledDataSource (:datasource pool))))
(.close ^ComboPooledDataSource (:datasource pool))
(when-let [ssh-tunnel (:ssh-tunnel pool)]
(.disconnect ^com.jcraft.jsch.Session ssh-tunnel))))
(defn db->pooled-connection-spec
"Return a JDBC connection spec that includes a cp30 `ComboPooledDataSource`.
......
src/metabase/driver/mongo.clj
+ 33
− 25
View file @ 2d1857d3
......@@ -14,6 +14,7 @@
[field :as field]
[table :as table]]
[metabase.sync-database.analyze :as analyze]
[metabase.util.ssh :as ssh]
[monger
[collection :as mc]
[command :as cmd]
......@@ -43,6 +44,12 @@
#"^Password can not be null when the authentication mechanism is unspecified$"
(driver/connection-error-messages :password-required)
#"^com.jcraft.jsch.JSchException: Auth fail$"
(driver/connection-error-messages :ssh-tunnel-auth-fail)
#"j^ava.net.ConnectException: Connection refused (Connection refused)$"
(driver/connection-error-messages :ssh-tunnel-connection-fail)
#".*" ; default
message))
......@@ -174,31 +181,32 @@
:can-connect? (u/drop-first-arg can-connect?)
:describe-database (u/drop-first-arg describe-database)
:describe-table (u/drop-first-arg describe-table)
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 27017}
{:name "dbname"
:display-name "Database name"
:placeholder "carrierPigeonDeliveries"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"}
{:name "pass"
:display-name "Database password"
:type :password
:placeholder "******"}
{:name "authdb"
:display-name "Authentication Database"
:placeholder "Optional database to use when authenticating"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 27017}
{:name "dbname"
:display-name "Database name"
:placeholder "carrierPigeonDeliveries"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"}
{:name "pass"
:display-name "Database password"
:type :password
:placeholder "******"}
{:name "authdb"
:display-name "Authentication Database"
:placeholder "Optional database to use when authenticating"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}]))
:execute-query (u/drop-first-arg qp/execute-query)
:features (constantly #{:basic-aggregations :dynamic-schema :nested-fields})
:field-values-lazy-seq (u/drop-first-arg field-values-lazy-seq)
......
src/metabase/driver/mongo/util.clj
+ 30
− 27
View file @ 2d1857d3
......@@ -4,6 +4,7 @@
[metabase
[driver :as driver]
[util :as u]]
[metabase.util.ssh :as ssh]
[monger
[core :as mg]
[credentials :as mcred]]))
......@@ -45,33 +46,35 @@
"Run F with a new connection (bound to `*mongo-connection*`) to DATABASE.
Don't use this directly; use `with-mongo-connection`."
[f database]
(let [{:keys [dbname host port user pass ssl authdb]
:or {port 27017, pass "", ssl false}} (cond
(string? database) {:dbname database}
(:dbname (:details database)) (:details database) ; entire Database obj
(:dbname database) database ; connection details map only
:else (throw (Exception. (str "with-mongo-connection failed: bad connection details:" (:details database)))))
user (when (seq user) ; ignore empty :user and :pass strings
user)
pass (when (seq pass)
pass)
authdb (if (seq authdb)
authdb
dbname)
server-address (mg/server-address host port)
credentials (when user
(mcred/create user authdb pass))
connect (partial mg/connect server-address (build-connection-options :ssl? ssl))
conn (if credentials
(connect credentials)
(connect))
mongo-connection (mg/get-db conn dbname)]
(log/debug (u/format-color 'cyan "<< OPENED NEW MONGODB CONNECTION >>"))
(try
(binding [*mongo-connection* mongo-connection]
(f *mongo-connection*))
(finally
(mg/disconnect conn)))))
(let [details (cond
(string? database) {:dbname database}
(:dbname (:details database)) (:details database) ; entire Database obj
(:dbname database) database ; connection details map only
:else (throw (Exception. (str "with-mongo-connection failed: bad connection details:" (:details database)))))]
(ssh/with-ssh-tunnel [details-with-tunnel details]
(let [{:keys [dbname host port user pass ssl authdb tunnel-host tunnel-user tunnel-pass]
:or {port 27017, pass "", ssl false}} details-with-tunnel
user (when (seq user) ; ignore empty :user and :pass strings
user)
pass (when (seq pass)
pass)
authdb (if (seq authdb)
authdb
dbname)
server-address (mg/server-address host port)
credentials (when user
(mcred/create user authdb pass))
connect (partial mg/connect server-address (build-connection-options :ssl? ssl))
conn (if credentials
(connect credentials)
(connect))
mongo-connection (mg/get-db conn dbname)]
(log/debug (u/format-color 'cyan "<< OPENED NEW MONGODB CONNECTION >>"))
(try
(binding [*mongo-connection* mongo-connection]
(f *mongo-connection*))
(finally (mg/disconnect conn)
(log/debug (u/format-color 'cyan "<< CLOSED MONGODB CONNECTION >>"))))))))
(defmacro with-mongo-connection
"Open a new MongoDB connection to DATABASE-OR-CONNECTION-STRING, bind connection to BINDING, execute BODY, and close the connection.
......
src/metabase/driver/mysql.clj
+ 27
− 23
View file @ 2d1857d3
......@@ -8,7 +8,10 @@
[util :as u]]
[metabase.db.spec :as dbspec]
[metabase.driver.generic-sql :as sql]
[metabase.util.honeysql-extensions :as hx]))
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]]))
;;; # IMPLEMENTATION
......@@ -154,28 +157,29 @@
driver/IDriver
(merge (sql/IDriverSQLDefaultsMixin)
{:date-interval (u/drop-first-arg date-interval)
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 3306}
{:name "dbname"
:display-name "Database name"
:placeholder "birds_of_the_word"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "additional-options"
:display-name "Additional JDBC connection string options"
:placeholder "tinyInt1isBit=false"}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 3306}
{:name "dbname"
:display-name "Database name"
:placeholder "birds_of_the_word"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "additional-options"
:display-name "Additional JDBC connection string options"
:placeholder "tinyInt1isBit=false"}]))
:humanize-connection-error-message (u/drop-first-arg humanize-connection-error-message)})
sql/ISQLDriver
......
src/metabase/driver/oracle.clj
+ 22
− 19
View file @ 2d1857d3
......@@ -8,7 +8,9 @@
[util :as u]]
[metabase.driver.generic-sql :as sql]
[metabase.driver.generic-sql.query-processor :as sqlqp]
[metabase.util.honeysql-extensions :as hx]))
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]]))
(def ^:private ^:const pattern->type
[;; Any types -- see http://docs.oracle.com/cd/B28359_01/server.111/b28286/sql_elements001.htm#i107578
......@@ -193,24 +195,25 @@
(merge (sql/IDriverSQLDefaultsMixin)
{:can-connect? (u/drop-first-arg can-connect?)
:date-interval (u/drop-first-arg date-interval)
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 1521}
{:name "sid"
:display-name "Oracle System ID"
:default "ORCL"}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 1521}
{:name "sid"
:display-name "Oracle System ID"
:default "ORCL"}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}]))
:execute-query (comp remove-rownum-column sqlqp/execute-query)})
sql/ISQLDriver
......
src/metabase/driver/postgres.clj
+ 30
− 27
View file @ 2d1857d3
......@@ -8,7 +8,9 @@
[util :as u]]
[metabase.db.spec :as dbspec]
[metabase.driver.generic-sql :as sql]
[metabase.util.honeysql-extensions :as hx])
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]])
(:import java.util.UUID))
(def ^:private ^:const column->base-type
......@@ -200,32 +202,33 @@
driver/IDriver
(merge (sql/IDriverSQLDefaultsMixin)
{:date-interval (u/drop-first-arg date-interval)
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 5432}
{:name "dbname"
:display-name "Database name"
:placeholder "birds_of_the_word"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}
{:name "additional-options"
:display-name "Additional JDBC connection string options"
:placeholder "prepareThreshold=0"}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 5432}
{:name "dbname"
:display-name "Database name"
:placeholder "birds_of_the_word"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}
{:name "additional-options"
:display-name "Additional JDBC connection string options"
:placeholder "prepareThreshold=0"}]))
:humanize-connection-error-message (u/drop-first-arg humanize-connection-error-message)})
sql/ISQLDriver PostgresISQLDriverMixin)
......
src/metabase/driver/presto.clj
+ 49
− 44
View file @ 2d1857d3
......@@ -17,7 +17,9 @@
[table :as table]]
[metabase.query-processor.util :as qputil]
[metabase.sync-database.analyze :as analyze]
[metabase.util.honeysql-extensions :as hx])
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]])
(:import java.util.Date
[metabase.query_processor.interface DateTimeValue Value]))
......@@ -64,28 +66,30 @@
(parser value))))))
(defn- fetch-presto-results! [details {prev-columns :columns, prev-rows :rows} uri]
(let [{{:keys [columns data nextUri error]} :body} (http/get uri (assoc (details->request details) :as :json))]
(when error
(throw (ex-info (or (:message error) "Error running query.") error)))
(let [rows (parse-presto-results columns data)
results {:columns (or columns prev-columns)
:rows (vec (concat prev-rows rows))}]
(if (nil? nextUri)
results
(do (Thread/sleep 100) ; Might not be the best way, but the pattern is that we poll Presto at intervals
(fetch-presto-results! details results nextUri))))))
(ssh/with-ssh-tunnel [details-with-tunnel details]
(let [{{:keys [columns data nextUri error]} :body} (http/get uri (assoc (details->request details-with-tunnel) :as :json))]
(when error
(throw (ex-info (or (:message error) "Error running query.") error)))
(let [rows (parse-presto-results columns data)
results {:columns (or columns prev-columns)
:rows (vec (concat prev-rows rows))}]
(if (nil? nextUri)
results
(do (Thread/sleep 100) ; Might not be the best way, but the pattern is that we poll Presto at intervals
(fetch-presto-results! details-with-tunnel results nextUri)))))))
(defn- execute-presto-query! [details query]
(let [{{:keys [columns data nextUri error]} :body} (http/post (details->uri details "/v1/statement")
(assoc (details->request details) :body query, :as :json))]
(when error
(throw (ex-info (or (:message error) "Error preparing query.") error)))
(let [rows (parse-presto-results (or columns []) (or data []))
results {:columns (or columns [])
:rows rows}]
(if (nil? nextUri)
results
(fetch-presto-results! details results nextUri)))))
(ssh/with-ssh-tunnel [details-with-tunnel details]
(let [{{:keys [columns data nextUri error]} :body} (http/post (details->uri details-with-tunnel "/v1/statement")
(assoc (details->request details-with-tunnel) :body query, :as :json))]
(when error
(throw (ex-info (or (:message error) "Error preparing query.") error)))
(let [rows (parse-presto-results (or columns []) (or data []))
results {:columns (or columns [])
:rows rows}]
(if (nil? nextUri)
results
(fetch-presto-results! details-with-tunnel results nextUri))))))
;;; Generic helpers
......@@ -291,29 +295,30 @@
:describe-database (u/drop-first-arg describe-database)
:describe-table (u/drop-first-arg describe-table)
:describe-table-fks (constantly nil) ; no FKs in Presto
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 8080}
{:name "catalog"
:display-name "Database name"
:placeholder "hive"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database"
:default "metabase"}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 8080}
{:name "catalog"
:display-name "Database name"
:placeholder "hive"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database"
:default "metabase"}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}]))
:execute-query (u/drop-first-arg execute-query)
:features (constantly (set/union #{:set-timezone
:basic-aggregations
......
src/metabase/driver/redshift.clj
+ 25
− 22
View file @ 2d1857d3
......@@ -11,7 +11,9 @@
[metabase.driver
[generic-sql :as sql]
[postgres :as postgres]]
[metabase.util.honeysql-extensions :as hx]))
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]]))
(defn- connection-details->spec [details]
(dbspec/postgres (merge details postgres/ssl-params))) ; always connect to redshift over SSL
......@@ -64,27 +66,28 @@
(merge (sql/IDriverSQLDefaultsMixin)
{:date-interval (u/drop-first-arg date-interval)
:describe-table-fks (u/drop-first-arg describe-table-fks)
:details-fields (constantly [{:name "host"
:display-name "Host"
:placeholder "my-cluster-name.abcd1234.us-east-1.redshift.amazonaws.com"
:required true}
{:name "port"
:display-name "Port"
:type :integer
:default 5439}
{:name "db"
:display-name "Database name"
:placeholder "toucan_sightings"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "cam"
:required true}
{:name "password"
:display-name "Database user password"
:type :password
:placeholder "*******"
:required true}])
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:placeholder "my-cluster-name.abcd1234.us-east-1.redshift.amazonaws.com"
:required true}
{:name "port"
:display-name "Port"
:type :integer
:default 5439}
{:name "db"
:display-name "Database name"
:placeholder "toucan_sightings"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "cam"
:required true}
{:name "password"
:display-name "Database user password"
:type :password
:placeholder "*******"
:required true}]))
:format-custom-field-name (u/drop-first-arg str/lower-case)})
sql/ISQLDriver
......
src/metabase/driver/sqlserver.clj
+ 32
− 32
View file @ 2d1857d3
......@@ -4,9 +4,8 @@
[driver :as driver]
[util :as u]]
[metabase.driver.generic-sql :as sql]
[metabase.util.honeysql-extensions :as hx]))
; need to import this in order to load JDBC driver
[metabase.util.honeysql-extensions :as hx]
[metabase.util.ssh :as ssh]))
(defn- column->base-type
"See [this page](https://msdn.microsoft.com/en-us/library/ms187752.aspx) for details."
......@@ -147,35 +146,36 @@
driver/IDriver
(merge (sql/IDriverSQLDefaultsMixin)
{:date-interval (u/drop-first-arg date-interval)
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 1433}
{:name "db"
:display-name "Database name"
:placeholder "BirdsOfTheWorld"
:required true}
{:name "instance"
:display-name "Database instance name"
:placeholder "N/A"}
{:name "domain"
:display-name "Windows domain"
:placeholder "N/A"}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}])})
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 1433}
{:name "db"
:display-name "Database name"
:placeholder "BirdsOfTheWorld"
:required true}
{:name "instance"
:display-name "Database instance name"
:placeholder "N/A"}
{:name "domain"
:display-name "Windows domain"
:placeholder "N/A"}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}
{:name "ssl"
:display-name "Use a secure connection (SSL)?"
:type :boolean
:default false}]))})
sql/ISQLDriver
(merge (sql/ISQLDriverDefaultsMixin)
......
src/metabase/driver/vertica.clj
+ 23
− 20
View file @ 2d1857d3
......@@ -7,7 +7,9 @@
[driver :as driver]
[util :as u]]
[metabase.driver.generic-sql :as sql]
[metabase.util.honeysql-extensions :as hx]))
[metabase.util
[honeysql-extensions :as hx]
[ssh :as ssh]]))
(def ^:private ^:const column->base-type
"Map of Vertica column types -> Field base types.
......@@ -103,25 +105,26 @@
(merge (sql/IDriverSQLDefaultsMixin)
{:date-interval (u/drop-first-arg date-interval)
:describe-database describe-database
:details-fields (constantly [{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 5433}
{:name "dbname"
:display-name "Database name"
:placeholder "birds_of_the_word"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}])})
:details-fields (constantly (ssh/with-tunnel-config
[{:name "host"
:display-name "Host"
:default "localhost"}
{:name "port"
:display-name "Port"
:type :integer
:default 5433}
{:name "dbname"
:display-name "Database name"
:placeholder "birds_of_the_word"
:required true}
{:name "user"
:display-name "Database username"
:placeholder "What username do you use to login to the database?"
:required true}
{:name "password"
:display-name "Database password"
:type :password
:placeholder "*******"}]))})
sql/ISQLDriver
(merge (sql/ISQLDriverDefaultsMixin)
{:column->base-type (u/drop-first-arg column->base-type)
......
src/metabase/util/ssh.clj 0 → 100644
+ 114
− 0
View file @ 2d1857d3
(ns metabase.util.ssh
(:require [clojure.tools.logging :as log]
[clojure.string :as string]
[metabase.util :as u])
(:import com.jcraft.jsch.JSch))
(def ^:private default-ssh-timeout 30000)
(defn start-ssh-tunnel
"Opens a new ssh tunnel and returns the connection along with the dynamically
assigned tunnel entrance port. It's the callers responsibility to call .disconnect
on the returned connection object."
[{:keys [tunnel-host tunnel-port tunnel-user tunnel-pass host port]}]
(let [connection (doto ^com.jcraft.jsch.Session (.getSession (new com.jcraft.jsch.JSch)
^String tunnel-user
^String tunnel-host
tunnel-port)
(.setPassword ^String tunnel-pass)
(.setConfig "StrictHostKeyChecking" "no")
(.connect default-ssh-timeout)
(.setPortForwardingL 0 host port))
input-port (some-> (.getPortForwardingL connection)
first
(string/split #":")
first
(Integer/parseInt))]
(assert (number? input-port))
(log/info (u/format-color 'cyan "creating ssh tunnel %s@%s:%s -L %s:%s:%s" tunnel-user tunnel-host tunnel-port input-port host port))
[connection input-port]))
(def ssh-tunnel-preferences
"Configuration parameters to include in the add driver page on drivers that
support ssh tunnels"
[{:name "tunnel-enabled"
:display-name "Use SSH tunnel"
:placeholder "Enable this ssh tunnel?"
:type :boolean
:default false}
{:name "tunnel-host"
:display-name "SSH tunnel host"
:placeholder "What hostname do you use to connect to the SSH tunnel?"
:required true}
{:name "tunnel-port"
:display-name "SSH tunnel port"
:type :integer
:default 22
:required false}
{:name "tunnel-user"
:display-name "SSH tunnel username"
:placeholder "What username do you use to login to the SSH tunnel?"
:required true}
{:name "tunnel-pass"
:display-name "SSH tunnel password"
:type :password
:placeholder "******"
:required true}
#_{:name "tunnel-private-key"
:display-name "SSH private key to connect to the tunnel"
:type :string
:placeholder "Paste the contents of an ssh private key here"}
#_{:name "tunnel-private-key-file-name"
:display-name "Path on the Metabase server to a SSH private key file to connect to the tunnel"
:type :string
:placeholder "/home/YOUR-USERNAME/.ssh/id_rsa"}])
(defn with-tunnel-config
"Add preferences for ssh tunnels to a drivers :details-fields"
[driver-options]
(concat driver-options ssh-tunnel-preferences))
(defn use-ssh-tunnel?
"Is the SSH tunnel currently turned on for these connection details"
[details]
(:tunnel-enabled details))
(defn include-ssh-tunnel
"Updates connection details for a data warehouse to use the ssh tunnel host and port
For drivers that enter hosts including the protocol (https://host), copy the protocol over as well"
[details]
(if (use-ssh-tunnel? details)
(let [[_ proto host] (re-find #"(.*://)?(.*)" (:host details))
[connection tunnel-entrance-port] (start-ssh-tunnel (assoc details :host host)) ;; don't include L7 protocol in ssh tunnel
details-with-tunnel (assoc details
:port tunnel-entrance-port ;; This parameter is set dynamically when the connection is established
:host (str proto (:tunnel-host details))
:tunnel-entrance-port tunnel-entrance-port ;; the input port is not known until the connection is opened
:tunnel-connection connection)]
details-with-tunnel)
details))
(defn with-ssh-tunnel*
"Starts an SSH tunnel, runs the supplied function with the tunnel open, then closes it"
[{:keys [host port tunnel-host tunnel-user tunnel-pass] :as details} f]
(if (use-ssh-tunnel? details)
(let [details-with-tunnel (include-ssh-tunnel details)]
(log/errorf "\nbefore:\n%s\n" (with-out-str (clojure.pprint/pprint details)))
(log/errorf "\nafter:\n%s\n" (with-out-str (clojure.pprint/pprint details-with-tunnel)))
(try
(log/info (u/format-color 'cyan "<< OPENED SSH TUNNEL >>"))
(f details-with-tunnel)
(catch Exception e
(throw e))
(finally
(.disconnect ^com.jcraft.jsch.Session (:tunnel-connection details-with-tunnel))
(log/info (u/format-color 'cyan "<< CLOSED SSH TUNNEL >>")))))
(f details)))
(defmacro with-ssh-tunnel
"Starts an ssh tunnel, and binds the supplied name to a database
details map with it's values adjusted to use the tunnel"
[[name details] & body]
`(with-ssh-tunnel* ~details
(fn [~name]
~@body)))
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment