Skip to content
Snippets Groups Projects
Unverified Commit 37f3ab43 authored by Jeff Bruemmer's avatar Jeff Bruemmer Committed by GitHub
Browse files

docs - driver guide (#20567)

parent b0b8288a
No related branches found
No related tags found
No related merge requests found
Showing
with 864 additions and 116 deletions
# Working with Clojure
Check out [Clojure for the Brave and True](https://www.braveclojure.com/clojure-for-the-brave-and-true/). It's free online.
If you don't feel like reading a whole book, [Mark Volkmann's Clojure tutorial](https://objectcomputing.com/resources/publications/sett/march-2009-clojure-functional-programming-for-the-jvm) is another good starting point.
# Development Environment
# Development environment
If you plan to work on the Metabase code and make changes then you'll need to understand a few more things.
The Metabase application has two basic components:
## Overview
1. A backend written in Clojure which contains a REST API as well as all the relevant code for talking to databases and processing queries.
2. A frontend written as a Javascript single-page application which provides the web UI.
The Metabase application has two basic components:
Both components are built and assembled together into a single JAR file. In the directory where you run the JAR, you can create a JAR file (if Metabase hasn't already created it) and add drivers in there (the drivers are also JARs).
1. a backend written in Clojure which contains a REST API as well as all the relevant code for talking to databases and processing queries.
2. a frontend written as a Javascript single-page application which provides the web UI.
## Quick start
Both components are built and assembled together into a single jar file which runs the entire application.
To spin up a development environment, you'll need to start two terminal sessions: one for the [frontend](#frontend) and one for the [backend](#backend).
## Third-party dependencies
### Frontend
Metabase depends on lots of third-party libraries to run, so you'll need to keep those up to date. The Clojure CLI will automatically fetch the dependencies when needed. With JavaScript dependencies, however, you'll need to kick off the installation process manually.
Metabase depends on third-party libraries to run, so you'll need to keep those up to date. The Clojure CLI will automatically fetch the dependencies when needed. With JavaScript dependencies, however, you'll need to kick off the installation process manually.
```sh
# javascript dependencies
$ yarn
```
## Development server (quick start)
Start the frontend build process with
```
yarn build-hot
```
See [Frontend development](#frontend-development).
### Backend
Run your backend development server with
clojure -M:run
```
clojure -M:run
Start the frontend build process with
```
yarn build-hot
See [backend development](#backend-development).
## Frontend development
......@@ -76,7 +86,7 @@ $ FS_CACHE=true yarn build-hot
When using `FS_CACHE=true` you may need to remove the `node_modules/.cache` directory to fix scenarios where the build may be improperly cached, and you must run `rm -rf node_modules/.cache` in order for the build to work correctly when alternating between open source and enterprise builds of the codebase.
## Frontend testing
### Frontend testing
Run all unit and Cypress end-to-end tests with
......@@ -88,7 +98,7 @@ Cypress tests and some unit tests are located in `frontend/test` directory. New
If you are using `FS_CACHE=true`, you can also use `FS_CACHE=true` with `yarn test`.
## Frontend debugging
### Frontend debugging
By default, we use a simple source mapping option that is optimized for speed.
......@@ -123,7 +133,9 @@ Clojure REPL is the main development tool for the backend. There are some direct
And of course your Jetty development server is available via
clojure -M:run
```
clojure -M:run
```
### Building drivers
......@@ -153,125 +165,61 @@ drivers' dependencies and source paths into the Metabase project:
clojure -P -X:dev:ci:drivers:drivers-dev
```
#### Unit Tests / Linting
### Running unit tests
Run unit tests with
# OSS tests only
clojure -X:dev:test
```
# OSS tests only
clojure -X:dev:test
# OSS + EE tests
clojure -X:dev:ee:ee-dev:test
# OSS + EE tests
clojure -X:dev:ee:ee-dev:test
```
or a specific test (or test namespace) with
# run tests in only one namespace (pass in a symbol)
clojure -X:dev:test :only metabase.api.session-test
# run one specific test (pass in a qualified symbol)
clojure -X:dev:test :only metabase.api.session-test/my-test
# run tests in one specific folder (test/metabase/util in this example)
# pass arg in double-quotes so Clojure CLI interprets it as a string;
# our test runner treats strings as directories
clojure -X:dev:test :only '"test/metabase/util"'
By default, the tests only run against the `h2` driver. You can specify which drivers to run tests against with the env var `DRIVERS`:
DRIVERS=h2,postgres,mysql,mongo clojure -X:dev:drivers:drivers-dev:test
Some drivers require additional environment variables when testing since they are impossible to run locally (such as Redshift and Bigquery). The tests will fail on launch and let you know what parameters to supply if needed.
##### Run the linters:
`clj-kondo` must be installed separately; see https://github.com/clj-kondo/clj-kondo/blob/master/doc/install.md for
instructions.
# Run Eastwood
clojure -X:dev:ee:ee-dev:drivers:drivers-dev:eastwood
# Run the namespace checker
clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:namespace-checker
# Run clj-kondo
clj-kondo --parallel --lint src shared/src enterprise/backend/src --config lint-config.edn
### Developing with Emacs
`.dir-locals.el` contains some Emacs Lisp that tells `clojure-mode` how to indent Metabase macros and which arguments are docstrings. Whenever this file is updated,
Emacs will ask you if the code is safe to load. You can answer `!` to save it as safe.
By default, Emacs will insert this code as a customization at the bottom of your `init.el`.
You'll probably want to tell Emacs to store customizations in a different file. Add the following to your `init.el`:
```emacs-lisp
(setq custom-file (concat user-emacs-directory ".custom.el")) ; tell Customize to save customizations to ~/.emacs.d/.custom.el
(ignore-errors ; load customizations from ~/.emacs.d/.custom.el
(load-file custom-file))
```
# run tests in only one namespace (pass in a symbol)
clojure -X:dev:test :only metabase.api.session-test
## Developing with Visual Studio Code
### Debugging
First, install the following extension:
* [Debugger for Firefox](https://marketplace.visualstudio.com/items?itemName=firefox-devtools.vscode-firefox-debug)
_Note_: Debugger for Chrome has been deprecated. You can safely delete it as Visual Studio Code now has [a bundled JavaScript Debugger](https://github.com/microsoft/vscode-js-debug) that covers the same functionality.
Before starting the debugging session, make sure that Metabase is built and running. Choose menu _View_, _Command Palette_, search for and choose _Tasks: Run Build Task_. Alternatively, use the corresponding shortcut `Ctrl+Shift+B`. The built-in terminal will appear to show the progress, wait a few moment until webpack indicates a complete (100%) bundling.
To begin debugging Metabase, switch to the Debug view (shortcut: `Ctrl+Shift+D`) and then select one of the two launch configurations from the drop-down at the top:
* Debug with Firefox, or
* Debug with Chrome
After that, begin the debugging session by choosing menu _Run_, _Start Debugging_ (shortcut: `F5`).
# run one specific test (pass in a qualified symbol)
clojure -X:dev:test :only metabase.api.session-test/my-test
For more details, please refer to the complete VS Code documentation on [Debugging](https://code.visualstudio.com/docs/editor/debugging).
### Docker-based Workflow
These instructions allow you to work on Metabase codebase on Windows, Linux, or macOS using [Visual Studio Code](https://code.visualstudio.com/), **without** manually installing the necessary dependencies. This is possible by leveraging Docker container and the Remote Containers extension from VS Code.
For more details, please follow the complete VS Code guide on [Developing inside a Container](https://code.visualstudio.com/docs/remote/containers).
Requirements:
* [Visual Studio Code](https://code.visualstudio.com/) (obviously)
* [Docker](https://www.docker.com/)
* [Remote - Containers extension](vscode:extension/ms-vscode-remote.remote-containers) for VS Code
# run tests in one specific folder (test/metabase/util in this example)
# pass arg in double-quotes so Clojure CLI interprets it as a string;
# our test runner treats strings as directories
clojure -X:dev:test :only '"test/metabase/util"'
```
_Important_: Ensure that Docker is running properly and it can be used to download an image and launch a container, e.g. by running:
#### Testing drivers
```
$ docker run hello-world
```
If everything goes well, you should see the following message:
By default, the tests only run against the `h2` driver. You can specify which drivers to run tests against with the env var `DRIVERS`:
```
Hello from Docker!
This message shows that your installation appears to be working correctly.
DRIVERS=h2,postgres,mysql,mongo clojure -X:dev:drivers:drivers-dev:test
```
Steps:
1. Clone Metabase repository
Some drivers require additional environment variables when testing since they are impossible to run locally (such as Redshift and Bigquery). The tests will fail on launch and let you know what parameters to supply if needed.
2. Launch VS Code and open your cloned Metabase repository
### Running the linters
3. From the _View_ menu, choose _Command Palette..._ and then find _Remote-Container: Reopen in Container_. (VS Code may also prompt you to do this with an "Open in container" popup).
**Note**: VS Code will create the container for the first time and it may take some time. Subsequent loads should be much faster.
`clj-kondo` must be [installed separately](https://github.com/clj-kondo/clj-kondo/blob/master/doc/install.md).
4. Use the menu _View_, _Command Palette_, search for and choose _Tasks: Run Build Task_ (alternatively, use the shortcut `Ctrl+Shift+B`).
```
# Run Eastwood
clojure -X:dev:ee:ee-dev:drivers:drivers-dev:eastwood
5. After a while (after all JavaScript and Clojure dependencies are completely downloaded), open localhost:3000 with your web browser.
# Run the namespace checker
clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:namespace-checker
See [here](dev-branch-docker.md) for more on running development branches of Metabase using Docker.
# Run clj-kondo
clj-kondo --parallel --lint src shared/src enterprise/backend/src --config lint-config.edn
```
## Continuous integration
All front-end and back-end linters and tests can be executed with
All frontend and backend linters and tests can be executed with
```sh
$ yarn ci
......
# Database driver basics
A Metabase driver:
- **Provides Metabase with basic information for the database** such as the database's capabilities, connection properties, and so on.
- **Provides Metabase with information about the schema of the database** -- the tables (or equivalent), the fields in those tables, the foreign key relationships (for databases that support foreign keys).
- This functionality is used by the Metabase _sync process_ and stored in the application database.
- The stored information is used in the visual Query Builder and other places to show users what tables/columns/etc. are available
- **Compiles our in-house query language, [MBQL](https://github.com/metabase/metabase/wiki/Query-Language-'98), into native queries.**
- MBQL queries are generated by the visual query builder.
- The Metabase **query processor\*** converts MBQL queries to native queries
- **Executes native queries and returns results**.
## Write your driver as a module and package it as a plugin
Metabase drivers are organized into modules and packaged as plugins. Modules are the source code; plugins are the JARs built from that source code.
A Metabase plugin is a JAR file that contains the compiled class files and a Metabase [plugin manifest](#plugin-manifest) that lists details about the driver. In most cases, plugins are [lazily loaded](#lazy-loading), which means that Metabase won't initialize the drivers until it connects to a database that would use the driver.
For Metabase to use your driver, all you need to do is put the driver JAR you built into the `/plugin` directory, which you'll find in the same directory where you run your metabase.jar. Something like this:
```
/Users/cam/metabase/metabase.jar
/Users/cam/metabase/plugins/my-plugin.jar
```
You can change the plugin directory by setting the [environment variable][env-var] `MB_PLUGINS_DIR`.
## Example module directory
Let's take a high-level look at the [SQLite driver](https://github.com/metabase/metabase/tree/master/modules/drivers/sqlite):
```
|-- deps.edn
|-- resources
| `-- metabase-plugin.yaml
|-- src
| `-- metabase
| `-- driver
| `-- sqlite.clj
`-- test
`-- metabase
|-- driver
| `-- sqlite_test.clj
`-- test
`-- data
`-- sqlite.clj
```
There are three files to call out here:
### `deps.edn`
The `deps.edn` file specifies the driver's dependencies.
### `resources/metabase-plugin.yaml`
Your driver's [manifest](plugins.md#plugin-manifests) includes details about your driver.
### `src/metabase/driver/sqlite.cljl`
This is the core file for your driver. We'll talk more about it in [Implementing multimethods](multimethods.md).
## Next Up
We'll learn more about [plugin manifests](plugins.md).
[env-var]: ../../operations-guide/environment-variables.html
# Submitting a PR for a new driver
If you want to submit a PR to add a driver plugin to the [Metabase repo](https://github.com/metabase/metabase) (as opposed to keeping it in a separate repo), you'll need to:
- Be able to run your database locally with Docker.
- Make sure your driver passes Metabase's core test suite.
## Testing your driver
To test your driver, you'll need to:
- Move your plugin into the [`modules/drivers`](https://github.com/metabase/metabase/tree/master/modules/drivers) directory in the Metabase repository.
- Add _test extensions_ to your driver.
- Edit [`.circleci/config.yml`](https://github.com/metabase/metabase/blob/master/.circleci/config.yml) to tell CircleCI how to set up a Docker image for your database and run tests against it.
## Add test extensions to your driver
Test extensions do things like create new databases and load data for given _database definitions_. Metabase defines a huge suite of tests that automatically run against all drivers, including your new driver.
To run the test suite with your driver, you'll need to write a series of method implementations for special _test extension_ multimethods. Test extensions do things like create new databases and load data for _database definitions_.
These test extensions will tell Metabase how to create new databases and load them with test data, and provide information about what Metabae can expect from the created database. Test extensions are simply additional multimethods used only by tests. Like the core driver multimethods, they dispatch on the driver name as a keyword, e.g. `:mysql`.
## File organization
Test extensions for a driver usually live in a namespace called `metabase.test.data.<driver>`. If your driver is for SQLite, your files should look something like:
```clj
metabase/modules/drivers/sqlite/deps.edn ; <- deps go in here
metabase/modules/drivers/sqlite/resources/metabase-plugin.yaml ; <- plugin manifest
metabase/modules/drivers/sqilte/src/metabase/driver/sqlite.clj ; <- main driver namespace
```
So you'll create a new directory and file to house your text extension method implementations.
```clj
metabase/modules/drivers/sqlite/test/metabase/test/data/sqlite.clj ; <- test extensions
```
## Where are test extension methods defined?
Metabase test extensions live in the [`metabase.test.data.interface`](https://github.com/metabase/metabase/blob/master/test/metabase/test/data/interface.clj) namespace. Like the core driver methods, `:sql` and `:jdbc-sql` implement some of the test extensions themselves, but define additional methods you must implement to use them; see the [`metabase.test.data.sql`](https://github.com/metabase/metabase/blob/master/test/metabase/test/data/sql.clj)
and [`metabase.test.data.sql-jdbc`](https://github.com/metabase/metabase/blob/master/test/metabase/test/data/sql_jdbc.clj) namespaces.
You'll need to require the following namespaces, aliased like so:
```clj
(require '[metabase.test.data.interface :as tx]) ; tx = test extensions
(require '[metabase.test.data.sql :as sql.tx]) ; sql test extensions
(require '[metabase.test.data.sql-jdbc :as sql-jdbc.tx])
```
## Registering test extensions
Like the driver itself, you need to register the fact that your driver has test extensions, so Metabase knows it doesn't need to try to load them a second time. (If they're not loaded yet, Metabase will load them when needed by looking for a namespace named `metabase.test.data.<driver>`, which is why you need to follow that naming pattern.) The `:sql` and `:sql-jdbc` drivers have their own sets of test extensions, so depending on which parent you're using for your driver, register test extensions with:
```clj
# Non-SQL drivers
(tx/add-test-extensions! :mongo)
# non-JDBC SQL
(sql/add-test-extensions! :bigquery)
# JDBC SQL
(sql-jdbc.tx/add-test-extensions! :mysql)
```
You only need one call -- there's no need to do all three for a `:sql-jdbc` driver. This call should go at the beginning of your test extension namespace, like this:
```clj
(ns metabase.test.data.mysql
(:require [metabase.test.data.sql-jdbc :as sql-jdbc.tx]))
(sql-jdbc.tx/register-test-extensions! :mysql)
```
## Anatomy of a Metabase test
Let's look at an real-life Metabase test so we can understand how it works and what exactly we need to do to power it:
```clj
;; expect-with-non-timeseries-dbs = run against all drivers listed in `DRIVERS` env var except timeseries ones like Druid
(expect-with-non-timeseries-dbs
;; expected results
[[ 5 "Brite Spot Family Restaurant" 20 34.0778 -118.261 2]
[ 7 "Don Day Korean Restaurant" 44 34.0689 -118.305 2]
[17 "Ruen Pair Thai Restaurant" 71 34.1021 -118.306 2]
[45 "Tu Lan Restaurant" 4 37.7821 -122.41 1]
[55 "Dal Rae Restaurant" 67 33.983 -118.096 4]]
;; actual results
(-> (data/run-mbql-query venues
{:filter [:ends-with $name "Restaurant"]
:order-by [[:asc $id]]})
rows formatted-venues-rows))
```
Let's say we launch tests with
```
DRIVERS=mysql clojure -X:dev:drivers:drivers-dev:test`.
```
1. Metabase will check and see if test extensions for `:mysql` are loaded. If not, it will `(require 'metabase.test.data.mysql)`.
2. Metabase will check to see if the default `test-data` database has been created for MySQL, loaded with data, and synced. If not, it will call the test extension method `tx/load-data!` to create the `test-data` database and load data into it. After loading the data, Metabase syncs the test database. (This is discussed in more detail below.)
3. Metabase runs an MBQL query against the `venues` table of the MySQL `test-data` database. The `run-mbql-query` macro is a helper for writing tests that looks up Field IDs based on names for symbols that have `$` in from of them. Don't worry too much about that right now; just know the actual query that is ran will look something like:
```clj
{:database 100 ; ID of MySQL test-data database
:type :query
:query {:source-table 20 ; Table 20 = MySQL test-data.venues
:filter [:ends-with [:field-id 555] "Restaurant"] ; Field 555 = MySQL test-data.venues.name
:order-by [[:asc [:field-id 556]]]}} ; Field 556 = MySQL test-data.venues.id
```
4. The results are ran through helper functions `rows` and `formatted-venues-rows` which return only the parts of the query results we care about
5. Those results are compared against the expected results.
That's about as much as you'd need to know about the internals of how Metabase tests work; now that we've covered that, let's take a look at how we can empower Metabase to do what it needs to do.
## Loading Data
In order to ensure consistent behavior across different drivers, the Metabase test suite creates new databases and load datas into them from a set of shared _Database Definitions_. That means whether we're running a test against MySQL, Postgres, SQL Server, or MongoDB, a single test can check that we get the exact same results for every driver!
Most of these database definitions live in [EDN](https://github.com/edn-format/edn) files; the majority of tests run against a test database named "test data", whose definition can be found [here](https://github.com/metabase/metabase/blob/master/test/metabase/test/data/dataset_definitions/test-data.edn). Take a look at that file -- it's just a simple set of tables names, column names and types, and then a few thousand rows of data to load into those tables.
Like test extension method definitions, schemas for `DatabaseDefinition` live in [`metabase.test.data.interface`](https://github.com/metabase/metabase/blob/master/test/metabase/test/data/interface.clj) -- you can take a look and see exactly what a database definition is supposed to look like.
**Your biggest job as a writer of test definitions is to write the methods needed to take a database definition, create a new database with the appropriate tables and columns, and load data into it.** For non-SQL drivers, you'll need to implement `tx/load-data!`; `:sql` and `:sql-jdbc` have a shared implementation used by child drivers, but define their own set of test extension methods. For example, `:sql` (and `:sql-jdbc`) will handle the DDL statements for creating tables, but need to know what type it should use for the primary key, so you'll need to implement `sql.tx/pk-sql-type`:
```clj
(defmethod sql.tx/pk-sql-type :mysql [_] "INTEGER NOT NULL AUTO_INCREMENT")
```
I'd like to document every single test extension method in detail here, but until I find the time to do that, the methods are all documented in the codebase itself; take a look at the appropriate test extension namespaces and see which methods you'll need to implement. You can also refer to the test extensions written for other similar drivers to get a picture of what exactly it is you need to be doing.
## Connection Details
Of course, Metabase also needs to know how it can connect to your newly created database. Specifically, it needs to know what it should save as part of the connection `:details` map when it saves the newly created database as a `Database` object. All drivers with test extensions need to implement `tx/dbdef->connection-details` to return an appropriate set of `:details` for a given database definition. For example:
```clj
(defmethod tx/dbdef->connection-details :mysql [_ context {:keys [database-name]}]
(merge
{:host (tx/db-test-env-var :mysql :host "localhost")
:port (tx/db-test-env-var :mysql :port 3306)
:user (tx/db-test-env-var :mysql :user "root")
;; :timezone :America/Los_Angeles
:serverTimezone "UTC"}
(when-let [password (tx/db-test-env-var :mysql :password)]
{:password password})
(when (= context :db)
{:db database-name})))
```
Let's take a look at what's going on here.
### Connection context
`tx/dbdef->connection-details` is called in two different contexts:
- When creating a database,
- And when loading data into one and syncing.
Most databases won't let you connect to a database that hasn't been created yet, meaning something like a `CREATE DATABASE "test-data";` statement would have to be ran _without_ specifying `test-data` as part of the connection. Thus, the `context` parameter. `context` is either `:server`, meaning "give me details for connecting to the DBMS server, but not to a specific database", or `:db`, meaning "give me details for connecting to a specific database". In MySQL's case, it adds the `:db` connection property whenever context is `:db`.
### Getting connection properties from env vars
You'll almost certainly be running your database in a local Docker container. Rather than hardcode the connection details (the username, host, port...) for the Docker container, we'd like to be flexible, and let people specify those in environment variables, in case they're running against a different container or are just running the database outside of a container, or on another computer entirely. You can use `tx/db-test-env-var` to get details from environment variables. For example,
```clj
(tx/db-test-env-var :mysql :user "root")
```
Tells Metabase to look for the environment variable `MB_MYSQL_TEST_USER`; if not found, default to `"root"`. The name of the environment variable follows the pattern `MB_<driver>_TEST_<property>`, as passed into the function as first and second args, respectively. You don't need to specify a default value for `tx/db-test-env-var`; perhaps `user` is an optional parameter; and if `MB_MYSQL_TEST_USER` isn't specified, you don't need to specify it in the connection details.
But what about properties you want to require, but do not have sane defaults? In those cases, you can use `tx/db-test-env-var-or-throw`. It the corresponding enviornment variable isn't set, these will throw an Exception, ultimately causing tests to fail.
```clj
;; If MB_SQLSERVER_TEST_USER is unset, the test suite will quit with a message saying something like
;; "MB_SQLSERVER_TEST_USER is required to run tests against :sqlserver"
(tx/db-test-env-var-or-throw :sqlserver :user)
```
Note that `tx/dbdef->connection-details` won't get called in the first place for drivers you aren't running tests against (i.e., drivers not listed in the `DRIVERS` env var), so you wouldn't see that SQL Server error message when running tests against Mongo, for example.
Besides `tx/db-test-env-var`, `metabase.test.data.interface` has several other helpful utility functions. Take a good look at that namespace as well as `metabase.test.data.sql` if your database uses SQL and `metabase.test.data.sql-jdbc` if your database uses a JDBC driver.
## Other Test Extensions
There's a few other things Metabase needs to know when comparing test results. For example, different databases name tables and columns in different ways; methods exist to let Metabase know it should expect something like the `venues` table in the `test-data` Database Definition to come back as `VENUES` for a database that uppercases everything. (We consider such minor variations in naming to still mean the same thing.) Take a look at `tx/format-name` and other methods like that and see which ones you need to implement.
## What about DBMSes that don't let you create new databases programatically?
This is actually a common problem, and luckily we have figured out how to work around it. The solution is usually something like using different _schemas_ in place of different databases, or prefixing table names with the database name, and creating everything in the same database. For SQL-based databases, you can implement `sql.tx/qualified-name-components` to have tests use a different identifier instead of what they would normally use, for example `"shared_db"."test-data_venues".id` instead of `"test-data".venues.id`. The SQL Server and Oracle test extensions are good examples of such black magic in action.
# Running Tests and Linters
See [Running unit tests](devenv.md#running-unit-tests) and [Testing drivers](devenv.md#testing-drivers).
## Running the linter
See [Run the linters](devenv.md#run-the-linters).
# Setting up CI
Once you have all the tests passing, you'll need to set up CircleCI to run those tests against your driver. You'll need to add a step to [`./circleci/config.yml`](https://github.com/metabase/metabase/blob/master/.circleci/config.yml) to run tests against your database, then add that step to the "workflows" at the bottom of the config file.
Here is an example configuration for PostgreSQL.
```yaml
postgres-latest:
working_directory: /home/circleci/metabase/metabase/
docker:
- image: metabase/ci:circleci-java-11-clj-1.10.3.929-07-27-2021-node-browsers
environment:
MB_DB_TYPE: postgres
MB_DB_PORT: 5432
MB_DB_HOST: localhost
MB_DB_DBNAME: metabase_test
MB_DB_USER: metabase_test
MB_POSTGRESQL_TEST_USER: metabase_test
- image: circleci/postgres:latest
environment:
POSTGRES_USER: metabase_test
POSTGRES_DB: metabase_test
POSTGRES_HOST_AUTH_METHOD: trust
```
and the steps:
```
- test-driver:
name: be-tests-postgres-ee
description: "(9.6)"
requires:
- be-tests-java-8-ee
e: postgres-9-6
driver: postgres
- test-driver:
name: be-tests-postgres-latest-ee
description: "(Latest)"
requires:
- be-tests-java-8-ee
e: postgres-latest
driver: postgres
extra-env: >-
MB_POSTGRES_SSL_TEST_SSL=true
MB_POSTGRES_SSL_TEST_SSL_MODE=verify-full
MB_POSTGRES_SSL_TEST_SSL_ROOT_CERT_PATH=/home/circleci/metabase/metabase/test-resources/certificates/us-east-2-bundle.pem
```
For more on what it is you're doing here and how all this works, see [CircleCI 2.0 Workflows](https://circleci.com/docs/2.0/workflows/).
# Implementing multimethods for your driver
Implementing multimethods lets you take advantage of Metabase's existing driver code by extending those methods to work for your particular database.
Let's first focus on the main driver file for our Fox Pro '98 `src/metabase/driver/foxpro98.clj`. Take a look at this sample code:
```clj
;; Define a namespace for the driver
(ns com.mycompany.metabase.driver.foxpro98
(:require [metabase.driver :as driver]))
;; NOTE FOR CAM: this should be in the plugin manifest, no?
;; Can you include a different method here as an example?
(defmethod driver/display-name :foxpro98 [_]
"Visual FoxPro '98")
```
Let's walk through each code block.
## Driver namespaces
```
;; Define a namespace for the driver
(ns com.mycompany.metabase.driver.foxpro98
(:require [metabase.driver :as driver]))
```
### Each Metabase driver lives in its own namespace
In this case, the namespace is`com.mycompany.metabase.driver.foxpro98`.
All core Metabase drivers live in `metabase.driver.<name-goes-here>` namespaces. It's probably best to use names that follow the [Java package naming conventions](https://en.wikipedia.org/wiki/Java_package#Package_naming_conventions).
### Many drivers are further broken out into additional namespaces
Especially larger drivers. Commonly, a driver will have a `query-processor` namespace (e.g., `com.mycompany.metabase.driver.foxpro98.query-processor`) that contains the logic for converting MBQL queries (queries built using Metabase's graphical query builder) into native queries (like SQL). The query processor is often the most complicated part of a driver, so keeping that logic separate can help make things easier to work with. Some drivers also have a separate `sync` namespace that has implementations for methods used by Metabase's [database synchronization](https://www.metabase.com/docs/latest/administration-guide/01-managing-databases.html#database-syncing).
## Driver initialization
All drivers can include additional code to be executed once (and only once) using `metabase.driver/initialize!` when Metabase initializes the driver, that is, before the driver establishes a connection to a database for the first time. (In fact, Metabase uses `metabase.driver/initialize!` to lazy-load the driver.) There are only a few cases where you should use `metabase.driver/initialize`, such as allocating resources or setting certain system properties.
## `metabase.driver` multimethods
The [`metabase.driver` namespace](https://github.com/metabase/metabase/blob/master/src/metabase/driver.clj) defines a series of [multimethods](https://clojure.org/reference/multimethods), and drivers provide implementations for them, as in our example:
```clj
(defmethod driver/display-name :foxpro98 [_]
"Visual FoxPro '98")
```
The four main features of a Metabase driver described above are all implemented by multimethods. These methods dispatch on the driver's keyword, `:foxpro98` in our case. In fact, that's all a Metabase driver is -- a keyword! There are no classes or objects to be seen -- just a single keyword.
You can browse the [`metabase.driver` namespace](https://github.com/metabase/metabase/blob/master/src/metabase/driver.clj) for a complete list of multimethods that you could implement. Read the docstring for each method and decide whether you need to implement it. Most methods are optional.
## Listing the available driver multimethods
To quickly look up a list of all driver multimethods, you can run the command
```
clojure -M:run driver-methods
```
which will print a list of all driver namespaces and multimethods. This includes many things like `sql` and `sql-jdbc` multimethods, as well as test extension multimethods.
If you want to see the docstrings for the methods as well, run:
```
clojure -M:run driver-methods docs
```
## Parent drivers
Many drivers share implementation details, and writing complete implementations for sync methods and the like would involve a lot of duplicate code. Thus **many high-level features are partially or fully implemented in shared "parent" drivers**, such as the most common parent, `:sql-jdbc`. A "parent" driver is analogous to a superclass in object-oriented programming.
You can define a driver parent by listing the parent in the [plugin manifest](plugin.md).
Parents like `:sql-jdbc` are intended as a common abstract "base class" for drivers that can share much of their implementation; in the case of `:sql-jdbc`, it's intended for SQL-based drivers that use a JDBC driver under the hood.`:sql-jdbc` and other parents provide implementations for many of the methods needed to power the four main features of a Metabase driver. In fact, `:sql-jdbc` provides implementations of things like `driver/execute-query`, so a driver using it as a parent does not need to provide one itself. However, various parent drivers define their own multimethods to implement.
## Notable parent drivers
These parents are kind of a big deal.
- `:sql-jdbc` can be used as the parent for SQL-based databases with a JDBC driver.
- `:sql-jdbc` implements most of the four main features, but instead you must implement `sql-jdbc` multimethods found in `metabase.driver.sql-jdbc.*` namespaces, as well as some methods in `metabase.driver.sql.*` namespaces.
- `:sql` is itself the parent of `:sql-jdbc`; it can be used for SQL-based databases that _do not_ have a JDBC driver, such as BigQuery.
- `:sql` implements a significant chunk of driver functionality, but you must implement some methods found in `metabase.driver.sql.*` namespaces to use it.
- Drivers that use Google's API, such as BigQuery and Google Analytics, can use the `:google` driver as a parent.
- Some drivers use other "concrete" drivers as their parent -- for example, `:redshift` uses `:postgres` as a parent, only supplying method implementations to override postgres ones where needed.
### Calling parent driver implementations
You can get a parent driver's implementation for a method by using `get-method`:
```clj
(defmethod driver/mbql->native :bigquery [driver query]
((get-method driver/mbql-native :sql) driver query))
```
This is the equivalent of calling `super.someMethod()` in object-oriented programming.
You must pass the driver argument to the parent implementation as-is so any methods called by that method used the correct implementation. Here's two ways of calling parents that you should avoid:
```clj
(defmethod driver/mbql->native :bigquery [_ query]
;; BAD! If :sql's implementation of mbql->native calls any other methods, it won't use the :bigquery implementation
((get-method driver/mbql->native :sql) :sql query))
```
also avoid:
```clj
(defmethod driver/mbql->native :bigquery [_ query]
;; BAD! If someone else creates a driver using :bigquery as a parent, any methods called by :sql's implementation
;; of mbql->native will use :bigquery method implementations instead of custom ones for that driver
((get-method driver/mbql->native :sql) :bigquery query))
```
### Multiple parents
Astute readers may have noticed that BigQuery is mentioned as having both `:sql` and `:google` as a parent. This multiple inheritance is allowed and helpful! You can define a driver with multiple parents as follows:
```clj
(driver/register! :bigquery, :parent #{:sql :google})
```
In some cases, both parents may provide an implementation for a method; to fix this ambiguity, simply provide an implementation for your driver and pass them to the preferred parent driver's implementation as described above.
For drivers shipped as a plugin, you'll register methods in the plugin manifest.
## Working with the driver from the REPL and in CIDER
Having to install `metabase-core` locally and build driver uberjars would be obnoxious, especially if you had to repeat it to test every change. Luckily, you can run commands as if everything was part of one giant project:
To start a REPL.
```bash
clojure -A:dev:drivers:drivers-dev
```
You'll need to rebuild the driver and install it in your `./plugins` directory, and restart Metabase when you make changes.
# Plugin manifests
Metabase plugin JARs contain a _plugin manifest_ -- a top-level file named `metabase-plugin.yaml`. When Metabase launches, it iterates over every JAR in the plugins directory, and looks for the manifest in each. This manifest tells Metabase what the plugin provides and how to initialize it.
## Example manifest
```yaml
info:
name: Metabase SQLite Driver
version: 1.0.0-SNAPSHOT-3.25.2
description: Allows Metabase to connect to SQLite databases.
driver:
name: sqlite
display-name: SQLite
lazy-load: true
parent: sql-jdbc
connection-properties:
- name: db
display-name: Filename
placeholder: /home/camsaul/toucan_sightings.sqlite
required: true
init:
- step: load-namespace
namespace: metabase.driver.sqlite
- step: register-jdbc-driver
class: org.sqlite.JDBC
```
The `driver` section tells Metabase that the plugin defines a driver named `:sqlite` that has `:sql-jdbc` as a parent. Metabase's plugin system uses these details to call `driver/register!`. The plugin also lists the display name and connection properties for the driver, which Metabase's plugin system uses to creates implementations for `driver/display-name` and `driver/connection-properties`.
## Lazy loading
The driver in the [example above](#example-manifest) is listed as `lazy-load: true`, which means that, while the method implementation mentioned above are created when Metabase launches, Metabase won't initialize the driver until the first time someone attempts to connect to a database that uses that driver.
You _can_ (but shouldn't) set a driver to `lazy-load: false`, as this will make Metabase take longer to launch and eat up more memory.
## Plugin initialization
Metabase will initialize plugins automatically as needed. Initialization goes something like this: Metabase adds the driver to the classpath, then it performs ea `init` section of the plugin manifest, in order. In the [example manifest above](#example-manifest), there are two steps, a `load-namespace` step, and a `register-jdbc-driver` step:
```yaml
init:
- step: load-namespace
namespace: metabase.driver.sqlite
- step: register-jdbc-driver
class: org.sqlite.JDBC
```
## Loading namespaces
You'll need to add one or more `load-namespace` steps to your driver manifest to tell Metabase which namespaces contain your driver method implementations. In the example above, the namespace is `metabase.driver.sqlite`. `load-namespace` calls `require` the [normal Clojure way, meaning it will load other namespaces listed in the `:require` section of its namespace declaration as needed. If your driver's method implementations are split across multiple namespaces, make sure they'll get loaded as well -- you can either have the main namespace handle this (e.g., by including them in the `:require` form in the namespace declaration) or by adding additional `load-namespace` steps.
For some background on namespaces, see [Clojure namespaces][clojure-namespace].
## Registering JDBC Drivers
Drivers that use a JDBC driver under the hood will need to add a `register-jdbc-driver` step as well.
The if-you're-interested reason is that Java's JDBC `DriverManager` won't use JDBC drivers loaded with something other than the system `ClassLoader`, which effectively only means `Drivermanager` will only use JDBC driver classes that are packaged as part of the core Metabase uberjar. Since the system classloader doesn't allow you to load the classpath at runtime, Metabase uses a custom `ClassLoader` to initialize plugins. To work around this limitation, Metabase ships with a JDBC proxy driver class that can wrap other JDBC drivers. When Metabase calls `register-jdbc-driver`, Metabase actually registers a new instance of the proxy class that forwards method calls to the actual JDBC driver. `DriverManager` is perfectly fine with this.
## Building the driver
To build a driver as a plugin JAR, check out the [Build-driver scripts README](https://github.com/metabase/metabase/tree/master/bin/build-drivers).
Place the JAR you built in your Metabase's `/plugins` directory, and you're off to the races.
## The Metabase plugin manifest reference
Here's an example plugin manifest with comments to get you started on writing your own.
```
# Basic user-facing information about the driver goes under the info: key
info:
# Make sure to give your plugin a name. In the future, we can show
# this to the user in a 'plugin management' admin page.
name: Metabase SQLite Driver
# For the sake of consistency with the core Metabase project you
# should use semantic versioning. It's not a bad idea to include the
# version of its major dependency (e.g., a JDBC driver) when
# applicable as part of the 'patch' part of the version, so we can
# update dependencies and have that reflected in the version number
#
# For now core Metabase modules should have a version
# 1.0.0-SNAPSHOT-x until version 1.0 ships and the API for plugins
# is locked in
version: 1.0.0-SNAPSHOT-3.25.2
# Describe what your plugin does. Not used currently, but in the
# future we may use this description in a plugins admin page.
description: Allows Metabase to connect to SQLite databases.
# You can list any dependencies needed by the plugin by specifying a
# list of dependencies. If all dependencies are not met, the plugin
# will not be initialized.
#
# A dependency may be either a 'class' or (in the future) a 'plugin' dependency
dependencies:
# A 'class' dependency checks whether a given class is available on
# the classpath. It doesn't initialize the class; Metabase defers initialization
# until it needs to use the driver
# Don't use this for classes that ship as part of the plugin itself;
# only use it for external dependencies.
- class: oracle.jdbc.OracleDriver
# You may optionally add a message that will be displayed for
# information purposes in the logs, and possibly in a plugin
# management page as well in the future
message: >
Metabase requires the Oracle JDBC driver to connect to JDBC databases. See
https://metabase.com/docs/latest/administration-guide/databases/oracle.html
for more details
# A 'plugin' dependency checks whether a given plugin is available.
# The value for 'plugin' is whatever that plugin has as its 'name:' -- make
# sure you match it exactly!
#
# If the dependency is not available when this module is first loaded, the module
# will be tried again later after more modules are loaded. This means things will
# still work the way we expect even if, say, we initially attempt to load the
# BigQuery driver *before* loading its dependency, the shared Google driver. Once
# the shared Google driver is loaded, Metabase will detect that BigQuery's
# dependencies are now satisfied and initialize the plugin.
#
# In the future, we'll like add version restrictions as well, but for now we only match
# by name.
- plugin: Metabase SQLHeavy Driver
# If a plugin adds a driver it should define a driver: section.
#
# To define multiple drivers, you can pass a list of maps instead. Note
# that multiple drivers currently still must share the same dependencies
# set and initialization steps. Thus registering multiple drivers is most
# useful for slight variations on the same driver or including an abstract
# parent driver. Note that init steps will get ran once for each driver
# that gets loaded. This can result in duplicate driver instances registered
# with the DriverManager, which is certainly not ideal but does not actually
# hurt anything.
#
# In the near future I think I might move init steps into driver itself (or
# at least allow them there)
driver:
# Name of the driver; corresponds to the keyword (e.g. :sqlite) used
# in the codebase
name: sqlite
# Nice display name shown to admins when connecting a database
display-name: SQLite
# Whether loading this driver can be deferred until the first
# attempt to connect to a database of this type. Default: true. Only
# set this to false if absolutely neccesary.
lazy-load: true
# Parent driver, if any.
parent: sql-jdbc
# You may alternatively specify a list of parents for drivers with
# more than one:
parent:
- google
- sql
# Whether this driver is abstract. Default: false
abstract: false
# List of connection properties to ask users to set to connect to
# this driver.
connection-properties:
# Connection properties can be one of the defaults found in
# metabase.driver.common, listed by name:
- dbname
- host
# Or a full map for a custom option. Complete schema for custom
# options can be found in metabase.driver. NOTE: these are not
# currently translated for i18n; I'm working on a way to translate
# these. Prefer using one of the defaults from
# metabase.driver.common if possible.
- name: db
display-name: Filename
placeholder: /home/camsaul/toucan_sightings.sqlite
required: true
# Finally, you can use merge: to merge multiple maps. This is
# useful to override specific properties in one of the defaults.
- merge:
- port
- placeholder: 1433
# You can also tell Metabase to include SSL tunnel configuration options with
# connection-properties-include-tunnel-config (default: false)
connection-properties-include-tunnel-config: true
# Steps to take to initialize the plugin. For lazy-load drivers, this
# is delayed until the driver is initialized the first time we connect
# to a database with it
init:
# load-namespace tells Metabase to require a namespace from the JAR,
# you can do whatever Clojurey things you need to do inside that
# namespace
- step: load-namespace
namespace: metabase.driver.sqlite
# register-jdbc-driver tells Metabase to register a JDBC driver that
# will be used by this driver. (It actually registers a proxy
# driver, because DriverManager won't allow drivers that are loaded
# by different classloaders than it was loaded by (i.e., the system
# classloader); don't worry to much about this, but know for
# JDBC-based drivers you need to include your dependency here)
- step: register-jdbc-driver
class: org.sqlite.JDBC
```
## Next up
[Implementing multimethods](multimethods.md) for your driver.
# Guide to writing a Metabase driver
So here's the scenario: you love Metabase. It's changed your life. But you have some data in a Visual Fox Pro '98 database and you need to make charts with it, and it might be a while before the core Metabase team writes a driver for Visual Fox Pro '98. No problem! Writing a driver can be fun.
## Does a driver for your data source already exist?
Before you start building a driver from scratch, see if one already exists that you could contribute to:
- [Officially supported databases](https://www.metabase.com/docs/latest/administration-guide/01-managing-databases.html#officially-supported-databases)
- [Community databases](https://www.metabase.com/docs/latest/developers-guide-drivers.html#how-to-use-a-community-built-driver)
## Setting up
Before you start working on a driver, you'll need to set up your [development environment](../devenv.md).
Having an in-depth understanding of Clojure is less important when writing JDBC-based drivers because their implementation is simpler -- much of the work is already done for you -- but it would still be helpful to understand what things like [multimethods](https://clojure.org/reference/multimethods) are. See [Working with Clojure](../clojure.md).
## Writing a driver
Try to avoid skipping right to whichever page you think will give you the code you'll need to copy-pasta. While Metabase drivers are often fairly small (some as little as fifty lines of code), you should put some careful thought into deciding what goes into those fifty lines. You'll have an easier time writing the driver, and end up with a better one.
1. [Driver basics](basics.md)
2. [Plugin manifests](plugins.md)
3. [Implementing multimethods for your driver](multimethods.md)
4. [Submitting a PR for your driver](driver-tests.md)
## Example drivers
- [Sample driver](https://github.com/metabase/sample-driver)
- [Metabase driver modules](https://github.com/metabase/metabase/tree/master/modules/drivers)
- [A sample sudoku driver](https://github.com/metabase/sudoku-driver)
# Developing Metabase with Emacs
`.dir-locals.el` contains some Emacs Lisp that tells `clojure-mode` how to indent Metabase macros and which arguments are docstrings. Whenever this file is updated,
Emacs will ask you if the code is safe to load. You can answer `!` to save it as safe.
By default, Emacs will insert this code as a customization at the bottom of your `init.el`.
You'll probably want to tell Emacs to store customizations in a different file. Add the following to your `init.el`:
```emacs-lisp
(setq custom-file (concat user-emacs-directory ".custom.el")) ; tell Customize to save customizations to ~/.emacs.d/.custom.el
(ignore-errors ; load customizations from ~/.emacs.d/.custom.el
(load-file custom-file))
```
......@@ -16,6 +16,7 @@ This guide contains detailed information on how to work on Metabase codebase.
- [How to work with the frontend code](frontend.md)
- [How to add support in Metabase for other languages](internationalization.md)
- [Working with Clojure](clojure.md)
## Testing
......@@ -27,3 +28,9 @@ This guide contains detailed information on how to work on Metabase codebase.
- [Autogenerated API docs](../api-documentation.md)
- [Working with the Metabase API](https://www.metabase.com/learn/administration/metabase-api.html)
- [Driver Interface Changelog](driver-changelog.md)
## Database drivers
Learn how to write a driver to connect Metabase to your database.
- [Guide to writing a driver](drivers/start.md)
## Developing with Visual Studio Code
### Debugging
First, install the following extension:
* [Debugger for Firefox](https://marketplace.visualstudio.com/items?itemName=firefox-devtools.vscode-firefox-debug)
_Note_: Debugger for Chrome has been deprecated. You can safely delete it as Visual Studio Code now has [a bundled JavaScript Debugger](https://github.com/microsoft/vscode-js-debug) that covers the same functionality.
Before starting the debugging session, make sure that Metabase is built and running. Choose menu _View_, _Command Palette_, search for and choose _Tasks: Run Build Task_. Alternatively, use the corresponding shortcut `Ctrl+Shift+B`. The built-in terminal will appear to show the progress, wait a few moment until webpack indicates a complete (100%) bundling.
To begin debugging Metabase, switch to the Debug view (shortcut: `Ctrl+Shift+D`) and then select one of the two launch configurations from the drop-down at the top:
* Debug with Firefox, or
* Debug with Chrome
After that, begin the debugging session by choosing menu _Run_, _Start Debugging_ (shortcut: `F5`).
For more details, please refer to the complete VS Code documentation on [Debugging](https://code.visualstudio.com/docs/editor/debugging).
### Docker-based Workflow
These instructions allow you to work on Metabase codebase on Windows, Linux, or macOS using [Visual Studio Code](https://code.visualstudio.com/), **without** manually installing the necessary dependencies. This is possible by leveraging Docker container and the Remote Containers extension from VS Code.
For more details, please follow the complete VS Code guide on [Developing inside a Container](https://code.visualstudio.com/docs/remote/containers).
Requirements:
* [Visual Studio Code](https://code.visualstudio.com/) (obviously)
* [Docker](https://www.docker.com/)
* [Remote - Containers extension](vscode:extension/ms-vscode-remote.remote-containers) for VS Code
_Important_: Ensure that Docker is running properly and it can be used to download an image and launch a container, e.g. by running:
```
$ docker run hello-world
```
If everything goes well, you should see the following message:
```
Hello from Docker!
This message shows that your installation appears to be working correctly.
```
Steps:
1. Clone Metabase repository
2. Launch VS Code and open your cloned Metabase repository
3. From the _View_ menu, choose _Command Palette..._ and then find _Remote-Container: Reopen in Container_. (VS Code may also prompt you to do this with an "Open in container" popup).
**Note**: VS Code will create the container for the first time and it may take some time. Subsequent loads should be much faster.
4. Use the menu _View_, _Command Palette_, search for and choose _Tasks: Run Build Task_ (alternatively, use the shortcut `Ctrl+Shift+B`).
5. After a while (after all JavaScript and Clojure dependencies are completely downloaded), open localhost:3000 with your web browser.
See [here](dev-branch-docker.md) for more on running development branches of Metabase using Docker.
docs/users-guide/images/pulses/02-name-it.png

46.1 KiB

docs/users-guide/images/pulses/03-pick-your-data.png

22.5 KiB

docs/users-guide/images/pulses/04-transformation.png

74.4 KiB

docs/users-guide/images/pulses/05-email-schedule.png

46.9 KiB

docs/users-guide/images/pulses/08-delete.png

15.3 KiB

docs/users-guide/images/pulses/attachments/attach-button.png

39.4 KiB

docs/users-guide/images/pulses/attachments/attached.png

41.7 KiB

docs/users-guide/images/pulses/attachments/email.png

77.9 KiB

docs/users-guide/images/pulses/table.png

86.3 KiB

......@@ -116,11 +116,13 @@
((resolve 'metabase.cmd.endpoint-dox/generate-dox!)))
(defn ^:command driver-methods
"Print a list of all multimethods a available for a driver to implement. A useful reference when implementing a new
driver."
[]
(classloader/require 'metabase.cmd.driver-methods)
((resolve 'metabase.cmd.driver-methods/print-available-multimethods)))
"Print a list of all multimethods available for a driver to implement, optionally with their docstrings."
([]
(classloader/require 'metabase.cmd.driver-methods)
((resolve 'metabase.cmd.driver-methods/print-available-multimethods) false))
([docs]
(classloader/require 'metabase.cmd.driver-methods)
((resolve 'metabase.cmd.driver-methods/print-available-multimethods) true)))
(defn- cmd-args->map
[args]
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment