-
John Swanson authored
* Do not cache all token check failures We want to cache token checks to avoid an issue where we repeatedly ask the store "hey, is this token valid?? is this token valid?? is this token valid??" for the same token. However, transient errors can also occur. For example, maybe a network issue causes the HTTP request to fail entirely. In this case, if we cache the result, the user needs to restart metabase (or wait 5 minutes until the cache is cleared) before they can attempt to validate their token again. This PR moves the cache logic deeper into the stack. We want to cache "successful" responses from the store API - cases where the store has told us categorically that the token is or is not valid. We don't want or need to cache other things that might happen. Maybe your token isn't the right length - we can recalculate that, it's ok. Maybe you get a 503 error from the Store - we should let you retry. Maybe your network is having issues and you can't contact the Store at all - again, we should let you retry. The one potential issue I see here is that if the store goes down, we'll massively increase the number of requests we send to the store, potentially making it harder to recover. If this is a concern, I can add a circuit breaker: if we repeatedly get errors back from the store, back off and stop making requests for a while. * Add a circuit breaker to store API requests In the pathological case where the store goes down for >5 minutes, the cache will expire and all instances everywhere will start repeatedly making requests for token validation at once. This might make recovering from an outage more difficult. This adds a circuit breaker around the API request. If the call repeatedly throws (5XX errors, socket timeouts, etc.) then we'll pause for 1 minute, during which time all calls to token validation will immediately fail without making any request to the API. After one minute, we'll allow one request through to the API. If it succeeds, we'll go back to normal operation. Otherwise, we'll wait another minute.
John Swanson authored* Do not cache all token check failures We want to cache token checks to avoid an issue where we repeatedly ask the store "hey, is this token valid?? is this token valid?? is this token valid??" for the same token. However, transient errors can also occur. For example, maybe a network issue causes the HTTP request to fail entirely. In this case, if we cache the result, the user needs to restart metabase (or wait 5 minutes until the cache is cleared) before they can attempt to validate their token again. This PR moves the cache logic deeper into the stack. We want to cache "successful" responses from the store API - cases where the store has told us categorically that the token is or is not valid. We don't want or need to cache other things that might happen. Maybe your token isn't the right length - we can recalculate that, it's ok. Maybe you get a 503 error from the Store - we should let you retry. Maybe your network is having issues and you can't contact the Store at all - again, we should let you retry. The one potential issue I see here is that if the store goes down, we'll massively increase the number of requests we send to the store, potentially making it harder to recover. If this is a concern, I can add a circuit breaker: if we repeatedly get errors back from the store, back off and stop making requests for a while. * Add a circuit breaker to store API requests In the pathological case where the store goes down for >5 minutes, the cache will expire and all instances everywhere will start repeatedly making requests for token validation at once. This might make recovering from an outage more difficult. This adds a circuit breaker around the API request. If the call repeatedly throws (5XX errors, socket timeouts, etc.) then we'll pause for 1 minute, during which time all calls to token validation will immediately fail without making any request to the API. After one minute, we'll allow one request through to the API. If it succeeds, we'll go back to normal operation. Otherwise, we'll wait another minute.
Code owners
Assign users and groups as approvers for specific file changes. Learn more.
deps.edn 46.98 KiB
;; -*- comment-column: 80; -*-
{:deps
;; !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
;; !! PLEASE KEEP THESE ORGANIZED ALPHABETICALLY !!
;; !! AND ADD A COMMENT EXPLAINING THEIR PURPOSE !!
;; !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
{amalloy/ring-buffer {:mvn/version "1.3.1" ; fixed length queue implementation, used in log buffering
:exclusions [org.clojure/clojure
org.clojure/clojurescript]}
amalloy/ring-gzip-middleware {:mvn/version "0.1.4"} ; Ring middleware to GZIP responses if client can handle it
babashka/fs {:mvn/version "0.5.20"} ; Portable filesystem operations
bigml/histogram {:mvn/version "4.1.4" ; Histogram data structure
:exclusions [junit/junit]}
buddy/buddy-core {:mvn/version "1.11.423" ; various cryptographic functions
:exclusions [commons-codec/commons-codec
org.bouncycastle/bcpkix-jdk15on
org.bouncycastle/bcprov-jdk15on
org.bouncycastle/bcpkix-jdk18on
org.bouncycastle/bcprov-jdk18on]}
buddy/buddy-sign {:mvn/version "3.5.351"} ; JSON Web Tokens; High-Level message signing library
camel-snake-kebab/camel-snake-kebab {:mvn/version "0.4.3"} ; util functions for converting between camel, snake, and kebob case
cheshire/cheshire {:mvn/version "5.12.0"} ; fast JSON encoding (used by Ring JSON middleware)
clj-bom/clj-bom {:mvn/version "0.1.2"} ; handle BOMs in imported CSVs
clj-commons/iapetos {:mvn/version "0.1.13"} ; prometheus metrics
clj-http/clj-http {:mvn/version "3.12.3" ; HTTP client
:exclusions [commons-codec/commons-codec
commons-io/commons-io
slingshot/slingshot]}
clojure.java-time/clojure.java-time {:mvn/version "1.4.2"} ; java.time utilities
clojurewerkz/quartzite {:mvn/version "2.2.0" ; scheduling library
:exclusions [c3p0/c3p0
org.quartz-scheduler/quartz]}
colorize/colorize {:mvn/version "0.1.1" ; string output with ANSI color codes (for logging)
:exclusions [org.clojure/clojure]}
com.clearspring.analytics/stream {:mvn/version "2.9.8" ; Various sketching algorithms
:exclusions [it.unimi.dsi/fastutil
org.slf4j/slf4j-api]}
com.draines/postal {:mvn/version "2.0.5"} ; SMTP library
; Detect the charset in uploaded CSV files
com.github.albfernandez/juniversalchardet {:mvn/version "2.5.0"}
com.github.seancorfield/honeysql {:mvn/version "2.6.1126"} ; Honey SQL 2. SQL generation from Clojure data maps
com.github.seancorfield/next.jdbc {:mvn/version "1.3.925"} ; Talk to JDBC DBs
com.github.steffan-westcott/clj-otel-api {:mvn/version "0.2.6"} ; Telemetry library
com.github.vertical-blank/sql-formatter {:mvn/version "2.0.4"} ; Java SQL formatting library https://github.com/vertical-blank/sql-formatter
com.google.guava/guava {:mvn/version "33.1.0-jre"} ; dep for BigQuery, Spark, and GA. Require here rather than letting different dep versions stomp on each other — see comments on #9697
com.fasterxml.jackson.core/jackson-databind
{:mvn/version "2.17.0"} ; JSON processor used by snowplow-java-tracker
com.fasterxml.woodstox/woodstox-core {:mvn/version "6.6.1"} ; trans dep of commons-codec
com.h2database/h2 {:mvn/version "2.1.214"} ; embedded SQL database
com.gfredericks/test.chuck {:mvn/version "0.2.14"} ; generating strings from regex
com.snowplowanalytics/snowplow-java-tracker
{:mvn/version "1.0.1" ; Snowplow analytics
:exclusions [com.fasterxml.jackson.core/jackson-databind]}
com.taoensso/nippy {:mvn/version "3.4.2"} ; Fast serialization (i.e., GZIP) library for Clojure
com.vladsch.flexmark/flexmark {:mvn/version "0.64.8"} ; Markdown parsing
com.vladsch.flexmark/flexmark-ext-autolink
{:mvn/version "0.64.8"} ; Flexmark extension for auto-linking bare URLs
commons-fileupload/commons-fileupload {:mvn/version "1.5"} ; ring/ring-core 1.9.6 uses v1.4, but we want 1.5 due to a CVE. When we upgrade to the forthcoming ring/ring-core 1.10.0 we can remove this.
commons-codec/commons-codec {:mvn/version "1.16.1"} ; Apache Commons -- useful codec util fns
commons-io/commons-io {:mvn/version "2.15.1"} ; Apache Commons -- useful IO util fns
commons-net/commons-net {:mvn/version "3.10.0"} ; Apache Commons -- useful network utils. Transitive dep of Snowplow, pinned due to CVE-2021-37533
commons-validator/commons-validator {:mvn/version "1.8.0" ; Apache Commons -- useful validation util fns
:exclusions [commons-beanutils/commons-beanutils
commons-digester/commons-digester
commons-logging/commons-logging]}
compojure/compojure {:mvn/version "1.7.1" ; HTTP Routing library built on Ring
:exclusions [ring/ring-codec]}
crypto-random/crypto-random {:mvn/version "1.2.1"} ; library for generating cryptographically secure random bytes and strings
diehard/diehard {:mvn/version "0.11.12"}
dk.ative/docjure {:mvn/version "1.19.0" ; excel export :exclusions [org.apache.poi/poi
org.apache.poi/poi-ooxml]}
environ/environ {:mvn/version "1.2.0"} ; env vars/Java properties abstraction
hiccup/hiccup {:mvn/version "1.0.5"} ; HTML templating
inflections/inflections {:mvn/version "0.14.1"} ; Clojure/Script library used for prularizing words
instaparse/instaparse {:mvn/version "1.4.12"} ; Make your own parser
junegunn/grouper {:mvn/version "0.1.1"} ; Batch processing helper
clj-commons/clj-yaml {:mvn/version "1.0.27"} ; Clojure wrapper for YAML library SnakeYAML
io.github.camsaul/toucan2 {:mvn/version "1.0.538"}
io.github.eerohele/pp {:git/tag "2024-01-04.60" ; super fast pretty-printing library
:git/sha "a428751"
:git/url "https://github.com/eerohele/pp"}
io.github.metabase/macaw {:mvn/version "0.2.10"} ; Parse native SQL queries
;; The 2.X line of Resilience4j requires Java 17, so we cannot upgrade this dependency until that is our minimum JVM version
io.github.resilience4j/resilience4j-retry {:mvn/version "1.7.1" #_"must be 1.7.1"} ; Support for retrying operations
io.prometheus/simpleclient_hotspot {:mvn/version "0.16.0"} ; prometheus jvm collector
io.prometheus/simpleclient_jetty {:mvn/version "0.16.0"} ; prometheus jetty collector
javax.servlet/servlet-api {:mvn/version "2.5"} ; used by ring's multipart-params (file upload)
kixi/stats {:mvn/version "0.5.5" ; Various statistic measures implemented as transducers
:exclusions [org.clojure/data.avl]}
lambdaisland/uri {:mvn/version "1.19.155"} ; Used by openai-clojure
medley/medley {:mvn/version "1.4.0"} ; lightweight lib of useful functions
metabase/connection-pool {:mvn/version "1.2.0"} ; simple wrapper around C3P0. JDBC connection pools
metabase/saml20-clj {:mvn/version "2.2.7.172"
:exclusions [; EE SAML integration TODO: bump version when we release the library
org.bouncycastle/bcpkix-jdk15on
org.bouncycastle/bcprov-jdk15on
org.bouncycastle/bcpkix-jdk18on
org.bouncycastle/bcprov-jdk18on]}
metabase/throttle {:mvn/version "1.0.2"} ; Tools for throttling access to API endpoints and other code pathways
methodical/methodical {:mvn/version "1.0.123"} ; drop-in replacements for Clojure multimethods and adds several advanced features
metosin/malli {:mvn/version "0.16.3"} ; Data-driven Schemas for Clojure/Script and babashka
nano-id/nano-id {:mvn/version "1.1.0"} ; NanoID generator for generating entity_ids
net.cgrand/macrovich {:mvn/version "0.2.2"} ; utils for writing macros for both Clojure & ClojureScript
net.clojars.wkok/openai-clojure {:mvn/version "0.16.0"
:exclusions [lambdaisland/uri]} ; OpenAI
net.i2p.crypto/eddsa {:mvn/version "0.3.0"} ; ED25519 key support (optional dependency for org.apache.sshd/sshd-core)
net.redhogs.cronparser/cron-parser-core {:mvn/version "3.5" ; describe Cron schedule in human-readable language
:exclusions [org.slf4j/slf4j-api]}
net.sf.cssbox/cssbox {:mvn/version "5.0.1" ; HTML / CSS rendering
:exclusions [org.slf4j/slf4j-api
junit/junit]}
net.thisptr/jackson-jq {:mvn/version "1.0.0-preview.20240207"} ; Java implementation of the JQ json query language
org.apache.commons/commons-compress {:mvn/version "1.26.1"} ; compression utils
org.apache.commons/commons-lang3 {:mvn/version "3.14.0"} ; helper methods for working with java.lang stuff
org.apache.logging.log4j/log4j-1.2-api {:mvn/version "2.23.1"} ; apache logging framework
org.apache.logging.log4j/log4j-api {:mvn/version "2.23.1"} ; add compatibility with log4j 1.2
org.apache.logging.log4j/log4j-core {:mvn/version "2.23.1"} ; apache logging framework
org.apache.logging.log4j/log4j-jcl {:mvn/version "2.23.1"} ; allows the commons-logging API to work with log4j 2
org.apache.logging.log4j/log4j-jul {:mvn/version "2.23.1"} ; java.util.logging (JUL) -> Log4j2 adapter
org.apache.logging.log4j/log4j-slf4j2-impl
{:mvn/version "2.23.1"} ; allows the slf4j2 API to work with log4j 2
org.apache.logging.log4j/log4j-layout-template-json
{:mvn/version "2.23.1"} ; allows the custom json logging format
org.apache.poi/poi {:mvn/version "5.2.5"} ; Work with Office documents (e.g. Excel spreadsheets) -- newer version than one specified by Docjure
org.apache.poi/poi-ooxml {:mvn/version "5.2.5"
:exclusions [org.bouncycastle/bcpkix-jdk15on
org.bouncycastle/bcprov-jdk15on]}
org.apache.sshd/sshd-core {:mvn/version "2.12.1" ; ssh tunneling and test server
:exclusions [org.slf4j/slf4j-api
org.slf4j/jcl-over-slf4j]}
org.apache.tika/tika-core {:mvn/version "2.9.2"}
org.apache.xmlgraphics/batik-all {:mvn/version "1.17"} ; SVG -> image
org.bouncycastle/bcpkix-jdk18on {:mvn/version "1.78"} ; Bouncy Castle crypto library -- explicit version of BC specified to resolve illegal reflective access errors
org.bouncycastle/bcprov-jdk18on {:mvn/version "1.78"}
org.clj-commons/claypoole {:mvn/version "1.2.2"} ; Threadpool tools for Clojure
org.clj-commons/hickory {:mvn/version "0.7.4" ; Parse HTML into Clojure data structures
:exclusions [org.jsoup/jsoup]}
org.clojars.pntblnk/clj-ldap {:mvn/version "0.0.17"} ; LDAP client
org.clojure/clojure {:mvn/version "1.12.0"} org.clojure/core.async {:mvn/version "1.6.681"
:exclusions [org.clojure/tools.reader]}
org.clojure/core.logic {:mvn/version "1.1.0"} ; optimized pattern matching library for Clojure
org.clojure/core.match {:mvn/version "1.1.0"}
org.clojure/core.memoize {:mvn/version "1.1.266"} ; useful FIFO, LRU, etc. caching mechanisms
org.clojure/data.csv {:mvn/version "1.1.0"} ; CSV parsing / generation
org.clojure/data.xml {:mvn/version "0.0.8"} ; XML parsing / generation
org.clojure/java.classpath {:mvn/version "1.1.0"} ; examine the Java classpath from Clojure programs
org.clojure/java.jdbc {:mvn/version "0.7.12"} ; basic JDBC access from Clojure
org.clojure/java.jmx {:mvn/version "1.1.0"} ; JMX bean library, for exporting diagnostic info
org.clojure/math.combinatorics {:mvn/version "0.3.0"} ; combinatorics functions
org.clojure/math.numeric-tower {:mvn/version "0.1.0"} ; math functions like `ceil`
org.clojure/tools.cli {:mvn/version "1.1.230"} ; command-line argument parsing
org.clojure/tools.logging {:mvn/version "1.3.0"} ; logging framework
org.clojure/tools.macro {:mvn/version "0.2.0"} ; local macros
org.clojure/tools.namespace {:mvn/version "1.5.0"}
org.clojure/tools.reader {:mvn/version "1.4.1"}
org.clojure/tools.trace {:mvn/version "0.8.0"} ; function tracing
;; v. 12 of jetty-server triggers "a org/eclipse/jetty/io/EofException has been compiled by a more recent version of
;; the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions
;; up to 55.0"
org.eclipse.jetty/jetty-server {:mvn/version "11.0.20" #_"must be 11"} ; web server
org.eclipse.jetty.websocket/websocket-jetty-server {:mvn/version "11.0.20"} ; ring-jetty-adapter needs that
org.flatland/ordered {:mvn/version "1.15.11"} ; ordered maps & sets
org.graalvm.js/js {:mvn/version "22.3.5"} ; JavaScript engine
org.jsoup/jsoup {:mvn/version "1.17.2"} ; required by hickory
org.liquibase/liquibase-core {:mvn/version "4.26.0" ; migration management (Java lib)
:exclusions [ch.qos.logback/logback-classic]}
;; The 3.X line of development for mariadb-java-client only supports the jdbc:mariadb protocol, so use 2.X for now.
org.mariadb.jdbc/mariadb-java-client {:mvn/version "2.7.10"} ; MySQL/MariaDB driver
org.mindrot/jbcrypt {:mvn/version "0.4"} ; Crypto library
org.postgresql/postgresql {:mvn/version "42.7.3"} ; Postgres driver
org.quartz-scheduler/quartz {:mvn/version "2.3.2"} ; Quartz job scheduler, provided by quartzite but this is a newer version.
org.slf4j/slf4j-api {:mvn/version "2.0.12"} ; abstraction for logging frameworks -- allows end user to plug in desired logging framework at deployment time
org.tcrawley/dynapath {:mvn/version "1.1.0"} ; Dynamically add Jars (e.g. Oracle or Vertica) to classpath
org.threeten/threeten-extra {:mvn/version "1.7.2"} ; extra Java 8 java.time classes like DayOfMonth and Quarter
potemkin/potemkin {:mvn/version "0.4.7" ; utility macros & fns
:exclusions [riddley/riddley]}
pretty/pretty {:mvn/version "1.0.5"} ; protocol for defining how custom types should be pretty printed
;; Dependency below is our fork of redux - utility functions for building and composing transducers. The
;; group/artifact name is the same as with the upstream so that it overrides the original dependency everywhere.
redux/redux {:git/url "https://github.com/metabase/redux"
:sha "4a37feaf817a2a6b5ef688c927f6fd4375964433"}
riddley/riddley {:mvn/version "0.2.0"} ; code walking lib -- used interally by Potemkin, manifold, etc.
ring/ring-core {:mvn/version "1.12.0"} ; HTTP abstraction
ring/ring-jetty-adapter {:mvn/version "1.12.0" ; Jetty adapter
:exclusions [org.eclipse.jetty/jetty-server
org.eclipse.jetty.websocket/websocket-jetty-server]}
ring/ring-json {:mvn/version "0.5.1"} ; Ring middleware for reading/writing JSON automatically
slingshot/slingshot {:mvn/version "0.12.2"} ; enhanced throw/catch, used by other deps
stencil/stencil {:mvn/version "0.5.0"} ; Mustache templates for Clojure
user-agent/user-agent {:mvn/version "0.1.1"} ; User-Agent string parser, for Login History page & elsewhere
weavejester/dependency {:mvn/version "0.2.1"}} ; Dependency graphs and topological sorting
;; !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
;; !! PLEASE KEEP NEW DEPENDENCIES ABOVE ALPHABETICALLY ORGANIZED AND ADD COMMENTS EXPLAINING THEM. !!
;; !! *PLEASE DO NOT* ADD NEW ONES TO THE BOTTOM OF THE LIST. !!
;; !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
:paths
["src" "resources"]
;; These are needed for the Athena and Redshift drivers if you are developing against them locally. If those drivers'
;; dependencies are not included (i.e., if we don't have the `:drivers` profile), these repos are effectively
;; ignored.
;;
;; 1. Maven repos from subprojects do not get copied over -- see
;; https://ask.clojure.org/index.php/10726/deps-manifest-dependencies-respect-repos-dependent-project
;;
;; 2. You cannot include `:mvn/repos` inside of an alias -- see ;; https://ask.clojure.org/index.php/12367/support-mvn-repos-inside-an-alias -- if we could, this could go in the
;; `:drivers` alias instead.
:mvn/repos
{"athena" {:url "https://s3.amazonaws.com/maven-athena"}
"redshift" {:url "https://s3.amazonaws.com/redshift-maven-repository/release"}
;; for metabase/saml20-clj
"opensaml" {:url "https://build.shibboleth.net/nexus/content/repositories/releases/"}}
:aliases
{;;; Local Dev & test profiles
;; for local development: start a REPL with
;;
;; clojure -A:dev (basic dev REPL that includes test namespaces)
;; clojure -A:dev:drivers:drivers-dev (dev REPL w/ drivers + tests)
;; clojure -A:dev:ee:ee-dev (dev REPL w/ EE code including tests)
;;
;; You can start a web server from this REPL with
;;
;; (require 'dev)
;; (dev/start!)
:dev
{:extra-deps
{com.clojure-goes-fast/clj-async-profiler
{:mvn/version "1.2.0"} ; Enables local profiling and heat map generation
com.clojure-goes-fast/clj-memory-meter
{:mvn/version "0.3.0"} ; Enables easy memory measurement
clj-http-fake/clj-http-fake {:mvn/version "1.0.4"
:exclusions [slingshot/slingshot]}
clj-kondo/clj-kondo {:mvn/version "2024.08.01"} ; included here mainly to facilitate working on the stuff in `.clj-kondo/src`
cloverage/cloverage {:mvn/version "1.2.4"}
com.gfredericks/test.chuck {:mvn/version "0.2.14"} ; generating strings from regexes (useful with malli)
djblue/portal {:mvn/version "0.56.0"} ; ui for inspecting values
hashp/hashp {:mvn/version "0.2.2"} ; debugging/spying utility
io.github.camsaul/humane-are {:mvn/version "1.0.2"}
io.github.metabase/hawk {:mvn/version "1.0.0"}
jonase/eastwood {:mvn/version "1.4.2" ; inspects namespaces and reports possible problems using tools.analyzer
:exclusions
[org.ow2.asm/asm-all]}
criterium/criterium {:mvn/version "0.4.6"} ; benchmarking library
lambdaisland/deep-diff2 {:mvn/version "2.11.216"} ; way better diffs
org.clojure/algo.generic {:mvn/version "1.0.1"}
peridot/peridot {:git/url "https://github.com/piranha/peridot.git"
:sha "999d0a02425c906c35bace749654dc095ecf3e6a"} ; mocking Ring requests; waiting for upstream release of commit 0fc7c01 (explicit charset)
pjstadig/humane-test-output {:mvn/version "0.11.0"}
reifyhealth/specmonstah {:mvn/version "2.1.0"
:exclusions [org.clojure/clojure
org.clojure/clojurescript]} ; lets you write test fixtures that are clear, concise, and easy to maintain (clojure.spec)
ring/ring-mock {:mvn/version "0.4.0"} ; creating Ring request maps for testing purposes
talltale/talltale {:mvn/version "0.5.14"}} ; generates fake data, useful for prototyping or load testing
:extra-paths ["dev/src" "local/src" "test" "test_resources" ".clj-kondo/src" ".clj-kondo/test"]
:jvm-opts ["-Dmb.run.mode=dev"
"-Dmb.field.filter.operators.enabled=true"
"-Dmb.test.env.setting=ABCDEFG"
"-Duser.timezone=UTC"
"-Dfile.encoding=UTF-8"
"-Duser.language=en"
"-Duser.country=US"
;; Allow clojure goes fast tooling to work
"-Djdk.attach.allowAttachSelf"
;; This will suppress the warning about dynamically loaded agents (like clj-memory-meter)
"-XX:+EnableDynamicAgentLoading"
;; set the logging properties set in metabase.bootstrap. calling (dev) will load it but putting here to be sure
"-Dlog4j2.contextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector"
"-Dclojure.tools.logging.factory=clojure.tools.logging.impl/log4j2-factory"
;; If Clojure fails to start (e.g. because of a compilation error somewhere) print the error
;; report/stacktrace to stderr rather than to a random EDN file in /tmp/
"-Dclojure.main.report=stderr"
;; Exceptions that get thrown repeatedly are created without stacktraces as a performance ;; optimization in newer Java versions. This makes debugging pretty hard when working on stuff
;; locally -- prefer debuggability over performance for local dev work.
"-XX:-OmitStackTraceInFastThrow"
;; prevent Java icon from randomly popping up in macOS dock
"-Djava.awt.headless=true"
;; ignore options that aren't present in older versions of Java, like the one below:
"-XX:+IgnoreUnrecognizedVMOptions"
;; include more details for debugging NPEs (Java 14+)
"-XX:+ShowCodeDetailsInExceptionMessages"]}
;; includes test code as source paths. Run tests with clojure -X:dev:test
:test
{:extra-paths ["test_config"]
:exec-fn metabase.test-runner/find-and-run-tests-cli
:jvm-opts ["-Dmb.run.mode=test"
"-Dmb.db.in.memory=true"
"-Dmb.jetty.join=false"
"-Dmb.field.filter.operators.enabled=true"
"-Dmb.api.key=test-api-key"
;; Different port from normal `:dev` so you can run tests on a different server.
;; TODO -- figure out how to do a random port like in the old project.clj?
"-Dmb.jetty.port=3001"]}
;; run the dev server with
;; clojure -M:run
;; clojure -M:run:drivers (include all drivers)
;; clojure -M:run:ee (include EE code)
:run
{:main-opts ["-m" "metabase.bootstrap"]
:jvm-opts ["-Dmb.run.mode=dev"
"-Djava.awt.headless=true"]} ; prevent Java icon from randomly popping up in macOS dock
;; alias for CI-specific options.
:ci
{:jvm-opts ["-Xmx12g"
"-Xms12g"
;; normally CircleCI sets `CI` as an env var, so this is mostly to replicate that locally. Hawk will not
;; print the progress bar in output when this is set. Progress bars aren't very CI friendly
"-Dci=TRUE"]}
;; include EE source code.
:ee
{:extra-paths ["enterprise/backend/src"]}
;; Include EE tests.
;; for EE dev: `clojure -X:dev:ee:ee-dev`
;; for EE tests: `clojure -X:dev:ee:ee-dev:test`
:ee-dev
{:extra-paths ["enterprise/backend/test"]}
;; these aliases exist for symmetry with the ee aliases. Empty for now.
:oss
{}
:oss-dev
{}
;; Generate BE documentation with
;; clojure -M:marginalia
:marginalia
{:extra-deps
{marginalia/marginalia {:mvn/version "0.9.2"}}
:main-opts ["-m" "marginalia.main" "-n" "Metabase" "-d" "backend-docs" "-D"
"The simplest, fastest way to get business intelligence and analytics to everyone in your company 😋"
"dev" "src" "enterprise/backend/src"]}
;; Generate a single file's docs with
;; clojure -M:marginalia/one src/metabase/some/namespace.clj
;; which creates `backend-docs/metabase.some.namespace.html`
:marginalia/one {:extra-deps
{marginalia/marginalia {:mvn/version "0.9.2"}}
:main-opts ["-m" "marginalia.main" "-m" "-n" "Metabase" "-d" "backend-docs" "-D"
"The simplest, fastest way to get business intelligence and analytics to everyone in your company 😋"]}
;; Find outdated versions of dependencies. Run with `clojure -M:outdated`
:outdated {;; Note that it is `:deps`, not `:extra-deps`
:deps {com.github.liquidz/antq {:mvn/version "RELEASE"}}
:main-opts ["-m" "antq.core" "--skip=github-action"]}
:cljs
{:extra-paths ["test"]
:extra-deps
{binaryage/devtools {:mvn/version "1.0.7"}
cider/cider-nrepl {:mvn/version "0.47.0"}
cider/piggieback {:mvn/version "0.5.3"}
cljs-bean/cljs-bean {:mvn/version "1.9.0"}
com.lambdaisland/glogi {:mvn/version "1.3.169"}
io.github.metabase/hawk {:git/url "https://github.com/metabase/hawk"
:git/sha "490c9b04fbf5861dbd47c0c5281dad2d113bd285"}
org.clojars.mmb90/cljs-cache {:mvn/version "0.1.4"}
refactor-nrepl/refactor-nrepl {:mvn/version "3.10.0"}
thheller/shadow-cljs {:mvn/version "2.28.12"}}}
;; for local dev -- include the drivers locally with :dev:drivers
:drivers
{:extra-deps
{metabase/driver-modules {:local/root "modules/drivers"}}}
;; for local dev: include drivers as well as their tests.
;;
;; clojure -X:dev:drivers:drivers-dev:test
;;
;; or
;;
;; clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test (for EE)
:drivers-dev
{:extra-paths
["modules/drivers/athena/test"
"modules/drivers/bigquery-cloud-sdk/test"
"modules/drivers/databricks/test"
"modules/drivers/druid/test"
"modules/drivers/druid-jdbc/test"
"modules/drivers/mongo/test"
"modules/drivers/oracle/test"
"modules/drivers/presto-jdbc/test"
"modules/drivers/redshift/test"
"modules/drivers/snowflake/test"
"modules/drivers/sparksql/test"
"modules/drivers/sqlite/test"
"modules/drivers/sqlserver/test"
"modules/drivers/vertica/test"]}
;;; Linters
;; clojure -M:kondo --lint src test
;;
;; clojure -M:kondo --version
;;
;; clojure -M:kondo --copy-configs --dependencies --lint "$(clojure -A:dev -Spath)" --skip-lint --parallel
;;
;; Run Kondo from the JVM using the pinned version. Preferable to running the installed command since we can pin the
;; version here which may be different from the version installed on your computer.
;;
;; Use this to only run Kondo against specific files.
:kondo
{:replace-deps
{clj-kondo/clj-kondo {:mvn/version "2024.08.29"}}
:main-opts ["-m" "clj-kondo.main"]}
;; clojure -M:kondo:kondo/all
;;
;; Like the command above but includes arguments so it lints everything.
:kondo/all
{:main-opts
["-m" "clj-kondo.main"
;; disabled for now since this seems to cause random failures -- see https://github.com/clj-kondo/clj-kondo/issues/2218
#_"--parallel"
;; enable this if Kondo isn't working as expected or if LSP jump-to-definition is being fussy
#_"--debug"
"--lint"
"src"
"test"
"enterprise/backend/src"
"enterprise/backend/test"
"bin/build/src"
"bin/build/test"
"bin/lint-migrations-file/src"
"bin/lint-migrations-file/test"
"bin/release-list/src"
"bin/release-list/test"
"modules/drivers/athena/src"
"modules/drivers/athena/test"
"modules/drivers/bigquery-cloud-sdk/src"
"modules/drivers/bigquery-cloud-sdk/test"
"modules/drivers/druid-jdbc/src"
"modules/drivers/druid-jdbc/test"
"modules/drivers/druid/src"
"modules/drivers/druid/test"
"modules/drivers/mongo/src"
"modules/drivers/mongo/test"
"modules/drivers/oracle/src"
"modules/drivers/oracle/test"
"modules/drivers/presto-jdbc/src"
"modules/drivers/presto-jdbc/test"
"modules/drivers/redshift/src"
"modules/drivers/redshift/test"
"modules/drivers/snowflake/src"
"modules/drivers/snowflake/test"
"modules/drivers/sparksql/src"
"modules/drivers/sparksql/test"
"modules/drivers/sqlite/src"
"modules/drivers/sqlite/test"
"modules/drivers/sqlserver/src"
"modules/drivers/sqlserver/test"
"modules/drivers/vertica/src"
"modules/drivers/vertica/test"]}
;; clojure -M:ee:drivers:check
;;
;; checks that all the namespaces we actually ship can be compiled, without any dependencies that we don't
;; ship (such as `:dev` dependencies). See #27009 for more context.
:check
{:extra-deps {athos/clj-check {:git/url "https://github.com/athos/clj-check.git"
:sha "518d5a1cbfcd7c952f548e6dbfcb9a4a5faf9062"}}
:main-opts ["-m" "clj-check.check"
"src"
"enterprise/backend/src"
"modules/drivers/athena/src"
"modules/drivers/bigquery-cloud-sdk/src"
"modules/drivers/databricks/src"
"modules/drivers/druid/src"
"modules/drivers/druid-jdbc/src"
"modules/drivers/mongo/src"
"modules/drivers/oracle/src"
"modules/drivers/presto-jdbc/src"
"modules/drivers/redshift/src"
"modules/drivers/snowflake/src" "modules/drivers/sparksql/src"
"modules/drivers/sqlite/src"
"modules/drivers/sqlserver/src"
"modules/drivers/vertica/src"]
:jvm-opts ["-Dclojure.main.report=stderr"]}
;; clojure -X:dev:ee:ee-dev:drivers:drivers-dev:test:eastwood
:eastwood
{:exec-fn metabase.linters.eastwood/eastwood
:exec-args {;; manually specify the source paths for the time being (exclude test paths) until we fix Eastwood
;; errors in the test paths (once PR #17193 is merged)
:source-paths ["src"
"enterprise/backend/src"
"modules/drivers/athena/src"
"modules/drivers/bigquery-cloud-sdk/src"
"modules/drivers/databricks/src"
"modules/drivers/druid/src"
"modules/drivers/druid-jdbc/src"
"modules/drivers/mongo/src"
"modules/drivers/oracle/src"
"modules/drivers/presto-jdbc/src"
"modules/drivers/redshift/src"
"modules/drivers/snowflake/src"
"modules/drivers/sparksql/src"
"modules/drivers/sqlite/src"
"modules/drivers/sqlserver/src"
"modules/drivers/vertica/src"]
:exclude-linters [;; Turn this off temporarily until we finish removing
;; self-deprecated functions & macros
:deprecations
;; this has a fit in libs that use Potemkin `import-vars` such
;; as `java-time`
:implicit-dependencies
;; too many false positives for now
:unused-ret-vals
;; Kondo lints this for us anyway, and this isn't as easy to configure.
:wrong-arity
;; Kondo lints this for us anyway, and this isn't as easy to configure.
:suspicious-expression]
;; Snowplow has a dynamic dependency on okhttp3.CookieJar that we
;; don't use but eastwood detects. This is discussed in Slack here
;; https://clojurians.slack.com/archives/C03S1KBA2/p1667925853699669
;; An issue has been filed here https://github.com/jonase/eastwood/issues/444
;; and a simple reproduction repo can be found here https://github.com/metabase/snowplow-eastwood-issue
:exclude-namespaces [metabase.analytics.snowplow]}}
;; clojure -T:whitespace-linter
:whitespace-linter
{:deps {com.github.camsaul/whitespace-linter {:sha "e35bc252ccf5cc74f7d543ef95ad8a3e5131f25b"}}
:ns-default whitespace-linter
:exec-fn whitespace-linter/lint
:exec-args {:paths ["./.dir-locals.el"
"./deps.edn"
"./package.json"
"./shadow-cljs.edn"
".clj-kondo"
".github"
"bin"
"enterprise"
"frontend"
"resources"
"src"
"test"]
:include-patterns ["\\.clj.?$"
"\\.edn$"
"\\.el$"
"\\.html$"
"\\.json$"
"\\.jsx?$"
"\\.sh$" "\\.yaml$"
"\\.yml$"]
:exclude-patterns [".clj-kondo/better-cond/.*"
".clj-kondo/com.github.seancorfield/.*"
"resources/i18n/.*\\.edn$"
"resources/frontend_client"
"resources/frontend_shared"
"resources/html-entities.edn"
"resources/openapi/.*"
"frontend/src/cljs"
"frontend/test/metabase/lib/urls\\.unit\\.spec\\.js$"
"frontend/test/metabase/lib/formatting\\.unit\\.spec\\.js$"
"src/metabase/shared/util/currency\\.cljc$"
"test/metabase/pulse/render/png_test.clj"
"#.+#$"
"\\.transit\\.json$"]}}
;; clojure -X:dev:ee:ee-dev:test:cloverage
:cloverage
{:exec-fn metabase.cloverage-runner/run-project
:exec-args {:fail-threshold 69
:codecov? true
;; don't instrument logging forms, since they won't get executed as part of tests anyway
;; log calls expand to these
:exclude-call
[clojure.tools.logging/logf
clojure.tools.logging/logp
metabase.util.log/logf
metabase.util.log/logp]
:src-ns-path
["src" "enterprise/backend/src"]
:test-ns-path
["test" "enterprise/backend/test"]
:ns-regex
["^metabase\\..*" "^metabase-enterprise\\..*"]
;; don't instrument Postgres/MySQL driver namespaces, because we don't current run tests for them
;; as part of recording test coverage, which means they can give us false positives.
;;
;; regex literals aren't allowed in EDN. We parse them in `./test/cloverage.clj`
:ns-exclude-regex
["metabase\\.driver\\.mysql.*"
"metabase\\.driver\\.postgres.*"]}
;; different port from `:test` so you can run it at the same time as `:test`.
:jvm-opts ["-Dmb.jetty.port=3002"]}
;; to fix indentation in source files:
;;
;; clj -T:cljfmt fix
;;
;; to fix errors in specific source files:
;;
;; clj -T:cljfmt fix '{:paths ["src/metabase/query_processor/card.clj"]}'
;;
;; to check for errors without fixing them:
;;
;; clj -T:cljfmt check
:cljfmt
{:deps {#_io.github.weavejester/cljfmt
#_{:git/tag "0.12.0", :git/sha "434408f6909924f524c8027b37422d32bb49622d"}
;; using my fork of cljfmt until these PRs are merged upstream:
;; - https://github.com/weavejester/cljfmt/pull/346
;; - https://github.com/weavejester/cljfmt/pull/347
;; - https://github.com/weavejester/cljfmt/pull/348
;; - https://github.com/weavejester/cljfmt/pull/350
io.github.camsaul/cljfmt
{:git/sha "77ede1e49c8718c9e1c9ac35cb6f34d0b02a65f3"}} :ns-default cljfmt.tool
:exec-fn cljfmt.tool/fix}
;;; building Uberjar, build and release scripts
:build
{:extra-paths
["bin/build/resources"
"bin/build/src"]
:extra-deps
{com.bhauman/spell-spec {:mvn/version "0.1.2"} ; used to find misspellings in YAML files
com.github.seancorfield/depstar {:mvn/version "2.1.303"}
expound/expound {:mvn/version "0.9.0"} ; better output of spec validation errors
io.github.borkdude/grasp {:mvn/version "0.1.4"}
io.github.clojure/tools.build {:mvn/version "0.10.0"}
org.clojure/data.xml {:mvn/version "0.2.0-alpha8"}
org.clojure/tools.deps.alpha {:mvn/version "0.15.1254"}
org.fedorahosted.tennera/jgettext {:mvn/version "0.15.1"}}
:jvm-opts
["-Dclojure.main.report=stderr"
"-XX:-OmitStackTraceInFastThrow"
"-XX:+IgnoreUnrecognizedVMOptions"
"-XX:+ShowCodeDetailsInExceptionMessages"]}
;; Build everything:
;;
;; clojure -X:drivers:build:build/all
;; clojure -X:drivers:build:build/all :edition :ee
;;
;; Run just a specific build step:
;;
;; clojure -X:drivers:build:build/all :steps '[:version]'
;;
;; the various steps available are:
;;
;; :version :translations :frontend :licenses :drivers :uberjar
:build/all
{:exec-fn build/build-cli}
;; build just the uberjar (without i18n, drivers, etc.)
;;
;; clojure -X:build:build/uberjar
;; clojure -X:build:build/uberjar :edition :ee
:build/uberjar
{:exec-fn build.uberjar/uberjar}
;; List dependencies without a license. Not really 100% sure why this needs `:dev`
;;
;; clojure -X:build:build/list-without-license
:build/list-without-license
{:exec-fn build/list-without-license}
;; Build a single driver.
;;
;; clojure -X:build:drivers:build/driver :driver :sqlserver :edition :oss
:build/driver
{:exec-fn build-driver/build-driver}
;; Build all of the drivers.
;;
;; clojure -X:build:drivers:build/drivers
;; clojure -X:build:drivers:build/drivers :edition :ee
:build/drivers
{:exec-fn build-drivers/build-drivers}
;; Verify that a driver JAR looks correct.
;;
;; clojure -X:build:build/verify-driver :driver :mongo :build/verify-driver
{:exec-fn verify-driver/verify-driver}
;; Build i18n artifacts.
;;
;; clojure -X:build:build/i18n
:build/i18n
{:exec-fn i18n.create-artifacts/create-all-artifacts!}
;; extra paths and deps for working on build scripts, or running the tests.
:build-dev
{:extra-paths ["bin/build/test"]
:extra-deps
{org.clojure/data.json {:mvn/version "2.5.0"}}}
;; Run tests for the build scripts:
;;
;; clj -X:dev:drivers:build:build-dev:build-test
:build-test
{:exec-fn mb.hawk.core/find-and-run-tests-cli
:exec-args {:only ["bin/build/test"]}}
;;; Other misc convenience aliases
;; Profile Metabase start time with clojure -M:profile
:profile
{:main-opts ["-m" "metabase.core" "profile"]
:jvm-opts ["-XX:+CITime" ; print time spent in JIT compiler
"-Xlog:gc"]}
;; get the H2 shell with clojure -X:dev:h2
:h2
{:extra-paths ["dev/src"]
:exec-fn dev.h2-shell/shell
:java-opts ["-Dfile.encoding=UTF-8"]}
:generate-automagic-dashboards-pot
{:main-opts ["-m" "metabase.xrays.automagic-dashboards.dashboard-templates"]}
;; Start a Network REPL (nrepl) that you can connect your editor to.
;;
;; clojure -M:dev:nrepl (etc.)
:nrepl
{:extra-deps {nrepl/nrepl {:mvn/version "1.1.1"}}
:main-opts ["-m" "nrepl.cmdline" "-p" "50605"]}
;; - start a Socket REPL on port 50505 that you can connect your editor to:
:socket {:jvm-opts ["-Dclojure.server.repl={:address,\"0.0.0.0\",:port,50505,:accept,clojure.core.server/repl}"]}
;; Liquibase CLI:
;;
;; clojure -M:liquibase <command>
;;
;; e.g.
;;
;; clojure -M:liquibase dbDoc target/liquibase
:liquibase
{:extra-deps {ch.qos.logback/logback-classic {:mvn/version "1.5.3"}}
:extra-paths ["dev/src"]
:main-opts ["-m" "dev.liquibase"]}
;; Migrate CLI:
;; clojure -M:migrate <command>
;; E.g.
;; clojure -M:migrate up ;; migrate up to the latest
;; clojure -M:migrate rollback count 2 ;; rollback 2 migrations
;; clojure -M:migrate rollback id "v40.00.001" ;; rollback to a specific migration with id
;; clojure -M:migrate status ;; print the latest migration id
:migrate {:extra-deps {io.github.camsaul/humane-are {:mvn/version "1.0.2"}}
:extra-paths ["dev/src"]
:main-opts ["-m" "dev.migrate"]}
;; run tests against MLv2. Very fast since this is almost all ^:parallel
:test/mlv2
{:exec-args {:only ["test/metabase/lib"]}}
;; test the Query Processor
;;
;; clj -X:dev:ee:ee-dev:test:test/qp
:test/qp
{:exec-args {:only ["test/metabase/driver"
"test/metabase/query_processor"
"test/metabase/query_processor_test"
metabase-enterprise.advanced-permissions.models.permissions.block-permissions-test
metabase-enterprise.sandbox.query-processor.middleware.row-level-restrictions-test]}}
;; test all MBQL related stuff: MLv2, the legacy shared `metabase.legacy-mbql` code, and the QP
;;
;; clj -X:dev:ee:ee-dev:test:test/mbql
:test/mbql
{:exec-args {:only ["test/mbql"
"test/metabase/lib"
"test/metabase/legacy_mbql"
"test/metabase/driver"
"test/metabase/query_processor"
"test/metabase/query_processor_test"
metabase-enterprise.advanced-permissions.models.permissions.block-permissions-test
metabase-enterprise.sandbox.query-processor.middleware.row-level-restrictions-test]}}
;; test search-related stuff.
;;
;; clj -X:dev:ee:ee-dev:test:test/search
:test/search
{:exec-args {:only [metabase.api.search-test
"test/metabase/search"]}}
;; test custom clj-kondo linters and hooks.
;;
;; clj -X:dev:test:test/kondo
:test/kondo
{:exec-args {:only [".clj-kondo/test"]}}
;; watch and reload clojure namespaces
:watch {:extra-deps
{com.nextjournal/beholder {:mvn/version "1.0.2"} ;; watcher
io.github.tonsky/clj-reload {:mvn/version "0.7.0"}} ;; reloader
:extra-paths "dev/src/watch"
:main-opts ["-m" "watch.watcher"]}}}
;; TODO -- consider creating an alias that includes the `./bin` build-drivers & release code as well so we can work
;; on them all from a single REPL process.