Make sure built drivers do not contain clojure core classes. Handle either string or keyword args to build scripts (#17608)

* No Clojure core classes please

* Don't try to make paths absolute twice. Build scripts should handle keyword args like ':driver'

* parse-as-keyword should return keywords as is

* Test fix 

* Test fix 

* Test fix part 2

* Test init code: read driver parents from driver deps.edn

* Rename :metabase.build-driver/parents -> :metabase.driver/parents

* Determine MB project root directory relative to source directory of build scripts

* Support calling build-driver! as a -X fn, and support dirs outside of modules/drivers

* Support loading 3rd-party driver manifest files at (dev) launch

* Fix log4j not working on latest version of CIDER (unrelated)

* Fix dissoc

* project-root-directory is supposed to be a string

New BigQuery Driver

Create new :bigquery-cloud-sdk driver using the google-cloud-bigquery library instead, and whose source is adapted from the :bigquery driver

https://cloud.google.com/bigquery/docs/reference/libraries

Marking existing :bigquery driver as deprecated, and superseded-by the new one (bigquery-cloud-sdk)

Update new driver and query processor code to use newer Google SDK

Switch test data loading over to use new API

Add project-id connection property to override the value from the service account JSON, and use it as part of qualified names in the query processor if set

Updating google driver so its libraries are compatible with the newer ones used in BigQuery

Update date bucketing tests to skip :bigquery-cloud-sdk (new driver) where :bigquery is skipped

Update `with-bigquery-fks` to take in the driver, since there are now multiple ones

Adding test to confirm that overriding project-id for a public BQ project works (sync and query)

Fixing a bunch of whitespace alignment errors in tests

* Backend SVG rendering proof of concept [ci skip]

* Update cssbox to 5.0.0

* Render bar, line, and pie charts in js to svg

sparkline is now done in js, bar is now recognized and done in js, new
:categorical/donut as well

* Remove api route for render

* pass along render-type, not hardcoded to :inline

* Move bar chart above sparkline and remove line check

In order to introduce the bar chart type need it above the sparkline
check since it is otherwise the same except for display property of
the card. But lots of tests assume that this will get hit with a nil
display type set in testing so remove checking for `:line` allows all
the testing cases to hit the right type

* Fix tests now that bar graphs aren't html but images

* Include attachments for bar charts

* Move over to in-tree bundle

* Force everything [ci noskip]

trying to ensure that the built jar includes the newer
"resources/frontend_client/app/dist/lib-static-viz.bundle.js"

* Run `yarn build-static-viz` in backend-deps in CI

this js file is now a hard dependency of the backend so it fits in
this tsk. All such things that depend on the backend sources will need
it. Makes me think perhaps we want a checked in version but i'm not
sure yet.

* Look on classpath not filesystem for js bundle [ci noskip]

* Move yarn build-static-viz into the checkout step

* License information for antlr4-runtime

* create attachment for categorical donuts

* add ordinal legend to donuts (#17177)

* set widths of html image and svg image to 1200

* Revert "add ordinal legend to donuts (#17177)"

This reverts commit 1eb81d2e.

* Helper functions to render html easily

* readme in dev

* readme ensure that static viz bundle exists

* Cleanup ns after removing proxy

* Donut chart colors and legend (#17251)

* use external color map for fill per dimension

* Add support new color legend for donut

* Ensure text doesn't appear as link

entire thing is actually the body of a link tag for emails but we want
a decent text color rather than a default link color

* use chart colors from https://stats.metabase.com/_internal/colors



* Make checkers happy

- remove unused imports
- add a docstring
- don't shadow fn with a local

* cleanup ns import

* Remove reflective call

* Cleanup ns on correct branch

Co-authored-by: dan sutton <dan@dpsutton.com>

* X-axis: just use (approx) 5 ticks to avoid overlapping labels (#17287)

* increase gap between arcs (#17271)

* Set rendering hints on html->image

* ignore width for now and make them larger

* Ns deprecation and some cleanup

* make namespace checker happy

* Simple tests for detecting chart type

* Rename from poc

* Tests for scalar/smartscalar

* cleanup js svg namespace a bit

* Tests of svg engine

* ns sorting after renaming

* Unify our two different js engine usages

settled on the js context. Has typed returns `(.asString ^Value ...)`
instead of perhaps capturing std out?
https://www.graalvm.org/sdk/javadoc/org/graalvm/polyglot/Value.html



Context is a bit more friendly for getting source into it. One
downside is that the invocable bit isn't quite as nice. The old way
would return a java.util.functionFunction but the difference is

(.apply function (object-array args))

vs

(.execute fn-ref (object-array args))

* Don't io/resource the io/resource

* js engine tests

* Ns cleanup in js-svg

type hints in the js-engine ns mean we don't need as many classes from
polyglot here

* Cleanup of text, ns docstrings, alignment

* Fix fill->fill-opacity with parsed doc, not regex

* Make a single helper that loads a static viz bundle context

* Docstrings and make private in js-svg

* Sort imported classes in js-svg

* Make width passed down through rendering aparatus

- svgs are always rendered at 1200 for quality
- slack images of html are rendered at 1200 so that they can be zoomed
  in in the ui but slack automatically scales down
- email sends html and the full svg but includes img width tags so
  that is handled appropriately

* docstring

Co-authored-by: dan sutton <dan@dpsutton.com>
Co-authored-by: Kyle Doherty <5248953+kdoh@users.noreply.github.com>
Co-authored-by: Ariya Hidayat <ariya@metabase.com>

* Add the whitespace linter

* Fix whitespace linter errors [except for one file]

* Add a line that will intentionally break stuff to verify the linter is working.

* Ok, remove the line that caused the linter to fail.

* Use latest version of the linter

* Fix missing newline

* Bump some more backend dependencies

* Add license info for com.sun.activation/jakarta.activation

* Fix reflection warnings

* Revert bump of MariaDB for now

* Remove `project.clj` from Percy workflows

* Replace `project.clj` with `deps.edn` in `build-for-test` script

* Update reference to the `project.clj` in the comment

* Intermediate commit

* Add the step to include .platform dir into the bundle

Only build currently active drivers

Update `all-drivers` fn so that it only "finds" drivers that have either a `project.clj` or `deps.edn` file (i.e. some build file), and ignores the directory otherwise. This is to deal with cached artifacts hanging around between builds in a shared executor type environment (ex: CircleCI), where even though the checkout won't bring in some driver directory that doesn't exist on the target branch, some other files (such as `pom.xml` or a `target` directory), might still be hanging around

* Update babel dependencies to latest

* Remove some old babel dependencies

The dependencies in this commit include dependencies that have been
renamed, such as babel-cli which is now @babel/cli. It also includes
dependencies that have been replaced, such as babel-preset-stage-0 and
its ilk, which have been replaced with @babel/preset-env.

* Remove babel-register dependency

This is referenced in our webpack.config.js file. I don't think we need
Babel in our Node envionment, but I could be wrong, so leaving this as a
separate commit.

* Remove @babel/standalone and dependent code

We're using @babel/standalone to support the writing of JSX in our
internal-only ScratchApp editor. Unfortunately, this dependency is
large--1.5mb not gzipped--and our current build does not intelligently
split bundles, meaning our users our burdened by the size of this
dependency whenever they must redownload the Metabase JS bundle.

I'm removing it. We should still be able to write JS in the ScratchApp,
but JSX no longer.

* Add @babel/node to dev deps

Needed by a cypress command

* Add browserslist config

Babel's preset-env dep needs this config to determine how to transform
our JavaScript. By default I believe it ends up defaulting to
"defaults."

"defaults" is equivalent to the following rules:
> 0.5%, last 2 versions, Firefox ESR, not dead

A comma between rules represents a UNION operator.

Per
https://www.metabase.com/docs/latest/faq/general/supported-browsers.html


we have deprecated IE 11 so I am including that explicitly in our
browserslist.

* Update babelrc plugins and presets

- Replaced presets with preset-env and preset-react
- Updated the names of some plugins, like transform-decorators-legacy,
  which is now @babel/plugin-proposal-decorators with an explicit
  "legacy: true" passed in the config.
- Removed add-react-displayname because it is included in preset-react
- Removed transform-builtin-extend because it is included in preset-env*
- Removed syntax-trailing-function-commas because it is supported by
  all browsers we support

* note: There is a "loose" config option for handling the extension of builtin
classes. We may need it, but I am not sure.

* Update babel cli args

Replace -q with --quiet because -q no longer exists

* Fix circular dep in auth.js by making it async

Our app unfortunately contains MANY circular dependencies. Some, like
this one, caused problems when the app instantiates because the
"refreshCurrentUser" method is undefined when auth.js is initially run.

While we should in the future avoid having circular deps in the first
place, making them run async by requiring the dep and function run-time
fixes this particular issue.

* Fix circular dependency in metabase/redux/requests.js

The handleActions and createAction methods imported from
"metabase/lib/redux" are actually just methods from the "redux-actions"
dependency... so importing those methods directly from that dep fixes
this circular dependency problem.

* Remove references to global.services and exports in metabase/services

This line breaks the build. Git blame says the line is 5 years old, and
I wouldn't expect such code to be used with how our app now requires
deps.

* Fix MBQLArrayClause problem caused by Array extension

* Explicitly pass args from flatMap to function via anonymous cb

I think the old Array.prototype.flatMap prototype must've had different
args. They are now "currentValue, index, array" so when we passed it
"enumeratePaths" as a callback, the second arg of enumeratePaths gets
set to a number. This breaks the logic of enumeratePaths because it is a
recursive function that depends on that second arg getting initialized
as an empty array by default.

* Remove code that spreads non-iterable value in array

New Babel is stricter about handling this sort of code that doesn't
match the JS spec. You can't spread an undefined value inside of an
iterator.

* Update jest and babel-jest to latest

Updating Babel breaks our unit tests. Updating nothing causes every test
to fail due to a Babel version mismatch. Updating only babel-jest causes
the following error:

TypeError: Jest: a transform must export a `process` function

There may be a way around updating Jest, but updating it does not appear
to be too difficult.

* Set the testEnvironment to jsdom

This was the default testEnvironment in our old version of jest. At some
point the default was changed to node.

* Remove unfinished "pending" unit tests

I'm unfamiliar with this pending function, but the function no longer
exists, causing the tests to fail.

* don't pass async callback to describe

* Update name of runTimersToTime to advanceTimersByTime

* Fix incorrectly nested unit test

* mock SVGElement.prototype.getBBox method

* Don't return values in describe callback

* Move variable declaration to above usage

* Fix prettier error

* Upgrade `documentation` library to the latest version (13.x.x)

This library was the only one left that was still using Babel 6.

* Return an MBQLClause instance after using Array methods

* remove 'not IE 11' from browserslist

* Remove mock that breaks unit test

Co-authored-by: Nemanja <31325167+nemanjaglumac@users.noreply.github.com>

* Fix some Eastwood failures

* Fix a lot of Eastwood errors now that it runs against test namespaces [ci skip]

* Run Eastwood against test namespaces [WIP]

* Bump Eastwood version to fix some errors in Eastwood itself

* Fix another lint warning

* Fix test failure

* More test fixes 

* Fix all the warnings!

* Test fix 

* Bump Eastwood GH action timeout to 20 minutes (!)

* yarn add @visx/axis@1.8.0 @visx/grid@1.16.0 @visx/group@1.7.0 @visx/scale@1.7.0 @visx/shape@1.8.0

* Import static viz charting code

* Format code with yarn prettier

* Silence linting violations

* Build static-viz bundle

* Remove `image-diff` library

* Remove `compare-screenshots` script

* Easy checking for missing library licenses

* include bin/build-mb readme

on java 16 there's a handy (newFilesystem path) but i don't think we
want to require java 16 yet. On previous versions we can use
(newFileSystem Path ClassLoader) which is what it was before. Really
just need to typehint the nil, but nils don't carry
metadata (sensibly) so would have to introduce a local binding

```clojure
(let [^ClassLoader cl nil]
  (FileSystems/newFileSystem jar-path cl))
```

feels a bit weird, so just send the proper classloader through.

* wip for licenses

* Lots of changes

- bump clojure.data.xml to prevent illegal access
- had to change navigators because of this
- switch to full text of license
- algo is now:
  1. check for a license file in jar
  2. check for an override pointing to a license file locally in
     resources
  3. Check pom file for license information and just use that

can be invoked as:
```bash
% clj -X mb.licenses/process :classpath \"$(cd ../.. && lein classpath)\" :backfill "\"overrides.edn\"" :output-filename "\"backend-licenses.txt\""
License information written to  backend-licenses.txt
/Users/dan/.m2/repository/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar  : No license information found.
org.clojars.brenton:google-diff-match-patch  : No license information found.

% wc backend-licenses.txt
   33638  299118 2092948 backend-licenses.txt
```

* xml/parse without the strings

* Add tests

* remove comment form

* namespace comments

* readme

* instructions to create backend license files

* Add version information to the build script

need to either yank it out of the separate project or just use it as a
local root. not sure which

todos: move to s3 once i get access and figure out the proper bucket
       copy to a place so it ends up in the jar
       generate frontend versions as well

* Build frontend and backend license files

put them in the jar at
META-INF/leiningen/metabase-core/metabase-core/license-frontend-third-party
META-INF/leiningen/metabase-core/metabase-core/license-backend-third-party

need to be named ^license.* for lein uberjar to pick them up

* Rename and reformat

* remove old stuff

* combine with a newline

* Tests ensure all jars have an identifiable license

* Extra information if backfill is malformed or missing

* correct name for apache2_0 backfill

* print out classpaths in CI

* print

* see whole classpath. it's empty on ee for some reason

* assert they are there

* Get deps before attempting classpath

for some reason the first one fails. but subsequent re-runs work.

* loop on deps until satisfied

* Include message if cannot satisfy

* include all drivers when getting deps

* rearrange ns docstring

* Hoist functions on pom inside of `with-open` on the input streams

* Update to inversion of pom parsing

* Move to nio from java.io

Co-authored-by: Yoshiki Iida <lhclrt176@gmail.com>

* Separate steps for faster build

* Fix typo

Co-authored-by: Paul Rosenzweig <paulrosenzweig@users.noreply.github.com>

* Update .ebextensions to send memory usage to Cloudwatch and also install inspector agent to actively monitor threats. Added full Cloudwatch agent with metrics config, simplified config file. Fix #9021

* Adding a few items to documentation

* Apply suggestions from code review

Co-authored-by: Jeff Bruemmer <jeff@metabase.com>

* Add a few more changes to finish this PR

Co-authored-by: Jeff Bruemmer <jeff@metabase.com>

* Various improvements to version update checks

***************************
* Release project changes *
***************************

Updating version_info.clj code to build version-info-ee.json file for Enterprise Edition build

Adding test for both oss and ee version info file generation logic

Decoupling GitHub milestone from version and setting it to be the OSS value for :ee release build

Adding nREPL alias for release deps.edn project and usage note

************************
* Core product changes *
************************

Adding new :ee version of :mb-version-info-url (:mb-version-info-ee-url) with default value to match the file name generated by the release code

Making enterprise-edition? public from core.clj, since it's now needed from the upgrade check code to switch between the URLs

Adding new setting to store when updates were last checked (since it's on a 12-hour interval)

Changing update check code to both set this new timestamp setting (always) to the current time, and set version-info itself to nil if the update check fails (so that users won't see "latest and greatest" if the update check succeeded at some point in the past, but then started failing

Frontend changes:

Showing the "last checked" timestamp (in user's browser timezone) when we show the "Sorry" text because of being unable to check for updates, so they know when the last check was made

Removing custom implementation of MetabaseSettings.versionInfo from EE frontend code since it's no longer needed (and never worked), because the EE backend will now be hitting a different URL with the version info specifically for EE releases

Linking to https://www.metabase.com/docs/<newVersion>/operations-guide/upgrading-metabase.html URL instead of simply /start

Port the metabase.mbql utility namespaces to ./shared/ so they can be used on both the frontend and backend.

* Port code for building i18n resources to Clojure build scripts; remove dep on gettext

* Use Metabase classloader

* Test fixes 



* Bump adoptopenjdk version

* Delete gettext dependency from Dockerfile

* Update developers-guide.md

* Address PR feedback

* Fix frontend singular msgstr format

Co-authored-by: Luis Paolini <paoliniluis@gmail.com>

* Add container build for metabase-enterprise-head

* Add build argument

* Delete script to build images

* Shared CLJ/CLJS lib (PoC/WIP)

* PoC 2.0

* Fixes 

* More test fixes 

* Bump shadow-cljs version

* Fix more stuff

* Need to ^:export the exports

* CI fixes 

* Add eslintignore

* Ignore cljs files for FE code coverage

* Try prefixing CLJS -> JS import with goog:

* Revert indentation change

* No goog:

* Add .prettierignore

* Use advanced build for now for JS tests unit we can figure out how to make it work

* Update the guide for deploying Metabase in ElasticBeanstalk

* parent bb5f28dee441ca09e610a6918890920592d215a2
author Luis Paolini <paoliniluis@gmail.com> 1610502973 -0300
committer Luis Paolini <paoliniluis@gmail.com> 1614898920 -0300

* Please work CircleCI