Samza’s unit tests are written on top of JUnit, and license checking is done with Apache Rat. Samza’s integration tests are written on top of Zopkio. As of Samza 0.10, the integration tests are based on Zopkio 0.1.17.
Running Unit Tests Locally
To run all tests, and license checks:
./gradlew clean check
To run a single test:
./gradlew clean :samza-core:test -Dtest.single=TestSamzaContainer
Test results are located in:
Testing Scala and Java
Samza’s unit tests can also be run against all supported permutations of Scala and Java.
To run the tests against a specific combination:
./gradlew -PscalaSuffix=2.11 -PyarnVersion=2.4.0 clean check
To run Samza’s unit tests against all permutations, run:
When run on Linux, this command requires you to set two environment variables:
JAVA8_HOME is not set.
On Mac, check-all.sh will default to the appropriate path for each environment variable if it’s not already set:
JAVA8_HOME is not set. JAVA8_HOME defaulted to /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home
Running Integration Tests Locally
Samza uses Zopkio to deploy and execute its integration tests. Integration tests can be executed by running:
./bin/integration-tests.sh /tmp/samza-tests [yarn-integration-tests|standalone-integration-tests]
The parameter defines where the integration tests should install packages both locally and on remote systems. Executing this command will:
- Build a samza-test job tarball.
- Download and install YARN, Kafka, and ZooKeeper.
- Deploy the samza-test job tarball to all YARN NM machines.
- Start all Samza integration test jobs.
- Feed input data to the jobs, and verify the results.
- Open a report, and aggregate all remote logs.
The default configurations that ship with Samza deploy all software, and run all tests locally on the machine from which the
integration-tests.sh command was executed.
If you have difficulties (
connection refused error or
JAVA_HOME is not set and could not be found. error), take a look the
Prerequisites for Running the Integration Tests Locally section.
The integration tests use SSH to interact with remote machines (and localhost). This means that you need an authentication mechanism when connecting to the machines. The two authentication mechanisms provided are:
- Public key
Zopkio will prompt you for a password by default. This password will be used as the SSH password when trying to log into remote systems.
Zopkio supports public key authentication if you prefer to use it, or if your environment doesn’t allow interactive authentication. To use public key authentication, add your public SSH key to ~/.ssh/authorized_keys, and SSH to all of the machines that you’ll be deploying to (localhost by default). See here for details.
Once this is done, you can run Zopkio with the –nopassword parameter:
./bin/integration-tests.sh /tmp/samza-tests [yarn-integration-tests|standalone-integration-tests] --nopassword
This will skip the password prompt, and force Zopkio to try public key authentication.
The integration-tests.sh script will set the console log level to INFO by default. The level can be changed with:
./bin/integration-tests.sh /tmp/samza-tests [yarn-integration-tests|standalone-integration-tests] --console-log-level DEBUG
Changing this setting will define how verbose Zopkio is during test execution. It does not affect any of the log4j.xml settings in Samza, YARN, Kafka, or ZooKeeper.
Prerequisites for Running the Integration Tests Locally
Before running the integration tests locally make sure that:
ssh localhost 'echo $JAVA_HOME'prints the correct path to Java installation on your system
ssh localhost does not works, you need to install an SSH server (enable
Remote login on Mac or install
openssh-server on Linux for example).
ssh localhost 'echo $JAVA_HOME' does not prints the path to Java installation on your system, you need to configure your login shell to set the
JAVA_HOME environment variable regardless in non-interactive mode.
Zopkio is doing the deployment in non-interactive mode therefore no configuration files (including
/etc/profile) will be taken into account.
For example if your system uses
bash shell, you can insert
export JAVA_HOME=/path/to/java/home at the beginning of the
~/.bashrc file before the line
# If not running interactively, don't do anything.
More info on this subject can be found here.