Deploying a Samza job from HDFS

This tutorial uses hello-samza to illustrate how to run a Samza job if you want to publish the Samza job’s .tar.gz package to HDFS.

Build a new Samza job package

Build a new Samza job package to include the hadoop-hdfs-version.jar.

  • Add dependency statement in pom.xml of samza-job-package
  • Add the following code to src/main/assembly/src.xml in samza-job-package.
  • Create .tar.gz package
mvn clean pacakge
  • Make sure hadoop-common-version.jar has the same version as your hadoop-hdfs-version.jar. Otherwise, you may still have errors.

Upload the package

hadoop fs -put ./samza-job-package/target/samza-job-package-0.7.0-dist.tar.gz /path/for/tgz

Add HDFS configuration

Put the hdfs-site.xml file of your cluster into ~/.samza/conf directory. (The same place as the yarn-site.xml)

Change properties file

Change the yarn.package.path in the properties file to your HDFS location.

yarn.package.path=hdfs://<hdfs name node ip>:<hdfs name node port>/path/to/tgz

Then you should be able to run the Samza job as described in hello-samza.