How to Install and configure Apache Storm

Learn | Teach Open Source Technologies

How to Install and configure Apache Storm

STORM INSTALLATION SETUP

Before setting up Apache Storm, Zookeeper server must be setup in the cluster, which takes the main responsibility of running Storm cluster. Zookeeper is not used for message passing, so the load Storm places on Zookeeper is quite low. Single node Zookeeper clusters should be sufficient for most cases, but if you want failover or are deploying large Storm clusters you may want larger Zookeeper clusters. Installing and Setting up Zookeeper:

# cd /usr/local
 # wget http://mirror.nexcess.net/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz
 # tar -xvf zookeeper-3.4.6.tar.gz
 # mv zookeeper-3.4.6 zookeeper

Setting up the path in .bashrc file:

#vi /root/.bashrc
add following quotes:
 export ZOOKEEPER_PREFIX=/usr/local/zookeeper
 export PATH=$PATH:$ZOOKEEPER_PREFIX/bin

Now we would need to create a configuration file for Zookeeper under its ‘conf’ directory:

# cd /usr/local/zookeeper/conf
 # vi zoo.cfg
Here you need to add the following mentioned parameters:
tickTime=2000
 initLimit=10
 syncLimit=5
 dataDir=/var/zookeeper
 clientPort=2181

Now save the file and start Zookeeper Server as follows:

# cd /usr/local/zookeeper
 # bin/zkServer.sh start
It will give following output after starting the service:
# bin/zkServer.sh start
 JMX enabled by default
 Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
 Starting zookeeper ... STARTED

You must verify Zookeeper service using ‘jps’ command in your server:

[root@graylog2 ~]# jps
 2129 Jps
 991 NettyServer
 1935 nimbus
 1092 ElasticSearch
 1982 core
 1529 QuorumPeerMain

Now time to setup Storm: Downloading and setting up storm:

# cd /usr/local
 # wget https://archive.apache.org/dist/incubator/storm/apache-storm-0.9.2-incubating/apache-storm-0.9.2-incubating.tar.gz
 # tar -xvf apache-storm-0.9.2-incubating.tar.gz
 # mv apache-storm-0.9.2-incubating storm

Now we need to edit configuration file of Storm and edit as follows:

# cd /usr/local/storm/conf/
 # im storm.yaml
After opening file we need to write following quotes in it:
#Zookeeper configuration file
 yaml storm.zookeeper.servers:
 - "localhost"
 yaml storm.local.dir: "/var/storm"
 yaml nimbus.host: "graylog2"
 yaml supervisor.slots.ports:
 - 6700 - 6701 - 6702 - 6703

After this, save the file and start Storm services which include 1. Nimbus (main server node), 2. Supervisors (worker nodes) and 3. Storm UI (GUI for Storm)

# cd /usr/local/storm/
 [root@graylog2 storm]# bin/storm nimbus
where you’ll see following output:
Running: java -server -Dstorm.options= -Dstorm.home=/usr/local/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm/lib/commons-fileupload-1.2.1.jar:/usr/local/storm/lib/commons-logging-1.1.3.jar:/usr/local/storm/lib/asm-4.0.jar:/usr/local/storm/lib/json-simple-1.1.jar:/usr/local/storm/lib/httpclient-4.3.3.jar:/usr/local/storm/lib/netty-3.2.2.Final.jar:/usr/local/storm/lib/slf4j-api-1.6.5.jar:/usr/local/storm/lib/servlet-api-2.5-20081211.jar:/usr/local/storm/lib/tools.macro-0.1.0.jar:/usr/local/storm/lib/logback-core-1.0.6.jar:/usr/local/storm/lib/core.incubator-0.1.0.jar:/usr/local/storm/lib/jgrapht-core-0.9.0.jar:/usr/local/storm/lib/curator-client-2.4.0.jar:/usr/local/storm/lib/chill-java-0.3.5.jar:/usr/local/storm/lib/ring-core-1.1.5.jar:/usr/local/storm/lib/hiccup-0.3.6.jar:/usr/local/storm/lib/jline-2.11.jar:/usr/local/storm/lib/clout-1.0.1.jar:/usr/local/storm/lib/objenesis-1.2.jar:/usr/local/storm/lib/commons-codec-1.6.jar:/usr/local/storm/lib/storm-core-0.9.2-incubating.jar:/usr/local/storm/lib/carbonite-1.4.0.jar:/usr/local/storm/lib/clj-stacktrace-0.2.4.jar:/usr/local/storm/lib/joda-time-2.0.jar:/usr/local/storm/lib/curator-framework-2.4.0.jar:/usr/local/storm/lib/tools.logging-0.2.3.jar:/usr/local/storm/lib/tools.cli-0.2.4.jar:/usr/local/storm/lib/compojure-1.1.3.jar:/usr/local/storm/lib/clojure-1.5.1.jar:/usr/local/storm/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm/lib/disruptor-2.10.1.jar:/usr/local/storm/lib/reflectasm-1.07-shaded.jar:/usr/local/storm/lib/clj-time-0.4.1.jar:/usr/local/storm/lib/httpcore-4.3.2.jar:/usr/local/storm/lib/zookeeper-3.4.5.jar:/usr/local/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm/lib/jetty-util-6.1.26.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/commons-io-2.4.jar:/usr/local/storm/lib/jetty-6.1.26.jar:/usr/local/storm/lib/commons-exec-1.1.jar:/usr/local/storm/lib/commons-lang-2.5.jar:/usr/local/storm/lib/snakeyaml-1.11.jar:/usr/local/storm/lib/guava-13.0.jar:/usr/local/storm/lib/ring-servlet-0.3.11.jar:/usr/local/storm/lib/minlog-1.2.jar:/usr/local/storm/lib/logback-classic-1.0.6.jar:/usr/local/storm/lib/ring-devel-0.3.11.jar:/usr/local/storm/lib/netty-3.6.3.Final.jar:/usr/local/storm/lib/kryo-2.21.jar:/usr/local/storm/conf -Xmx1024m -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/usr/local/storm/logback/cluster.xml backtype.storm.daemon.nimbus

Now Start Supervisor:

[root@graylog2]# cd /usr/local/storm/
 [root@graylog2 storm]# bin/storm supervisor
Running: java -server -Dstorm.options= -Dstorm.home=/usr/local/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm/lib/commons-fileupload-1.2.1.jar:/usr/local/storm/lib/commons-logging-1.1.3.jar:/usr/local/storm/lib/asm-4.0.jar:/usr/local/storm/lib/json-simple-1.1.jar:/usr/local/storm/lib/httpclient-4.3.3.jar:/usr/local/storm/lib/netty-3.2.2.Final.jar:/usr/local/storm/lib/slf4j-api-1.6.5.jar:/usr/local/storm/lib/servlet-api-2.5-20081211.jar:/usr/local/storm/lib/tools.macro-0.1.0.jar:/usr/local/storm/lib/logback-core-1.0.6.jar:/usr/local/storm/lib/core.incubator-0.1.0.jar:/usr/local/storm/lib/jgrapht-core-0.9.0.jar:/usr/local/storm/lib/curator-client-2.4.0.jar:/usr/local/storm/lib/chill-java-0.3.5.jar:/usr/local/storm/lib/ring-core-1.1.5.jar:/usr/local/storm/lib/hiccup-0.3.6.jar:/usr/local/storm/lib/jline-2.11.jar:/usr/local/storm/lib/clout-1.0.1.jar:/usr/local/storm/lib/objenesis-1.2.jar:/usr/local/storm/lib/commons-codec-1.6.jar:/usr/local/storm/lib/storm-core-0.9.2-incubating.jar:/usr/local/storm/lib/carbonite-1.4.0.jar:/usr/local/storm/lib/clj-stacktrace-0.2.4.jar:/usr/local/storm/lib/joda-time-2.0.jar:/usr/local/storm/lib/curator-framework-2.4.0.jar:/usr/local/storm/lib/tools.logging-0.2.3.jar:/usr/local/storm/lib/tools.cli-0.2.4.jar:/usr/local/storm/lib/compojure-1.1.3.jar:/usr/local/storm/lib/clojure-1.5.1.jar:/usr/local/storm/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm/lib/disruptor-2.10.1.jar:/usr/local/storm/lib/reflectasm-1.07-shaded.jar:/usr/local/storm/lib/clj-time-0.4.1.jar:/usr/local/storm/lib/httpcore-4.3.2.jar:/usr/local/storm/lib/zookeeper-3.4.5.jar:/usr/local/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm/lib/jetty-util-6.1.26.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/commons-io-2.4.jar:/usr/local/storm/lib/jetty-6.1.26.jar:/usr/local/storm/lib/commons-exec-1.1.jar:/usr/local/storm/lib/commons-lang-2.5.jar:/usr/local/storm/lib/snakeyaml-1.11.jar:/usr/local/storm/lib/guava-13.0.jar:/usr/local/storm/lib/ring-servlet-0.3.11.jar:/usr/local/storm/lib/minlog-1.2.jar:/usr/local/storm/lib/logback-classic-1.0.6.jar:/usr/local/storm/lib/ring-devel-0.3.11.jar:/usr/local/storm/lib/netty-3.6.3.Final.jar:/usr/local/storm/lib/kryo-2.21.jar:/usr/local/storm/conf -Xmx256m -Dlogfile.name=supervisor.log -Dlogback.configurationFile=/usr/local/storm/logback/cluster.xml backtype.storm.daemon.supervisor

Now Start UI of Apache Storm:

[root@graylog2 ~]# cd /usr/local/storm/
 [root@graylog2 storm]# bin/storm ui
 Running: java -server -Dstorm.options= -Dstorm.home=/usr/local/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm/lib/commons-fileupload-1.2.1.jar:/usr/local/storm/lib/commons-logging-1.1.3.jar:/usr/local/storm/lib/asm-4.0.jar:/usr/local/storm/lib/json-simple-1.1.jar:/usr/local/storm/lib/httpclient-4.3.3.jar:/usr/local/storm/lib/netty-3.2.2.Final.jar:/usr/local/storm/lib/slf4j-api-1.6.5.jar:/usr/local/storm/lib/servlet-api-2.5-20081211.jar:/usr/local/storm/lib/tools.macro-0.1.0.jar:/usr/local/storm/lib/logback-core-1.0.6.jar:/usr/local/storm/lib/core.incubator-0.1.0.jar:/usr/local/storm/lib/jgrapht-core-0.9.0.jar:/usr/local/storm/lib/curator-client-2.4.0.jar:/usr/local/storm/lib/chill-java-0.3.5.jar:/usr/local/storm/lib/ring-core-1.1.5.jar:/usr/local/storm/lib/hiccup-0.3.6.jar:/usr/local/storm/lib/jline-2.11.jar:/usr/local/storm/lib/clout-1.0.1.jar:/usr/local/storm/lib/objenesis-1.2.jar:/usr/local/storm/lib/commons-codec-1.6.jar:/usr/local/storm/lib/storm-core-0.9.2-incubating.jar:/usr/local/storm/lib/carbonite-1.4.0.jar:/usr/local/storm/lib/clj-stacktrace-0.2.4.jar:/usr/local/storm/lib/joda-time-2.0.jar:/usr/local/storm/lib/curator-framework-2.4.0.jar:/usr/local/storm/lib/tools.logging-0.2.3.jar:/usr/local/storm/lib/tools.cli-0.2.4.jar:/usr/local/storm/lib/compojure-1.1.3.jar:/usr/local/storm/lib/clojure-1.5.1.jar:/usr/local/storm/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm/lib/disruptor-2.10.1.jar:/usr/local/storm/lib/reflectasm-1.07-shaded.jar:/usr/local/storm/lib/clj-time-0.4.1.jar:/usr/local/storm/lib/httpcore-4.3.2.jar:/usr/local/storm/lib/zookeeper-3.4.5.jar:/usr/local/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm/lib/jetty-util-6.1.26.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/commons-io-2.4.jar:/usr/local/storm/lib/jetty-6.1.26.jar:/usr/local/storm/lib/commons-exec-1.1.jar:/usr/local/storm/lib/commons-lang-2.5.jar:/usr/local/storm/lib/snakeyaml-1.11.jar:/usr/local/storm/lib/guava-13.0.jar:/usr/local/storm/lib/ring-servlet-0.3.11.jar:/usr/local/storm/lib/minlog-1.2.jar:/usr/local/storm/lib/logback-classic-1.0.6.jar:/usr/local/storm/lib/ring-devel-0.3.11.jar:/usr/local/storm/lib/netty-3.6.3.Final.jar:/usr/local/storm/lib/kryo-2.21.jar:/usr/local/storm:/usr/local/storm/conf -Xmx768m -Dlogfile.name=ui.log -Dlogback.configurationFile=/usr/local/storm/logback/cluster.xml backtype.storm.ui.core
Go to web browser
 http://192.168.48.11:8080/index.html

stormui