How to Install and configure Apache Storm
STORM INSTALLATION SETUP
Before setting up Apache Storm, Zookeeper server must be setup in the cluster, which takes the main responsibility of running Storm cluster. Zookeeper is not used for message passing, so the load Storm places on Zookeeper is quite low. Single node Zookeeper clusters should be sufficient for most cases, but if you want failover or are deploying large Storm clusters you may want larger Zookeeper clusters. Installing and Setting up Zookeeper:
# cd /usr/local # wget http://mirror.nexcess.net/apache/zookeeper/stable/zookeeper-3.4.6.tar.gz # tar -xvf zookeeper-3.4.6.tar.gz # mv zookeeper-3.4.6 zookeeper
Setting up the path in .bashrc file:
#vi /root/.bashrc
add following quotes: export ZOOKEEPER_PREFIX=/usr/local/zookeeper export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
Now we would need to create a configuration file for Zookeeper under its ‘conf’ directory:
# cd /usr/local/zookeeper/conf # vi zoo.cfg
Here you need to add the following mentioned parameters:
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/var/zookeeper clientPort=2181
Now save the file and start Zookeeper Server as follows:
# cd /usr/local/zookeeper # bin/zkServer.sh start
It will give following output after starting the service:
# bin/zkServer.sh start JMX enabled by default Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
You must verify Zookeeper service using ‘jps’ command in your server:
[root@graylog2 ~]# jps 2129 Jps 991 NettyServer 1935 nimbus 1092 ElasticSearch 1982 core 1529 QuorumPeerMain
Now time to setup Storm: Downloading and setting up storm:
# cd /usr/local # wget https://archive.apache.org/dist/incubator/storm/apache-storm-0.9.2-incubating/apache-storm-0.9.2-incubating.tar.gz # tar -xvf apache-storm-0.9.2-incubating.tar.gz # mv apache-storm-0.9.2-incubating storm
Now we need to edit configuration file of Storm and edit as follows:
# cd /usr/local/storm/conf/ # im storm.yaml
After opening file we need to write following quotes in it:
#Zookeeper configuration file yaml storm.zookeeper.servers: - "localhost" yaml storm.local.dir: "/var/storm" yaml nimbus.host: "graylog2" yaml supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703
After this, save the file and start Storm services which include 1. Nimbus (main server node), 2. Supervisors (worker nodes) and 3. Storm UI (GUI for Storm)
# cd /usr/local/storm/ [root@graylog2 storm]# bin/storm nimbus
where you’ll see following output:
Running: java -server -Dstorm.options= -Dstorm.home=/usr/local/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm/lib/commons-fileupload-1.2.1.jar:/usr/local/storm/lib/commons-logging-1.1.3.jar:/usr/local/storm/lib/asm-4.0.jar:/usr/local/storm/lib/json-simple-1.1.jar:/usr/local/storm/lib/httpclient-4.3.3.jar:/usr/local/storm/lib/netty-3.2.2.Final.jar:/usr/local/storm/lib/slf4j-api-1.6.5.jar:/usr/local/storm/lib/servlet-api-2.5-20081211.jar:/usr/local/storm/lib/tools.macro-0.1.0.jar:/usr/local/storm/lib/logback-core-1.0.6.jar:/usr/local/storm/lib/core.incubator-0.1.0.jar:/usr/local/storm/lib/jgrapht-core-0.9.0.jar:/usr/local/storm/lib/curator-client-2.4.0.jar:/usr/local/storm/lib/chill-java-0.3.5.jar:/usr/local/storm/lib/ring-core-1.1.5.jar:/usr/local/storm/lib/hiccup-0.3.6.jar:/usr/local/storm/lib/jline-2.11.jar:/usr/local/storm/lib/clout-1.0.1.jar:/usr/local/storm/lib/objenesis-1.2.jar:/usr/local/storm/lib/commons-codec-1.6.jar:/usr/local/storm/lib/storm-core-0.9.2-incubating.jar:/usr/local/storm/lib/carbonite-1.4.0.jar:/usr/local/storm/lib/clj-stacktrace-0.2.4.jar:/usr/local/storm/lib/joda-time-2.0.jar:/usr/local/storm/lib/curator-framework-2.4.0.jar:/usr/local/storm/lib/tools.logging-0.2.3.jar:/usr/local/storm/lib/tools.cli-0.2.4.jar:/usr/local/storm/lib/compojure-1.1.3.jar:/usr/local/storm/lib/clojure-1.5.1.jar:/usr/local/storm/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm/lib/disruptor-2.10.1.jar:/usr/local/storm/lib/reflectasm-1.07-shaded.jar:/usr/local/storm/lib/clj-time-0.4.1.jar:/usr/local/storm/lib/httpcore-4.3.2.jar:/usr/local/storm/lib/zookeeper-3.4.5.jar:/usr/local/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm/lib/jetty-util-6.1.26.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/commons-io-2.4.jar:/usr/local/storm/lib/jetty-6.1.26.jar:/usr/local/storm/lib/commons-exec-1.1.jar:/usr/local/storm/lib/commons-lang-2.5.jar:/usr/local/storm/lib/snakeyaml-1.11.jar:/usr/local/storm/lib/guava-13.0.jar:/usr/local/storm/lib/ring-servlet-0.3.11.jar:/usr/local/storm/lib/minlog-1.2.jar:/usr/local/storm/lib/logback-classic-1.0.6.jar:/usr/local/storm/lib/ring-devel-0.3.11.jar:/usr/local/storm/lib/netty-3.6.3.Final.jar:/usr/local/storm/lib/kryo-2.21.jar:/usr/local/storm/conf -Xmx1024m -Dlogfile.name=nimbus.log -Dlogback.configurationFile=/usr/local/storm/logback/cluster.xml backtype.storm.daemon.nimbus
Now Start Supervisor:
[root@graylog2]# cd /usr/local/storm/ [root@graylog2 storm]# bin/storm supervisor
Running: java -server -Dstorm.options= -Dstorm.home=/usr/local/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm/lib/commons-fileupload-1.2.1.jar:/usr/local/storm/lib/commons-logging-1.1.3.jar:/usr/local/storm/lib/asm-4.0.jar:/usr/local/storm/lib/json-simple-1.1.jar:/usr/local/storm/lib/httpclient-4.3.3.jar:/usr/local/storm/lib/netty-3.2.2.Final.jar:/usr/local/storm/lib/slf4j-api-1.6.5.jar:/usr/local/storm/lib/servlet-api-2.5-20081211.jar:/usr/local/storm/lib/tools.macro-0.1.0.jar:/usr/local/storm/lib/logback-core-1.0.6.jar:/usr/local/storm/lib/core.incubator-0.1.0.jar:/usr/local/storm/lib/jgrapht-core-0.9.0.jar:/usr/local/storm/lib/curator-client-2.4.0.jar:/usr/local/storm/lib/chill-java-0.3.5.jar:/usr/local/storm/lib/ring-core-1.1.5.jar:/usr/local/storm/lib/hiccup-0.3.6.jar:/usr/local/storm/lib/jline-2.11.jar:/usr/local/storm/lib/clout-1.0.1.jar:/usr/local/storm/lib/objenesis-1.2.jar:/usr/local/storm/lib/commons-codec-1.6.jar:/usr/local/storm/lib/storm-core-0.9.2-incubating.jar:/usr/local/storm/lib/carbonite-1.4.0.jar:/usr/local/storm/lib/clj-stacktrace-0.2.4.jar:/usr/local/storm/lib/joda-time-2.0.jar:/usr/local/storm/lib/curator-framework-2.4.0.jar:/usr/local/storm/lib/tools.logging-0.2.3.jar:/usr/local/storm/lib/tools.cli-0.2.4.jar:/usr/local/storm/lib/compojure-1.1.3.jar:/usr/local/storm/lib/clojure-1.5.1.jar:/usr/local/storm/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm/lib/disruptor-2.10.1.jar:/usr/local/storm/lib/reflectasm-1.07-shaded.jar:/usr/local/storm/lib/clj-time-0.4.1.jar:/usr/local/storm/lib/httpcore-4.3.2.jar:/usr/local/storm/lib/zookeeper-3.4.5.jar:/usr/local/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm/lib/jetty-util-6.1.26.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/commons-io-2.4.jar:/usr/local/storm/lib/jetty-6.1.26.jar:/usr/local/storm/lib/commons-exec-1.1.jar:/usr/local/storm/lib/commons-lang-2.5.jar:/usr/local/storm/lib/snakeyaml-1.11.jar:/usr/local/storm/lib/guava-13.0.jar:/usr/local/storm/lib/ring-servlet-0.3.11.jar:/usr/local/storm/lib/minlog-1.2.jar:/usr/local/storm/lib/logback-classic-1.0.6.jar:/usr/local/storm/lib/ring-devel-0.3.11.jar:/usr/local/storm/lib/netty-3.6.3.Final.jar:/usr/local/storm/lib/kryo-2.21.jar:/usr/local/storm/conf -Xmx256m -Dlogfile.name=supervisor.log -Dlogback.configurationFile=/usr/local/storm/logback/cluster.xml backtype.storm.daemon.supervisor
Now Start UI of Apache Storm:
[root@graylog2 ~]# cd /usr/local/storm/ [root@graylog2 storm]# bin/storm ui Running: java -server -Dstorm.options= -Dstorm.home=/usr/local/storm -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/storm/lib/commons-fileupload-1.2.1.jar:/usr/local/storm/lib/commons-logging-1.1.3.jar:/usr/local/storm/lib/asm-4.0.jar:/usr/local/storm/lib/json-simple-1.1.jar:/usr/local/storm/lib/httpclient-4.3.3.jar:/usr/local/storm/lib/netty-3.2.2.Final.jar:/usr/local/storm/lib/slf4j-api-1.6.5.jar:/usr/local/storm/lib/servlet-api-2.5-20081211.jar:/usr/local/storm/lib/tools.macro-0.1.0.jar:/usr/local/storm/lib/logback-core-1.0.6.jar:/usr/local/storm/lib/core.incubator-0.1.0.jar:/usr/local/storm/lib/jgrapht-core-0.9.0.jar:/usr/local/storm/lib/curator-client-2.4.0.jar:/usr/local/storm/lib/chill-java-0.3.5.jar:/usr/local/storm/lib/ring-core-1.1.5.jar:/usr/local/storm/lib/hiccup-0.3.6.jar:/usr/local/storm/lib/jline-2.11.jar:/usr/local/storm/lib/clout-1.0.1.jar:/usr/local/storm/lib/objenesis-1.2.jar:/usr/local/storm/lib/commons-codec-1.6.jar:/usr/local/storm/lib/storm-core-0.9.2-incubating.jar:/usr/local/storm/lib/carbonite-1.4.0.jar:/usr/local/storm/lib/clj-stacktrace-0.2.4.jar:/usr/local/storm/lib/joda-time-2.0.jar:/usr/local/storm/lib/curator-framework-2.4.0.jar:/usr/local/storm/lib/tools.logging-0.2.3.jar:/usr/local/storm/lib/tools.cli-0.2.4.jar:/usr/local/storm/lib/compojure-1.1.3.jar:/usr/local/storm/lib/clojure-1.5.1.jar:/usr/local/storm/lib/math.numeric-tower-0.0.1.jar:/usr/local/storm/lib/disruptor-2.10.1.jar:/usr/local/storm/lib/reflectasm-1.07-shaded.jar:/usr/local/storm/lib/clj-time-0.4.1.jar:/usr/local/storm/lib/httpcore-4.3.2.jar:/usr/local/storm/lib/zookeeper-3.4.5.jar:/usr/local/storm/lib/ring-jetty-adapter-0.3.11.jar:/usr/local/storm/lib/jetty-util-6.1.26.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/commons-io-2.4.jar:/usr/local/storm/lib/jetty-6.1.26.jar:/usr/local/storm/lib/commons-exec-1.1.jar:/usr/local/storm/lib/commons-lang-2.5.jar:/usr/local/storm/lib/snakeyaml-1.11.jar:/usr/local/storm/lib/guava-13.0.jar:/usr/local/storm/lib/ring-servlet-0.3.11.jar:/usr/local/storm/lib/minlog-1.2.jar:/usr/local/storm/lib/logback-classic-1.0.6.jar:/usr/local/storm/lib/ring-devel-0.3.11.jar:/usr/local/storm/lib/netty-3.6.3.Final.jar:/usr/local/storm/lib/kryo-2.21.jar:/usr/local/storm:/usr/local/storm/conf -Xmx768m -Dlogfile.name=ui.log -Dlogback.configurationFile=/usr/local/storm/logback/cluster.xml backtype.storm.ui.core
Go to web browser http://192.168.48.11:8080/index.html
