본문 바로가기

Cloud & Backing services/Distributed System

Apache Storm 설치하기

프로젝트에서 Apache Storm을 활용할 부분이 생겨서 기능 검증이 필요한 시점이다. AWS에 Apache Storm을 설치한 과정을 정리하고자 한다. EC2의 Instance Type 및 용도는 아래와 같다.

  • Zookeeper #1 + Nimbus + Supervisor + Storm UI
  • Zookeeper #2 + Supervisor
  • Zookeeper #3 + Supervisor

 

#공통 (Each Node) 각 Instance에 JDK를 설치한다.

 

$sudo apt-add-repository ppa:webupd8team/java$sudo apt-get update

$sudo apt-get install oracle-java8-installer

 

#Zookeeper Cluster 설치 (Each Node)아래의 command를 수행한다.

$wget http://apache.mirror.cdnetworks.com/zookeeper/stable/zookeeper-3.4.8.tar.gz$tar xzvf zookeeper-3.4.8.tar.gz$mv zookeeper-3.4.8 zookeeper$cd zookeeper$mkdir data$cd conf

$cp zoo_sample.cfg zoo.cfg

 

zoo.cfg파일을 수정한다. Zookeeper #1 설정

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/ubuntu/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=0.0.0.0:2888:3888server.2=2.2.2.2:2888:3888server.3=3.3.3.3:2888:3888# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to “0” to disable auto purge feature

#autopurge.purgeInterval=1

 

Zookeeper #2 설정

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/ubuntu/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=1.1.1.1:2888:3888server.2=0.0.0.0:2888:3888server.3=3.3.3.3:2888:3888# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to “0” to disable auto purge feature

#autopurge.purgeInterval=1

 

Zookeeper #3 설정

# The number of milliseconds of each ticktickTime=2000# The number of ticks that the initial# synchronization phase can takeinitLimit=10# The number of ticks that can pass between# sending a request and getting an acknowledgementsyncLimit=5# the directory where the snapshot is stored.# do not use /tmp for storage, /tmp here is just# example sakes.dataDir=/home/ubuntu/zookeeper/data# the port at which the clients will connectclientPort=2181server.1=1.1.1.1:2888:3888server.2=2.2.2.2:2888:3888server.3=0.0.0.0:2888:3888# the maximum number of client connections.# increase this if you need to handle more clients#maxClientCnxns=60## Be sure to read the maintenance section of the# administrator guide before turning on autopurge.## http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance## The number of snapshots to retain in dataDir#autopurge.snapRetainCount=3# Purge task interval in hours# Set to “0” to disable auto purge feature

#autopurge.purgeInterval=1

 

설정이 완료되었으면 Zookeeper를 실행한다. (각 서버별 수행)

$cd /home/ubuntu/zookeeper/bin
$./zkServer.sh start

 

#Apache Storm 설치 아래의 명령어를 모든 노드에서 수행한다.

$wget http://apache.mirror.cdnetworks.com/storm/apache-storm-1.0.1/apache-storm-1.0.1.tar.gz
$tar
 xzvf apache-storm-1.0.1.tar.gz$mv apache-storm-1.0.1 stormStorm configuration을 설정한다.
$vi storm/conf/storm.yaml

 

각 노드별로 설정이 완료되었으면, 아래의 과정을 수행한다.

Start Nimbus

$nohup ./storm/bin/storm nimbus >> nimbus.log 2>&1 &

 

Start Supervisor 슈퍼바이저는 각 노드에서 수행한다.

$nohup ./storm/bin/storm supervisor >> supervisor.log 2>&1 &

 

Start Storm UI 스톰 UI는 Nimbus 노드에서 수행한다. (Worker 노드에서 수행 금지, 접속은 브라우저에서 IP:8080)

$nohup ./storm/bin/storm ui >> ui.log 2>&1 &

 

Start Logviwer 각 노드에서 수행한다.

$nohup ./storm/bin/storm logviewer > logviewer.log &

 

Start Example Topology Nimbus 노드에서 수행한다.

$./storm/bin/storm jar ../examples/storm-starter/storm-starter-topologies-0.10.0.jar org.apache.storm.starter.RollingTopWords production-topology remote

 

Kill Example Topology Nimbus 노드에서 수행한다.

$./storm/bin/storm kill production-topology 1

 

이제 갖고 놀면 된다.