在 Windows 上运行 Apache Storm 2.0.0

🏷️ Storm

本文主要参考 Running Apache Storm on Windows（其使用的是 0.9.1-incubating 版）。

前置需要安装 Java 和 Python，Storm 2.0.0 需要安装 Java 8。

安装后不要忘记配置环境变量。

变量名	变量值
`JAVA_HOME`	C:\Progra~1\Java\jdk1.8.0_111
`Path` （追加）	C:\Python37\Scripts;C:\Python37;%JAVA_HOME%\bin;
`PATHEXT` （追加）	.PY;

安装后可使用命令 java -version 和 py -V 确认安装的版本。

bash

C:\Users\liujiajia>java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

bash

C:\Users\liujiajia>py -V
Python 3.7.3

安装 Zookeeper

下载 zookeeper-3.4.14.tar.gz 解压到 D:\zookeeper\zookeeper-3.4.14 目录后执行如下命令。

bash

D:
cd D:\zookeeper\zookeeper-3.4.14
copy conf\zoo_sample.cfg conf\zoo.cfg
bin\zkServer.cmd

执行成功的窗口打印日志：

txt

D:\zookeeper\zookeeper-3.4.14>call "C:\Progra~1\Java\jdk1.8.0_111"\bin\java "-Dz
ookeeper.log.dir=D:\zookeeper\zookeeper-3.4.14\bin\.." "-Dzookeeper.root.logger=
INFO,CONSOLE" -cp "D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookeep
er\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\*;D:
\zookeeper\zookeeper-3.4.14\bin\..\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\co
nf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper\zookeeper-3.
4.14\bin\..\conf\zoo.cfg"
2019-09-05 14:13:04,838 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading co
nfiguration from: D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg
2019-09-05 14:13:04,856 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopu
rge.snapRetainCount set to 3
2019-09-05 14:13:04,857 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopu
rge.purgeInterval set to 0
2019-09-05 14:13:04,857 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge
 task is not scheduled.
2019-09-05 14:13:04,860 [myid:] - WARN  [main:QuorumPeerMain@116] - Either no co
nfig or no quorum defined in config, running  in standalone mode
2019-09-05 14:13:04,935 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading co
nfiguration from: D:\zookeeper\zookeeper-3.4.14\bin\..\conf\zoo.cfg
2019-09-05 14:13:04,936 [myid:] - INFO  [main:ZooKeeperServerMain@98] - Starting
 server
2019-09-05 14:13:04,951 [myid:] - INFO  [main:Environment@100] - Server environm
ent:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on
03/06/2019 16:18 GMT
2019-09-05 14:13:04,951 [myid:] - INFO  [main:Environment@100] - Server environm
ent:host.name=EDG2GYKVF8G5P6Z.octopus.888ly.cn
2019-09-05 14:13:04,952 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.version=1.8.0_111
2019-09-05 14:13:04,952 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.vendor=Oracle Corporation
2019-09-05 14:13:04,953 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.home=C:\Progra~1\Java\jdk1.8.0_111\jre
2019-09-05 14:13:04,953 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.class.path=D:\zookeeper\zookeeper-3.4.14\bin\..\build\classes;D:\zookee
per\zookeeper-3.4.14\bin\..\build\lib\*;D:\zookeeper\zookeeper-3.4.14\bin\..\zoo
keeper-3.4.14.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\audience-annotations-
0.5.0.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\jline-0.9.94.jar;D:\zookeeper
\zookeeper-3.4.14\bin\..\lib\log4j-1.2.17.jar;D:\zookeeper\zookeeper-3.4.14\bin\
..\lib\netty-3.10.6.Final.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\slf4j-api
-1.7.25.jar;D:\zookeeper\zookeeper-3.4.14\bin\..\lib\slf4j-log4j12-1.7.25.jar;D:
\zookeeper\zookeeper-3.4.14\bin\..\conf
2019-09-05 14:13:04,953 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.library.path=C:\Progra~1\Java\jdk1.8.0_111\bin;C:\Windows\Sun\Java\bin;
C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\sy
stem32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell
\v1.0\;C:\Windows\system32\config\systemprofile\.dnx\bin;C:\Program Files\Micros
oft DNX\Dnvm\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program F
iles\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Git\cmd;C:\Program Fi
les\dotnet\;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Microsoft SQL Serv
er\120\DTS\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools
\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\;C:\Program Fi
les (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio\;C:\Program Files
 (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files\Microsoft SQL Server\
Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\PuTTY\;C:\PROGRA~2\Groovy\GROOV
Y~1.0\bin;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\Window
sPowerShell\v1.0\;C:\Program Files\Redis\;D:\hadoop-3.2.0\bin;D:\hadoop-3.2.0\sb
in;C:\Users\liujiajia\AppData\Local\Programs\Python\Python37\Scripts\;C:\Users\l
iujiajia\AppData\Local\Programs\Python\Python37\;C:\Users\liujiajia\AppData\Loca
l\Programs\Microsoft VS Code\bin;C:\Users\liujiajia\.dotnet\tools;;C:\Program Fi
les\Docker Toolbox;.
2019-09-05 14:13:04,954 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.io.tmpdir=C:\Users\LIUJIA~1\AppData\Local\Temp\
2019-09-05 14:13:04,955 [myid:] - INFO  [main:Environment@100] - Server environm
ent:java.compiler=<NA>
2019-09-05 14:13:04,956 [myid:] - INFO  [main:Environment@100] - Server environm
ent:os.name=Windows 7
2019-09-05 14:13:04,956 [myid:] - INFO  [main:Environment@100] - Server environm
ent:os.arch=amd64
2019-09-05 14:13:04,957 [myid:] - INFO  [main:Environment@100] - Server environm
ent:os.version=6.1
2019-09-05 14:13:04,957 [myid:] - INFO  [main:Environment@100] - Server environm
ent:user.name=liujiajia
2019-09-05 14:13:04,957 [myid:] - INFO  [main:Environment@100] - Server environm
ent:user.home=C:\Users\liujiajia
2019-09-05 14:13:04,958 [myid:] - INFO  [main:Environment@100] - Server environm
ent:user.dir=D:\zookeeper\zookeeper-3.4.14
2019-09-05 14:13:04,988 [myid:] - INFO  [main:ZooKeeperServer@836] - tickTime se
t to 2000
2019-09-05 14:13:04,988 [myid:] - INFO  [main:ZooKeeperServer@845] - minSessionT
imeout set to -1
2019-09-05 14:13:04,989 [myid:] - INFO  [main:ZooKeeperServer@854] - maxSessionT
imeout set to -1
2019-09-05 14:13:05,200 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org
.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-09-05 14:13:05,202 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding
 to port 0.0.0.0/0.0.0.0:2181

安装 Storm

下载 apache-storm-2.0.0.zip 解压到 D:\storm\apache-storm-2.0.0 目录

配置环境变量

变量名	变量值
`STORM_HOME`	D:\storm\apache-storm-2.0.0
`Path` （追加）	%STORM_HOME%\bin;

启动 Nimbus、Supervisor 和 Storm UI

Nimbus
bash
```
D:
cd %STORM_HOME%
storm.py nimbus
```
1
2
3
Supervisor
bash
```
D:
cd %STORM_HOME%
storm.py supervisor
```
1
2
3
Storm UI
bash
```
D:
cd %STORM_HOME%
storm.py ui
```
1
2
3

若启动失败，可在 logs 目录下查看对应的日志文件。

访问 StormUI

默认地址：http://localhost:8080/

若 8080 端口被占用，可在 conf/storm.yaml 指定端口号。

yaml

ui.port: 8071

这里指定为 8071 端口，然后访问 http://localhost:8071/ 。

部署 Storm 自带的 examples 中的 "WordCount" Topology

代码在 D:\storm\apache-storm-2.0.0\examples\storm-starter 目录，需要自己打包。
打包后生成 storm-starter-2.0.0.jar 文件（172MB - 这么大难怪下载包里不提供）。

执行如下命令部署：

bash

storm.py jar D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar org.apache.storm.starter.WordCountTopology WordCount -c nimbus.host=localhost

控制台输出：

txt

D:\storm\apache-storm-2.0.0>storm.py jar D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar org.apache.storm.starter.WordCountTopology WordCount -c nimbus.host=localhost
Running: C:\Progra~1\Java\jdk1.8.0_111\bin\java.exe -client -Ddaemon.name= -Dstorm.options=nimbus.host%3Dlocalhost -Dstorm.home=D:\storm\apache-storm-2.0.0 -Dstorm.log.dir=D:\storm\apache-storm-2.0.0\logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp D:\storm\apache-storm-2.0.0\*;D:\storm\apache-storm-2.0.0\lib\*;D:\storm\apache-storm-2.0.0\extlib\*;D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar;D:\storm\apache-storm-2.0.0\conf;D:\storm\apache-storm-2.0.0\bin; -Dstorm.jar=D:\storm\apache-storm-2.0.0\examples\storm-starter\target\storm-starter-2.0.0.jar -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={} org.apache.storm.starter.WordCountTopology WordCount
b'10:17:19.107 [main] INFO  o.a.s.StormSubmitter - Generated ZooKeeper secret payload for MD5-digest: -8881801371475247195:-5380596067515307374\r\n10:17:19.153 [main] WARN  o.a.s.v.ConfigValidation - task.heartbeat.frequency.secs is a deprecated config please see class org.apache.storm.Config.TASK_HEARTBEAT_FREQUENCY_SECS for more information.\r\n10:17:19.227 [main] INFO  o.a.s.u.NimbusClient - Found leader nimbus : EDG2GYKVF8G5P6Z.octopus.888ly.cn:6627\r\n10:17:19.228 [main] INFO  o.a.s.s.a.ClientAuthUtils - Got AutoCreds []\r\n10:17:19.256 [main] INFO  o.a.s.StormSubmitter - Uploading dependencies - jars...\r\n10:17:19.257
[main] INFO  o.a.s.StormSubmitter - Uploading dependencies - artifacts...\r\n10:17:19.257 [main] INFO  o.a.s.StormSubmitter - Dependency Blob keys - jars : [] / artifacts : []\r\n10:17:19.267 [main] INFO  o.a.s.StormSubmitter - Uploading topology jar D:\\storm\\apache-storm-2.0.0\\examples\\storm-starter\\target\\storm-starter-2.0.0.jar to assigned location: D:\\storm\\apache-storm-2.0.0\\storm-local\\nimbus\\inbox/stormjar-ebe35cf8-cbc5-4821-a95f-f604bcebe442.jar\r\n10:17:26.576 [main] INFO  o.a.s.StormSubmitter - Successfully uploaded topology jar to assigned location: D:\\storm\\apache-storm-2.0.0\\storm-local\\nimbus\\inbox/stormjar-ebe35cf8-cbc5-4821-a95f-f604bcebe442.jar\r\n10:17:26.576 [main] INFO  o.a.s.StormSubmitter - Submitting topology WordCount in distributed mode with conf {"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-8881801371475247195:-5380596067515307374","nimbus.host":"localhost","topology.workers":3,"topology.debug":true}\r\n10:17:27.604 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: WordCount\r\n'

在 Storm UI 上查看 Topology summary

在 Windows 上运行 Apache Storm 2.0.0 ​

安装 Zookeeper ​

安装 Storm ​

配置环境变量 ​

启动 Nimbus、Supervisor 和 Storm UI ​

访问 StormUI ​

部署 Storm 自带的 examples 中的 "WordCount" Topology ​

在 Windows 上运行 Apache Storm 2.0.0

安装 Zookeeper

安装 Storm

配置环境变量

启动 Nimbus、Supervisor 和 Storm UI

访问 StormUI

部署 Storm 自带的 examples 中的 "WordCount" Topology