How to debug local topology?

How to debug a JStorm topology locally

JStorm has two modes: local and distributed. Local mode is very helpful for developing and debugging your topologies.

This document briefs how to debug locally without installing JStorm on your computer.

Use LocalCluster in your code

```java
import backtype.storm.LocalCluster;

LocalCluster cluster = new LocalCluster();

// set spout/bolt max parallelism
conf.put(Config.TOPOLOGY_MAX_TASK_PARALLELISM, 1);

// submit a local topology
cluster.submitTopology("SequenceTest", conf, builder.createTopology());

// wait for 1 min before killing the topology and local cluster, you can change this value accordingly
Thread.sleep(60000);		
		
// kill the topology
cluster.killTopology("SequenceTest");

// shutdown local cluster
cluster.shutdown();
```

A LocalCluster puts nimbus, supervisor and worker into a single process. You can use `submitTopology` to 
submit topologies just like `StormSubmitter`.

You can also call `killTopology`, `active`, `deactive`, `rebalance`, etc. to perform other operations.

Modify pom.xml

Let's say you are using JStorm 2.2.1

```xml
<dependency>
  <groupId>com.alibaba.jstorm</groupId>
  <artifactId>jstorm-core</artifactId>
  <version>2.2.1</version>
  <!-- keep jstorm out of the jar-with-dependencies -->
  <!-- <scope>provided</scope> -->
</dependency>
```

`<scope>provided</scope>` is unset, but **remember to change it back when submitting to a distributed cluster!** 
otherwise you'll get errors complaining about multiple `defaults.yaml`

Run main class。

Below is a snapshot of local debug

image

For better code organization, it's recommended to separate local running and distributed running code into two 
 methods, and put running mode into your conf. 
 
Please refer to [SequenceTopology](https://github.com/alibaba/jstorm/blob/master/example/sequence-split-merge/src/main/java/com/alipay/dw/jstorm/example/sequence/SequenceTopology.java)

Run sequence-split-merge examples

example/sequence-split-merge in JStorm source code comes with plenty of examples, unfortunately, due to maven-shade-plugin in jstorm-core, if you want to run the examples locally, a few modification is required.

Take the simplest SequenceTopology as an example, you’ll find errors when building with javac or within your IDE. Please change as follows:

In TotalCount class, comment import com.alibaba.jstorm.metrics.Gauge; and import classes in codahale dependency, i.e., import com.codahale.metrics.Gauge;

In PairSerializer and TradeCustomerSerializer, comment packages starting with com.alibaba.jstorm, and uncomment classes starting with com.esotericsoftware below.

Now you should be fine to run SequenceTopology.

Finally, if you have a better solution for this, please don’t hesitate to issue a PR.

Limitation

Local debug is mainly used to test application logic, so there’re limitations like user defined classloader is not supported.

Moreover, pay attention to your log4j dependency in your application, if a log4j.properties is within your dependency or application resource, it might overlap log4j conf of JStorm, resulting in nil output in your console.

FAQ

No output in console

A few reasons may cause this problem

1. If your application depends slf4j-log4j12, this will conflict with log4j-over-slf4j(introduced by jstorm-core),
please exclude slf4j-log4j12

2. Make sure your logs are printed using slf4j-api, i.e., `LoggerFactory` rather than `Logger`.