storm

Apache Storm is a distributed real-time computation system for processing large volumes of data with high fault tolerance and guaranteed data processing. It is designed to handle unbounded streams of data and can be used for real-time analytics, online machine learning, continuous computation, and more.

Pros

• Scalable and fault-tolerant architecture
• Low latency processing with high throughput
• Supports multiple programming languages
• Easy to set up and operate

Cons

• Steep learning curve for complex topologies
• Limited built-in support for stateful processing
• Can be resource-intensive for large-scale deployments
• Requires careful tuning for optimal performance

Code Examples

1. Creating a basic topology:

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("word-spout", new RandomSentenceSpout());
builder.setBolt("word-splitter", new SplitSentenceBolt()).shuffleGrouping("word-spout");
builder.setBolt("word-counter", new WordCountBolt()).fieldsGrouping("word-splitter", new Fields("word"));

2. Defining a custom bolt:

public class WordCountBolt extends BaseBasicBolt {
    Map<String, Integer> counts = new HashMap<String, Integer>();

    @Override
    public void execute(Tuple tuple, BasicOutputCollector collector) {
        String word = tuple.getString(0);
        Integer count = counts.get(word);
        if (count == null)
            count = 0;
        count++;
        counts.put(word, count);
        collector.emit(new Values(word, count));
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("word", "count"));
    }
}

3. Submitting a topology to a Storm cluster:

Config conf = new Config();
conf.setDebug(true);
conf.setNumWorkers(3);

StormSubmitter.submitTopology("word-count-topology", conf, builder.createTopology());

Getting Started

1. Add Storm dependency to your project:

<dependency>
    <groupId>org.apache.storm</groupId>
    <artifactId>storm-core</artifactId>
    <version>2.4.0</version>
</dependency>

2. Create a topology with spouts and bolts:

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout-id", new YourCustomSpout());
builder.setBolt("bolt-id", new YourCustomBolt()).shuffleGrouping("spout-id");

3. Configure and submit the topology:

Config conf = new Config();
conf.setNumWorkers(2);
StormSubmitter.submitTopology("topology-name", conf, builder.createTopology());