hadoop

Pros of Spark

• Faster processing speed due to in-memory computation
• More versatile with support for SQL, streaming, and machine learning
• Simpler API and easier to use for developers

Cons of Spark

• Higher memory requirements, which can be costly for large datasets
• Less mature ecosystem compared to Hadoop
• Not as efficient for batch processing of very large datasets

Code Comparison

Hadoop MapReduce (Java):

public static class MapClass extends Mapper<LongWritable, Text, Text, IntWritable> {
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            context.write(new Text(tokenizer.nextToken()), new IntWritable(1));
        }
    }
}

Spark (Scala):

val textFile = spark.read.textFile("hdfs://...")
val counts = textFile.flatMap(line => line.split(" "))
                     .map(word => (word, 1))
                     .reduceByKey(_ + _)
counts.saveAsTextFile("hdfs://...")

The Spark code is more concise and easier to read, demonstrating its simpler API. However, Hadoop's MapReduce offers more fine-grained control over the mapping and reducing processes, which can be beneficial for complex data processing tasks.

Pros of Flink

• Faster processing for streaming data and real-time analytics
• More flexible API with support for multiple programming languages
• Lower latency and higher throughput for stream processing tasks

Cons of Flink

• Smaller community and ecosystem compared to Hadoop
• Less mature for batch processing workloads
• Steeper learning curve for developers new to stream processing

Code Comparison

Flink (Java):

DataStream<String> stream = env.addSource(new FlinkKafkaConsumer<>("topic", new SimpleStringSchema(), properties));
stream.map(s -> s.toUpperCase())
      .filter(s -> s.startsWith("A"))
      .addSink(new FlinkKafkaProducer<>("output-topic", new SimpleStringSchema(), properties));

Hadoop (Java):

Job job = Job.getInstance(conf, "WordCount");
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

The Flink code example shows a simple streaming pipeline, while the Hadoop example demonstrates a typical MapReduce job setup. Flink's API is more concise and expressive for stream processing tasks, whereas Hadoop's MapReduce model is better suited for batch processing operations.

Pros of Hive

• Easier to use for data analysts with SQL-like query language (HiveQL)
• Better suited for data warehousing and analytics on structured data
• Faster query execution for complex analytical queries

Cons of Hive

• Limited real-time querying capabilities compared to Hadoop
• Less flexible for unstructured data processing
• Dependent on Hadoop for underlying storage and processing

Code Comparison

Hive (HiveQL):

SELECT customer_id, SUM(order_total)
FROM orders
GROUP BY customer_id
HAVING SUM(order_total) > 1000;

Hadoop (MapReduce in Java):

public static class MapClass extends Mapper<LongWritable, Text, Text, DoubleWritable> {
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String[] fields = value.toString().split(",");
        context.write(new Text(fields[0]), new DoubleWritable(Double.parseDouble(fields[1])));
    }
}

public static class ReduceClass extends Reducer<Text, DoubleWritable, Text, DoubleWritable> {
    public void reduce(Text key, Iterable<DoubleWritable> values, Context context) throws IOException, InterruptedException {
        double sum = 0;
        for (DoubleWritable val : values) {
            sum += val.get();
        }
        if (sum > 1000) {
            context.write(key, new DoubleWritable(sum));
        }
    }
}