flink-learning

flink learning blog. http://www.54tianzhisheng.cn/ 含 Flink 入门、概念、原理、实战、性能调优、源码解析等内容。涉及 Flink Connector、Metrics、Library、DataStream API、Table API & SQL 等内容的学习案例，还有 Flink 落地应用的大型项目案例（PVUV、日志存储、百亿数据实时去重、监控告警）分享。欢迎大家支持我的专栏《大数据实时计算引擎 Flink 实战与性能优化》

14,703

3,927

14,703

View on GitHub

Top Related Projects

flink

24,422

Apache Flink

flink-cdc

5,999

Flink CDC is a streaming data integration tool

beam

7,959

Apache Beam is a unified programming model for Batch and Streaming data processing.

spark

40,785

Apache Spark - A unified analytics engine for large-scale data processing

storm

6,609

Apache Storm

Quick Overview

The zhisheng17/flink-learning repository is a comprehensive learning resource for Apache Flink, a distributed stream processing framework. It provides examples, tutorials, and best practices for various Flink features and use cases, making it an excellent resource for both beginners and experienced Flink developers.

Pros

Extensive coverage of Flink features and concepts
Includes practical examples and real-world use cases
Regularly updated with new content and examples
Well-organized structure for easy navigation and learning

Cons

Primarily in Chinese, which may be a barrier for non-Chinese speakers
Some examples may not be up-to-date with the latest Flink versions
Limited documentation on advanced topics and optimizations
Lack of comprehensive testing for all examples

Code Examples

Basic Flink DataStream example:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStream<String> text = env.socketTextStream("localhost", 9999);

DataStream<Tuple2<String, Integer>> counts = text
    .flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
        @Override
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
            for (String word : value.split("\\s")) {
                out.collect(new Tuple2<>(word, 1));
            }
        }
    })
    .keyBy(0)
    .sum(1);

counts.print();
env.execute("Word Count Example");

This example demonstrates a basic word count application using Flink's DataStream API.

Flink Table API example:

StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);

Table table = tableEnv.fromDataStream(stream);
Table result = table
    .groupBy("word")
    .select("word, count(1) as count");

DataStream<Tuple2<Boolean, Row>> resultStream = tableEnv.toRetractStream(result, Row.class);

This example shows how to use Flink's Table API to perform a word count operation on a stream.

Flink State Backend configuration:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStateBackend(new RocksDBStateBackend("hdfs://namenode:40010/flink/checkpoints"));
env.enableCheckpointing(1000);
env.getCheckpointConfig().setCheckpointTimeout(60000);

This code snippet demonstrates how to configure a RocksDB state backend with checkpointing enabled.

Getting Started

To get started with the flink-learning project:

Clone the repository:

git clone https://github.com/zhisheng17/flink-learning.git

Navigate to the project directory:
```
cd flink-learning
```
Build the project using Maven:
```
mvn clean package -DskipTests
```

Run a specific example, e.g., the word count example:

flink run -c com.zhisheng.examples.streaming.WordCount flink-learning-examples/target/flink-learning-examples-1.0-SNAPSHOT.jar

For more detailed instructions and examples, refer to the project's README and documentation.

Competitor Comparisons

flink

24,422

Apache Flink

Pros of Flink

Official Apache project with extensive community support and regular updates
Comprehensive documentation and extensive feature set for production use
Robust ecosystem with connectors, libraries, and tools for various data processing scenarios

Cons of Flink

Steeper learning curve due to its complex architecture and extensive features
Requires more resources and configuration for deployment and management
Less focused on learning and experimentation compared to flink-learning

Code Comparison

flink-learning:

DataStream<String> text = env.socketTextStream("localhost", 9999);
DataStream<Tuple2<String, Integer>> counts = text
    .flatMap(new FlatMapFunction<String, Tuple2<String, Integer>>() {
        public void flatMap(String value, Collector<Tuple2<String, Integer>> out) {
            for (String word : value.split("\\s")) {
                out.collect(new Tuple2<String, Integer>(word, 1));
            }
        }
    })
    .keyBy(0)
    .sum(1);

Flink:

DataStream<String> text = env.socketTextStream("localhost", 9999);
DataStream<Tuple2<String, Integer>> counts = text
    .flatMap((String value, Collector<Tuple2<String, Integer>> out) -> {
        for (String word : value.split("\\s")) {
            out.collect(new Tuple2<>(word, 1));
        }
    })
    .returns(Types.TUPLE(Types.STRING, Types.INT))
    .keyBy(value -> value.f0)
    .sum(1);

flink-cdc

5,999

Flink CDC is a streaming data integration tool

Pros of Flink CDC

Official Apache project with strong community support and regular updates
Focused specifically on Change Data Capture (CDC) for Flink, providing specialized functionality
Supports multiple database systems out-of-the-box (MySQL, PostgreSQL, MongoDB, etc.)

Cons of Flink CDC

More limited in scope compared to the broader Flink learning resources
May have a steeper learning curve for beginners due to its specialized nature
Less comprehensive documentation and examples for general Flink usage

Code Comparison

Flink CDC example:

MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
    .hostname("localhost")
    .port(3306)
    .databaseList("inventory")
    .tableList("inventory.products")
    .username("user")
    .password("password")
    .deserializer(new JsonDebeziumDeserializationSchema())
    .build();

Flink Learning example:

DataStream<String> stream = env.addSource(new FlinkKafkaConsumer<>("topic", new SimpleStringSchema(), properties));
stream.map(new MapFunction<String, String>() {
    @Override
    public String map(String value) throws Exception {
        return "Processed: " + value;
    }
}).print();

The code comparison shows that Flink CDC focuses on database source configuration, while Flink Learning provides more general Flink usage examples.

beam

7,959

Apache Beam is a unified programming model for Batch and Streaming data processing.

Pros of Beam

Supports multiple execution engines (Flink, Spark, etc.) vs Flink-learning's focus on Flink
More comprehensive and mature project with broader community support
Offers a unified programming model for batch and streaming

Cons of Beam

Steeper learning curve due to more abstract concepts
Less focused on Flink-specific optimizations and features
May have overhead when used solely with Flink compared to native Flink APIs

Code Comparison

Beam pipeline example:

Pipeline p = Pipeline.create();
p.apply(TextIO.read().from("input.txt"))
 .apply(ParDo.of(new DoFn<String, String>() {
   @ProcessElement
   public void processElement(ProcessContext c) {
     c.output(c.element().toUpperCase());
   }
 }))
 .apply(TextIO.write().to("output.txt"));

Flink-learning example:

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
env.fromElements("hello", "world")
   .map(String::toUpperCase)
   .print();
env.execute();

The Beam example shows a more verbose, but flexible approach that can run on multiple engines. The Flink-learning example demonstrates a more concise, Flink-specific implementation.

spark

40,785

Apache Spark - A unified analytics engine for large-scale data processing

Pros of Spark

More mature and widely adopted in industry
Extensive ecosystem with libraries for ML, graph processing, etc.
Better support for batch processing workloads

Cons of Spark

Less efficient for real-time stream processing
Higher latency compared to Flink for certain use cases
More complex configuration and tuning required

Code Comparison

Spark (Scala):

val lines = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "host1:port1,host2:port2").option("subscribe", "topic1").load()
val words = lines.as[String].flatMap(_.split(" "))
val wordCounts = words.groupBy("value").count()

Flink (Java):

DataStream<String> lines = env.addSource(new FlinkKafkaConsumer<>("topic1", new SimpleStringSchema(), properties));
DataStream<Tuple2<String, Integer>> counts = lines
    .flatMap(new Splitter())
    .keyBy(0)
    .sum(1);

Both examples show stream processing from Kafka, but Flink's API is more concise and intuitive for streaming scenarios. Spark requires more boilerplate code for similar functionality. However, Spark's unified API allows for easier transitions between batch and streaming workloads.

storm

6,609

Apache Storm

Pros of Storm

Mature and battle-tested distributed real-time computation system
Simpler programming model with lower learning curve
Better support for stateless processing and micro-batch operations

Cons of Storm

Less efficient for stateful processing compared to Flink
Limited support for event time processing and out-of-order events
Lacks advanced features like savepoints and native machine learning libraries

Code Comparison

Storm topology definition:

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout("spout", new RandomSentenceSpout(), 5);
builder.setBolt("split", new SplitSentence(), 8).shuffleGrouping("spout");
builder.setBolt("count", new WordCount(), 12).fieldsGrouping("split", new Fields("word"));

Flink job definition:

DataStream<String> text = env.addSource(new FlinkKafkaConsumer<>("topic", new SimpleStringSchema(), properties));
DataStream<Tuple2<String, Integer>> wordCounts = text
    .flatMap(new Tokenizer())
    .keyBy(0)
    .sum(1);
wordCounts.print();

Both Storm and Flink-learning repositories provide examples and learning resources for their respective stream processing frameworks. Storm offers simplicity and ease of use, while Flink-learning showcases Flink's advanced features and stateful processing capabilities. The choice between them depends on specific use cases and requirements.

Convert designs to code with AI

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

Flink å¦ä¹

Stargazers over time

æ¬é¡¹ç®ç»æ

How to build

Maybe your Maven conf file settings.xml mirrors can add aliyun central mirror :

<mirror>
  <id>alimaven</id>
  <mirrorOf>central</mirrorOf>
  <name>aliyun maven</name>
  <url>https://maven.aliyun.com/repository/central</url>
</mirror>

then you can run the following command :

mvn clean package -Dmaven.test.skip=true

you can see following result if build success.

Flink ç³»ç»ä¸æ

é¦åå°åï¼http://www.54tianzhisheng.cn/2019/11/15/flink-in-action/

ä¸æ å°åï¼https://gitbook.cn/gitchat/column/5dad4a20669f843a1a37cb4f

Change

2022/02/26 å°èªå·± ãFlink å®æä¸æ§è½ä¼åãä¸æ æ¾å¨ GitHubï¼åè§ books ç®å½

2021/12/18 å°è¯¥é¡¹ç®ç Flink çæ¬åçº§è³ 1.14.2ï¼å¦ææéè¦å¯ä»¥å»èçåæ¯æ¥çã

2021/08/15 å°è¯¥é¡¹ç®ç Flink çæ¬åçº§è³ 1.13.2ï¼API åçéå¤§æ¹åï¼æä»¥ä»£ç ç»æä¹åäºç¸åºçè°æ´ï¼é¨åä»£ç å¨ master åæ¯å·²ç»å é¤ï¼åæ¶å°ä¹åçä»£ç åå° feature/flink-1.10.0 ä¸äºï¼å¦ææéè¦å¯ä»¥å»èçåæ¯æ¥çï¼ã

2019/09/06 å°è¯¥é¡¹ç®ç Flink çæ¬åçº§å° 1.9.0ï¼æä¸äºåå¨ï¼Flink 1.8.0 çæ¬çä»£ç ç»ç¾¤éè®¨è®ºä¿åå¨åæ¯ feature/flink-1.8.0 ä»¥ä¾¿é¨ååå¦éè¦ã

2019/06/08 åæ¬ Flink ä¹¦ç±ï¼

Introduction_to_Apache_Flink_book.pdf è¿æ¬ä¹¦æ¯è¾èï¼å¤äºä»ç»é¶æ®µï¼å½åæè¿æ¬çç¿»è¯ä¹¦ç±
Learning Apache Flink.pdf è¿æ¬ä¹¦æ¯è¾åºç¡ï¼åå¦çè¯å¯ä»¥å¤çç
Stream Processing with Apache Flink.pdf è¿æ¬ä¹¦æ¯ Flink PMC åç
Streaming System.pdf è¿æ¬ä¹¦è¯ä»·ä¸æ¯ä¸è¬çé«

2019/06/09 æ°å¢æµå¤çå¼æç¸å³ç Paperï¼å¨ paper ç®å½ä¸ï¼