Convert Figma logo to code with AI

lancopku logopkuseg-python

pkuseg多领域中文分词工具; The pkuseg toolkit for multi-domain Chinese word segmentation

6,504
984
6,504
129
View on GitHub

Top Related Projects

fxsjy's avatar

jieba

33,063

结巴中文分词

thunlp's avatar

THULAC-Python

2,003

An Efficient Lexical Analyzer for Chinese

hankcs's avatar

HanLP

33,448

Natural Language Processing for the next decade. Tokenization, Part-of-Speech Tagging, Named Entity Recognition, Syntactic & Semantic Dependency Parsing, Document Classification

baidu's avatar

lac

3,840

百度NLP：分词，词性标注，命名实体识别，词重要性

isnowfy's avatar

snownlp

6,397

Python library for processing Chinese text

blmoistawinde's avatar

HarvestText

2,381

文本挖掘和预处理工具（文本清洗、新词发现、情感分析、实体识别链接、关键词抽取、知识抽取、句法分析等），无监督或弱监督方法

Quick Overview

PKUSeg-python is a Chinese word segmentation toolkit developed by the Peking University Language Computing Lab. It offers state-of-the-art performance in Chinese text segmentation tasks and is designed to be easy to use and integrate into various natural language processing applications.

Pros

  • High accuracy in Chinese word segmentation compared to other popular tools
  • Supports multiple domains, including news, web, medicine, and more
  • Easy to install and use with a simple Python API
  • Allows for custom model training on specific domains

Cons

  • Primarily focused on Chinese language, limiting its use for other languages
  • May require more computational resources compared to simpler segmentation tools
  • Documentation is primarily in Chinese, which might be challenging for non-Chinese speakers
  • Limited community support compared to more widely-used NLP libraries

Code Examples

  1. Basic word segmentation:
import pkuseg

seg = pkuseg.pkuseg()
text = "我爱北京天安门"
result = seg.cut(text)
print(result)
# Output: ['我', '爱', '北京', '天安门']
  1. Using a specific domain model:
seg = pkuseg.pkuseg(model_name='medicine')
text = "头孢霉素类抗生素可以治疗肺炎"
result = seg.cut(text)
print(result)
# Output: ['头孢霉素', '类', '抗生素', '可以', '治疗', '肺炎']
  1. Custom dictionary usage:
seg = pkuseg.pkuseg(user_dict=['北京大学'])
text = "北京大学是世界一流大学"
result = seg.cut(text)
print(result)
# Output: ['北京大学', '是', '世界', '一流', '大学']

Getting Started

To get started with PKUSeg-python, follow these steps:

  1. Install the library using pip:

    pip install pkuseg

  2. Import the library and create a segmenter:

    import pkuseg
seg = pkuseg.pkuseg()

  3. Segment Chinese text:

    text = "我在北京大学学习自然语言处理"
result = seg.cut(text)
print(result)

This will output the segmented words as a list. You can now use PKUSeg-python for various Chinese text processing tasks in your projects.

Competitor Comparisons

fxsjy logo

jieba

33,063

结巴中文分词

Pros of jieba

  • Faster processing speed for large-scale text segmentation
  • More extensive documentation and community support
  • Broader range of features, including keyword extraction and text summarization

Cons of jieba

  • Less accurate for specialized domains or formal texts
  • Requires more manual tuning for optimal performance
  • Larger memory footprint, especially for large dictionaries

Code Comparison

jieba:

import jieba
seg_list = jieba.cut("我来到北京清华大学", cut_all=False)
print("Default Mode: " + "/ ".join(seg_list))

pkuseg:

import pkuseg
seg = pkuseg.pkuseg()
text = seg.cut("我来到北京清华大学")
print(text)

Both libraries offer simple APIs for text segmentation, but pkuseg provides more flexibility in model selection and training. jieba offers additional features like keyword extraction, while pkuseg focuses on accurate segmentation for various domains.

jieba is generally faster and more suitable for large-scale processing, while pkuseg excels in accuracy, especially for formal texts or specific domains. The choice between the two depends on the specific requirements of the project, such as processing speed, accuracy, and domain specificity.

thunlp logo

THULAC-Python

2,003

An Efficient Lexical Analyzer for Chinese

Pros of THULAC-Python

  • Faster processing speed for large-scale text segmentation tasks
  • Includes part-of-speech tagging functionality out of the box
  • Supports both simplified and traditional Chinese characters

Cons of THULAC-Python

  • Less flexible in terms of customization and fine-tuning
  • May have lower accuracy on domain-specific or non-standard text
  • Requires more memory resources compared to pkuseg-python

Code Comparison

THULAC-Python usage:

import thulac

thu = thulac.thulac()
text = "我爱北京天安门"
result = thu.cut(text)
print(result)

pkuseg-python usage:

import pkuseg

seg = pkuseg.pkuseg()
text = "我爱北京天安门"
result = seg.cut(text)
print(result)

Both libraries offer simple APIs for text segmentation, but THULAC-Python provides additional features like part-of-speech tagging by default. pkuseg-python focuses on customizable segmentation and may be more suitable for specific domain applications.

hankcs logo

HanLP

33,448

Natural Language Processing for the next decade. Tokenization, Part-of-Speech Tagging, Named Entity Recognition, Syntactic & Semantic Dependency Parsing, Document Classification

Pros of HanLP

  • More comprehensive NLP toolkit with broader functionality beyond segmentation
  • Supports multiple languages, not just Chinese
  • Actively maintained with frequent updates and improvements

Cons of HanLP

  • Larger library size, potentially slower for simple segmentation tasks
  • May have a steeper learning curve due to more extensive features
  • Higher computational resource requirements for full functionality

Code Comparison

HanLP:

from hanlp_restful import HanLPClient

HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh')
print(HanLP.tokenize('我爱自然语言处理'))

pkuseg:

import pkuseg

seg = pkuseg.pkuseg()
text = seg.cut('我爱自然语言处理')
print(text)

Both libraries offer straightforward APIs for tokenization, but HanLP provides a more extensive set of NLP tools beyond simple segmentation. pkuseg is more focused on Chinese word segmentation specifically, while HanLP offers a broader range of language processing capabilities.

HanLP's code example demonstrates its client-server architecture, which allows for more advanced processing but may require additional setup. pkuseg, on the other hand, provides a simpler, more direct approach to Chinese word segmentation that can be quickly implemented in Python projects.

baidu logo

lac

3,840

百度NLP：分词，词性标注，命名实体识别，词重要性

Pros of LAC

  • Supports both word segmentation and part-of-speech tagging
  • Offers pre-trained models for various domains (e.g., news, web)
  • Provides both Python and C++ interfaces for flexibility

Cons of LAC

  • Less customizable for specific domains compared to pkuseg
  • May have lower accuracy on certain text types or specialized content
  • Requires more dependencies and setup compared to pkuseg

Code Comparison

pkuseg usage:

import pkuseg
seg = pkuseg.pkuseg()
text = "我爱北京天安门"
print(seg.cut(text))

LAC usage:

from LAC import LAC
lac = LAC(mode='seg')
text = "我爱北京天安门"
print(lac.run(text))

Both libraries offer simple APIs for word segmentation, but LAC provides additional functionality for part-of-speech tagging and named entity recognition. pkuseg focuses primarily on customizable word segmentation for various domains.

pkuseg is generally easier to set up and use for basic word segmentation tasks, while LAC offers more comprehensive language processing capabilities at the cost of increased complexity.

isnowfy logo

snownlp

6,397

Python library for processing Chinese text

Pros of snownlp

  • More comprehensive NLP toolkit with sentiment analysis, text classification, and more
  • Simpler installation process and fewer dependencies
  • Faster processing speed for basic NLP tasks

Cons of snownlp

  • Less accurate for complex Chinese word segmentation tasks
  • Not actively maintained (last update in 2020)
  • Limited documentation and community support

Code Comparison

snownlp:

from snownlp import SnowNLP

s = SnowNLP(u'这是一个测试句子')
print(s.words)  # 分词
print(s.sentiments)  # 情感分析

pkuseg-python:

import pkuseg

seg = pkuseg.pkuseg()
text = "这是一个测试句子"
print(seg.cut(text))  # 分词

Both libraries offer Chinese word segmentation, but pkuseg-python focuses on providing more accurate segmentation, especially for domain-specific texts. snownlp, on the other hand, offers a broader range of NLP functionalities beyond just segmentation. pkuseg-python is more actively maintained and provides better documentation, while snownlp offers a simpler API for quick NLP tasks but may lack in accuracy for complex segmentation scenarios.

blmoistawinde logo

HarvestText

2,381

文本挖掘和预处理工具（文本清洗、新词发现、情感分析、实体识别链接、关键词抽取、知识抽取、句法分析等），无监督或弱监督方法

Pros of HarvestText

  • More comprehensive NLP toolkit with additional features like named entity recognition and sentiment analysis
  • Supports both Chinese and English text processing
  • Includes built-in dictionaries and regular expression patterns for common tasks

Cons of HarvestText

  • Less specialized for Chinese word segmentation compared to pkuseg-python
  • May have slower performance for large-scale text processing tasks
  • Requires more dependencies and setup compared to the more focused pkuseg-python

Code Comparison

HarvestText:

from harvesttext import HarvestText
ht = HarvestText()
text = "今天是个好日子"
words = ht.seg(text)
print(words)

pkuseg-python:

import pkuseg
seg = pkuseg.pkuseg()
text = "今天是个好日子"
words = seg.cut(text)
print(words)

Both libraries offer simple APIs for Chinese word segmentation, but HarvestText provides a more extensive set of NLP tools beyond just segmentation. pkuseg-python focuses specifically on accurate Chinese word segmentation, while HarvestText aims to be a more comprehensive toolkit for various NLP tasks in both Chinese and English.

Convert Figma logo designs to code with AI

Visual Copilot

Introducing Visual Copilot: A new AI model to turn Figma designs to high quality code using your components.

Try Visual Copilot

README

pkuseg：一个多领域中文分词工具包 (English Version)

pkuseg 是基于论文[Luo et. al, 2019]的工具包。其简单易用，支持细分领域分词，有效提升了分词准确度。

目录

主要亮点

pkuseg具有如下几个特点：

  1. 多领域分词。不同于以往的通用中文分词工具，此工具包同时致力于为不同领域的数据提供个性化的预训练模型。根据待分词文本的领域特点，用户可以自由地选择不同的模型。 我们目前支持了新闻领域，网络领域，医药领域，旅游领域，以及混合领域的分词预训练模型。在使用中，如果用户明确待分词的领域，可加载对应的模型进行分词。如果用户无法确定具体领域，推荐使用在混合领域上训练的通用模型。各领域分词样例可参考 example.txt。
  2. 更高的分词准确率。相比于其他的分词工具包，当使用相同的训练数据和测试数据，pkuseg可以取得更高的分词准确率。
  3. 支持用户自训练模型。支持用户使用全新的标注数据进行训练。
  4. 支持词性标注。

编译和安装

  • 目前仅支持python3
  • 为了获得好的效果和速度，强烈建议大家通过pip install更新到目前的最新版本

  1. 通过PyPI安装(自带模型文件)：

    pip3 install pkuseg
之后通过import pkuseg来引用

    **建议更新到最新版本**以获得更好的开箱体验：

    pip3 install -U pkuseg

  2. 如果PyPI官方源下载速度不理想，建议使用镜像源，比如：
    初次安装：

    pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple pkuseg

    更新：

    pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple -U pkuseg

  3. 如果不使用pip安装方式，选择从GitHub下载，可运行以下命令安装：

    python setup.py build_ext -i

    GitHub的代码并不包括预训练模型，因此需要用户自行下载或训练模型，预训练模型可详见release。使用时需设定"model_name"为模型文件。

注意：**安装方式1和2目前仅支持linux(ubuntu)、mac、windows 64 位的python3版本**。如果非以上系统，请使用安装方式3进行本地编译安装。

各类分词工具包的性能对比

我们选择jieba、THULAC等国内代表分词工具包与pkuseg做性能比较，详细设置可参考实验环境。

细领域训练及测试结果

以下是在不同数据集上的对比结果：

MSRAPrecisionRecallF-score
jieba87.0189.8888.42
THULAC95.6095.9195.71
pkuseg96.9496.8196.88
WEIBOPrecisionRecallF-score
jieba87.7987.5487.66
THULAC93.4092.4092.87
pkuseg93.7894.6594.21

默认模型在不同领域的测试效果

考虑到很多用户在尝试分词工具的时候，大多数时候会使用工具包自带模型测试。为了直接对比“初始”性能，我们也比较了各个工具包的默认模型在不同领域的测试效果。请注意，这样的比较只是为了说明默认情况下的效果，并不一定是公平的。

DefaultMSRACTB8PKUWEIBOAll Average
jieba81.4579.5881.8383.5681.61
THULAC85.5587.8492.2986.6588.08
pkuseg87.2991.7792.6893.4391.29

其中，All Average显示的是在所有测试集上F-score的平均。

更多详细比较可参见和现有工具包的比较。

使用方式

代码示例

以下代码示例适用于python交互式环境。

代码示例1：使用默认配置进行分词（如果用户无法确定分词领域，推荐使用默认模型分词）

import pkuseg

seg = pkuseg.pkuseg()           # 以默认配置加载模型
text = seg.cut('我爱北京天安门')  # 进行分词
print(text)

代码示例2：细领域分词（如果用户明确分词领域，推荐使用细领域模型分词）

import pkuseg

seg = pkuseg.pkuseg(model_name='medicine')  # 程序会自动下载所对应的细领域模型
text = seg.cut('我爱北京天安门')              # 进行分词
print(text)

代码示例3：分词同时进行词性标注，各词性标签的详细含义可参考 tags.txt

import pkuseg

seg = pkuseg.pkuseg(postag=True)  # 开启词性标注功能
text = seg.cut('我爱北京天安门')    # 进行分词和词性标注
print(text)

代码示例4：对文件分词

import pkuseg

# 对input.txt的文件分词输出到output.txt中
# 开20个进程
pkuseg.test('input.txt', 'output.txt', nthread=20)

其他使用示例可参见详细代码示例。

参数说明

模型配置

pkuseg.pkuseg(model_name = "default", user_dict = "default", postag = False)
	model_name		模型路径。
			        "default"，默认参数，表示使用我们预训练好的混合领域模型(仅对pip下载的用户)。
				"news", 使用新闻领域模型。
				"web", 使用网络领域模型。
				"medicine", 使用医药领域模型。
				"tourism", 使用旅游领域模型。
			        model_path, 从用户指定路径加载模型。
	user_dict		设置用户词典。
				"default", 默认参数，使用我们提供的词典。
				None, 不使用词典。
				dict_path, 在使用默认词典的同时会额外使用用户自定义词典，可以填自己的用户词典的路径，词典格式为一行一个词（如果选择进行词性标注并且已知该词的词性，则在该行写下词和词性，中间用tab字符隔开）。
	postag		        是否进行词性分析。
				False, 默认参数，只进行分词，不进行词性标注。
				True, 会在分词的同时进行词性标注。

对文件进行分词

pkuseg.test(readFile, outputFile, model_name = "default", user_dict = "default", postag = False, nthread = 10)
	readFile		输入文件路径。
	outputFile		输出文件路径。
	model_name		模型路径。同pkuseg.pkuseg
	user_dict		设置用户词典。同pkuseg.pkuseg
	postag			设置是否开启词性分析功能。同pkuseg.pkuseg
	nthread			测试时开的进程数。

模型训练

pkuseg.train(trainFile, testFile, savedir, train_iter = 20, init_model = None)
	trainFile		训练文件路径。
	testFile		测试文件路径。
	savedir			训练模型的保存路径。
	train_iter		训练轮数。
	init_model		初始化模型，默认为None表示使用默认初始化，用户可以填自己想要初始化的模型的路径如init_model='./models/'。

多进程分词

当将以上代码示例置于文件中运行时，如涉及多进程功能，请务必使用if __name__ == '__main__'保护全局语句，详见多进程分词。

预训练模型

从pip安装的用户在使用细领域分词功能时，只需要设置model_name字段为对应的领域即可，会自动下载对应的细领域模型。

从github下载的用户则需要自己下载对应的预训练模型，并设置model_name字段为预训练模型路径。预训练模型可以在release部分下载。以下是对预训练模型的说明：

  • news: 在MSRA（新闻语料）上训练的模型。

  • web: 在微博（网络文本语料）上训练的模型。

  • medicine: 在医药领域上训练的模型。

  • tourism: 在旅游领域上训练的模型。

  • mixed: 混合数据集训练的通用模型。随pip包附带的是此模型。

我们还通过领域自适应的方法，利用维基百科的未标注数据实现了几个细领域预训练模型的自动构建以及通用模型的优化，这些模型目前仅可以在release中下载：

  • art: 在艺术与文化领域上训练的模型。

  • entertainment: 在娱乐与体育领域上训练的模型。

  • science: 在科学领域上训练的模型。

  • default_v2: 使用领域自适应方法得到的优化后的通用模型，相较于默认模型规模更大，但泛化性能更好。

欢迎更多用户可以分享自己训练好的细分领域模型。

版本历史

详见版本历史。

开源协议

  1. 本代码采用MIT许可证。
  2. 欢迎对该工具包提出任何宝贵意见和建议，请发邮件至jingjingxu@pku.edu.cn。

论文引用

该代码包主要基于以下科研论文，如使用了本工具，请引用以下论文：


@article{pkuseg,
  author = {Luo, Ruixuan and Xu, Jingjing and Zhang, Yi and Zhang, Zhiyuan and Ren, Xuancheng and Sun, Xu},
  journal = {CoRR},
  title = {PKUSEG: A Toolkit for Multi-Domain Chinese Word Segmentation.},
  url = {https://arxiv.org/abs/1906.11455},
  volume = {abs/1906.11455},
  year = 2019
}

其他相关论文

  • Xu Sun, Houfeng Wang, Wenjie Li. Fast Online Training with Frequency-Adaptive Learning Rates for Chinese Word Segmentation and New Word Detection. ACL. 2012.
  • Jingjing Xu and Xu Sun. Dependency-based gated recursive neural network for chinese word segmentation. ACL. 2016.
  • Jingjing Xu and Xu Sun. Transfer learning for low-resource chinese word segmentation with a novel neural network. NLPCC. 2017.

常见问题及解答

  1. 为什么要发布pkuseg？
  2. pkuseg使用了哪些技术？
  3. 无法使用多进程分词和训练功能，提示RuntimeError和BrokenPipeError。
  4. 是如何跟其它工具包在细领域数据上进行比较的？
  5. 在黑盒测试集上进行比较的话，效果如何？
  6. 如果我不了解待分词语料的所属领域呢？
  7. 如何看待在一些特定样例上的分词结果？
  8. 关于运行速度问题？
  9. 关于多进程速度问题？

致谢

感谢俞士汶教授（北京大学计算语言所）与邱立坤博士提供的训练数据集！

作者

Ruixuan Luo （罗睿轩）, Jingjing Xu（许晶晶）, Xuancheng Ren（任宣丞）, Yi Zhang（张艺）, Zhiyuan Zhang（张之远）, Bingzhen Wei（位冰镇）， Xu Sun （孙栩）

北京大学 语言计算与机器学习研究组