From https://en.wikipedia.org/wiki/Tokenization_(lexical_analysis)

In lexical analysis, tokenization is the process of breaking a stream of text up into words, phrases, symbols, or other meaningful elements called tokens.

Usually, we call a function/tool which split text into words as tokenizer. So, is there any word used to denote split text into sentences?


1 Answer 1


Tokenization is the process of splitting a string into a list of pieces or tokens. (Words, sentences, phrases, chars or any symbol as you mentioned)

There are Sentence Tokenizers and Word Tokenizers [Source]

A Sentence tokenizer can be used to find the list of sentences and Word tokenizer can be used to find the list of words in strings.

There's no exact term that means "split text into sentences" but there are other possible names for that Sentence boundary disambiguation, Sentence boundary detection, Sentence segmentation, and Sentence breaking.

  • OK, it is a sad news.I ask this question is because tokenizer is translated as 分词器 (分 split 词 word 器 tool) in Chinese, which make me think maybe there is a corresponding word for split sentence tool in English . Sentence tokenizer have two long words looks a little cumbersome .
    – Mithril
    Apr 27, 2017 at 8:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .