全部

1·Why do I need a tokenizer for each language?

为什么我需要为每种语言的标记？

2·With the tokenizer, it is easy to decode the processing instructions.

有了 tokenizer，就很容易译码处理指令。

3·If the tokenizer encounters neither token, it returns a lexical error.

如果记号赋予器未遇到这两个记号，则返回词法错误。

4·A good lexer example can help a lot with learning how to write a tokenizer.

一个好的 lexer 例子会非常有助于学习如何编写断词器（tokenizer）。

5·What the parser fills in depends on the path it takes through the tokenizer.

解析器填入的内容取决于它通过记号赋予器所采用的路径。

6·Fortunately, gnosis.indexer.TextSplitter is a robust tokenizer. For example.

幸运的是，gnosis . indexer . TextSplitter是一个健壮的断词工具。

7·Not surprisingly, the part of the parser that's responsible for that task is called the scanner or tokenizer.

解析器中负责该任务的部件称为扫描器(scanner)或记号赋予器(tokenizer)一点也不稀奇。

8·The process that scans an input string and determines what the tokens are is called the tokenizer. In the query.

扫描输入字符串以及判断是何标记的过程称作记号赋予器(tokenizer)。

9·The tokenizer moves through the stream, inspecting characters one at a time until it encounters either an integer or the end of file.

记号赋予器穿过输入流，每次检查一个字符，直到它遇到一个整数或者直至文件结束。

10·Quantifiers can be used within the regular expressions of the Spark tokenizer, and can be simulated by recursion in parse expression grammars.

计量符可以在Spark记号赋予器(tokenizer)的正则表达式中使用，并可以用解析表达式语法中的递归来进行模拟。