C++
Python
SentencePiecegoogle/sentencepiece
A C++ library for unsupervised text tokenization and detokenization, widely used in modern NLP models.
Deep learningNLP
12.2k updated 4d ago
colibri-core
proycon/colibri-core
C++ library, command line tools, and Python binding for extracting and working with basic linguistic constructions such as n-grams and skipgrams in a quick and memory-efficient way.
- c-plus-plus
- python
- nlp
- ngrams
- skipgram
- ngram
- corpus
- linguistics
More in C++
MIT Information Extraction Toolkitmit-nlp/MITIE
C, C++, and Python tools for named entity recognition and relation extraction
NLP
3k updated 1y ago
MeTAmeta-toolkit/meta
MeTA : ModErn Text Analysis is a C++ Data Sciences Toolkit that facilitates mining big text data.
NLP
719 updated 3y ago
CRFsuitechokkan/crfsuite Deprecated
CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
NLP
674 updated 2y ago
CRF++taku910/crfpp Deprecated
Open source implementation of Conditional Random Fields (CRFs) for segmenting/labeling sequential data & other Natural Language Processing tasks.
NLP
514 updated 1y ago
BLLIP ParserBLLIP/bllip-parser
BLLIP Natural Language Parser (also known as the Charniak-Johnson parser).
NLP
226 updated 4y ago
frogLanguageMachines/frog
Memory-based NLP suite developed for Dutch: PoS tagger, lemmatiser, dependency parser, NER, shallow parser, morphological analyzer.
NLP
84 updated 14d ago
uctoLanguageMachines/ucto
Unicode-aware regular-expression based tokenizer for various languages. Tool and C++ library. Supports FoLiA format.
NLP
72 updated 2mo ago