Natural language processing libraries

144 libraries, 49 updated this year

Libraries for working with text. Hugging Face Transformers covers modern model-based NLP, spaCy is the production toolkit for tokenization, tagging, parsing and named entities, and NLTK remains the teaching standard with corpora and classic algorithms. gensim does topic modelling and word vectors, and SentencePiece and Hugging Face tokenizers split text into subwords for model training.

144 libraries

Transformershuggingface/transformers

A deep learning library containing thousands of pre-trained models on different tasks. The goto place for anything related to Large Language Models.

PythonDeep learningLLMsSpeech

168.1k updated today

jiebafxsjy/jieba

Chinese Words Segmentation Utilities.

Python

35.2k updated 2y ago

spaCyexplosion/spaCy

Industrial strength NLP with Python and Cython.

Python

34k updated 1d ago

Haystackdeepset-ai/haystack

A framework for building industrial-strength applications with Transformer models and LLMs.

PythonLLMs

26.8k updated today

CometLLMcomet-ml/comet-llm

Track, log, visualize and evaluate your LLM prompts and prompt chains.

PythonLLMs

22.9k updated today

RasaRasaHQ/rasa

A "machine learning framework to automate text-and voice-based conversations."

Python

21.4k updated 3mo ago

Weaviatesemi-technologies/weaviate

Weaviate is an open source vector search engine and vector database. Weaviate uses machine learning to vectorize and store data, and to find answers to natural language queries. With Weaviate you can also bring your custom ML models to production scale.

LLMsMLOps

16.9k updated today

gensimRaRe-Technologies/gensim

Topic Modelling for Humans.

PythonGeneral

16.5k updated 11mo ago

numpy-MLddbourgin/numpy-ml

Reference implementations of ML models written in numpy

PythonGeneralDeep learning

16.3k updated 2y ago

NLTKnltk/nltk

A leading platform for building Python programs to work with human language data.

Python

14.8k updated today

txtaineuml/txtai

Build semantic search applications and workflows.

LLMsMLOps

13k updated 2d ago

NLP Compromisespencermountain/compromise

Natural Language processing in the browser.

JavaScript

12.2k updated 5d ago

SentencePiecegoogle/sentencepiece

A C++ library for unsupervised text tokenization and detokenization, widely used in modern NLP models.

C++Deep learning

12.2k updated 4d ago

huggingface/tokenizershuggingface/tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

RustDeep learning

11.2k updated 1d ago

naturalNaturalNode/natural

General natural language facilities for node.

JavaScript

10.9k updated today

AutoGluonawslabs/autogluon

AutoML for Image, Text, Tabular, Time-Series, and MultiModal Data.

PythonGeneralDeep learningVisionMLOps

10.8k updated 2d ago

CoreNLPstanfordnlp/CoreNLP

Stanford CoreNLP provides a set of natural language analysis tools which can take raw English language text input and give the base forms of words.

Java

10.1k updated today

TextBlobsloria/TextBlob

Providing a consistent API for diving into common natural language processing (NLP) tasks. Stands on the giant shoulders of NLTK and Pattern, and plays nicely with both.

Python

9.6k updated 4d ago

Fuzzy Wuzzyseatgeek/fuzzywuzzy Archived

Fuzzy String Matching in Python.

Python

9.2k archived

Patternclips/pattern

A web mining module for the Python programming language. It has tools for natural language processing, machine learning, among others.

PythonGeneralData & viz

8.9k updated 2mo ago

DeepPavlovdeepmipt/DeepPavlov

Conversational AI library with many pre-trained Russian NLP models.

PythonDeep learning

7k updated 1y ago

pkuseg-pythonlancopku/pkuseg-python

A better version of Jieba, developed by Peking University.

Python

6.7k updated 3y ago

SnowNLPisnowfy/snownlp

A library for processing Chinese text.

Python

6.6k updated 6y ago

nlp.jsaxa-group/nlp.js

An NLP library built in node over Natural, with entity extraction, sentiment analysis, automatic language identify, and so more.

JavaScript

6.6k updated 1y ago

Smilehaifengl/smile

Statistical Machine Intelligence & Learning Engine.

GeneralData & viz

6.4k updated today

shimmyMichael-A-Kuykendall/shimmy

Python-free Rust inference server for NLP models with OpenAI API compatibility and hot model swapping.

RustLLMs

6k updated 5d ago

Knwl.jsloadfive/Knwl.js

A Natural Language Processor in JS.

JavaScript

5.2k updated 3y ago

Dedupededupeio/dedupe

A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

Python

4.5k updated 1y ago

DrQAfacebookresearch/DrQA Archived

Reading Wikipedia to answer open-domain questions.

Python

4.5k archived

Spark NLPJohnSnowLabs/spark-nlp

Natural language processing library built on top of Apache Spark ML to provide simple, performant, and accurate NLP annotations for machine learning pipelines, that scale easily in a distributed environment.

Scala

4.2k updated 1d ago

Snips NLUsnipsco/snips-nlu

Natural Language Understanding library for intent classification and entity extraction

Python

4k updated 3y ago

ScalaNLPscalanlp/breeze

ScalaNLP is a suite of machine learning and numerical computing libraries.

Scala

3.5k updated 1y ago

Twitter Text Javatwitter/twitter-text

A Java implementation of Twitter's text processing library.

3.1k updated 2y ago

rust-bertguillaume-be/rust-bert

Rust native ready-to-use NLP pipelines and transformer-based models (BERT, DistilBERT, GPT2,...)

RustDeep learning

3.1k updated 9mo ago

MIT Information Extraction Toolkitmit-nlp/MITIE

C, C++, and Python tools for named entity recognition and relation extraction

C++

3k updated 1y ago

Retextretextjs/retext

Extensible system for analyzing and manipulating natural language.

JavaScript

2.4k updated 1y ago

PolyglotaboSamoor/polyglot

Multilingual text (NLP) processing toolkit.

Python

2.4k updated 2y ago

textacychartbeat-labs/textacy

Higher-level NLP built on Spacy.

Python

2.2k updated 3y ago

jellyfishjamesturk/jellyfish

A python library for doing approximate and phonetic matching of strings.

Python

2.2k updated 8d ago

Spagonlpodyssey/spago

Self-contained Machine Learning and Natural Language Processing library in Go.

GoGeneralDeep learning

1.9k updated 1y ago

PyTorch Lightning BoltsPyTorchLightning/pytorch-lightning-bolts Archived

Toolbox of models, callbacks, and datasets for AI/ML researchers.

PythonGeneralDeep learningVision

1.8k archived

NeuroNERFranck-Dernoncourt/NeuroNER

Named-entity recognition using neural networks providing state-of-the-art-results

PythonDeep learning

1.7k updated 3y ago

NobodyWhonobodywho-ooo/nobodywho

The simplest way to run an LLM locally. Supports tool calling and grammar constrained sampling.

PythonLLMs

1.6k updated today

OpenNLPapache/opennlp

A machine learning based toolkit for the processing of natural language text.

Java

1.6k updated today

KoNLPykonlpy/konlpy

A Python package for Korean natural language processing.

Python

1.5k updated 3y ago

Tribouoracle/tribuo

A machine learning library written in Java by Oracle.

JavaGeneralDeep learning

1.4k updated 5mo ago

jieba-phpfukuball/jieba-php

Chinese Words Segmentation Utilities.

PHP

1.4k updated 10mo ago

Treatlouismullie/treat

Text Retrieval and Annotation Toolkit, definitely the most comprehensive toolkit I’ve encountered so far for Ruby.

Ruby

1.4k updated 1y ago

Quepymachinalis/quepy

A python framework to transform natural language questions to queries in a database query language.

Python

1.3k updated 5y ago

Datumboxdatumbox/datumbox-framework

Machine Learning framework for rapid development of Machine Learning and Statistical applications.

JavaGeneral

1.1k updated 2y ago