The Wayback Machine - https://web.archive.org/web/20200812025739/https://github.com/topics/tokenization

#

tokenization

Here are 219 public repositories matching this topic...

VKCOM / YouTokenToMe

Star

Unsupervised text tokenizer focused on computational efficiency

nlp natural-language-processing word-segmentation tokenization bpe

Updated Feb 13, 2020
C++

RavenProject / Ravencoin

Star

Ravencoin Core integration/staging tree

bitcoin raven blockchain token asset tokenization ravencoin

Updated Aug 7, 2020
C

yooper / php-text-analysis

Star

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

nlp php text-analysis php-language tokenization php-text-analysis

Updated Jul 27, 2020
PHP

cbaziotis / ekphrasis

Star

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization

Updated Aug 5, 2020
Python

adobe / NLP-Cube

Star

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

parse machine-translation embeddings information-extraction dependency-parser universal-dependencies part-of-speech-tagger dependency-parsing tokenization lemmatization sentence-splitting nlp-cube language-pipeline

Updated May 5, 2020
Python

macmade / ClangKit

Star

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

c syntax-highlighting c-plus-plus parsing objective-c code llvm static-analysis clang source diagnostics tokenization

Updated May 9, 2017
C

CodeChain-io / codechain

Star

CodeChain's official implementation in Rust.

rust blockchain asset tokenization digital-securities

Updated Aug 11, 2020
Rust

explosion / spacy-streamlit

Star

👑 spaCy building blocks and visualizers for Streamlit apps

nlp machine-learning natural-language-processing text-classification spacy visualizer named-entity-recognition ner dependency-parsing tokenization word-vectors visualizers streamlit part-of-speech-tagging

Updated Jul 26, 2020
Python

AmoDinho / datacamp-python-data-science-track

Star

All the slides, accompanying code and exercises all stored in this repo. 🎈

python nlp data-science natural-language-processing neural-network scikit-learn pandas datascience neural-networks bokeh machinelearning tokenization datacamp-course datacamp datacamp-exercises datacamp-projects datacamp-solutions-python datacamp-python datacamp-machine-learning

Updated Jul 9, 2020
Python

razdel

natasha / razdel

Star

Rule-based token, sentence segmentation for Russian language

python nlp russian tokenization sentence-segmentation sentence-boundary-detection

Updated Jul 2, 2020
Python

OpenNMT / Tokenizer

Star

Fast and customizable text tokenization library with BPE and SentencePiece support

python unicode natural-language-processing cpp icu tokenizer machine-translation tokenization bpe sentencepiece

Updated Aug 3, 2020
C++

rth / vtext

Star

Simple NLP in Rust with Python bindings

nlp information-retrieval bag-of-words tf-idf tokenization

Updated Jul 22, 2020
Rust

AlphaWallet / TokenScript

Star

TokenScript schema, specs and paper

security cryptography mobile xml blockchain tokens web3 tokenization tokenisation

Updated Aug 7, 2020
CSS

adamshamsudeen / Vaaku2Vec

Star

Language Modeling and Text Classification in Malayalam Language using ULMFiT

text-classification word2vec malayalam language-model fastai tokenization ulmfit

Updated Mar 31, 2020
Jupyter Notebook

sorami / sudachi.rs

Star

A Japanese morphological analyzer: An unofficial Sudachi clone in Rust 🦀

segmentation pos-tagging morphological-analysis tokenization sudachi nlp-libary

Updated Dec 20, 2019
Rust

wongnai / wongnai-corpus

Star

Collection of Wongnai's datasets

nlp datasets nlp-machine-learning tokenization

Updated Aug 26, 2019

JuliaText / WordTokenizers.jl

Star

High performance tokenizers for natural language processing and other related tasks

nlp information-retrieval data-mining lexer tokenization

Updated Jul 20, 2020
Julia

liuzl / ling

Star

Natural Language Processing Toolkit in Golang

nlp spacy opencc tokenization normalization lemmatization corenlp

Updated May 9, 2020
Go

neelkamath / spacy-server

Star

🦜 Containerized HTTP API for industial-strength NLP via spaCy and sense2vec

python nlp docker spacy named-entity-recognition sense2vec part-of-speech-tagger tokenization sentence-segmentation

Updated Jul 30, 2020
Python

GoogleCloudPlatform / dlp-dataflow-deidentification

Star

Multi Cloud Data Tokenization Solution By Using Dataflow and Cloud DLP

bigquery data beam dataflow tokenization dlp pii

Updated Aug 11, 2020
Java

vaulty-co / vaulty

Star

Tokenize, encrypt/decrypt, mask your data on the fly with Vaulty proxy

security encryption proxy tokenization

Updated Jul 24, 2020
Go

winkjs / wink-tokenizer

Star

Multilingual tokenizer that automatically tags each token with its type

multilingual german tokenizer tagging latin french hindi wink devanagari marathi tokenization konkani

Updated Jul 16, 2020
JavaScript

bastienbot / nlp-js-tools-french

Star

POS Tagger, lemmatizer and stemmer for french language in javascript

nlp tokenizer postgresql stemmer lemmatizer tokenization stemming lemmatization postagging

Updated Sep 13, 2017
JavaScript

rosette-api / python

Star

Rosette API Client Library for Python

python nlp machine-learning natural-language-processing text-mining sentiment-analysis text morphology text-analysis language-detection fuzzy-matching name-generation tokenization categorization lemmatization relation-extraction entity-extraction language-identification name-translation name-similarity

Updated Jun 16, 2020
Python

manorie / textoken

Star

Simple and customizable text tokenization gem.

ruby nlp tokenization rubynlp

Updated May 30, 2019
Ruby

alm

anyks / alm

Star

Smart Language Model

cpp tokenizer language-models arpa alm tokenization vocab-pruning

Updated Aug 6, 2020
C++

PyThaiNLP / attacut

Star

A Fast and Accurate Neural Thai Word Segmenter

nlp cnn tokenization

Updated Aug 8, 2020
Python

aatimofeev / spacy_russian_tokenizer

Star

Custom Russian tokenizer for spaCy

natural-language-processing russian tokenization spacy-pipeline

Updated May 14, 2019
Python

unicode-cookbook / cookbook

Star

The Unicode Cookbook for Linguists

python unicode r transliteration linguistics ipa phonetics transcription writing-systems tokenization

Updated Sep 14, 2018
TeX

clipperhouse / uax29

Star

A tokenizer based on Unicode text segmentation (UAX 29), for Go

unicode tokenization uax29

Updated Jul 9, 2020
Go

Improve this page

Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.