Language parsing

A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.

Repositories 339

theseer / tokenizer

A small library for converting tokenized PHP source code into XML (and potentially other formats)

php tokenizer xml

PHP Updated Apr 3, 2019

SAP / chevrotain

Parser Building Toolkit for JavaScript

javascript typescript parser-library parsing grammars tokenizer open-source

TypeScript Updated Apr 3, 2019 7 issues need help

mathewsanders / Mustard

684

🌭 Mustard is a Swift library for tokenizing strings when splitting by whitespace doesn't cut it.

swift substrings tokenizer

Swift Updated May 14, 2018

JohnSnowLabs / spark-nlp

602

Natural Language Understanding Library for Apache Spark.

nlp nlu natural-language-processing natural-language-understanding spark spark-ml pyspark machine-learning named-entity-recognition sentiment-analysis lemmatizer spell-checker tokenizer entity-extraction stemmer part-of-speech-tagger annotation-framework

Scala Updated Apr 2, 2019 2 issues need help

ikawaha / kagome

362

Self-contained Japanese Morphological Analyzer written in pure Go

japanese tokenizer nlp-library japanese-language pos-tagging segmentation morphological-analysis

Go Updated Feb 16, 2019

open-korean-text / open-korean-text

336

Open Korean Text Processor - An Open-source Korean Text Processor

korean korean-text-processing natural-language-processing text-processing tokenizer korean-tokenizer

Scala Updated Aug 7, 2018

lovit / soynlp

307

한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.

korean-nlp korean-text-processing nlp tokenizer postagging word-extraction

Python Updated Mar 8, 2019

glayzzle / php-parser

266

🌿 NodeJS PHP Parser - extract AST or tokens (PHP5 and PHP7)

php-parser ast lexer javascript parser tokenizer static-code-analysis development php php-ast

JavaScript Updated Apr 3, 2019 3 issues need help

jflex-de / jflex

256

The fast scanner generator for Java with full Unicode support

Java Updated Mar 12, 2019 9 issues need help

neurosnap / sentences

253

A multilingual command line sentence tokenizer in Golang

sentence-tokenizer tokenizer cli sentences

Go Updated Feb 7, 2019 1 issue needs help

timtadh / lexmachine

243

Lex machinary for go.

go regular-expression nfa lexer tokenizer lexical-analysis-engines lexical-analysis-framework dfa lex

Go Updated Feb 4, 2019

lionsoul2014 / friso

207

High performance chinese tokenizer with both GBK and UTF-8 charset support developed by ANSI C

c tokenizer chinese-tokenizer php-tokenizer full-text-search chinese-word-segmentation

C Updated Sep 8, 2017

cbaziotis / ekphrasis

177

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis…

nlp text-processing nlp-library spelling-correction tokenizer tokenization word-segmentation word-normalization spell-corrector text-segmentation semeval

Python Updated Nov 22, 2018

no-context / moo

167

Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.

javascript lexer tokenizer regexp

JavaScript Updated Feb 27, 2019

ku-nlp / jumanpp

143

Juman++ (a Morphological Analyzer Toolkit)

juman nlp japanese morphological-analysis morphological-analyser pos-tagging pos-tagger part-of-speech-tagger word-segmentation cjk tokenizer

C++ Updated Jan 23, 2019 5 issues need help

ropensci / tokenizers

126

Fast, Consistent Tokenization of Natural Language Text

text-mining tokenizer rstats nlp r r-package peer-reviewed

R Updated Oct 22, 2018

nik-garmash / works-for-me

126

Collection of developer toolkits

parser tokenizer developer-tools development-environment development-workflow lexer developer-toolkit devtools

JavaScript Updated May 25, 2018

the-new-sky / Kadot

104

Kadot, the unsupervised natural language processing library.

natural-language-processing tokenizer word-embeddings text-classification text-generation

Python Updated Jun 22, 2018

bnosac / udpipe

102

R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural…

udpipe nlp natural-language-processing r r-package pos-tagging dependency-parser tokenizer rcpp r-pkg conll lemmatization text-mining

C++ Updated Feb 15, 2019

lydell / js-tokens

101

A regex that tokenizes JavaScript.

regex javascript tokenizer ecmascript

JavaScript Updated Jun 14, 2018

foonathan / lex

100

C++14 fast and efficient tokenizer

tokenizer lexer cplusplus

C++ Updated Feb 12, 2019

bevacqua / megamark

😻 Markdown with easy tokenization, a fast highlighter, and a lean HTML sanitizer

markdown tokenizer

JavaScript Updated Jul 12, 2016

nette / tokenizer

Source code tokenizer

nette nette-framework php tokenizer regular-expression

PHP Updated Mar 27, 2019

venturachrisdev / djurl

Simple yet helpful library for writing Django urls by an easy, short and intuitive way.

python django web url tokenizer regex routing django-routing urls

Python Updated Sep 24, 2018

HippoPHP / Hippo

PHP standards checker.

php ast code-quality code-style tokenizer

PHP Updated Aug 12, 2017

ropensci / hunspell

High-Performance Stemmer, Tokenizer, and Spell Checker for R

spellchecker hunspell stemmer tokenizer spell-check r rstats r-package

C++ Updated Dec 21, 2018

clipperhouse / jargon

Tokenizers and lemmatizers for Go

lemmatizer nlp data-science tokenizer go

Go Updated Nov 7, 2018

howl-anderson / MicroTokenizer

一个微型&算法全面的中文分词引擎 | A micro tokenizer for Chinese

chinese-nlp tokenizer chinese-tokenizer dag-network nlp-machine-learning

Python Updated Dec 13, 2018

janlelis / wirb

Don't use an IRB without WIRB!

irb ruby syntax-highlighting terminal tokenizer stdlib

Ruby Updated Feb 19, 2018

GerHobbelt / jison

bison / YACC / LEX in JavaScript (LALR(1), SLR(1), etc. lexer/parser generator)

lexer parser lalr lr lr1 slr slr1 lalr1 flex yacc bison lex parsing tokenizer

Forked from zaach/jison JavaScript Updated Jan 19, 2019

Learn about parsing

Wikipedia: Wikipedia

Loading…