Questions tagged [lexical-analysis]
Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.
73 questions
9
votes
4
answers
1k
views
Basic, general lexer for a programming language
I've been working on a general lexer for a programming language for a couple days now. I don't know if the code I've written is overcomplicated, if there is a better way to parse the code into tokens ...
- 113
11
votes
6
answers
2k
views
Efficient C Tokenizer/Lexer in C++
I wanted to create my own compiler for C and it's the first prototype of the Lexer. Any thoughts on improvements?
Note: I forgot to add comment lines in header files while I was coding, but I assure ...
- 113
8
votes
1
answer
351
views
C++ - Lexer for the Monkey Programming Language from the book "Writing An Interpreter In Go"
Below is a C++ version of a lexer for the Monkey programming language, originally written in Go, as presented in the book Writing An Interpreter In Go.
LOC: 262. (Excluding the tests.)
Language:
Copy ...
- 10.1k
4
votes
1
answer
163
views
C99 - Lexer for the Monkey Programming Language from the book "Writing An Interpreter In Go"
Below is a C version of a lexer for the Monkey programming language, originally written in Go, as presented in the book Writing An Interpreter In Go.
LOC: 371. (Excluding the tests.)
Language:
Copy ...
- 10.1k
-4
votes
2
answers
96
views
How to refactor "long" if statment in lexer? [closed]
I have a Tokenizer doing lexer thing.
It is used like
String line = "class Main {"
...
- 1
3
votes
1
answer
283
views
ANTLR4 grammar for Conventional Commits spec
I would like to create a grammar for the Conventional Commits spec and I would love to hear any feedback for what I wrote.
The spec has some ambiguities, I think, hence my usage of "island ...
- 133
6
votes
2
answers
235
views
Programming Language Source Lexer
I have a lexer (program that turns source code into tokens) written in Go that I am re-writing in Rust.
The lexer should take in a sequence of tokens, generally different special characters, integers, ...
- 163
2
votes
1
answer
190
views
Rust state-machine lexer
I tried implementing a lexer in rust that peeks ahead at the next character and makes a decision based on that.
However, i am told that this is bad practice, and instead i should be using finite-state-...
- 67
0
votes
1
answer
194
views
supporting unicode in rust lexer
I am working on a lexer in rust.
It has been through many different iterations, going from iterating over a Vec<char> to ...
- 67
4
votes
1
answer
466
views
lexer for programming languages in rust
I'm relatively new to rust, so you don't need to go easy on the criticism.
right now, this lexes 1MB of text (well, trims a 1MB-long word) in 30ms.
is there any way I can make this faster?
cursor.rs:
<...
- 67
0
votes
1
answer
158
views
top-down lexer in rust
the other day, i decided to build a top-down lexer in rust, just for fun.
this is what i have so far:
...
- 37
3
votes
1
answer
336
views
Generating indent and dedent tokens for python-style indenting code
I found lexy, a C++ template-based parser combinator, that helps a lot to build a language parser.
The problem is that lexy can only deal with context free grammar, and python indentation is not ...
- 293
2
votes
1
answer
174
views
Simple Manual Lexer in Haskell
As a beginner exercise, I made small manual lexer that recognizes three types of inputs:
integers: /[-]?[0-9]+/
strings, inside double quotes, with backslash ...
- 121
2
votes
1
answer
130
views
Lexer for shell-like language [rust]
I am working on writing a rust implementation of a sh-like language.
Rather than posting 600 lines of code here, you can just go to the GitHub repository.
Is this a ...
- 212
2
votes
1
answer
246
views
C# Language Lexer
Here is a Lexer for a programming language I am working on. Any feedback would be appreciated. I only started learning C# a couple of days ago, so please excuse my newbie code :)
...
- 135