I Built My Own Programming Language and its Virtual Machine In Python!

#python #learning #showdev #opensource

Ever looked at your code and wondered what really happens when you hit "run"? I mean, deep down, past the libraries and frameworks, how does your if statement actually... if? That curiosity bug bit me hard, and it led me down a rabbit hole that ended with me building my own programming language and virtual machine from scratch, using nothing but Python. I call it Pyle.

So, Why Build a Whole Language?

Pyle isn't here to compete with Python, JavaScript, or any of the giants. The world probably doesn't need another programming language from me (not to mention written in Python). But I needed to build one. I wanted to tear away the layers of abstraction and see the guts. How does text turn into something a computer understands? How does a + b actually work?

Pyle is my personal laboratory for exploring these questions. It’s an educational project, first and foremost.

But this isn't my first time creating a language. My journey started with CupScript, a classic tree-walk interpreter. I remember piecing it together, trying to absorb every detail from a YouTube series, making tweaks here and there but never feeling quite satisfied. Then came FCL – more of a joke, really – where I took "functional programming language" so literally that everything was a function. Good times.

All of these were in Python, the language I knew best back then. But the itch to see how these concepts would perform in something faster was always there. That led to my third attempt, DUSL. I had ambitions for DUSL, maybe for scripting or small automation tasks. Life, as it does, got in the way, and a million bugs remained unfixed.

Through all this, I was aware of stack-based virtual machines. I knew they were the engine behind many interpreted languages and generally faster than just walking an AST. But I hadn't dived deep. After finally taking that plunge and learning more, I started prototyping in Python again. And that’s how Pyle came to be. It's a stepping stone, really. The grand plan is to eventually rewrite Pyle in Go, both for that performance boost and to deepen my own understanding of Go.

From Text File to Execution

The Lexer (or Tokenizer): This is the first stop. Imagine it as a meticulous librarian who reads your .pyle script character by character and groups them into meaningful "words" or tokens. So, let x = 10; becomes something like KEYWORD(let), IDENTIFIER(x), OPERATOR(=), NUMBER(10), PUNCTUATION(;). No understanding of what it means yet, just breaking it down.
The Parser: Now that we have tokens, the parser steps in. This stage is like a grammar expert. It takes the stream of tokens and tries to build a structured representation, an Abstract Syntax Tree (AST). Think of it as an upside-down tree where each node represents a part of your code's structure – an assignment, an expression, a function call. If your code doesn't make grammatical sense (like let = x 10), the parser throws a fit.
The Compiler: This is where the magic starts to get really interesting. The compiler walks through the AST and translates it into a simpler, more machine-friendly language: Pyle Bytecode. This isn't the machine code your CPU runs directly, but a custom set of instructions I designed specifically for Pyle's own little virtual brain. For instance, echo(a + b) might become a sequence of opcodes like PUSH_VAR(a), PUSH_VAR(b), ADD, CALL_BUILTIN(echo). Seeing my high-level Pyle code turn into these compact instructions was a huge "aha!" moment.
The Stack-Based Virtual Machine (VM): This is the engine that actually runs the bytecode. My Pyle VM is a stack-based machine. Imagine a stack of plates: it pushes values onto this stack, performs operations (like ADD which takes the top two values, adds them, and pushes the result back), manages function calls, and generally brings the bytecode to life. It reads each bytecode instruction and does what it says.

What Can Pyle Actually Do?

It's still a learning project, but Pyle can handle the basics:

Variables (let for mutable, const for immutable)
Numbers, strings, booleans, and even arrays (lists)
The usual arithmetic (+, -, *, /), logical (and, or, not), and comparison operators
if/else statements, while loops, and for-in loops (with ranges for now)
Defining and calling your own functions, complete with return statements and even keyword arguments!
A really fun one: importpy("module_name") lets you import actual Python modules. So, const math = importpy("math"); echo(math.pi); totally works! Most of this interactivity with python types is possible because of pythons dynamicity.
Some built-ins like echo(), len(), scan() for input, and perf_counter().

A Quick Showcase

This example defines a function to check for prime numbers and then uses importpy to get a random number to test.
Using JavaScript syntax highlighter as the syntax is very similar.

// Function to check if a number is prime
fn is_prime(n) {
    if n <= 1 {
        return false;
    }
    if n <= 3 {
        return true;
    }
    if n % 2 == 0 or n % 3 == 0 {
        return false;
    }
    let i = 5;
    while i * i <= n {
        if n % i == 0 or n % (i + 2) == 0 {
            return false;
        }
        i = i + 6;
    }
    return true;
}

// Let's use a Python module to get a random number
const random = importpy("random");
let test_num = random.randint(1, 100);

echo("Checking if", test_num, "is prime...");

if is_prime(test_num) {
    echo(test_num, "is a prime number!");
} else {
    echo(test_num, "is not a prime number.");
}

echo("--- Primes up to 20 ---");
for i in 0:21 {
    if is_prime(i) {
        echo(i);
    }
}

The Real Payoff: Understanding

Building Pyle has been an incredible learning experience. It’s one thing to read about compilers or VMs, and another entirely to wrestle with operator precedence in your parser or debug your VM's stack handling. It’s given me a much deeper appreciation for the tools and languages I use every day.

If you've ever been curious about what goes on under the hood, I can't recommend a project like this enough. It doesn't have to be perfect or feature-complete. The goal is the journey and the understanding you gain along the way.

The whole thing is up on GitHub if you want to check it out: https://github.com/Fus3n/pyle-python