Language Explorer is a tool built to help developers explore how programming languages actually work under the hood. Instead of just writing code and seeing the output, this project lets you define tokens and grammar rules, parse a source file, and visualize the resulting Abstract Syntax Tree (AST).
The idea came from tools like Compiler Explorer, which let developers inspect assembly generated by compilers. We wanted to build something similar, but focused more on the language parsing stage. By visualizing how code gets parsed into a syntax tree, you can better understand how programming languages interpret structure, grammar, and syntax.
Features
Custom Token Definitions
Users can define tokens using regular expressions, which represent the smallest pieces of a language (identifiers, numbers, keywords, etc.).
Grammar Rule Input
Users can define grammar rules that describe how tokens combine to form valid language constructs.
Source Code Parsing
Once tokens and grammar rules are defined, the tool parses a provided source file using those rules.
AST Visualization
The parsed result is displayed as an interactive Abstract Syntax Tree, allowing users to visually explore how their source code is structured internally.
Full Stack Architecture
The project connects a modern frontend with a Python backend to create an interactive learning experience for language parsing.
How It Works
- Define Tokens: Users create tokens using regular expressions.
- Define Grammar Rules: Grammar rules describe how tokens combine to form expressions or statements.
- Provide Source Code: A source file is submitted to the parser.
- Parse the Code: The backend processes the input and generates an AST.
- Visualize the Tree: The frontend renders the AST so users can explore the structure interactively.
This process allows developers to experiment with language design concepts and see how different rules affect the structure of parsed code.
Tech Stack
Language Explorer uses a simple but powerful full stack setup:
Frontend
- Vite
- React
- Interactive UI for token and grammar inputs
- AST visualization
Backend
- Python
- FastAPI
- Lark parsing library
The backend generates a grammar from the user inputs and parses the provided source file into an Abstract Syntax Tree, which is then returned to the frontend for visualization.
Thoughts
When Building:
One of the hardest parts of building this project was rendering the interactive AST tree in a way that felt smooth and understandable. Trees can get complex quickly, and finding a good way to display them visually took a lot of experimentation.
Another challenge was generating a grammar from user input. Since users can write tokens and rules in many different ways, the backend had to be flexible enough to interpret those inputs correctly.
Overall this was a really fun project because it touches on an area of programming that most developers don’t normally see — how languages are actually parsed. It was also a great opportunity to work on a full stack system with React, FastAPI, and Python parsing libraries, and it definitely gave me a deeper appreciation for how compilers and interpreters work.
Future Improvements
There are a few ideas that could make Language Explorer even better:
- Adding syntax highlighting to the source code editor
- Displaying more information inside each AST node
- Highlighting the exact section of source code that corresponds to each AST node
- Improving the interactivity of the tree visualization
Overall, Language Explorer has a lot of potential as a learning tool for understanding language parsing, grammars, and syntax trees.