Domain-Specific Languages in Theia
Build a Cloud-Based DSL Workbench With Open Source
--
This article is outdated. For an updated version also covering VS Code please read on here.
Domain-specific languages (DSLs) allow to formally capture your knowledge about a certain domain and transform it directly into executable code. A DSL is a very powerful tool that lets you define abstractions even across technological boundaries. Frameworks like Xtext help you to implement your own textual DSLs including a smart editor, a validator, a code generator, and a lot of additional sugar with little effort. Still DSLs are kind of heavy, so please consider them the last resort for solving your abstraction problems.
This article shows how to build a cloud-based DSL tool. We will exclusively use open-source technologies (all hosted at Eclipse), in particular
- Xtext to define the textual DSL and generate a Language Server (LS) for it,
- Theia as the a Cloud-IDE framework,
- Sprotty to visualize the DSL in diagrams, and
- the Eclipse Layout Kernel for auto-layouting these diagrams.
The final workbench looks like this:
You can find the code on GitHub. The repo has the individual steps nicely sorted in separate commits. There are four major stages, each of them marked as a branch. Consider that repository a comprehensive example showing how to embed Xtext in Theia, or how to implement diagrams on top of Xtext with Sprotty.
I’ve added Gitpod links, so you can checkout, browse, build, run, and modify each step of this tutorial in the cloud right from your browser, without having to install any prerequisite software on your local machine. Just click on the link, wait for the build to finish and open the new IDE by clicking on the Open in Browser button. To run it without Gitpod, e.g. as a desktop tool, have a look at the readme.
Stage 1: Define a Language Server For the Textual DSL
We are using Xtext to define a simple domain-specific language for statemachines. In Xtext, you capture the syntax, the structure of the AST, and the cross-references of the language in an enhanced grammar notation. From that, the framework generates a lot of stuff, including a parser, a linker, smart editing support, stubs for services like code-generation and validation, and, in this case, a language server (LS) that allows to plug in all of that into various IDEs.
We create a new Xtext project (in Eclipse or using the CLI tooling) with LSP support and write a grammar for our language.
We feed that grammar into Xtext’s code generator and… get a working LS already! You can customize almost everything in Xtext. In this case, we add a parser test, change the linking to only look at elements in the same file, implement a code generator to transform a statemachine model into a Java class, and add a few model constraints to be checked by the validator.
Stage 2: Hook the Language Server Up To Theia
Theia is an open-source cloud and desktop IDE framework written in TypeScript. Like VS Code, Theia relies on the Language Server Protocol to support new languages. Theia clearly separates a backend (the language servers, the filesystem, the dev tools, etc.) from a frontend (UI), which is a prerequisite to run in the cloud. Each Theia application is composed of extensions, which means we have to wrap our language server into a Theia extension.
We use the yeoman generator to create the skeleton for a new Theia extension and an application launcher for each, a cloud IDE or a desktop IDE. We copy our LS from the previous stage into the new extension and register the language to Theia’s frontend and backend, such that the LS is automatically started when the user opens a statemachine file in an editor. Syntax coloring, folding, and bracket matching is not covered by the LSP, so we add that manually via a TextMate grammar and some JSON configuration.
The resulting IDE (run it in Gitpod) is already pretty cool: You can edit statemachine files (file extension .sm) with syntax highlighting, validation of syntax, linking, and constraint errors, content assist, navigation, automatic incremental code generation, etc. Thanks to other standard Theia extensions, we can version our textual DSL files with Git, or integrate the generated code in a bigger Java application.
Now, let’s make it even better add graphics.
Stage 3: Visualizing Statemachines in Diagrams
Statemachines are a lot about the connections (transitions) between the elements (states), and thereby a natural fit for graphs. With the Sprotty framework, we can extend the LS to also provide a graphical model of the statemachines and display that in a diagram widget in the UI. This way, we keep the LSP’s separation of the generic frontend and the language-aware backend.
Enhancing the language server is mostly about changing some base classes, adding custom launcher, and implementing a diagram generator, that transforms a statemachine model into a Sprotty graph model. In the client, we add a diagram manager which acts as a factory for the diagram widget, and a stylesheet. Sprotty uses the same dependency injection framework as Theia, but while Theia has a single DI container for the entire application, Sprotty has one for each diagram instance. This is why we have to separate the diagram configuration from the application contribution.
In the next second step, we use the ELK framework to place the nodes and route the edges in a nice way. ELK is written in Java, so we do that in the backend. With its generic action flow, Sprotty allows to execute such tasks arbitrarily on the client or the server.
We then add arrow heads and fine-tune the visual appearance beyond the capabilities of CSS. By Sprotty’s defaults, the diagram is already automatically updated on changes in the associated text. To add even more coupling, we synchronize the selections in the editor and the diagram by adding trace information to the graphical model. Finally we show errors from the text as marker bubbles in the diagram.
Take it for a spin (run it in Gitpod), open the .sm file in the diagram by choosing Open in diagram from the context menu, and don’t miss the smooth animations that Sprotty automatically uses to morph the diagram to a new state.
The diagram is still read only. You should stop here. Don’t read any further. Dragons ahead.
Stage 4: Diagram Editing (beta)
Once you see the nice graphics you may ask yourself why you cannot edit the model in the graphical notation. Or, as you have the textual editor already, in both notations at the same time, whatever fits best.
Implementing such combined editing is hard. Really hard. A textual editor takes the text as the source of truth and derives a model from it. A diagram editor usually goes the other way around. There is a lot of friction caused by different handling of (syntax) errors, of element identities, of transactions etc. This will result in all kinds of problems ranging from small usability glitches to complete data loss.
The best chance to manage editing text and graphics consistently is to choose the same leading model for both editors. Given its big advantages in terms of versioning, editing speed, and fixing a broken model we choose the text. Our approach is to map all user editing actions to text modifications, and, if possible, leverage LSP operations. The diagram is always updated from changes of the text, never the other way around.
Even with these simplifications, there still are some glitches in the current implementation, most likely because missing synchronization step in the Sprotty/Theia architecture. This is why we consider this still beta.
So for our last stage, we add a popup palette that uses LSP’s code actions to create new states and events and a facility to delete elements. We enable Sprotty’s edge editing features, and allow to rename transition labels via LSP code assist and state names via LSP rename.
Try it youself (run it in Gitpod): The palette to create new elements appears when you hover over the diagram. Selected elements can be deleted using DEL or BACKSPACE. To reroute/reconnect an edge, use the handles that pop up when you select it. New edges can be added by dragging the small triangles next to the nodes. Labels can be edited with double-click.
That’s all for today. I hope I could demonstrate what is possible with the combination of these frameworks, and that the repo guides you to set up your own DSL tools. For questions, there are forums for Xtext, Theia, Sprotty, and ELK, or just ask us directly.
This example has also been presented at EclipseCon Europe 2018 (recording, slides).
