Textual Editing Framework (TEF) Tutorial

In this tutorial we show how to install TEF and how to create your first DSL editor with it. TEF includes an editor template that features a very simple DSL for defining mathematical functions. You learn how the syntax for this language is described, how semantic aspects like name resolution and code-completion are realised. We will extent the pre-defined template with additional features, so that you get an understanding on how to create own editors for your own meta-models.

Installing TEF

This is easy if your familiar with eclipse. You need an eclipse europa with EMF 2.3.0 installed. TEF can be installed using our update site. Use Help/Software Updates/Find and Install/Search for new features to install, create a new remote site, and use http://tef.berlios.de/updatesite as URL and choose a arbitrary name for the site. Make sure that the newly created site is checked and press finish. Select the TEF feature, press next, accept our license, and next, and finish. Confirm all dialogues with yes, especially the restart eclipse dialogue. TEF is now installed

Creating the Example TEF Editor

We want to create a new eclipse text editor. Therefore, we need to create a new eclipse plug-in. Since you are interested in TEF, you probably have written an eclipse plug-in before. Use the eclipse plug-in creation wizard with all default options to create a project called Functions: File/New/Project.../Plug-in Development/Plug-in Project, then press next, set the name, next, next, finish. You can change the perspective if you like. This will create a new eclipse project, with an activator class (a standard implementation that controls the plug-in lifecycle) and a plug-in manifest.

One function of this plug-in manifest is to determine, which other plug-ins your plug-in uses. Open the manifest and switch to the Extension tab. This tab lists all the extension points of other plug-ins that you want to use. We have to extend the eclipse ui plug-ins to create a new eclipse editor. You don't have to create this extension manually, since TEF provides an extension wizard for this task. Click Add..., select the Extension Wizards tab, select TEF Templates and TEF Text Editor. Press next. (You can experiment with the other TEF templates later.)

Usually you would have to manually create some extensions to the eclipse ui, create a meta-model, a syntax definition, you would have to connect everything by extending one of the TEF classes, etc. The extension wizard allows you to skip all this and simply create fully functional example editor. The idea is that you use this example editor as a starting point. You have something that works, and that you use as a template for your own editors. Based on this template you can replace the example meta-model with your own, replace the syntax definition with your own notation, etc. The wizard allows you to customize some of the properties of the example editor. This are thinks like class and package names, project and file names, the name for the file extensions that the editor is started for, etc. For the beginning it is recommended that you leave everything as it is. Press finish. If you are asked to save the manifest file, confirm.

The extension wizard creates a lot of files for you. The picture shows how your project should be look like by now. The wizard generated the main editor class, a few Java programmed extension to the editors syntax, a resource directory with meta-model, a generator model for the meta-model, and a syntax description file. It also created all the extension and dependencies in the plug-in manifest and plug-in xml file.

If you look at the dependencies tab in the plug-in manifest you see all the dependencies: we are using some eclipse ui (actually we are using decorated eclipse text editors) and eclipse core functionalities. Of course the TEF plug-ins are used, as well as a plug-in that provides a model repository for our example language. This is marked as an error, since we have not yet generated this plug-in. The extensions tab of the manifest now shows a our example editor as an extension to the org.eclipse.ui.editors extension point. We are using the normal eclipse ui extension point for text editors, but the editor class the we provide is an extension of an abstract TEF text editor class, providing all the language specific editor functionality based on a meta-model and syntax description.

We still have to generate the model repository plug-in. Simply open the gen-model, right-click the top node and first select Generate Model Code and than Generate Edit Code. This generates a new plug-in and all the eclipse errors should vanish. The generated model code allows to program with instances of the meta-model. The generated edit code provides the elements used in the outline view. This are the same elements that are usually used in the normal EMF tree-based editors.

Running the Example Editor

We need to start a new eclipse instance in order to run the newly created plug-in. If you don't have a launch configuration for plug-ins already, create one. You can right-click the project and choose Run as/Eclipse Application, which implicitly creates a launch configuration. A new eclipse instance starts.

Select a place to create a new file. Either choose one of your existing projects or create a new one (File/New/Project.../General/Project/Enter a name/Finish). Create a new file by clicking right on the selected project, New/File.... Select a name that ends with the extension of our editor: ".ext". Eclipse creates the file and should automatically open our editor. Now enter something like:

function fact(number) = fact(number-1)*number;

Try the different editor features shown in the figure on the left.

The Editor Explained

The Meta-Model

If you open the meta-model you will see a ecore model that corresponds to the meta-model on the left. This is the abstract syntax that our example editor uses. The meta-model is pretty straight forward. We basically have a model consisting of functions with parameters and expressions. Within an expressions we can use integer values, different arithmetic operators, parameters, and function calls.

The Syntax Definition

While the meta-model is a description for the abstract syntax, the .etsl-file contains a description for the concrete syntax. This extendent textual syntax language file is written in a specific syntax that allows you to describe how instances of the meta-model are to be represented as texts. This file basically contains a EBNF-grammar. Each textual model representation has to fit to this grammar. Writing a grammar, you can use the usual grammar features: you write rules, with non terminals, terminals, morphems, etc.

Besides regular grammar elements, the file also contains meta-model bindings. The syntax description defines a top-level meta-model class and refers to the used meta-model. Furthermore, you can bind rules to meta-model classes (element binding) and right-hand-side symbols to meta-model properties (property binding). There are two possible property bindings: composite bindings and reference bindings. The first reflects the compositional structure of models, the user is used to define references. References in textual model representations are usually realised with identifiers like names.

The Main Editor Class

This Java class is the editor. The eclipse ui extension for editors lets you configure editor id, name, extension, etc. (check the manifest for details), and it lets you determine an editor class. This class has to implement ITextEdior. Eclipse itself provides some abstract implementations for this interface. These implementations provide functionality for reconciliation, decorations, content-assist, highlight, etc. TEF provides its own abstract editor implementations that specialise the eclipse decorated text editor implementations. For our example editor we used TextEditor. The contents of this class is pretty much self explanatory. In later tutorials, we will see: how to customise the regular TEF editor behaviour by overwriting more methods in this class.

Summary

Ok, this tutorial should have helped you to get started with TEF. You can now create an editor from the templates and have the basic knowledge needed to modify this editor. In later tutorials we will also cover the other editors: model editor and embedded editor. In later tutorials, we will also see how we can customise the editor behaviour: implement better content assist, more error annotations, use our own morphem classes (primitive types), and much more. You can also download further example projects from our svn repository or explore the Textual Editing Framework (TEF) Examples feature that you find on the update site. Subscribe to my blog if you want to be informed about new tutorials.

Additional Concepts

This is just a conceptual documentation and we cannot go into all the technical details because these are currently subject to continuous changes. After the first major release of TEF, we will provide more tutorials that explain all the TEF features in detail.

TEF and Eclipse

TEF editors are eclipse decorated text editors. That means they use the same facilities than, for example, the Java editor. They use the rulers to mark errors and occurences and provide according hovers for this markings. Syntax highlighting and code-completion is realised based on the same eclipse given framework.

How such eclipse editors work and are to be programmed has not to concern you, because TEF abstracts from the eclipse editor framework. You simply use TEF to define a textual syntax that describes your editor, the rest is automatically done by TEF.

TEF itself is an eclipse plugin and each TEF editor that you create is a eclipse plugin too. TEF currently works with eclipse 3.2 and required the EMF installed.

The Reconciliation Process

On first sight, a TEF editor works like a regular text editor: the user enters text and the text is stored in memory as a sequence of characters. But of course that is only half the story. Each time the user stops typing for a few seconds a background process is running that transforms the user text into a model. Because this is usually done to replace an old model that become invalid through the recent user typing, we like to call this process reconciliation.

The image on the right shows this reconciliation process. It starts with the user text. This text is parsed according the context free syntax of our language. The result is a tree-based representation of the text. Since a model contains non context free features. This are mainly cross references: for example, the use of a variable references the variable definition. The next step is therefore to resolve all those references. This step will convert the abstract syntax tree into an actual model that is corresponding to the language's meta-model. After that the model is checked against the constraints in the meta-model. During all this three steps, errors can be discovered: the parser can fail parsing due to an syntax error; a name might be wrong, so it cannot be resolved; a model invariant may be violated. All these errors are reported back to the user, by placing a red line with a error message hover under the corresponding places within the text.

Reconciliation tries to regularly update the model represented by the text. All the information about parsing, reference resolution, and constraint checking have to be providing by a language engineer. TEF uses a textual syntax definition language, that can be used to define a concrete textual notation. These syntax definitions include all the necessary information and refer to the languages meta-model. In other words, these syntax definitions describe a TEF editor.

Semantic Features

Within the syntax defintion, a language engineer can describe several non-syntax aspects of the language. This concludes reference resolution and code completion, syntax highlighting, constraint checking, etc. We will add additional information soon.