This is an example for a Rule describing something called an entity :

Entity :
  "entity" name=ID "{"
    (features+=Feature)+
  "}"
 

Entity is both the name of the rule and the name of the metatype corresponding to this rule. After the colon the description of the rule is following. A description is made up of tokens. The first token is a KeywordToken which says that a description of an entity starts with the keyword entity. A so called Assignment follows (name=ID).

The left hand refers to a property of the Metatype (in this case it's the property name of type Entity). The left hand side is a call to the built-in token ID. Which means Identifier and allows character sequences of the form ('a-zA-Z_' ('a-zA-Z_0-9)*). The parser will assign ('=') the Identifier to the specified property (name).

Then (enclosed in curly brackets ("{" and "}" both are essentially keyword tokens)) one or more features can be declared ((features+=Feature)+). This one again is an assignment. This time the token points to another rule (called Feature) and each feature is added (note += operator) to the Entity's reference called features.

The Feature rule itself could be described like this:

Feature :
  type=ID name=ID ";"
 

so that the following description of an entity would be valid according to the grammar:

entity Customer {
  String name;
  String street;
  Integer age;
  Boolean isPremiumCustomer;
}
 

Note, that the types (String, Integer, Boolean) used in this description of a customer, are simple identifiers, they don't have been mapped to e.g. Java types or something else. So according to the grammar this would also be valid, so far:

entity X {
  X X;
  X X;
  X X;
  cjbdlfjerifuerfijerf dkjdhferifheirhf;
}
 

As stated before the grammar is not only used as input for the parser generator but it is also used to compute a meta model for your DSL. We will first talk about how an Xtext parser works in general, before we look at how a meta model is beeing constructed.

The parsing of text is devided in two separate tasks the lexing and the parsing.

The lexer is responsible of creating a sequence of tokens from a character stream. Such tokens are identifiers, keywords, whitespace, comments, operators, etc. Xtext comes with a set of built-in lexer rules which can be extended or overwritten if necessary. You have already seen some of them (e.g. ID).

The parser get's the stream of tokens and creates a parse tree out of them. The type rules from the example are essentially parser rules.

Now let's have a look at how the meta model is constructed.

We've already seen how the Type Rules works in general. The rule's name is used as the name of the meta type generated by Xtext.

Entity :
  (isAbstract?="abstract")? "entity" name=ID "{"
     (features+=Feature)*
  "}";
 

Entity : 
  "entity" name=ID ("extends" superType=[Entity])?
  "{" 
     (features+=Feature)* 
  "}";

 ... ("extends" superType=[Entity|MyComplexTokenRule])? ...

Feature :
  Attribute | Reference;

Attribute :
  type=ID name=ID ";";

Reference :
  "ref" (containment?"+")? type=ID name=ID ("<->" oppositeName=ID)? ";";
 

Feature :
  type=ID name=ID ";" | Reference;

Reference :
  "ref" (containment?"+")? type=ID name=ID ("<->" oppositeName=ID)? ";";
 

Model : TypeA TypeA TypeC;

TypeA : "A" | TypeB;
TypeB : "B";
TypeC : "C";

CommonSuper : TypeB | TypeC; // just for the typehierarchy
 

The enum rule is used to define enumerations. For example if you would like to hardwire the possible datatypes for attributes into the language you could just write:

Attribute :
  type=DataType name=ID ";";

Enum DataType :
  String="string"|Integer="int"|Boolean="bool";
 

So that this would be valid:

entity Customer {
  string name; 
  string street;
  int age;
  bool isPremiumCustomer;
}
 

but this would not

entity Customer {
  X name; // type X is not known
  String street; // type String is not known (case sensitivity!)
}
 

Xtend provides built-in Tokens (we have already seen the

IdentifierToken

and the

KeywordToken

). Sometimes this is not sufficient, so we might want to create our own Tokens. Therefore we have the so called

String Rule

, which is implemented as a parser rule (it's not a lexer rule!).

Example

String JavaIdentifier :
  ID ("." ID)*;
 

The contents of the String Rule is simply concatenated and returned as a string. One can refer to a String Rule in the same manner we refere to any other rule.

So just for the case you want to declare datatypes using your DSL and therein specify how it is mapped to Java (not Platform independent, i know, but expressive and pragmatic), you could do so using the following rules.

Attribute :
  type=DataType name=ID ";";

DataType :
  "datatype" name=ID "mappedto" javaType=JavaIdentifier;

String JavaIdentifier :
  ID ("." ID)*;
 

A respective model could look like this:

entity Customer {
  string name;
  string street;
  int age;
  bool isPremiumCustomer;
}

datatype string mappedto java.util.String
datatype int mappedto int
datatype bool mappedto boolean
 

You could of course point to a custom typemapping implementation, if you need to support multiple platforms (like e.g. SQL, WSDL, Java,...). Additionally you should consider to define the datatypes in a separate file, so the users of your DSL can import and use them.

All static characters or words (keywords) can be specified directly in the grammar using the usual string literal syntax. We never need the value of keyword because we know it (it's static). But sometimes there are optional keywords like e.g. the modifiers in Java. The existence of a keyword can be assigned using the boolean assignment operator "?". However, if you want to assign the value of the keyword to a property just use the assignment operator '='.

We also have seen the identifier token (ID). This is the token rule expressed in Antlr grammar syntax:

('^')?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*
 

So an identifier is a word starting with a character or underscore followed by optionally additional characters, underscores or digits. The return value of the Identifier token is a String. So if you use the usual assignment operator „=“, the feature the value is assigned to will be of type String. You could also use the boolean operator and the type will be Boolean.

If an identifier conflicts with a keyword or another lexer rule, it can be escaped with the '^' character.

There is also a built-in String Token. Here is an example:

Attribute :
   type=DataType name=ID (description=STRING)? ";";
 

With this one can optionally specify a description for an entity like this:

entity Customer {
  string name ;
  string street "should include the street number, too.";
  int age;
  bool isPremiumCustomer;
}

By default the two string literal syntaxes "my text" and 'my text' are supported. Note that unlike in Java also multiline strings are supported:

entity Customer {
  string name ;
  string street "should include the street number, too.
                 And if you don't want to specify it, you
                 should consider specifying it somewhere else.";
  int age;
  bool isPremiumCustomer;
}
 

Sometimes you want to assign Integers. Xtext supports it with the built-in lexer rule INT.

Index:
  "#" index=INT;
 

The default pattern is :

('-')?('0'..'9')+

It can be overwritten (see next section), but you have to take car that the coercion (Integer.valueOf(String) is used) works.

There are two different kinds of comments automatically available in any Xtext language.

  // single-line comments and
  
  /*
    mutli-line comments
  */
 

Note that those comments are ignored by the language's parser by default (i.e. they are not contained in the AST returned from the parser).

If you don't want ignored comments, or you ant to have a different syntax you need to overwrite the default implementation (name is SL_COMMENT resp. ML_COMMENT).

Every textual model contains whitespace. As most languages simply ignore whitespace, Xtext does so by default, too. If you want to have semantic whitespace in your language (like e.g. python), you have to overwrite the builtin whitespace rule (name is WS).

If you want to overwrite one or more of the built-in lexer rules or add an additional one, the so called native rule is your friend.

Example:

// overwriting SL_COMMENTS we don't want Java syntax (//) but bash syntax (#)
Native SL_COMMENT :
  "'#' ~('\n'|'\r')* '\r'? '\n' {$channel=HIDDEN;}";

// fully qualified names as a lexer rule
Native FQN :
  "ID ('.' ID)*";

The syntax is :

"Native" ID ":" 
   STRING // The string contains a valid ANTLR 3 lexer rule expression (see http://www.antlr.org/wiki/display/ANTLR3/ANTLR+v3+documentation)
   ";"
 

The generator can be configured with the following properties definied in generate.properties:

Any textual artifacts located in the src dir (of any project) will always stay untouched. The generator just creates them the first time when they don't exist.

Files generated to the src-gen directory should never be touched! The whole directory will be wiped out the next time one starts the generator.

Xtext generates artifact into two different projects.

	Void link_featureName(my::MetaType this) : 
	   (let ents = this.allElements().typeSelect(my::ReferredType) : 
	   this.setFeatureName(ents.select(e|e.id() 
	     == this.parsed_featureName).first()) );
	 

Code Completion is controlled using oAW extensions. The default implementation provides keyword proposals as well as proposals for cross references.

Have a look at the extension file ContentAssist.ext. Therein a comment describes how to customize the default behaviour:

/*
 * There are two types of extensions one can define
 *
 * 1) completeMetaType_feature(ModelElement ele, String prefix)
 * This one is called for assignments only. It gets the underlying Modelelement and the current 
 * prefix passed in.
 *
 * 2) completeMetaType(xtext::Element grammarEle, ModelElement ele, String prefix)
 * This one gets the grammarElement which should be completed passed in as the first parameter.
 * an xtext::Element can be of the following types : 
 *  - xtext::RuleName (a call to a lexer rule (e.g. ID)), 
 *  - xtext::Keyword, 
 *  - xtext::Assignment
 *
 * Overwrite rules are as follows:
 * 1) if the first one returns null for a given xtext::Assignment or does not exist the second one 
 *    is called.
 * 2) if the second one returns null for a given xtext::Keyword or does not exist a default keyword 
 *    proposal will be added. 
 *
 * Note that only propals with wich match (case-in-sensitive) the current prefix will be proposed 
 * in the editor
 */

The implementation for the navigation actions is implemented via extensions, too. As for Code completion the same pattern applies: There is a GenNavigation.ext extension file in the src-gen folder which can be overwritten or specialized using the Navigation.ext file in the src folder (reexporting the generated extensions).

There are two different Actions supported by Xtext:

List[UIContentNode] findReferences(String s, Object grammarelement, Object element) : ...;

package tree;


eclass UIContentNode {
   UIContentNode parent;
   UIContentNode[] children;
   String label;
   String image;
   emf::EObject context;
}

emf::EObject findDeclaration(String identifier, emf::EObject grammarElement, 
   emf::EObject modelElement) : ...;

The outline view is constructed using a tree of UIContentNode (see above).

Each time the outline view is created the following extension is called:

UIContentNode outlineTree(emf::EObject model)

It is expected to be declared in Outline.ext which by default exports a generic implementation from GenOutline.ext (the same pattern again).

You can either reuse the generic extension and just provide a label() and image() extension for your model elements (should be added in EditorExtensions.ext).

However, if you want to control the structure of the outline tree you can overwrite the extension outlineTree(emf::EObject model) in Outline.ext.

The default syntax highlighting distincts between comments, string literals, keywords and the rest.

If you just want to specify which words to be coloured as keywords you can extend the [basepackage.][Languagename]Utilities.java class from the editor plugin. You need to overwrite the following method (DON'T CHANGE IT DIRECTLY, BECAUSE IT WILL BE OVERWRITTEN THE NEXT TIME YOU START THE GENERATOR!).

public String[] allKeywords()

Each String returned by the method represents a keyword.

The utiltities method is created within the [LanguageName]EditorPlugin.java. So make sure that you change the following lines, too :

   // OLD -> private MyLangUtilities utilities = new MyLangUtilities();
   private MyCustomUtilities utilities = new MyCustomUtilities();
   public LanguageUtilities getUtilities() {
      return utilities;
   }

If you want to change the syntax of comments and string literals you have to provide an alternative implementation of GeneratedPartitionScanner.

Don't touch the class directly but, use the Utilities method to return a different instance.

If you have uge models or want to separate parts of your models from others for other reasons (e.g. provide a kind of library). You need to tweak Xtext a bit, because there is no first class support for this as for now. However, Xtext has a built-in registry where you can get parsed models by name.

MyModel : 
   (imports+=Import)*
   (types+=Type)*;

Import :
   "import" name=ID; 

Type : ...

 extension mydsl::GenExtensions reexport;
 extension org::openarchitectureware::xtext::registry::Reg;

 allElements(emf::EObject this) :
    wholeModel().union(
       (((MyModel)eRootContainer)
        .imports                   // for each import
        .getRootNode(i)            // get the root from the registry
        .wholeModel()              // get the whole model as a list
    );

 wholeModel(emf::EObject this) :
    {eRootContainer}.union(eRootContainer.eAllContents);