2.28. Macros¶

In Daslang, macros are the machinery that allow direct manipulation of the syntax tree.

Macros are exposed via the daslib/ast module and daslib/ast_boost helper module.

Macros are evaluated at compilation time during different compilation passes. Macros assigned to a specific module are evaluated as part of the module every time that module is included.

2.28.1. Compilation passes¶

The Daslang compiler performs compilation passes in the following order for each module (see Modules):

Parser transforms das program to AST
1. If there are any parsing errors, compilation stops
apply is called for every function or structure
1. If there are any errors, compilation stops
Infer pass repeats itself until no new transformations are reported
1. Built-in infer pass happens
  transform macros are called for every function or expression
2. Macro passes happen
If there are still any errors left, compilation stops
finish is called for all functions and structure macros
Lint pass happens
1. If there are any errors, compilation stops
Optimization pass repeats itself until no new transformations are reported
1. Built-in optimization pass happens
2. Macro optimization pass happens
If there are any errors during optimization passes, compilation stops
If the module contains any macros, simulation happens
1. If there are any simulation errors, compilation stops
2. Module macro functions (annotated with _macro) are invoked
  If there are any errors, compilation stops

Modules are compiled in require order.

2.28.2. Invoking macros¶

The [_macro] annotation is used to specify functions that should be evaluated at compilation time . Consider the following example from daslib/ast_boost:

[_macro,private]
def setup
    if is_compiling_macros_in_module("ast_boost")
        add_new_function_annotation("macro", new MacroMacro())

The setup function is evaluated after the compilation of each module, which includes ast_boost. The is_compiling_macros_in_module function returns true if the currently compiled module name matches the argument. In this particular example, the function annotation macro would only be added once: when the module ast_boost is compiled.

Macros are invoked in the following fashion:

Class is derived from the appropriate base macro class
Adapter is created
Adapter is registered with the module

For example, this is how this lifetime cycle is implemented for the reader macro:

def add_new_reader_macro ( name:string; someClassPtr )
    var ann <- make_reader_macro(name, someClassPtr)
    this_module() |> add_reader_macro(ann)
    unsafe
        delete ann

2.28.3. AstFunctionAnnotation¶

The AstFunctionAnnotation macro allows you to manipulate calls to specific functions as well as their function bodies. Annotations can be added to regular or generic functions.

add_new_function_annotation adds a function annotation to a module. There is additionally the [function_macro] annotation which accomplishes the same thing.

AstFunctionAnnotation allows several different manipulations:

class AstFunctionAnnotation
    def abstract transform ( var call : smart_ptr<ExprCallFunc>; var errors : das_string ) : ExpressionPtr
    def abstract verifyCall ( var call : smart_ptr<ExprCallFunc>; args,progArgs:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract apply ( var func:FunctionPtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract finish ( var func:FunctionPtr; var group:ModuleGroup; args,progArgs:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract patch ( var func:FunctionPtr; var group:ModuleGroup; args,progArgs:AnnotationArgumentList; var errors : das_string; var astChanged:bool& ) : bool
    def abstract fixup ( var func:FunctionPtr; var group:ModuleGroup; args,progArgs:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract lint ( var func:FunctionPtr; var group:ModuleGroup; args,progArgs:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract complete ( var func:FunctionPtr; var ctx:smart_ptr<Context> ) : void
    def abstract isCompatible ( var func:FunctionPtr; var types:VectorTypeDeclPtr; decl:AnnotationDeclaration; var errors:das_string ) : bool
    def abstract isSpecialized : bool
    def abstract appendToMangledName ( func:FunctionPtr; decl:AnnotationDeclaration; var mangledName:das_string ) : void

transform lets you change calls to the function and is applied at the infer pass. Transform is the best way to replace or modify function calls with other semantics.

verifyCall is called durng the lint phase on each call to the function and is used to check if the call is valid.

apply is applied to the function itself before the infer pass. Apply is typically where global function body modifications or instancing occurs.

finish is applied to the function itself after the infer pass. It’s only called on non-generic functions or instances of the generic functions. finish is typically used to register functions, notify C++ code, etc. After this, the function is fully defined and inferred, and can no longer be modified.

patch is called after the infer pass. If patch sets astChanged to true, the infer pass will be repeated.

fixup is called after the infer pass. It’s used to fixup the function’s body.

lint is called during the lint phase on the function itself and is used to verify that the function is valid.

complete is called during the simulate portion of context creation. At this point Context is available.

isSpecialized must return true if the particular function matching is governed by contracts. In that case, isCompatible is called, and the result taken into account.

isCompatible returns true if a specialized function is compatible with the given arguments. If a function is not compatible, the errors field must be specified.

appendToMangledName is called to append a mangled name to the function. That way multiple functions with the same type signature can exist and be differentiated between.

Lets review the following example from ast_boost of how the macro annotation is implemented:

class MacroMacro : AstFunctionAnnotation
    def override apply ( var func:FunctionPtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
        compiling_program().flags |= ProgramFlags needMacroModule
        func.flags |= FunctionFlags init
        var blk <- new [[ExprBlock() at=func.at]]
        var ifm <- new [[ExprCall() at=func.at, name:="is_compiling_macros"]]
        var ife <- new [[ExprIfThenElse() at=func.at, cond<-ifm, if_true<-func.body]]
        push(blk.list,ife)
        func.body <- blk
        return true

During the apply pass the function body is appended with the if is_compiling_macros() closure. Additionally, the init flag is set, which is equivalent to a _macro annotation. Functions annotated with [macro] are evaluated during module compilation.

2.28.4. AstBlockAnnotation¶

AstBlockAnnotation is used to manipulate block expressions (blocks, lambdas, local functions):

class AstBlockAnnotation
    def abstract apply ( var blk:smart_ptr<ExprBlock>; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract finish ( var blk:smart_ptr<ExprBlock>; var group:ModuleGroup; args,progArgs:AnnotationArgumentList; var errors : das_string ) : bool

add_new_block_annotation adds a function annotation to a module. There is additionally the [block_macro] annotation which accomplishes the same thing.

apply is called for every block expression before the infer pass.

finish is called for every block expression after infer pass.

2.28.5. AstStructureAnnotation¶

The AstStructureAnnotation macro lets you manipulate structure or class definitions via annotation:

class AstStructureAnnotation
    def abstract apply ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract finish ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract patch ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string; var astChanged:bool& ) : bool
    def abstract complete ( var st:StructurePtr; var ctx:smart_ptr<Context> ) : void

add_new_structure_annotation adds a function annotation to a module. There is additionally the [structure_macro] annotation which accomplishes the same thing.

AstStructureAnnotation allows 2 different manipulations:

class AstStructureAnnotation
    def abstract apply ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract finish ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool

apply is invoked before the infer pass. It is the best time to modify the structure, generate some code, etc.

finish is invoked after the successful infer pass. Its typically used to register structures, perform RTTI operations, etc. After this, the structure is fully inferred and defined and can no longer be modified afterwards.

patch is invoked after the infer pass. If patch sets astChanged to true, the infer pass will be repeated.

complete is invoked during the simulate portion of context creation. At this point Context is available.

An example of such annotation is SetupAnyAnnotation from daslib/ast_boost.

2.28.6. AstEnumerationAnnotation¶

The AstStructureAnnotation macro lets you manipulate enumerations via annotation:

class AstEnumerationAnnotation
    def abstract apply ( var st:EnumerationPtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool

add_new_enumeration_annotation adds a function annotation to a module. There is additionally the [enumeration_macro] annotation which accomplishes the same thing.

apply is invoked before the infer pass. It is the best time to modify the enumeration, generate some code, etc.

2.28.7. AstVariantMacro¶

AstVariantMacro is specialized in transforming is, as, and ?as expressions.

add_new_variant_macro adds a variant macro to a module. There is additionally the [variant_macro] annotation which accomplishes the same thing.

Each of the 3 transformations are covered in the appropriate abstract function:

class AstVariantMacro
    def abstract visitExprIsVariant     ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprIsVariant> ) : ExpressionPtr
    def abstract visitExprAsVariant     ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprAsVariant> ) : ExpressionPtr
    def abstract visitExprSafeAsVariant ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprSafeAsVariant> ) : ExpressionPtr

Let’s review the following example from daslib/ast_boost:

// replacing ExprIsVariant(value,name) => ExprOp2('==",value.__rtti,"name")
// if value is ast::Expr*
class BetterRttiVisitor : AstVariantMacro
    def override visitExprIsVariant(prog:ProgramPtr; mod:Module?;expr:smart_ptr<ExprIsVariant>) : ExpressionPtr
        if isExpression(expr.value._type)
            var vdr <- new [[ExprField() at=expr.at, name:="__rtti", value <- clone_expression(expr.value)]]
            var cna <- new [[ExprConstString() at=expr.at, value:=expr.name]]
            var veq <- new [[ExprOp2() at=expr.at, op:="==", left<-vdr, right<-cna]]
            return veq
        return [[ExpressionPtr]]

// note the following ussage
class GetHintFnMacro : AstFunctionAnnotation
    def override transform ( var call : smart_ptr<ExprCall>; var errors : das_string ) : ExpressionPtr
        if call.arguments[1] is ExprConstString     // HERE EXPRESSION WILL BE REPLACED
            ...

Here, the macro takes advantage of the ExprIsVariant syntax. It replaces the expr is TYPENAME expression with an expr.__rtti = "TYPENAME" expression. The isExpression function ensures that expr is from the ast::Expr* family, i.e. part of the Daslang syntax tree.

2.28.8. AstReaderMacro¶

AstReaderMacro allows embedding a completely different syntax inside Daslang code.

add_new_reader_macro adds a reader macro to a module. There is additionally the [reader_macro] annotation, which essentially automates the same thing.

Reader macros accept characters, collect them if necessary, and return an ast::Expression:

class AstReaderMacro
    def abstract accept ( prog:ProgramPtr; mod:Module?; expr:ExprReader?; ch:int; info:LineInfo ) : bool
    def abstract visit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprReader> ) : ExpressionPtr

Reader macros are invoked via the % READER_MACRO_NAME ~ character_sequence syntax. The accept function notifies the correct terminator of the character sequence:

var x = %arr~\{\}\w\x\y\n%% // invoking reader macro arr, %% is a terminator

Consider the implementation for the example above:

[reader_macro(name="arr")]
class ArrayReader : AstReaderMacro
    def override accept ( prog:ProgramPtr; mod:Module?; var expr:ExprReader?; ch:int; info:LineInfo ) : bool
        append(expr.sequence,ch)
        if ends_with(expr.sequence,"%%")
            let len = length(expr.sequence)
            resize(expr.sequence,len-2)
            return false
        else
            return true
    def override visit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprReader> ) : ExpressionPtr
        let seqStr = string(expr.sequence)
        var arrT <- new [[TypeDecl() baseType=Type tInt]]
        push(arrT.dim,length(seqStr))
        var mkArr <- new [[ExprMakeArray() at = expr.at, makeType <- arrT]]
        for x in seqStr
            var mkC <- new [[ExprConstInt() at=expr.at, value=x]]
            push(mkArr.values,mkC)
        return mkArr

The accept function macro collects symbols in the sequence. Once the sequence ends with the terminator sequence %%, accept returns false to indicate the end of the sequence.

In visit, the collected sequence is converted into a make array [[int ch1; ch2; ..]] expression.

More complex examples include the JsonReader macro in daslib/json_boost or RegexReader in daslib/regex_boost.

2.28.9. AstCallMacro¶

AstCallMacro operates on expressions which have function call syntax or something similar. It occurs during the infer pass.

add_new_call_macro adds a call macro to a module. The [call_macro] annotation automates the same thing:

class AstCallMacro
    def abstract preVisit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprCallMacro> ) : void
    def abstract visit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprCallMacro> ) : ExpressionPtr
    def abstract canVisitArguments ( expr:smart_ptr<ExprCallMacro> ) : bool

apply from daslib/apply is an example of such a macro:

[call_macro(name="apply")]  // apply(value, block)
class ApplyMacro : AstCallMacro
    def override visit ( prog:ProgramPtr; mod:Module?; var expr:smart_ptr<ExprCallMacro> ) : ExpressionPtr
        ...

Note how the name is provided in the [call_macro] annotation.

preVisit is called before the arguments are visited.

visit is called after the arguments are visited.

canVisitArguments is called to determine if the macro can visit the arguments.

2.28.10. AstPassMacro¶

AstPassMacro is one macro to rule them all. It gets entire module as an input, and can be invoked at numerous passes:

class AstPassMacro
    def abstract apply ( prog:ProgramPtr; mod:Module? ) : bool

make_pass_macro registers a class as a pass macro.

add_new_infer_macro adds a pass macro to the infer pass. The [infer] annotation accomplishes the same thing.

add_new_dirty_infer_macro adds a pass macro to the dirty section of infer pass. The [dirty_infer] annotation accomplishes the same thing.

Typically, such macros create an AstVisitor which performs the necessary transformations.

2.28.11. AstTypeInfoMacro¶

AstTypeInfoMacro is designed to implement custom type information inside a typeinfo expression:

class AstTypeInfoMacro
    def abstract getAstChange ( expr:smart_ptr<ExprTypeInfo>; var errors:das_string ) : ExpressionPtr
    def abstract getAstType ( var lib:ModuleLibrary; expr:smart_ptr<ExprTypeInfo>; var errors:das_string ) : TypeDeclPtr

add_new_typeinfo_macro adds a reader macro to a module. There is additionally the [typeinfo_macro] annotation, which essentially automates the same thing.

getAstChange returns a newly generated ast for the typeinfo expression. Alternatively, it returns null if no changes are required, or if there is an error. In case of error, the errors string must be filled.

getAstType returns the type of the new typeinfo expression.

2.28.12. AstForLoopMacro¶

AstForLoopMacro is designed to implement custom processing of for loop expressions:

class AstForLoopMacro
    def abstract visitExprFor ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprFor> ) : ExpressionPtr

add_new_for_loop_macro adds a reader macro to a module. There is additionally the [for_loop_macro] annotation, which essentially automates the same thing.

visitExprFor is similar to that of AstVisitor. It returns a new expression, or null if no changes are required.

2.28.13. AstCaptureMacro¶

AstCaptureMacro is designed to implement custom capturing and finalization of lambda expressions:

class AstCaptureMacro
    def abstract captureExpression ( prog:Program?; mod:Module?; expr:ExpressionPtr; etype:TypeDeclPtr ) : ExpressionPtr
    def abstract captureFunction ( prog:Program?; mod:Module?; var lcs:Structure?; var fun:FunctionPtr ) : void

add_new_capture_macro adds a reader macro to a module. There is additionally the [capture_macro] annotation, which essentially automates the same thing.

captureExpression is called when an expression is captured. It returns a new expression, or null if no changes are required.

captureFunction is called when a function is captured. This is where custom finalization can be added to the final section of the function body.

2.28.14. AstCommentReader¶

AstCommentReader is designed to implement custom processing of comment expressions:

class AstCommentReader
    def abstract open ( prog:ProgramPtr; mod:Module?; cpp:bool; info:LineInfo ) : void
    def abstract accept ( prog:ProgramPtr; mod:Module?; ch:int; info:LineInfo ) : void
    def abstract close ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeStructure ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterStructure ( st:StructurePtr; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeStructureFields ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterStructureField ( name:string; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterStructureFields ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeFunction ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterFunction ( fn:FunctionPtr; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeGlobalVariables ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterGlobalVariable ( name:string; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterGlobalVariables ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeVariant ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterVariant ( name:string; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeEnumeration ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterEnumeration ( name:string; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract beforeAlias ( prog:ProgramPtr; mod:Module?; info:LineInfo ) : void
    def abstract afterAlias ( name:string; prog:ProgramPtr; mod:Module?; info:LineInfo ) : void

add_new_comment_reader adds a reader macro to a module. There is additionally the [comment_reader] annotation, which essentially automates the same thing.

open occurs when the parsing of a comment starts.

accept occurs for every character of the comment.

close occurs when a comment is over.

beforeStructure and afterStructure occur before and after each structure or class declaration, regardless of if it has comments.

beforeStructureFields and afterStructureFields occur before and after each structure or class field, regardless of if it has comments.

afterStructureField occurs after each field declaration.

beforeFunction and afterFunction occur before and after each function declaration, regardless of if it has comments.

beforeGlobalVariables and afterGlobalVariables occur before and after each global variable declaration, regardless of if it has comments.

afterGlobalVariable occurs after each individual global variable declaration.

beforeVariant and afterVariant occur before and after each variant declaration, regardless of if it has comments.

beforeEnumeration and afterEnumeration occur before and after each enumeration declaration, regardless of if it has comments.

beforeAlias and afterAlias occur before and after each alias type declaration, regardless or if it has comments.

2.28.15. AstSimulateMacro¶

AstSimulateMacro is designed to customize the simulation of the program:

class AstSimulateMacro
    def abstract preSimulate ( prog:Program?; ctx:Context? ) : bool
    def abstract simulate ( prog:Program?; ctx:Context? ) : bool

preSimulate occurs after the context has been simulated, but before all the structure and function annotation simulations.

simulate occurs after all the structure and function annotation simulations.

2.28.16. AstVisitor¶

AstVisitor implements the visitor pattern for the Daslang expression tree. It contains a callback for every single expression in prefix and postfix form, as well as some additional callbacks:

class AstVisitor
    ...
    // find
        def abstract preVisitExprFind(expr:smart_ptr<ExprFind>) : void          // prefix
        def abstract visitExprFind(expr:smart_ptr<ExprFind>) : ExpressionPtr    // postifx
    ...

Postfix callbacks can return expressions to replace the ones passed to the callback.

PrintVisitor from the ast_print example implements the printing of every single expression in Daslang syntax.

make_visitor creates a visitor adapter from the class, derived from AstVisitor. The adapter then can be applied to a program via the visit function:

var astVisitor = new PrintVisitor()
var astVisitorAdapter <- make_visitor(*astVisitor)
visit(this_program(), astVisitorAdapter)

If an expression needs to be visited, and can potentially be fully substituted, the visit_expression function should be used:

expr <- visit_expression(expr,astVisitorAdapter)