Diagrams
Overall Flex Compiler Design
MmxlDocument Containment
Subcompilers
The Flex compiler, which is often referred to as mxmlc, is a
collection of subcompilers, used to create a SWF. The two main
subcompilers are flex2.compiler.as3.Compiler, which compiles .as files,
and flex2.compiler.mxml.Compiler, which compiles .mxml files. The
flex2.compiler.abc.Compiler is used to extract type information from
precompiled ABC (ActionScript Byte Code), like from SWC files. The
flex2.compiler.css.Compiler is used to compile Runtime CSS SWF's. The
flex2.compiler.i18n.Compiler is used to compile translation files,
usually .properties files.
Each of the subcompilers implements a common interface,
flex2.compiler.Compiler. This interface declares a method for each
phase of compilation: preprocess, parse1, parse2, analyze1, analyze2,
analyze3, analyze4, generate, and postprocess. Historically most of
these phases line up with how ASC is designed. A flex2.compiler.Source
is passed into preprocess() and parse1(). It represents a single file
from the CompilerSwcContext (flex2.compiler.CompilerSwcContext),
FileSpec (flex2.compiler.FileSpec), SourceList
(flex2.compiler.SourceList), or SourcePath (flex2.compiler.SourcePath).
The parse1() is suppose to return a CompilationUnit, which is then
passed into each of the following phase methods. The analyze phase
methods register the type dependencies of the CompilationUnit. The
generate phase method emits byte code.
Most of the subcompilers wrap an instance of
flex2.compiler.as3.Compiler. In each of the phase methods, some work is
potentially done and then the same phase method is called on the
wrapped AS3 compiler. For example, in
flex2.compiler.mxml.InterfaceCompiler.parse1(), the mxml document is
parsed, an AS skeleton is generated, and then parse is called on the
wrapped As3 compiler with the skeleton.
API
The orchestration of the subcompilers is handled by API
(flex2.compiler.API). API has two algorithms, the original batch1,
which is used when -conservative is set to true, and the newer batch2,
which is the default. The batch1 algorithm starts by compiling the
Sources in the FileSpec. As new type dependencies are discovered, they
are added to the batch and caught up to CompilationUnit, which is
furtheset in the phases. For example, if we are compiling A.mxml using
batch1, API starts by parsing and analyzing A. If in analyze2, it's
discovered that A depends B, API will parse and analyze B through
analyze2. Then if in analyze4 of B, it's discovered by B depends on C,
API will compile C through analyze4 before continuing with A and B. The
main disadvantage with the batch1 algorithm is that it consumes alot of
memory, because the syntax tree and other information built up prior to
the generate phase can't be garbage collected until every
CompilationUnit reaches the generate phase.
The batch2 algorithm addresses this problem, by compiling one or
more of the dependent types to completion before continuing with the
rest of the batch. In the above example, batch2 would potentially
compile C through the generate phase before returning to B and it would
potentially compile B through the generate phase before returning to A.
It's just potentially, because the batch2 algorithm batches up
dependent types using a budget algorithm. The budget algorithm using a
configurable memory usage factor. This factor is a hidden configuration
option and it's default is 1000. The size of each Mxml Source is
multiplied by 4.5 and then divided by the factor. The size of each AS
Source is divided by the factor. The results are summed and each Source
is added to the batch until the maxBudget is reached. The default
maxBudget is 100 and it's not configurable. This is all quite
complicated, but it does a pretty good jump of compiling a set of
Sources fast without using an excessive amount of memory.
The API class also handles persisting CompilationUnits, loading
CompilationUnits, validating CompilationUnits, and resolving multinames.
CompilerSWCContext, FileSpec, ResourceContainer, SourceList, SourcePath
Each of these is a place where API looks when it tries to resolve a
definition/type. CompilerSWCContext wraps a flex2.compiler.swc.SwcGroup
and uses it to create Source objects for each definition. FileSpec
represents a list of files, which are not required to follow the single
public definition rule. It is used by compc's -include-sources option
and by Flex Builder. It's similar to ASC's -include option.
ResourceContainer represents a collection of the generated sources
during a compilation. An example of a generated source is the class
generated for an @Embed. SourceList represents a list of files,
following the single public definition rule, and an associated path
where dependencies can be found. For mxmlc, the SourceList will contain
one file, ie the one specified on the command line. SourcePath
represents a list of paths where dependencies, following the single
public definition rule, can be resolved.
Mxml Compilation
The Mxml subcompiler is divided into two subsubcompilers,
flex2.compiler.mxml.InterfaceCompiler and
flex2.compiler.mxml.ImplementationCompiler. The InterfaceCompiler
serves two purposes. First, it exposes the public signature at the same
point in the compiler workflow as a native AS would. The public
signature contains declarations for each Mxml child tag with an id
attribute and user script code, which might include public
declarations. Second, it forces all dependent types to be defined.
These dependent types are used by the ImplementationCompiler in the
second pass to disambiguate child tags.
As an example:
<mx:Button id="myButton" width="100" color="red"/>
The InterfaceCompiler will expose a variable named "myButton" with a
type of Button. It will also force Button to be resolved and analyzed.
This way ImplementationCompiler can lookup that width a public property
of Button and color is a Button style. This information is necessary
for ImplemenationCompiler to generate the full implementation.
During the parse1 phase, the InterfaceCompiler uses a JavaCC based
parser (Grammar.jj) and a SAX based scanner
(flex2.compiler.mxml.dom.Scanner) to create the syntax tree, which is
an instance of flex2.compiler.mxml.dom.ApplicationNode.
During the parse2 phase, the InterfaceCompiler uses an
InterfaceAnalyzer, which is a private inner class of InterfaceCompiler,
to populate the DocumentInfo. The DocumentInfo is used by
generateSkeleton(), which uses a Velocity template, InterfaceDef.vm, to create the generated ActionScript. As a performance optimization, the Velocity templates are precompiled by the build into .vms files.
During the generate phase, the InterfaceCompiler does not generate
byte code. This signals to flex2.compiler.API that it should try
compiling the CompilationUnit again. The second time through, the
ImplementationCompiler kicks in. The ImplementationCompiler generates
the full implementation.
The ImplementationCompiler reuses the ApplicationNode and
DocumentInfo created by the InterfaceCompiler, but instead uses an
flex2.compiler.mxml.builder.ApplicationBuilder to populate a
flex2.compiler.mxml.rep.MxmlDocument. See this diagram of the MxmlDocument. The DocumentInfo and MxmlDocument are used by generateImplementation(), which uses a Velocity
template, ClassDef.vm, to create the generated ActionScript.
ClassDef.vm uses numerous macros, which are defined in ClassDefLib.vm.
During the generate phase, the ImplementationCompiler does generate
byte code.
ApplicationBuilder is responsible for processing the root node
attributes, see processAttributes(), and child nodes, see
processChildren(). It does this by extending
flex2.compiler.mxml.builder.ComponentBuilder, adding support for some
special root attributes like backgroundColor, height, width, frameRate,
etc. Most of these are special, because they are used when encoding the
SWF and not by AS3 code. In processAttributes, the appropriate
flex2.compiler.mxml.lang.AttributeHandler is used to handle each
attribute. In processChildren(), the appropriate
flex2.compiler.mxml.lang.ChildNodeHandler is used to handle each child
node. One thing to note is that ApplicationBuilder handles every mxml
document, not just ones with an Application root node.
Compiler Extensions
Compiler extensions are used to add additional AST (Abstract Syntax
Tree) handling. They are added to instances of
flex2.compiler.as3.Compiler via the addCompilerExtension() method. A
compiler extension must implement flex2.compiler.as3.Extension which
includes a method for each compiler phase. One example of a compiler
extension is the flex2.compiler.as3.SignatureExtension.
Data Binding
There are two main aspects to data binding. The first is the inline
data binding expressions, ie the curly brace expressions, and the
<mx:Binding> tags. Each of these defines a source and
destination. They are processed by flex2.compiler.mxml.builders.Builder
and it's subclasses. The result of the processing is that a
flex2.compiler.mxml.rep.BindingExpression is added to the MxmlDocument.
In ImplementationCompiler.parse1(), we transfer the list of
BindingExpressions to the as3Unit's context, to make them available to
the flex2.compiler.as3.binding.DataBindingFirstPassEvaluator.
The second is the variables and properties decorated with Bindable
metadata. Bindable metadata with an event attribute advertise to the
DataBindingFirstPassEvaluator that data binding expressions, which
include the variable or property in the source, can listen for that
event as an indicator that a change has taken place. Bindable metadata
without an event attribute first indicates to the
flex2.compiler.as3.binding.BindableExtension that the variable or
property should be wrapped in a new getter/setter pair which dispatches
a change event when a change occurs. Once this takes place, the
Bindable metadata behaves just like it had originally had an event
attribute.
DataBindingFirstPassEvaluator is responsible for processing each
BindingExpression source by looking for things that can be watched for
changes. For each thing, a flex2.compiler.as3.binding.Watcher is
created. The flex2.compiler.as3.binding.DataBindingExtension later uses
these with the WatcherSetupUtil.vm Velocity template to create the
WatcherSetupUtil helper class.
Embed
The embedding of assets has many layers and entry points. For mxml's
@Embed, flex2.compiler.mxml.Builder.TextValueParser.embed() creates a
new flex2.compiler.mxml.rep.AtEmbed object and adds it to the
MxmlDocument. Then in ClassDef.vm, each AtEmbed is used to code gen a
variable with Embed metadata.
For CSS's Embed(), flex2.compiler.css.StyleDef.processEmbed()
creates a new AtEmbed object. If the CSS is inside a <mx:Style>
tag, the AtEmbed is added to the MxmlDocument and handled like @Embed.
If the CSS is inside of a global, theme, or defaults style sheet, it's
added to the flex2.compiler.css.StyleContainer. When the StyleContainer
generates it's code using the StyleDef.vm Velocity template, a variable
with Embed metadata is created for each AtEmbed.
For translation file's Embed(),
flex2.compiler.i18n.PropertyText.load() creates and stores an AtEmbed
object. Then in flex2.compiler.i18n.Compiler.codegenAtEmbeds() a
variable with Embed metadata is generated for each AtEmbed.
Downstream of each of those sits the
flex2.compiler.as3.EmbedExtension, which processes the Embed metadata
on each variable by creating a new Source using the EmbedClass.vm
Velocity template. The created class has Embed metadata at the class
level. Each of these new Source objects is added to the
CompilationUnit's generatedSources, which is handled by API in a later
batch. EmbedExtension also process the class level metadata. It does
this by transcoding the asset using flex2.compiler.as3.EmbedUtil, which
in turn uses a transcoder from the flex2.compiler.media package. The
generated class is associated with the transcoded asset when the SWF is
encoded.
CSS
CSS can be used in three types of places. The first is in global,
theme, and defaults style sheets. The second is in mxml
<mx:Style> tags. These are processed by StyleContainer. The third
is in flex2.compiler.css.Compiler. StyleContainer and
flex2.compiler.css.Compiler use a flash.css.StyleSheet to handle the
parsing. StyleSheet uses the Batik CSS parser. Once the CSS is parsed,
it is used to generate AS3. For global, theme, and defaults style
sheets, the code generation is done by the StyleDef.vm Velocity
template. For <mx:Style> tags, the code generation is done by the
ClassDefLib.vm Velocity template in the emitComponentStyleSettings
macro. For flex2.compiler.css.Compiler, the StyleModule.vm Velocity
template is used.
Compilation Unit Persistence
For incremental compilation, API uses
flex2.compiler.PersistenceStore to persist and load CompilationUnits,
the FileSpec, the SourceList, the SourcePath, the ResourceContainer,
the ResourceBundlePath, and some other information like checksums. The
PersistenceStore writes all this information out to a single file and
uses Java serialization to do it.
Incremental Compilation
Incremental compilation works by first trying to read in a
persistence store from a previous compile. If it's found, each
compilation unit is validated to see if it is out of date. Then only
the out of date compilation units and their dependent compilation units
are recompiled. We try to be smart about compiling dependency by only
recompiling them if the public signature or imports for a compilation
unit change. See API.validateCompilationUnits() and
API.computeSignatureChecksum(). If information from a previous
compilation is not found, a full compilation is done. In both cases,
after the compilation is done, a new persistence store is written out.