Dylan

LLBLGen - Customising the DotNetTemplateEngine

Dylan Lewis — Fri, 27 Mar 2009 09:41:55 GMT

Introduction

Customers of LLBLGen Pro have access to the LLBLGen Pro SDK. This contains source code for all task performers, the TDL interpreter, the TDL parser, the LPT Parser, plugins, TypeConverters, and all command line tools and all drivers. Light Speed IT Solutions has made use of this source code and has modified the DotNetTemplateEngine with several enhancements. This article describes each modification that we have made and also the steps needed to reproduce these modifications.

This article is based on the LLBLGen Pro - SDK version 2.6 released on the 8th of December 2008, i.e. LLBLGenProSDK_12082008.zip.

The DotNetTemplateEngine

A majority of the custom LLBLGen templates that we have written at Light Speed IT Solutions have been LPT templates that use the DotNetTemplateEngine template engine as the task performer. I personally am used to writing templates this way, as the LLBLGen LPT templates are very similar to CodeSmith and T4 templates, both of which I have had experience in writing before using LLBLGen. As a consequence, the ideas which I have had for extending the LLBLGen Pro SDK have focussed on what can be done with the DotNetTemplateEngine.

Using the DotNetTemplateEngine Source Code

The DotNetTemplateEngine class is in the file DotNetTemplateEngine.cs, in the project LptParserEngine.csproj, in the solution CodeGenerator 2.6.sln, which is included with the SDK source code. This is a Visual Studio 2005 solution. The first step in customising the SDK source code will be to upgrade the solution to a Visual Studio 2008. To support the enhancements planned, the projects in the solution will be converted to .Net version 3.5, which will require Visual Studio 2008.

When the solution is converted and all of the projects have been converted to .Net 3.5, the next step is to fix the broken references. The readme for the LLBLGen Pro SDK recommends that the contents of the zip file in which the LLBLGen Pro SDK is distributed is copied in to the installation folder for LLBLGen Pro. This does not guarantee that the references to the LLBLGen Pro assemblies used by the source code will work. These references use paths that rely on source code not distributed with the LLBLGen Pro SDK. The following references will be broken in all projects: SD.LLBLGen.Pro.ApplicationCore, SD.LLBLGen.Pro.DBDriverCore, and SD.LLBLGen.Pro.GeneratorCore. These can be fixed by referencing the appropriate assemblies in the installation folder for LLBLGen Pro.

Deploying the Task Performer Assemblies

The projects in the solution also have post build event commands setup to deploy the compiled assemblies to directories used by the LLBLGen build process. These commands can be altered to deploy the assemblies to the TaskPerformers directory in your LLBLGen Pro installation directory. Assuming that the LLBLGen Pro SDK source code has been extracted in to the LLBLGen Pro installation directory, the following commands can be used:

LptParserEngine:

copy /y "$(TargetDir)\SD.LLBLGen.Pro.LptParser.*" "..\..\..\Taskperformers\"

TaskPerformers:

copy /y "$(TargetDir)\SD.LLBLGen.Pro.TaskPerformers.*" "..\..\..\Taskperformers\"

TDLInterpreter:

copy /y "$(TargetDir)\SD.LLBLGen.Pro.TDLInterpreter.*" "..\..\..\Taskperformers\"

TDLParser:

copy /y "$(TargetDir)\SD.LLBLGen.Pro.TDLParser.*" "..\..\..\Taskperformers\"

Upgrading to .Net 3.5

The DotNetTemplateEngine uses the CodeDom to compile the LPT templates. The CodeDomProvider, i.e. the CSharpCodeProvider or VBCodeProvider, is constructed without specifying a target .Net framework version. This means that a default framework version is used. This framework version is determined by a call to the static method System.Runtime.InteropServices.RuntimeEnvironment.GetRuntimeDirectory() which even in a .Net 3.5 application on my computer returns C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\. Fortunately it is quite straightforward to pass a parameter to the CodeDomProvider that specifies a .Net framework version.

The CodeDomProvider is setup in the CompileTemplates method of the DotNetTemplateEngine. A switch statement is used to construct the specific CodeDomProvider. The start of this code looks like this:

CodeDomProvider compiler = null;

// now select the target compiler to use. Do this by using the templateLanguage set by the first templateID to compile.

string templateLanguageToUse = DetermineTemplateLanguage(executingGenerator, dotNetTemplateIDs);

string logicLanguageExtension = "cs";

switch(templateLanguageToUse)

{

In this part of the code, the parameters for the CodeDomProvider can be setup:

CodeDomProvider compiler = null;

// now select the target compiler to use. Do this by using the templateLanguage set by the first templateID to compile.

string templateLanguageToUse = DetermineTemplateLanguage(executingGenerator, dotNetTemplateIDs);

string logicLanguageExtension = "cs";

// Make the template engine use framework 3.5

Dictionary<string, string> providerOptions = new Dictionary<string, string>();

providerOptions.Add("CompilerVersion", "v3.5");

switch(templateLanguageToUse)

{

The providerOptions dictionary can be then passed as a parameter to the CodeDomProvider:

For C#:

case "C#":

// C# templates

ProduceCSCode(templateCode, referencedAssemblies, namespaces, executingGenerator, dotNetTemplateIDs);

compiler = new CSharpCodeProvider(providerOptions);

logicLanguageExtension = "cs";

break;

For VB.NET:

case "VB.NET":

// VB.NET templates

ProduceVBNETCode(templateCode, referencedAssemblies, namespaces, executingGenerator, dotNetTemplateIDs);

compiler = new VBCodeProvider(providerOptions);

logicLanguageExtension = "vb";

break;

All of this will change the framework version used by the LPT templates. But it is not enough to get the full benefit of .Net 3.5 in the templates. The System.Core assembly must be referenced in order to use Linq in LPT templates. Near the top of the CompileTemplates method in the DotNetTemplateEngine class, a hash table object is populated with assembly references:

Hashtable referencedAssemblies = new Hashtable();

referencedAssemblies.Add(typeof(IBindingList).Assembly.Location, null);

referencedAssemblies.Add(Path.Combine(Application.StartupPath, "SD.LLBLGen.Pro.GeneratorCore.dll"), null);

referencedAssemblies.Add(Path.Combine(Application.StartupPath, "SD.LLBLGen.Pro.ApplicationCore.dll"), null);

referencedAssemblies.Add(Path.Combine(Application.StartupPath, "SD.LLBLGen.Pro.DBDriverCore.dll"), null);

referencedAssemblies.Add(this.GetType().Assembly.Location, null);

Referencing System.Core can be done by adding the path to the System.Core.dll on your computer. The location I have used is the .Net 3.5 directory within the Reference Assemblies directory in Program Files. The following code uses the ‘ProgramFiles’ environment variable to figure out the location of the Reference Assemblies directory:

IDictionary environmentVariables = Environment.GetEnvironmentVariables();

string framework35Directory;

if (environmentVariables.Contains("ProgramFiles"))

{

framework35Directory = Path.Combine((string)environmentVariables["ProgramFiles"], @"Reference Assemblies\Microsoft\Framework\v3.5");

}

else

{

framework35Directory = @"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5";

}

Using this, the complete code for adding the assembly references will be:

Hashtable referencedAssemblies = new Hashtable();

referencedAssemblies.Add(typeof(IBindingList).Assembly.Location, null);

referencedAssemblies.Add(Path.Combine(Application.StartupPath, "SD.LLBLGen.Pro.GeneratorCore.dll"), null);

referencedAssemblies.Add(Path.Combine(Application.StartupPath, "SD.LLBLGen.Pro.ApplicationCore.dll"), null);

referencedAssemblies.Add(Path.Combine(Application.StartupPath, "SD.LLBLGen.Pro.DBDriverCore.dll"), null);

referencedAssemblies.Add(this.GetType().Assembly.Location, null);

// Add the location of System.Core.dll to the referencedAssemblies

IDictionary environmentVariables = Environment.GetEnvironmentVariables();

string framework35Directory;

if (environmentVariables.Contains("ProgramFiles"))

{

framework35Directory = Path.Combine((string)environmentVariables["ProgramFiles"], @"Reference Assemblies\Microsoft\Framework\v3.5");

}

else

{

framework35Directory = @"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5";

}

referencedAssemblies.Add(Path.Combine(framework35Directory, "System.Core.dll"), null);

Below these references is code that adds in the namespaces that are used by the template code, starting with:

namespaces.Add("System", null);

namespaces.Add("System.IO", null);

namespaces.Add("System.Collections", null);

namespaces.Add("System.Collections.Generic", null);

namespaces.Add("System.ComponentModel", null);

To this, the ‘System.Linq’ namespace should be added:

namespaces.Add("System", null);

namespaces.Add("System.IO", null);

namespaces.Add("System.Collections", null);

namespaces.Add("System.Collections.Generic", null);

namespaces.Add("System.ComponentModel", null);

namespaces.Add("System.Linq", null);

With these changes in place, LPT templates can now use both Linq and extension methods. Extension methods are a great way for un-complicating LPT templates. In our development team we have a class library which our templates reference that provides many extension methods for existing LLBLGen objects. These are two examples of these methods:

/// <summary>

/// Returns an entity name in camel case format, e.g. UserRole => userRole;

/// </summary>

/// <param name="entity"></param>

/// <returns></returns>

public static string CamelCaseName(this EntityDefinition entity)

{

return entity.Name.Substring(0, 1).ToLower() + entity.Name.Substring(1);

}

/// <summary>

/// Returns an entity name with spaces to separate all capitalised parts of the name, e.g. UserRole => User Role;

/// </summary>

/// <param name="entity"></param>

/// <returns></returns>

public static string SpacedName(this EntityDefinition entity)

{

string name = entity.Name;

StringBuilder spacedName = new StringBuilder();

for (int i = 0; i < name.Length; i++)

{

if (i > 0 && i < name.Length - 1 && name.Substring(i, 1).ToUpper() == name.Substring(i, 1))

{

spacedName.Append(" ");

}

spacedName.Append(name[i]);

}

return spacedName.ToString();

}

This is an example of using the CamelCaseName extension method with an entity and a field:

<%= entity.Name %>Entity <%= entity.CamelCaseName() %> = <%= entity.Name %>.Get(<%= primaryKey.CamelCaseName() %>);

Adding a Filter Task Parameter

Another improvement that we have made to the DotNetTemplateEngine is to add support for a new task parameter. This task parameter is a predicate expression used to filter which objects have code generated for them, when using an emit type that specifies a type of LLBLGen object. For example, when generating code for entities, it can be used to generate code only for entities that have a primary key using an expression ‘PrimaryKeyFields.Count > 0’. These expressions must resolve in to a Boolean value. The example expression is equivalent to the following lambda expression using ‘e’ as a parameter name for an entity:

e => e.PrimaryKeyFields.Count > 0

Linq Dynamic Expression API

To parse and apply this expression, the Linq Dynamic Expression API is used. This Dynamic Expression API is actually one of the C# samples distributed with Visual Studio 2008. If you have not already looked at these samples, they are contained in a zip file in the location <Program Files>\Microsoft Visual Studio 9.0\Samples\1033\CSharpSamples.zip. Once extracted, the Dynamic Expression API can be found in the directory LinqSamples\DynamicQuery.

The LptParserEngine project needs to reference the DynamicQuery project. If you have not already changed the LptParserEngine project target framework to version 3.5, adding a reference to this project in Visual Studio will prompt you to change the framework version to match the DynamicQuery project version. The DynamicQuery sample includes two C# files, Dynamic.cs, and Program.cs. Only Dynamic.cs is needed, so if Program.cs produces build errors, delete it. Changing the output type of the project from Console Application to Class Library would also be advisable. The output of the Dynamic Expression API also needs to be copied to the LLBLGen Pro task performers directory. This can be done by adding a command line instruction to the post build event of the DynamicQuery project of:

copy /y "$(TargetDir)\Dynamic.*" "..\..\..\Taskperformers\"

This command line assumes that the DynamicQuery project directory has been copied in to CodeGenerator solution folder, so it is side by side with the other projects in the solution.

The Dynamic.cs file contains a class called DynamicQueryable which contains extra Linq extension methods that accept text parameters instead of expression trees as parameters. It is the text based equivalent of the Queryable class in the System.Linq namespace in the System.Core assembly, but it does not support all of the Linq extension methods. For example ‘Join’, ‘GroupJoin’, and ‘SelectMany’ do not have a DynamicQueryable equivalent. In order to parse and apply the predicate expression, the DynamicQueryable equivalent of ‘Where’ will be used.

To get the DotNetTemplateEngine class ready to use Linq, two using statements should be added:

using System.Linq;

using System.Linq.Dynamic;

The second using statement brings in scope the DynamicQueryable class and makes its extension methods accessible.

Implementing the Filter Parameter

The first step in implementing the filter parameter is to define the string used for identifying this parameter. A string constant added to the top of the DynamicQueryable class can be used for this:

/// <summary>

/// Name of the filter expression task parameter

/// </summary>

private const string FilterParameterName = "filter";

A generic method can be used to return a filtered collection of objects to generate code for:

/// <summary>

/// Gets a filtered collecion of objects to emit code for.

/// </summary>

/// <typeparam name="T">The type of the object to filter</typeparam>

/// <param name="items">A collection of items to filter</param>

/// <param name="parameters">The task parameters</param>

/// <returns>

/// A filtered list of objects to emit code for, if the filter task parameter is present,

/// otherwise a list of all objects

/// </returns>

private static List<T> GetFilteredEmit<T>(IEnumerable items, Dictionary<string, TaskParameter> parameters)

{

if (parameters.ContainsKey(FilterParameterName))

{

string filterLambdaExpression = parameters[FilterParameterName].Value.Trim();

IQueryable<T> enumerableItems = items.Cast<T>().AsQueryable();

return enumerableItems.Where(filterLambdaExpression, null, null).ToList();

}

return items.Cast<T>().ToList();

}

The type parameter for this method is the type of object that is filtered. The method will filter the objects if the filter parameter is present, or just return a list of the original objects.

The DotNetTemplateEngine gets the objects to emit code for in the second overload of the Perform method; it is done within a switch statement that starts like this:

// start code emitting process.

string destinationFilename = "";

switch(emitTypeToPerform)

{

case EmitType.AllEntities:

There is a case statement for each of the emit types that can be passed as a parameter to the DotNetTemplateEngine. Each one, except for the Generic emit type, uses a ‘for’ loop to go through each object and generate code for it. This is the start of the loop for all entities:

case EmitType.AllEntities:

executingGenerator.SubTaskProgressInitHandler(executingGenerator.Entities.Count);

for(int i=0;i<executingGenerator.Entities.Count;i++)

{

EntityDefinition currentEntity = (EntityDefinition)executingGenerator.Entities[i];

Instead of using the Entities property for the ‘for’ loop, this can be changed to filter the entities before the code is generated:

case EmitType.AllEntities:

List<EntityDefinition> entitites = GetFilteredEmit<EntityDefinition>(executingGenerator.Entities, parameters);

executingGenerator.SubTaskProgressInitHandler(entitites.Count);

for (int i = 0; i < entitites.Count; i++)

{

EntityDefinition currentEntity = entitites[i];

For all of the other emit types, except generic, a similar change can be made to the for loops:

case EmitType.AllTypedLists:

List<TypedListDefinition> typedLists = GetFilteredEmit<TypedListDefinition>(executingGenerator.TypedLists, parameters);

executingGenerator.SubTaskProgressInitHandler(typedLists.Count);

for (int i = 0; i < typedLists.Count; i++)

{

TypedListDefinition currentTypedList = typedLists[i];

case EmitType.AllTypedViews:

List<TypedViewDefinition> typedViews = GetFilteredEmit<TypedViewDefinition>(executingGenerator.TypedViews, parameters);

executingGenerator.SubTaskProgressInitHandler(typedViews.Count);

for (int i = 0; i < typedViews.Count; i++)

{

TypedViewDefinition currentTypedView = typedViews[i];

case EmitType.ActionSPCalls:

List<SPCallDefinition> actionSPCalls = GetFilteredEmit<SPCallDefinition>(executingGenerator.ActionProcedures, parameters);

executingGenerator.SubTaskProgressInitHandler(actionSPCalls.Count);

for (int i = 0; i < actionSPCalls.Count; i++)

{

SPCallDefinition currentSPCall = actionSPCalls[i];

case EmitType.RetrievalSPCalls:

List<SPCallDefinition> retrievalSPCalls = GetFilteredEmit<SPCallDefinition>(executingGenerator.RetrievalProcedures, parameters);

executingGenerator.SubTaskProgressInitHandler(retrievalSPCalls.Count);

for (int i = 0; i < retrievalSPCalls.Count; i++)

{

SPCallDefinition currentSPCall = retrievalSPCalls[i];

Once these changes have been made, all objects can now be filtered using this new filter parameter.

Adding an All Relations Emit Type

Another type of object that is available from the LLBLGen project model is the EntityRelation object. The DotNetTemplateEngine can be enhanced to support a new emit type for generating code for each relation in the database. This can be quite useful if you want to generate objects specific to each relation but will also result in a lot of classes being generated for large, complex database.

The emit types are defined in a private enumeration nested in the DotNetTemplateEngine class:

#region Enums

private enum EmitType:int

{

AllEntities,

AllTypedLists,

AllTypedViews,

ActionSPCalls,

RetrievalSPCalls,

Generic

}

#endregion

A new emit type of AllRelations can be added to this enumeration:

#region Enums

private enum EmitType:int

{

AllEntities,

AllTypedLists,

AllTypedViews,

AllRelations,

ActionSPCalls,

RetrievalSPCalls,

Generic

}

#endregion

The Perform method is responsible for taking the string emit type passed as a parameter and converting it in to an enumeration:

EmitType emitTypeToPerform = EmitType.Generic;

switch(emitTypeSpecified)

{

case "allEntities":

emitTypeToPerform = EmitType.AllEntities;

break;

case "allTypedLists":

emitTypeToPerform = EmitType.AllTypedLists;

break;

case "allTypedViews":

emitTypeToPerform = EmitType.AllTypedViews;

break;

case "actionSPCalls":

emitTypeToPerform = EmitType.ActionSPCalls;

break;

case "retrievalSPCalls":

emitTypeToPerform = EmitType.RetrievalSPCalls;

break;

case "generic":

emitTypeToPerform = EmitType.Generic;

break;

default:

// error

_performResult = false;

return;

}

An extra case has to be added for AllRelations:

case "allRelations":

emitTypeToPerform = EmitType.AllRelations;

break;

And then in the emit code switch statement; an extra entry has to be added for emitting code for relations. The relations will be got by aggregating all of the relations belonging to the entities in the project. This entry will also use the filter parameter to filter the actual relations for which code is emitted.

case EmitType.AllRelations:

// Get all of the relations in one collection

List<EntityRelation> relations =

(from EntityDefinition e in executingGenerator.Entities

from EntityRelation r in e.Relations

select r).ToList();

relations = GetFilteredEmit<EntityRelation>(relations, parameters);

executingGenerator.SubTaskProgressInitHandler(relations.Count);

for (int i = 0; i < relations.Count; i++)

{

EntityRelation currentRelation = relations[i];

executingGenerator.SubTaskProgressTaskStartHandler("Generating code for relation : " + currentRelation.FullDescription);

// Create a usable name for the relation

string relationName = GetRelationFullName(currentRelation);

destinationFilename = Path.Combine(fullPath, CreateFilename(filenameFormat, relationName, executingGenerator.LanguageToUse.FileExtension));

// Add destination file to filecache

((ArrayList)executingGenerator.TaskCache["destinationFilenames"]).Add(destinationFilename);

if (!TestFileCanBeOverwritten(destinationFilename, failWhenExistent))

{

// skip

base.LogLine("No code generated code for relation '" + currentRelation.FullDescription + "' because file '" + destinationFilename + " could not be overwritten.", "DotNetTemplateEngine", true, true);

continue;

}

// all clear, generate code.

bool removeFile = false;

try

{

outputWriter = new StreamWriter(destinationFilename, false, _encodingToUse);

ITemplateClass templateObject = (ITemplateClass)compiledTemplatesAssembly.CreateInstance(templateID);

templateObject.___RUN(executingGenerator, parameters, outputWriter, currentRelation);

base.AddNewLogNode(_task.ElementLogNode, LogNodeType.ActionDescription, "Code generated for relation '{0}' into file '{1}' successfully.", currentRelation.FullDescription, destinationFilename);

base.LogLine("Generated code for relation '" + currentRelation.FullDescription + "' in file '" + destinationFilename + ".", "DotNetTemplateEngine", true, true);

}

catch (TemplateAbortException)

{

// signal that the template should be aborted.

removeFile = true;

// don't propagate this exception upwards, as it's not meant to be used that way.

}

finally

{

if (outputWriter != null)

{

outputWriter.Close();

}

executingGenerator.SubTaskProgressTaskCompletedHandler();

}

if (removeFile)

{

((ArrayList)executingGenerator.TaskCache["destinationFilenames"]).Remove(destinationFilename);

try

{

File.Delete(destinationFilename);

}

catch

{

// swallow

}

break;

This switch statement relies on an extra method called GetRelationFullName. Naming relations in a manner suitable for class names is potential problem with emitting code for relations. These names can be quite large. This implementation of GetRelationFullName generates a name similar to the value returned from the FullDescriptionWithFieldNames property of the EntityRelation class:

/// <summary>

/// Gets a name for the relation suitable for a type name.

/// </summary>

/// <param name="relation">The relation.</param>

/// <returns>A name for the relation suitable for a type name</returns>

private static string GetRelationFullName(EntityRelation relation)

{

// Create a usable name for the relation

StringBuilder relationName = new StringBuilder();

relationName.Append(relation.RelationStartName);

if (relation.RelationType != EntityRelationType.ManyToMany)

{

foreach (EntityFieldRelation fieldRelation in relation.FieldRelations)

{

relationName.Append("_");

relationName.Append(fieldRelation.RelationStartField.FieldName);

}

relationName.Append("_");

}

relationName.Append(relation.RelationEndName);

if (relation.RelationType != EntityRelationType.ManyToMany)

{

foreach (EntityFieldRelation fieldRelation in relation.FieldRelations)

{

relationName.Append("_");

relationName.Append(fieldRelation.RelationEndField.FieldName);

}

relationName.Append("_");

}

relationName.Append(relation.RelationType.ToString("f"));

if (relation.RelationsToWalk != null && relation.RelationsToWalk.Count > 1)

{

relationName.Append("Via");

relationName.Append(relation.RelationsToWalk[0].RelationEndName);

}

return relationName.ToString();

}

All of this will allow you to generate code files on a per relation basis and also will include the facility to filter which relations that you want to generate code for.

LLBLGen - Automating Code Generation with Team Foundation Server

Dylan Lewis — Wed, 04 Mar 2009 15:00:10 GMT

Introduction

At Light Speed IT Solutions we use the code generator LLBLGen to make much of our source code for our ASP.NET and SQL Server based applications. If source code can be derived from the schema of our application database, then we try to auto-generate it, and LLBLGen is our main tool for doing this with. We also use Team Foundation Server 2008 for our source control. This article describes the steps we have taken to automate the generation of source code, and is a guide as to how other development teams can automate the generation of their source code when using LLBLGen and Team Foundation Server.

The Manual Code Generation Process

Code generation with LLBLGen normally takes place outside of Visual Studio using the LLBLGen Pro application. The process of regenerating using the LLBLGen Pro application, in response to a schema change has many steps which a developer has to perform. Within the LLBLGen Pro application it is typical to do these steps after a schema change:

Refresh the catalogue that you have made schema changes to
Add in any extra entities from tables or views that you have added to the database
Regenerate all code created by LLBLGen

In our projects we use LLBLGen to create a data layer, business layer, and web user controls all derived from the database schema. We also use the Linq to SQL templates, generate unit tests and utility classes for making test data, and also generate code that makes use of the LLBLGen Authorizer system. This means that regenerating all of this code would require using the generate feature of LLBLGen Pro six times.

In addition to using the LLBLGen Pro application, we also have to make sure that the code we are regenerating is checked out and that any new files that are created are checked in to Team Foundation Server source control once the code generation is completed. As the code generation takes place outside of Visual Studio, Visual Studio does not add the new files created by LLBLGen in to source control. While checking out the code generated files is easy, adding in all the additional files, that get created in response to the schema changes, does take a lot longer. We also want to make sure that we are using the latest version of the LLBLGen project file, which we have under source control and any custom LLBLGen code generation templates and task files we are using. We don’t want a developer to generate code using templates that are weeks out of date, or to not incorporate the latest changes to the project file. We also have a class library containing utility classes used by the LLBLGen templates. Before code generation takes place, a developer needs to get the latest version of this class library, build it, and copy it to the LLBLGen Pro directory, from where it is referenced by the templates.

Automating the Code Generation

Performing all of these steps each time a schema change is made, and not missing any of them out has proved to be a challenge. Problems happen when even one of these steps is forgotten. Forgetting to add in a file to source control created by LLBLGen will not cause any build errors for the developer who does the code generation, but it will do for the next developer who gets the latest version of the affected projects. Build errors caused by not having the latest templates, tasks, and utility classes can be confusing for a developer who does not know what changes have been made to these since the last time the code was generated.

Fortunately LLBLGen Pro users can obtain the LLBLGen Pro SDK which contains two command line tools which can be used to automate the code generation process. Using these in combination with the Team Foundation Server command line tools, and the MSBuild command line tool, the code generation process can be automated to the point that a single click of a button can do all of these tasks.

LLBLGen Pro SDK

The LLBLGen Pro SDK is available to all people who have purchased LLBLGen Pro. It is distributed as a zip file that contains many different Visual Studio solutions. The two which are useful for automating the code generation process are the CommandLineGenerator and CliRefresher solutions. Both of these are solutions for building a command line tool. The CommandLineGenerator not surprisingly allows you to generate code using the command line. The actual program it compiles is called the CliGenerator. The CliRefresher allows you to refresh the catalogues in your LLBLGen project file from the command line.

Team Foundation Server Command Line Tools

To automate the code generation process, we use two different command line tools from Team Foundation Server. The first is the main Team Foundation Server command line tool, ‘TF.exe’. The second command line tool is distributed as part of the Team Foundation Server 2008 power tools (available for free from: http://msdn.microsoft.com/en-us/teamsystem/bb980963.aspx) called ‘TFPT.exe’. TF.exe can be used to perform many operations with Team Foundation Server including getting the latest version of files under source control, checking out files and checking them in. TFPT.exe has a command called ‘online’, which searches for offline changes made within a source controlled project, and notifies Team Foundation Server of these changes. It does the same job as the ‘Go Online’ button in Visual Studio, which is displayed when you work disconnected from Team Foundation Server.

Writing a Code Generation Batch File

Using a batch file, these command line tools can be used to automate the code generation process. If you are unfamiliar with batch files, this is a good guide for learning about them: http://commandwindows.com/batch.htm. We want the following steps to be automated when generating code with LLBLGen:

Get the latest version of the LLBLGen project file
Check out the LLBLGen project file
Refresh the schema information in the LLBLGen project file
Get the latest version of any custom LLBLGen templates and task files that are used
Get the latest version of any class libraries referenced by LLBLGen templates
Build and deploy the class libraries referenced by LLBLGen templates
Check out the files that we are going to regenerate
Generate the code with LLBLGen
Add any new files created in to Team Foundation Server source control

The command line tools already mentioned will be used to perform these steps. Getting the latest version of a file in source control is done with the TF.exe command. Using it in a batch file could look like this:

SET TFCMD="%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\tf"

SET ROOT=$/Your Team Project/Your Solution

%TFCMD% get "%ROOT%/Your LLBLGen Project file.lgp" /version:T

In the above batch file content, two variables are declared: TFCMD and ROOT. The TFCMD variable stores the location of the TF.exe command as it will be used quite frequently. The ROOT variable stores the location in Team Foundation Server source control of the solution for which code will be generated. This is also something that will be referred to frequently and so it is stored as a variable. TF.exe is called with the parameter ‘get’ to get a file from source control, the second parameter is the file which we want to get, and the third parameter is the version. The letter ‘T’ is used to indicate that it is the latest version that we want to get.

As we want to update the schema in the LLBLGen project file, first of all we need to check it out. This is also done with the TF.exe command and it will look like this:

%TFCMD% checkout "%ROOT%/Your LLBLGen Project file.lgp"

Next we want to update the schema using the CliRefresher command line tool:

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliRefresher" 1 "Your LLBLGen Project file.lgp"

The CliRefresher tool is called with the first parameter of 1 to indicate that the error log is shown, and the second parameter is of course the project file. This line assumes that the tool is in the LLBLGen directory. As the tool references key LLBLGen assemblies this is a good location for it.

The next step is to get the latest version of any custom templates and task files that are used for the solution. At Light Speed IT Solutions we use a Team Foundation Server Team Project called LSITS as a respository for various internal libraries and content shared between projects. This Team Project also contains the class library containing the utility classes which is used by our custom LLBLGen templates. Before generating any code it is important that we have the latest version of the tasks, templates and this class library. The part of the batch file for this is:

%TFCMD% get "$/LSITS/LLBLGen Tasks" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Templates" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Template Debugger" /version:T /recursive

%SystemRoot%\Microsoft.NET\Framework\v3.5\msbuild "D:\Projects\LSITS\LLBLGen Template Debugger\TemplateUtils\TemplateUtils.csproj"

if %ERRORLEVEL% == 0 GOTO End

echo Build Failure - Are you running LLBLGen?

pause

:End

TF.exe is used again to get the latest version of the files we need, this time using an extra parameter /recursive to make sure that all files under the specified source control folder are retrieved. The msbuild command line tool is called to build the TemplateUtils class library project, which has an extra MSBuild task that copies the assembly in to the LLBLGen directory. This is the task and target used:

</Target>

This step is done by the project file, not the batch file as we want this assembly to be copied in to the LLBLGen directory every time it is built, not just when it is built by this batch file.

After the assembly is compiled the %ERRORLEVEL% variable is checked to see if the return value from the msbuild command is 0, which means that everything was OK with compiling the project. If it was not 0 then a message is displayed and the batch file is paused. A common cause of the compilation failing is if a developer running the batch file also has the LLBLGen Pro application running, this assembly cannot be copied in to the LLBLGen directory. As soon as you generate code with LLBLGen Pro is loads any assemblies used by the code generation templates in to its app domain. This then prevents the files from being overwritten and cause the copying of the TemplateUtils assembly in to the LLBLGen directory to fail.

Next the files that will be regenerated have to be checked out. Which files depends entirely on what is being generated by LLBLGen. In the example below, it will be assumed that there are templates that generate code for a data layer, business layer, unit tests, and web application project, all of which are located in immediate subdirectories of the solution.

%TFCMD% checkout /recursive "%ROOT%/DataLayer"

%TFCMD% checkout /recursive "%ROOT%/BusinessLayer"

%TFCMD% checkout /recursive "%ROOT%/UnitTests"

%TFCMD% checkout /recursive "%ROOT%/Web"

These commands can be a specific as necessary only checking out directories that contain auto-generated code, but to make this example simple, all of the projects are being checked out.

After all of this preparation, the CliGenerator tool can be called to do the actual code generation. This is an example of one of the calls to CliGenerator:

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.DataLayer C# ".NET 3.5" Adapter SD.Presets.Adapter.General2008 DataLayer 0

There are several parameters that will vary depending on what code generation you actually will be doing. The project file is the first parameter. Immediately after is the root namespace that the code will be using. In the above example, ‘YourProject.DataLayer’ is used as the root namespace. Following this is the target language to use, in this case C#, followed by the target platform, .Net 3.5. After these are the template group and the preset name. Specifying the template bindings is not required by the CliGenerator as the template bindings will be inferred from the other parameters. The last two parameters are the target directory and whether to output to a log file (0 meaning don’t write to a log file). In this example, ‘DataLayer’ is a directory immediately under the solution folder where the data layer projects are kept. The paths you need to provide can be paths relative to the location of the batch file, in this case it is assumed to be in the root of the Visual Studio solution.

The CliGenerator should be called as many times as needed to generate all of the auto-generated code. After this, new files that have been created can be added in automatically to source control using the TFPT.exe command. Using the data layer again as an example, this statement can be used to add in the new files:

SET TFPTCMD="%ProgramFiles%\Microsoft Team Foundation Server 2008 Power Tools\tfpt"

%TFPTCMD% online /adds /noprompt DataLayer\*.cs /recursive

The /nopromt parameter tells the TFPT.exe command to pend the changes found automatically. Without it the TFPT.exe command displays a dialogue window with all of the changes it has found, just like the ‘Go Online’ button in Visual Studio. The /adds parameter tells the command to automatically add new files found. In the above example DataLayer\*.cs is used as the path of the items to check. In this case we are interested in finding new classes created by LLBLGen, so we are only interested in checking for new C# files. The *.cs part of the path describes a filter for the files checked.

Adding the files is the final step, so now a complete batch file can be assembled:

SET TFCMD="%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\tf"

SET ROOT=$/Your Team Project/Your Solution

%TFCMD% get "%ROOT%/Your LLBLGen Project file.lgp" /version:T

%TFCMD% checkout "%ROOT%/Your LLBLGen Project file.lgp"

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliRefresher" 1 "Your LLBLGen Project file.lgp"

%TFCMD% get "$/LSITS/LLBLGen Tasks" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Templates" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Template Debugger" /version:T /recursive

%SystemRoot%\Microsoft.NET\Framework\v3.5\msbuild "D:\Projects\LSITS\LLBLGen Template Debugger\TemplateUtils\TemplateUtils.csproj"

if %ERRORLEVEL% == 0 GOTO End

echo Build Failure - Are you running LLBLGen?

pause

:End

%TFCMD% checkout /recursive "%ROOT%/DataLayer”

%TFCMD% checkout /recursive "%ROOT%/BusinessLayer”

%TFCMD% checkout /recursive "%ROOT%/UnitTests”

%TFCMD% checkout /recursive "%ROOT%/Web”

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.DataLayer C# ".NET 3.5" Adapter SD.Presets.Adapter.General2008 DataLayer 0

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.BusinessLayer C# ".NET 3.5" Adapter BusinessLayer.Presets BusinessLayer 0

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.UnitTests C# ".NET 3.5" Adapter UnitTests.Presets UnitTests 0

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.Web C# ".NET 3.5" Adapter Web.Presets UnitTests 0

SET TFPTCMD="%ProgramFiles%\Microsoft Team Foundation Server 2008 Power Tools\tfpt"

%TFPTCMD% online /adds /noprompt DataLayer\*.cs /recursive

%TFPTCMD% online /adds /noprompt BusinessLayer\*.cs /recursive

%TFPTCMD% online /adds /noprompt UnitTests\*.cs /recursive

%TFPTCMD% online /adds /noprompt Web\*.cs /recursive

%TFPTCMD% online /adds /noprompt Web\*.aspx /recursive

%TFPTCMD% online /adds /noprompt Web\*.ascx /recursive

pause

In the web project, any new .aspx and .ascx files are also added in addition to new C# files.

One Click Code Generation

The batch file on its own can be used for regenerating all of the source code quite conveniently but Visual Studio can be configured to make the process better. By default, Team Foundation Server source control does not get the latest version of a file when it is checked out, but it can be configured to do so. When using Team Foundation Server source control, and generating code with LLBLGen, if many people in the project team are generating source code, this can lead to conflicts when checking the generated code in. If a developer generating the source code did not have the latest version of the files before checking them back in, the developer will be asked to resolve conflicts with all of the files. Team Foundation Server usually can resolve the conflicts automatically but with hundreds or more files this can take a while. If the option to get the latest version of a file when checked out is switched on, situations like this will be less frequent. This option can be found in the Tools->Options... menu item:

Getting the latest version of a file being generated also helps deal with another potential problem: having user code regions discarded. If the files being generated are up to date before the code generation takes place, then they will contain any user code regions added by other developers.

Another change to Visual Studio that can be made to improve the process is to make the batch file for generating the source code an external tool. The external tools used by Visual Studio can be found by using the Tools -> Externals Tools... menu item:

This is the Externals Tools dialogue window with a batch file as an additional external tool:

The batch file has been configured to use the solution directory as the working directory. This is the normal location for the batch file. It also has been configured to use an Output window. This has the advantage that all output from the batch file is contained within the Output window. This makes it easier to track down errors, as a normal command window that the batch file runs in has a limited buffer for the text it outputs. It does have the disadvantage that any pause statements in your batch file will be ignored.

This tool can be added as an item to a toolbar within Visual Studio. It may be tempting to make a new toolbar for all of your code generation batch files but in my experience new toolbars tend to disappear the next time Visual Studio is loaded. The best thing to do is to add the external tool to an existing toolbar. This can be done by dragging the external tool on to a toolbar from the ‘Customize’ dialogue window (from Tools -> Customize):

As the list of external commands is not named properly, i.e. they are just External Command 1, External Command 2 and so on; the external tool you added will be numbered by the position in the list of external tools in the Externals Tools dialogue window. As the GenerateAll batch file was the sixth external tool it is now ‘External Command 6’. Once the command is on a toolbar, it will be displayed with the original name you gave it when you added it as an external tool. And also, once it is on the toolbar, it can be run by clicking on it within Visual Studio.

Configuring the Catalogue Refreshing

By default new tables and views are not added as entities when they are detected during a refresh of a catalogue. In order to automate the code generation process further, LLBLGen can be configured to do this, allowing the CliRefresher tool to add new entities automatically. The CliRefresher tool seems to ignore the settings within an LLBLGen project file for the catalogue refresher, but it will use the preferences setup within a developer’s user profile. For LLBLGen 2.6, there is an XML preferences file stored within the user profile application data directory, i.e. C:\Documents and Settings\<Your User Name>\Application Data\LLBLGen Pro. In this directory are several files used by LLBLGen, the preferences are stored in the Preferences26.xml file. In this file is an XML element for the setting ‘AddNewElementsAfterRefresh’:

This is false by default but when set to true, tables will be added as entities automatically, as well as typed views and stored procedures. If you want views added automatically as entities then this setting can also be set to true: