LLBLGen

Automating Code Generation with Team Foundation Server

Introduction

At Light Speed IT Solutions we use the code generator LLBLGen to make much of our source code for our ASP.NET and SQL Server based applications. If source code can be derived from the schema of our application database, then we try to auto-generate it, and LLBLGen is our main tool for doing this with. We also use Team Foundation Server 2008 for our source control. This article describes the steps we have taken to automate the generation of source code, and is a guide as to how other development teams can automate the generation of their source code when using LLBLGen and Team Foundation Server.

The Manual Code Generation Process

Code generation with LLBLGen normally takes place outside of Visual Studio using the LLBLGen Pro application. The process of regenerating using the LLBLGen Pro application, in response to a schema change has many steps which a developer has to perform. Within the LLBLGen Pro application it is typical to do these steps after a schema change:

1. Refresh the catalogue that you have made schema changes to
2. Add in any extra entities from tables or views that you have added to the database
3. Regenerate all code created by LLBLGen

In our projects we use LLBLGen to create a data layer, business layer, and web user controls all derived from the database schema. We also use the Linq to SQL templates, generate unit tests and utility classes for making test data, and also generate code that makes use of the LLBLGen Authorizer system. This means that regenerating all of this code would require using the generate feature of LLBLGen Pro six times.

In addition to using the LLBLGen Pro application, we also have to make sure that the code we are regenerating is checked out and that any new files that are created are checked in to Team Foundation Server source control once the code generation is completed. As the code generation takes place outside of Visual Studio, Visual Studio does not add the new files created by LLBLGen in to source control. While checking out the code generated files is easy, adding in all the additional files, that get created in response to the schema changes, does take a lot longer. We also want to make sure that we are using the latest version of the LLBLGen project file, which we have under source control and any custom LLBLGen code generation templates and task files we are using. We don’t want a developer to generate code using templates that are weeks out of date, or to not incorporate the latest changes to the project file. We also have a class library containing utility classes used by the LLBLGen templates. Before code generation takes place, a developer needs to get the latest version of this class library, build it, and copy it to the LLBLGen Pro directory, from where it is referenced by the templates.

Automating the Code Generation

Performing all of these steps each time a schema change is made, and not missing any of them out has proved to be a challenge. Problems happen when even one of these steps is forgotten. Forgetting to add in a file to source control created by LLBLGen will not cause any build errors for the developer who does the code generation, but it will do for the next developer who gets the latest version of the affected projects. Build errors caused by not having the latest templates, tasks, and utility classes can be confusing for a developer who does not know what changes have been made to these since the last time the code was generated.

Fortunately LLBLGen Pro users can obtain the LLBLGen Pro SDK which contains two command line tools which can be used to automate the code generation process. Using these in combination with the Team Foundation Server command line tools, and the MSBuild command line tool, the code generation process can be automated to the point that a single click of a button can do all of these tasks.

LLBLGen Pro SDK

The LLBLGen Pro SDK is available to all people who have purchased LLBLGen Pro. It is distributed as a zip file that contains many different Visual Studio solutions. The two which are useful for automating the code generation process are the CommandLineGenerator and CliRefresher solutions. Both of these are solutions for building a command line tool. The CommandLineGenerator not surprisingly allows you to generate code using the command line. The actual program it compiles is called the CliGenerator. The CliRefresher allows you to refresh the catalogues in your LLBLGen project file from the command line.

Team Foundation Server Command Line Tools

To automate the code generation process, we use two different command line tools from Team Foundation Server. The first is the main Team Foundation Server command line tool, ‘TF.exe’. The second command line tool is distributed as part of the Team Foundation Server 2008 power tools (available for free from: http://msdn.microsoft.com/en-us/teamsystem/bb980963.aspx) called ‘TFPT.exe’. TF.exe can be used to perform many operations with Team Foundation Server including getting the latest version of files under source control, checking out files and checking them in. TFPT.exe has a command called ‘online’, which searches for offline changes made within a source controlled project, and notifies Team Foundation Server of these changes. It does the same job as the ‘Go Online’ button in Visual Studio, which is displayed when you work disconnected from Team Foundation Server.

Writing a Code Generation Batch File

Using a batch file, these command line tools can be used to automate the code generation process. If you are unfamiliar with batch files, this is a good guide for learning about them: http://commandwindows.com/batch.htm. We want the following steps to be automated when generating code with LLBLGen:

1. Get the latest version of the LLBLGen project file
2. Check out the LLBLGen project file
3. Refresh the schema information in the LLBLGen project file
4. Get the latest version of any custom LLBLGen templates and task files that are used
5. Get the latest version of any class libraries referenced by LLBLGen templates
6. Build and deploy the class libraries referenced by LLBLGen templates
7. Check out the files that we are going to regenerate
8. Generate the code with LLBLGen
9. Add any new files created in to Team Foundation Server source control

The command line tools already mentioned will be used to perform these steps. Getting the latest version of a file in source control is done with the TF.exe command. Using it in a batch file could look like this:

SET TFCMD="%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\tf"

SET ROOT=$/Your Team Project/Your Solution

%TFCMD% get "%ROOT%/Your LLBLGen Project file.lgp" /version:T

In the above batch file content, two variables are declared: TFCMD and ROOT. The TFCMD variable stores the location of the TF.exe command as it will be used quite frequently. The ROOT variable stores the location in Team Foundation Server source control of the solution for which code will be generated. This is also something that will be referred to frequently and so it is stored as a variable. TF.exe is called with the parameter ‘get’ to get a file from source control, the second parameter is the file which we want to get, and the third parameter is the version. The letter ‘T’ is used to indicate that it is the latest version that we want to get.

As we want to update the schema in the LLBLGen project file, first of all we need to check it out. This is also done with the TF.exe command and it will look like this:

%TFCMD% checkout "%ROOT%/Your LLBLGen Project file.lgp"

Next we want to update the schema using the CliRefresher command line tool:

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliRefresher" 1 "Your LLBLGen Project file.lgp"

The CliRefresher tool is called with the first parameter of 1 to indicate that the error log is shown, and the second parameter is of course the project file. This line assumes that the tool is in the LLBLGen directory. As the tool references key LLBLGen assemblies this is a good location for it.

The next step is to get the latest version of any custom templates and task files that are used for the solution. At Light Speed IT Solutions we use a Team Foundation Server Team Project called LSITS as a respository for various internal libraries and content shared between projects. This Team Project also contains the class library containing the utility classes which is used by our custom LLBLGen templates. Before generating any code it is important that we have the latest version of the tasks, templates and this class library. The part of the batch file for this is:

%TFCMD% get "$/LSITS/LLBLGen Tasks" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Templates" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Template Debugger" /version:T /recursive

%SystemRoot%\Microsoft.NET\Framework\v3.5\msbuild "D:\Projects\LSITS\LLBLGen Template Debugger\TemplateUtils\TemplateUtils.csproj"

if %ERRORLEVEL% == 0 GOTO End

echo Build Failure - Are you running LLBLGen?

pause

:End

TF.exe is used again to get the latest version of the files we need, this time using an extra parameter /recursive to make sure that all files under the specified source control folder are retrieved. The msbuild command line tool is called to build the TemplateUtils class library project, which has an extra MSBuild task that copies the assembly in to the LLBLGen directory. This is the task and target used:

<Target Name="AfterBuild">

<Copy SourceFiles="@(MainAssembly)" DestinationFolder="C:\Program Files\Solutions Design\LLBLGen Pro v2.6\" />

</Target>

This step is done by the project file, not the batch file as we want this assembly to be copied in to the LLBLGen directory every time it is built, not just when it is built by this batch file.

After the assembly is compiled the %ERRORLEVEL% variable is checked to see if the return value from the msbuild command is 0, which means that everything was OK with compiling the project. If it was not 0 then a message is displayed and the batch file is paused. A common cause of the compilation failing is if a developer running the batch file also has the LLBLGen Pro application running, this assembly cannot be copied in to the LLBLGen directory. As soon as you generate code with LLBLGen Pro is loads any assemblies used by the code generation templates in to its app domain. This then prevents the files from being overwritten and cause the copying of the TemplateUtils assembly in to the LLBLGen directory to fail.

Next the files that will be regenerated have to be checked out. Which files depends entirely on what is being generated by LLBLGen. In the example below, it will be assumed that there are templates that generate code for a data layer, business layer, unit tests, and web application project, all of which are located in immediate subdirectories of the solution.

%TFCMD% checkout /recursive "%ROOT%/DataLayer"

%TFCMD% checkout /recursive "%ROOT%/BusinessLayer"

%TFCMD% checkout /recursive "%ROOT%/UnitTests"

%TFCMD% checkout /recursive "%ROOT%/Web"

These commands can be a specific as necessary only checking out directories that contain auto-generated code, but to make this example simple, all of the projects are being checked out.

After all of this preparation, the CliGenerator tool can be called to do the actual code generation. This is an example of one of the calls to CliGenerator:

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.DataLayer C# ".NET 3.5" Adapter SD.Presets.Adapter.General2008 DataLayer 0

There are several parameters that will vary depending on what code generation you actually will be doing. The project file is the first parameter. Immediately after is the root namespace that the code will be using. In the above example, ‘YourProject.DataLayer’ is used as the root namespace. Following this is the target language to use, in this case C#, followed by the target platform, .Net 3.5. After these are the template group and the preset name. Specifying the template bindings is not required by the CliGenerator as the template bindings will be inferred from the other parameters. The last two parameters are the target directory and whether to output to a log file (0 meaning don’t write to a log file). In this example, ‘DataLayer’ is a directory immediately under the solution folder where the data layer projects are kept. The paths you need to provide can be paths relative to the location of the batch file, in this case it is assumed to be in the root of the Visual Studio solution.

The CliGenerator should be called as many times as needed to generate all of the auto-generated code. After this, new files that have been created can be added in automatically to source control using the TFPT.exe command. Using the data layer again as an example, this statement can be used to add in the new files:

SET TFPTCMD="%ProgramFiles%\Microsoft Team Foundation Server 2008 Power Tools\tfpt"

%TFPTCMD% online /adds /noprompt DataLayer\*.cs /recursive

The /nopromt parameter tells the TFPT.exe command to pend the changes found automatically. Without it the TFPT.exe command displays a dialogue window with all of the changes it has found, just like the ‘Go Online’ button in Visual Studio. The /adds parameter tells the command to automatically add new files found. In the above example DataLayer\*.cs is used as the path of the items to check. In this case we are interested in finding new classes created by LLBLGen, so we are only interested in checking for new C# files. The *.cs part of the path describes a filter for the files checked.

Adding the files is the final step, so now a complete batch file can be assembled:

SET TFCMD="%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\tf"

SET ROOT=$/Your Team Project/Your Solution

%TFCMD% get "%ROOT%/Your LLBLGen Project file.lgp" /version:T

%TFCMD% checkout "%ROOT%/Your LLBLGen Project file.lgp"

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliRefresher" 1 "Your LLBLGen Project file.lgp"

%TFCMD% get "$/LSITS/LLBLGen Tasks" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Templates" /version:T /recursive

%TFCMD% get "$/LSITS/LLBLGen Template Debugger" /version:T /recursive

%SystemRoot%\Microsoft.NET\Framework\v3.5\msbuild "D:\Projects\LSITS\LLBLGen Template Debugger\TemplateUtils\TemplateUtils.csproj"

if %ERRORLEVEL% == 0 GOTO End

echo Build Failure - Are you running LLBLGen?

pause

:End

%TFCMD% checkout /recursive "%ROOT%/DataLayer”

%TFCMD% checkout /recursive "%ROOT%/BusinessLayer”

%TFCMD% checkout /recursive "%ROOT%/UnitTests”

%TFCMD% checkout /recursive "%ROOT%/Web”

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.DataLayer C# ".NET 3.5" Adapter SD.Presets.Adapter.General2008 DataLayer 0

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.BusinessLayer C# ".NET 3.5" Adapter BusinessLayer.Presets BusinessLayer 0

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.UnitTests C# ".NET 3.5" Adapter UnitTests.Presets UnitTests 0

"%ProgramFiles%\Solutions Design\LLBLGen Pro v2.6\CliGenerator" "Your LLBLGen Project file.lgp" YourProject.Web C# ".NET 3.5" Adapter Web.Presets UnitTests 0

SET TFPTCMD="%ProgramFiles%\Microsoft Team Foundation Server 2008 Power Tools\tfpt"

%TFPTCMD% online /adds /noprompt DataLayer\*.cs /recursive

%TFPTCMD% online /adds /noprompt BusinessLayer\*.cs /recursive

%TFPTCMD% online /adds /noprompt UnitTests\*.cs /recursive

%TFPTCMD% online /adds /noprompt Web\*.cs /recursive

%TFPTCMD% online /adds /noprompt Web\*.aspx /recursive

%TFPTCMD% online /adds /noprompt Web\*.ascx /recursive

pause

In the web project, any new .aspx and .ascx files are also added in addition to new C# files.

One Click Code Generation

The batch file on its own can be used for regenerating all of the source code quite conveniently but Visual Studio can be configured to make the process better. By default, Team Foundation Server source control does not get the latest version of a file when it is checked out, but it can be configured to do so. When using Team Foundation Server source control, and generating code with LLBLGen, if many people in the project team are generating source code, this can lead to conflicts when checking the generated code in. If a developer generating the source code did not have the latest version of the files before checking them back in, the developer will be asked to resolve conflicts with all of the files. Team Foundation Server usually can resolve the conflicts automatically but with hundreds or more files this can take a while. If the option to get the latest version of a file when checked out is switched on, situations like this will be less frequent. This option can be found in the Tools->Options... menu item:

Getting the latest version of a file being generated also helps deal with another potential problem: having user code regions discarded. If the files being generated are up to date before the code generation takes place, then they will contain any user code regions added by other developers.

Another change to Visual Studio that can be made to improve the process is to make the batch file for generating the source code an external tool. The external tools used by Visual Studio can be found by using the Tools -> Externals Tools... menu item:

This is the Externals Tools dialogue window with a batch file as an additional external tool:

The batch file has been configured to use the solution directory as the working directory. This is the normal location for the batch file. It also has been configured to use an Output window. This has the advantage that all output from the batch file is contained within the Output window. This makes it easier to track down errors, as a normal command window that the batch file runs in has a limited buffer for the text it outputs. It does have the disadvantage that any pause statements in your batch file will be ignored.

This tool can be added as an item to a toolbar within Visual Studio. It may be tempting to make a new toolbar for all of your code generation batch files but in my experience new toolbars tend to disappear the next time Visual Studio is loaded. The best thing to do is to add the external tool to an existing toolbar. This can be done by dragging the external tool on to a toolbar from the ‘Customize’ dialogue window (from Tools -> Customize):

As the list of external commands is not named properly, i.e. they are just External Command 1, External Command 2 and so on; the external tool you added will be numbered by the position in the list of external tools in the Externals Tools dialogue window. As the GenerateAll batch file was the sixth external tool it is now ‘External Command 6’. Once the command is on a toolbar, it will be displayed with the original name you gave it when you added it as an external tool. And also, once it is on the toolbar, it can be run by clicking on it within Visual Studio.

Configuring the Catalogue Refreshing

By default new tables and views are not added as entities when they are detected during a refresh of a catalogue. In order to automate the code generation process further, LLBLGen can be configured to do this, allowing the CliRefresher tool to add new entities automatically. The CliRefresher tool seems to ignore the settings within an LLBLGen project file for the catalogue refresher, but it will use the preferences setup within a developer’s user profile. For LLBLGen 2.6, there is an XML preferences file stored within the user profile application data directory, i.e. C:\Documents and Settings\<Your User Name>\Application Data\LLBLGen Pro. In this directory are several files used by LLBLGen, the preferences are stored in the Preferences26.xml file. In this file is an XML element for the setting ‘AddNewElementsAfterRefresh’:

<addNewElementsAfterRefresh value="true" />

This is false by default but when set to true, tables will be added as entities automatically, as well as typed views and stored procedures. If you want views added automatically as entities then this setting can also be set to true:

<addNewViewsAsEntitiesAfterRefresh value="true" />