1. Code generation – Generating code in such a way that I would have more semantic information about the existing code and thus be able to generate code in a “smarter” way.
2. C# to Html syntax highlighting – There are a few solutions out there but for one reason or another, I’m just not happy with them.

At the time of this writing the Roslyn project does not support attributes or partial methods. For my code generation ideas, I need support for both these C# language features. The syntax highlighting “project” is more a way for getting my feet wet with Roslyn, since I’ve never really worked with a parser/ lexical scanner/compiler.

Here is a live online demo version you can use to colorize your C# code:

C# to Html Syntax Highlighter

In this post I’ll present a C# to Html syntax highlighter that I’ll publish online as a service so anyone can use it independent of an IDE. The primary point of focus (in terms of highlighting) in this project is the ability to highlight types that are unknown. This area is the biggest problem I have with other syntax highlighters.

The Roslyn Syntax APIs give you information about the syntactic structure of the code you provide it with. However, that’s not enough to do a good job of colorizing code the way we’d expect it to. The reason is that there are many cases in which names of types have no meaning unless the appropriate assemblies and namespaces are “in-scope” in order to glean more semantic information about the code. So for example, lets take a look at the snippet of code below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication27
{
  class Program
  {
    static void Main(string[] args)
    {
      Customer customer = new Customer();
    }
  }
}

Customer customer = new Customer();

Is syntactically legal C# code. However, without any semantic information about the rest of the code, the type Customer is unknown and as as result won’t get colored as an identifier (in teal) as we see above. In fact in VS the compiler will issue an error and put squiggles under the text Customer. The error would be: The type or namespace name 'Customer' could not be found (are you missing a using directive or an assembly reference?)

There are many such cases that present difficulties in proper colorizing when attempting to do this outside of the IDE, as we are since the project presented here is for online use or as a plug-in to other tools such as Windows Live Writer that won’t be able to provide the additional semantic information needed for this job.

When I initially started down this road I thought this would be a fairly simple matter because I found SyntaxFacts (as static class in the Roslyn.Compilers.CSharp namespace) that had methods such as:

• IsKeyword
• IsContextualKeyword
• IsTypeDecleration
• IsPredefinedType

So it would be a simple matter of iterating over all of the tokens in the syntax tree and processing each token differently. Instead of iterating over all of the tokens, we could Visit each token (Visitor Pattern).

As part of the Roslyn project we get a class called SyntaxWalker that descends an entire SyntaxNode graph visiting each SyntaxNode and its children nodes and SyntaxTokens in depth-first order. For our purposes this is perfect since the depth-first order will allow us to write out html as we’re walking the tree.

Thanks go out to Shyam Namboodiripad who is on the Roslyn team and without whose help this project would have take a heck of a lot longer and probably never have completed. So Thank you Shyam!

C# to Html Syntax Highlighter

The way you would use the classes presented in this post is really very simple.

var html = CSharpToHtmlSyntaxHighlighter.GetHtml("SomeCode"));

  .Keyword { color: #0000ff; }
  .StringLiteral { color: #a31515; }
  .CharacterLiteral { color: #d202fe; }
  .Identifier { color: #2b91af; }
  .Comment { color: #008000; }
  .Region { color: #e0e0e0; }

Introducing the CSharpToHtmlSyntaxHighlighter

using System.Text;
using System.Web;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;

namespace Matlus.SyntaxHighlighter
{
  public static class CSharpToHtmlSyntaxHighlighter
  {
    private static readonly AssemblyFileReference mscorlib = new AssemblyFileReference(typeof(object).Assembly.Location);

    private static SemanticModel GetSemanticModelForSyntaxTree(SyntaxTree syntaxTree)
    {      
      var compilation = Compilation.Create(
        outputName: "CSharpToHtmlSyntaxHighlighterCompilation",
        syntaxTrees: new[] { syntaxTree },
        references: new[] { mscorlib });

      return compilation.GetSemanticModel(syntaxTree);
    }

    public static string GetHtml(string snippetOfCode)
    {
      var syntaxTree = SyntaxTree.ParseCompilationUnit(snippetOfCode);
      var semanticModel = GetSemanticModelForSyntaxTree(syntaxTree);
      var htmlColorizerSyntaxWalker = new HtmlColorizerSyntaxWalker();

      var htmlBuilder = new StringBuilder();
      htmlColorizerSyntaxWalker.DoVisit(syntaxTree.Root, semanticModel, (tk, text) =>
        {
          switch (tk)
          {
            case TokenKind.None:
              htmlBuilder.Append(text);
              break;
            case TokenKind.Keyword:
            case TokenKind.Identifier:
            case TokenKind.StringLiteral:
            case TokenKind.CharacterLiteral:
            case TokenKind.Comment:
            case TokenKind.DisabledText:
            case TokenKind.Region:
              htmlBuilder.Append("<span class=\"" + tk.ToString() + "\">" + HttpUtility.HtmlEncode(text) + "</span>");
              break;
            default:
              break;
          }
        });
      return htmlBuilder.ToString();
    }
  }
}

Let's take a look at the GetHtml() static method. This is the method that kicks off the whole process.

• Given a snippet of code, we create a SyntaxTree using a helper method of the SyntaxTree class.
• Next, using the syntax tree we create a SemanticModel using the code in the GetSemanticModelForSyntaxTree method.
• And finally, we call the DoVisit() method of the HtmlColorizerSyntaxWalker class.

When we call the DoVisit() method we pass it an Action delegate that we implement as a lambda expression as shown above. Each time the HtmlColorizerSyntaxWalker class finds a token of "interest" while walking the tree it calls us back in this Action delegate (the lambda above) and it is in this method that we generate the html and colorize each of the tokens in the way we desire.

Introducing the HtmlColorizerSyntaxWalker

The code listing below shows the entire HtmlColorizerSyntaxWalker class. The primary method in this class is the DoVisit() method. This method initiates the “Visit” process, providing the SyntaxWalker with the root of the syntax tree that we want it to walk along with a couple of other parameters.

using System;
using Roslyn.Compilers.CSharp;

namespace Matlus.SyntaxHighlighter
{
  internal class HtmlColorizerSyntaxWalker : SyntaxWalker
  {
    private SemanticModel semanticModel;
    private Action<TokenKind, string> writeDelegate;

    internal void DoVisit(SyntaxNode token, SemanticModel semanticModel, Action<TokenKind, string> writeDelegate)
    {
      this.semanticModel = semanticModel;
      this.writeDelegate = writeDelegate;
      Visit(token);
    }

    // Handle SyntaxTokens
    protected override void VisitToken(SyntaxToken token)
    {
      base.VisitLeadingTrivia(token);

      var isProcessed = false;
      if (token.IsKeyword())
      {
        writeDelegate(TokenKind.Keyword, token.GetText());
        isProcessed = true;
      }
      else
      {
        switch (token.Kind)
        {
          case SyntaxKind.StringLiteralToken:
            writeDelegate(TokenKind.StringLiteral, token.GetText());
            isProcessed = true;
            break;
          case SyntaxKind.CharacterLiteralToken:
            writeDelegate(TokenKind.CharacterLiteral, token.GetText());
            isProcessed = true;
            break;
          case SyntaxKind.IdentifierToken:
            if (token.Parent is SimpleNameSyntax)
            {
              // SimpleName is the base type of IdentifierNameSyntax, GenericNameSyntax etc.
              // This handles type names that appear in variable declarations etc.
              // e.g. "TypeName x = a + b;"
              var name = (SimpleNameSyntax)token.Parent;
              var semanticInfo = semanticModel.GetSemanticInfo(name);
              if (semanticInfo.Symbol != null && semanticInfo.Symbol.Kind != SymbolKind.ErrorType)
              {
                switch (semanticInfo.Symbol.Kind)
                {
                  case SymbolKind.NamedType:
                    writeDelegate(TokenKind.Identifier, token.GetText());
                    isProcessed = true;
                    break;
                  case SymbolKind.Namespace:
                  case SymbolKind.Parameter:
                  case SymbolKind.Local:
                  case SymbolKind.Field:
                  case SymbolKind.Property:
                    writeDelegate(TokenKind.None, token.GetText());
                    isProcessed = true;
                    break;
                  default:
                    break;
                }
              }
            }
            else if (token.Parent is TypeDeclarationSyntax)
            {
              // TypeDeclarationSyntax is the base type of ClassDeclarationSyntax etc.
              // This handles type names that appear in type declarations
              // e.g. "class TypeName { }"
              var name = (TypeDeclarationSyntax)token.Parent;
              var symbol = semanticModel.GetDeclaredSymbol(name);
              if (symbol != null && symbol.Kind != SymbolKind.ErrorType)
              {
                switch (symbol.Kind)
                {
                  case SymbolKind.NamedType:
                    writeDelegate(TokenKind.Identifier, token.GetText());
                    isProcessed = true;
                    break;
                }
              }
            }
            break;
        }
      }

      if (!isProcessed)
        HandleSpecialCaseIdentifiers(token);

      base.VisitTrailingTrivia(token);
    }

    private void HandleSpecialCaseIdentifiers(SyntaxToken token)
    {
      switch (token.Kind)
      {
        // Special cases that are not handled because there is no semantic context/model that can truely identify identifiers.
        case SyntaxKind.IdentifierToken:
          if ((token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.Parameter)
            || (token.Parent.Kind == SyntaxKind.EnumDeclaration)
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.Attribute)
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.CatchDeclaration)
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.ObjectCreationExpression)
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.ForEachStatement && !(token.GetNextToken().Kind == SyntaxKind.CloseParenToken))
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Parent.Kind == SyntaxKind.CaseSwitchLabel && !(token.GetPreviousToken().Kind == SyntaxKind.DotToken))
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.MethodDeclaration)
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.CastExpression)
            //e.g. "private static readonly HashSet patternHashSet = new HashSet();" the first HashSet in this case
            || (token.Parent.Kind == SyntaxKind.GenericName && token.Parent.Parent.Kind == SyntaxKind.VariableDeclaration)
            //e.g. "private static readonly HashSet patternHashSet = new HashSet();" the second HashSet in this case
            || (token.Parent.Kind == SyntaxKind.GenericName && token.Parent.Parent.Kind == SyntaxKind.ObjectCreationExpression)
            //e.g. "public sealed class BuilderRouteHandler : IRouteHandler" IRouteHandler in this case
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.BaseList)
            //e.g. "Type baseBuilderType = typeof(BaseBuilder);" BaseBuilder in this case
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Parent.Parent.Kind == SyntaxKind.TypeOfExpression)
            // e.g. "private DbProviderFactory dbProviderFactory;" OR "DbConnection connection = dbProviderFactory.CreateConnection();"
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.VariableDeclaration)
            // e.g. "DbTypes = new Dictionary();" DbType in this case
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.TypeArgumentList)
            // e.g. "DbTypes.Add("int", DbType.Int32);" DbType in this case
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.MemberAccessExpression && token.Parent.Parent.Parent.Kind == SyntaxKind.Argument && !(token.GetPreviousToken().Kind == SyntaxKind.DotToken || Char.IsLower(token.GetText()[0])))
            // e.g. "schemaCommand.CommandType = CommandType.Text;" CommandType in this case
            || (token.Parent.Kind == SyntaxKind.IdentifierName && token.Parent.Parent.Kind == SyntaxKind.MemberAccessExpression && !(token.GetPreviousToken().Kind == SyntaxKind.DotToken || Char.IsLower(token.GetText()[0])))
            )
          {
            writeDelegate(TokenKind.Identifier, token.GetText());
          }
          else
          {
            if (token.GetText() == "HashSet")
            {
            }
            writeDelegate(TokenKind.None, token.GetText());
          }
          break;
        default:
          writeDelegate(TokenKind.None, token.GetText());
          break;
      }
    }

    // Handle SyntaxTrivia
    protected override void VisitTrivia(SyntaxTrivia trivia)
    {
      switch (trivia.Kind)
      {
        case SyntaxKind.MultiLineCommentTrivia:
        case SyntaxKind.SingleLineCommentTrivia:
          writeDelegate(TokenKind.Comment, trivia.GetText());
          break;
        case SyntaxKind.DisabledTextTrivia:
          writeDelegate(TokenKind.DisabledText, trivia.GetText());
          break;
        case SyntaxKind.DocumentationComment:
          writeDelegate(TokenKind.Comment, trivia.GetText());
          break;
        case SyntaxKind.RegionDirective:
        case SyntaxKind.EndRegionDirective:
          writeDelegate(TokenKind.Region, trivia.GetText());
          break;
        default:
          writeDelegate(TokenKind.None, trivia.GetText());
          break;
      }
      base.VisitTrivia(trivia);
    }
  }
}

The 3rd parameter to the DoVisit() method is an Action delegate or a callback that gets called each time the SyntaxWalker has determined that the token it is currently on is one we would be interested in. When it calls us back on this delegate it also lets us know the kind of token is has found using the TokenKind enum. The TokenKind enum is not a built in type. I couldn’t find a suitable type so I had to define one that worked best for this purpose (syntax highlighting). The definition of this enum is shown below:

namespace Matlus.SyntaxHighlighter
{
  internal enum TokenKind
  {
    None,
    Keyword,
    Identifier,
    StringLiteral,
    CharacterLiteral,
    Comment,
    DisabledText,
    Region    
  }
}

The rest of the code looks quite complicated but really isn't. It is basically a bunch of conditional statements.

The method that handles all of the (special) cases where *we* know the token is an identifier but Roslyn can't (because it lacks semantic information) is, HandleSpecialCaseIdentifiers. If you come across any token that falls through the cracks, you’ll need to add a conditional statement here to handle that case.

In my mind it is inevitable and as an entrepreneur or business if you want to stay ahead of the “S” curve, this is the time to get cracking. The World Wide Net is about getting out of and beyond the box.

But first let me level-set a couple of terms for this discussion.
Internet: Internet is a short form for the word Internetwork. It is essentially an infrastructure comprised of hardware communication devices providing connectivity between computers.

World Wide Web: The World Wide Web is web of interconnected documents and application that use the Internet as a transport medium. As a visual, what you see depicted in the image below is the current state.

You have:

• Computers connected via the Internet sharing documents and applications
• Humans primarily living in a virtual space
  

There are a few problems as I see it, with this status. Humans need more than “virtual”, we need to be able to share and feel more than we currently can. Essentially, we need to get out of and beyond the box. How is this going to happen and what is it going to take?

A Browser as we know it today is not going to cut it, for many reasons. Browsers confine us even more to the “box”.

Mobile phones and devices (Android and iPhone/iPad) have taken a step in the right direction and that is, native application (available from the store) that have access to hardware on the device such as the gyro, accelerometer, GPS etc. These differences make the big difference in our experience between what we can do over the Internet with our PCs and mobile devices. Gaming consoles such as the XBOX and Play Station provide other capabilities such as high performance HD graphics, and have bridged a gap by bringing the Internet to these devices.

World Wide Net is more than the two (mobile phones and gaming consoles) combined. First Gaming consoles and PCs will need to merge and the browser needs to go. Don’t get me wrong, I love the direction browsers are moving in and I spend virtually all of my time working with browsers. As a result I feel the constraints more so, I guess. Browsers provide us with a sandbox and that’s important but we need more then what this sandbox provides us.

So imagine a world where your PC has all of the capabilities of a PC and a gaming console and in addition the operating system rather than the browser provides the sandbox. The sandbox the operating system provides is similar to what mobile devices provide for applications downloaded from the app store. When applications need access to certain hardware (like the GPS system) they ask for permission.

But beyond the box goes further than the box (obviously). So the World Wide Net is about having access to hardware devices that may be connected directly to your PC via a USB port or serial port or via your home network. The devices I’m talking about don’t exist yet (for the most part). Some are around the corner but most of them are yet to be dreamed up. Devices that are around the corner are devices such as networked refrigerators, microwave ovens, music system etc..

These devices need to be programmable via an API and not just networked, such that any software engineer can program a certain class of device using this API.

Talking to peripheral and networked devices is already possible today and has been for many years now, but it requires native applications (written in C, C++, C#, Java etc.). User interfaces are going to be more “game like” than what we have today. All of this will require an operating system level sandbox rather than a browser based sandbox and the capabilities of programming languages such as those mentioned above, rather than HTML, SVG and JavaScript.

The image below shows you my vision of the World Wide Net. The Internet is still the transport/communications layer, but we move from the World Wide Web, to the World Wide Net, of connections out of and beyond the box, more "real". Where people you interact with on the World Wide Net can control devices connected to your computer via applications that run in the sandbox of the operating system.

Being an electronics engineer by qualification (It’s a hobby now), I can dream up a number of devices that I can build and hook up to my computer and share with friends and family and interact and socialize with them in the “real” world, beyond the box. But as I said, most devices (that I’m talking about) don’t exist yet. They are a new breed of devices we don’t have a need for today, primarily because we don’t have the means to use them.

We’ll still have browsers and a need for them in the era of the World Wide Net because we'll continue to share "documents", we just need go out of and beyond the box to share more the documents.

In WCF 4.0 (in contrast to earlier versions), things have gotten a lot easier in as far a configuration is concerned. WCF 4.0 stresses convention over configuration. So we'll first get our client and service up and running using conventions (no configuration at all) and then we'll take a look at how we would configure things if we need to. But rather than using the config file we'll programmatically configure things.

WCF Service Application

We'll start with the WCF service application first. So create a new console application and follow along. We'll need reference 2 assemblies in this project so lets go ahead and do that as well. The 2 assemblies are:

1. System.ServiceModel
2. System.Runtime.Serialization

In order to define what methods and properties our service has we define an interface that will represent our service contract. This is a regular interface that has the properties and methods we want to include in our service contract. The only additional thing to do here is to decorate the interface with the ServiceContract attribute. So go ahead and create a new class file define the service contract as shown below:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using WCFTcpServer;
using System.ServiceModel;
 
namespace WcfServer
{
  [ServiceContract]
  interface ICustomerService
  {
    [OperationContract]
    IEnumerable<Customer> GetCustomers();
    [OperationContract]
    Customer GetCustomer(int id);
    [OperationContract]
    int InsertCustomer(Customer customer);
    [OperationContract]
    void UpdateCustomer(Customer customer);
    [OperationContract]
    void DeleteCustomer(int customerId);
  }
}

Notice that the interface is decorated with the ServiceContract attribute and each of the methods we want to expose is decorated with the OperationContract attribute. That takes care of our service contract.

The next thing we need to do is define the Customer class that this service contract is operating on. So lets create a new class file and define our Customer class as shown below:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.Serialization;
 
namespace WcfServer
{
  [DataContract]
  public class Customer
  {
    [DataMember]
    public int Id { get; set; }
    [DataMember]
    public string FirstName { get; set; }
    [DataMember]
    public string LastName { get; set; }
 
    public override string ToString()
    {
      return String.Concat(
        "Id: " + Id.ToString(),
        "  Name: " + FirstName,
        " " + LastName);
    }
  }
}

Notice that the Customer class has been decorated with the DataContract attribute and each of the public properties we wish to expose as part of our service contract is decorated with the DataMember attribute. We've also overridden the ToString()method here just as a convenience (displaying in the console window).

Next, well need to implement our service contract. That is we'll need to define a class that implements our ICustomerService interface. The implementation is not important for this discussion as it is a regular class and indeed, the implementation is not specific to WCF.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
 
namespace WcfServer
{
  [ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple,
    InstanceContextMode = InstanceContextMode.Single)]
  public class CustomerService : ICustomerService
  {
    private List<Customer> customers;
 
    public CustomerService()
    {
      customers = new List<Customer> {
        new Customer { Id=1, FirstName="George", LastName="Washington"},
        new Customer { Id=2, FirstName="John", LastName="Adams"},
        new Customer { Id=3, FirstName="Thomas", LastName="Jefferson"},
        new Customer { Id=4, FirstName="James", LastName="Madison"}         
      };
    }
    
    public IEnumerable<Customer> GetCustomers()
    {
      Console.WriteLine("GetCustomers()");
      return customers;
    }
 
    public Customer GetCustomer(int id)
    {
      Console.WriteLine("GetCustomer(" + id.ToString() + ")");
      return customers.Where(c => c.Id == id).First();
    }
 
    public int InsertCustomer(Customer customer)
    {
      Console.WriteLine("InsertCustomer()");
      var newId = customers.Max(c => c.Id) + 1;
      customer.Id = newId;
      customers.Add(customer);
      return newId;
    }
 
    public void UpdateCustomer(Customer customer)
    {
      Console.WriteLine("UpdateCustomer()");
      Console.WriteLine(customer);
    }
 
    public void DeleteCustomer(int customerId)
    {
      Console.WriteLine("DeleteCustomer(" + customerId.ToString() + ")");
      var customerToDelete = customers.Where(c => c.Id == customerId).Select(c => c).First();
      if (customerToDelete != null)
        customers.Remove(customerToDelete);
    }
  }
}

Notice that the CustomerService class has been decorated with the ServiceBehavior. You don't need to specify any of the named parameters you see there.

We're now at the final part of our WCF Service application. WCF services can support multiple transport protocols (in the same service application without any code changes in our service), such as HTTP, HTTPS, TCP, P2P (Peer to Peer), IPC (Named Pipes), MSMQ (Message Queue). We can even choose to host our service in IIS or choose to host it in our own application (as we are doing here). In the service we're building we'll support 3 of these transport protocols (just for kicks).

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Description;
using System.ServiceModel.Channels;
 
namespace WcfServer
{
  class WcfServiceHost
  {
    static void Main(string[] args)
    {
      using (ServiceHost host = new ServiceHost(typeof(CustomerService),
        new Uri("http://localhost:8080"),
        new Uri("net.pipe://localhost"),
        new Uri("net.tcp://localhost:8081")))
      {
        host.AddServiceEndpoint(typeof(ICustomerService),
          new BasicHttpBinding(),
          "CustomerService");
 
        host.AddServiceEndpoint(typeof(ICustomerService),
          new NetNamedPipeBinding(),
          "CustomerService");
 
        host.AddServiceEndpoint(typeof(ICustomerService),
          new NetTcpBinding(),
          "CustomerService");
 
        host.Open();
 
        Console.WriteLine("Service is ready and waiting.\r\nPress <ENTER> to exit.");
        Console.ReadLine();
 
        host.Close();
      }
    }
  }
}

When we construct our ServiceHost instance (the host variable in the code above) we specify two parameters:

1. The Type of our service (that is the Type of the class that implements our service contract)
2. A params array Uri that is the array of BaseAddresses our service will support.

A BaseAddress is essentially the initial part of the Uri for the hosted service. It includes the transport protocol as well as a port number (if required by the transport protocol). So you can see above that we've provided 3 bases addresses, one for each of the transport protocols we'd like our service to support.

Once we have our ServiceHost instance, we add the endpoints for each of the transport protocols we've added (those BaseAddresses in the constructor).

The AdServiceEnpoint() method has 9 overloads, bur essentially we need to either provide and endpoint instance or provide the information that makes up an endpoint (see note above). In the overload we're using we provide the Contract (ICustomerService), the Binding (for example BasicHttpBinding) and the Address (the string "CustomerService"). The address can be an absolute url or a relative url. We're using a relative url since we've provided the BaseAddress for each of the transport protocols earlier. Effectively, the address for each of the endpoints is the concatenation of the BaseAddress and the relative url. For for the Http endpoint our url is http://localhost:8080/CustomerService.

Once we've configured our host we can call the Open() method on it and that starts the host listening for requests from clients. Our WCF service application is now complete.

WCF Client Application

The typical way to start with building a client for a service is to get the IDE to generate a proxy (just like with a SOAP service). With WCF you can do that, however, at the moment, our service does not support metadata discovery. With WCF there is a simpler way to create a proxy and that is to use the ChannelFactory<T> class. For the client we add a new console application project our solution. The code listing below shows the code for the WCF client.

using System;
using System.Linq;
using System.ServiceModel;
using WcfServer;
 
namespace WCFClient
{
  class WcfCustomerServiceClient
  {
    static void Main(string[] args)
    {
      var channelFactory = new ChannelFactory<ICustomerService>(new BasicHttpBinding(),

        "http://localhost:8080/CustomerService");
      var customerService = channelFactory.CreateChannel();
      try
      {
        Console.WriteLine("GetCustomers()");
        var customers = customerService.GetCustomers();
        foreach (var customer in customers)
          Console.WriteLine(customer);
      }
      finally
      {
        if (customerService != null)
          (customerService as IDisposable).Dispose();
        if (channelFactory != null)
          channelFactory.Close();
      }
 
      Console.ReadLine();
    }
  }
}

Before this code will compile, we need to have a definition for Customer and ICustomerService. There are a few ways to do this:

1. Ideally we would have a library project in our solution and these two classes/interfaces would be in this library project and both the server and client projects would reference this library project.
2. We could simply link to these classes/interface in our client project. That way if/when we make any changes to them, those changes are automatically visible in our client project as well.
3. We could simply redefine these classes/interfaces in our client project. That will work, but it's really bad idea.

In this project I've opted for option #2 above.

The ChannelFactory and the channel/proxy (the customerService variable) both need to be Disposed or Closed which is why we're using the try-finally here.

Our client only uses the BasicHttpBinding (as you can see above). Typically a client only every uses a single binding while servers may implement/support multiple bindings and since our server does support multiple binding, let's see how we'd use those from our client. It's simple really. You can see the two other variations below.

var channelFactory = new ChannelFactory<ICustomerService>(new NetTcpBinding(),
  "net.tcp://localhost:8081/CustomerService");

var channelFactory = new ChannelFactory<ICustomerService>(new NetNamedPipeBinding(),
  "net.pipe://localhost/CustomerService");

At this point we're really done. We could run our service and then run our client and we should see a list of customers output to the console window. In the rest of this post, we'll look at:

1. Building a WCF client proxy by hand
2. Getting our service to support metadata publishing so we can use the VS.NET IDE or svcutil.exe to have our proxy class generated for us.

WCF Proxy by hand

We could if we wanted to, create a proxy for our ICustomerService contract by hand. Essentially, we'll be doing what the ChannelFactory<T> is doing behind the scenes.

Every proxy descends from ClientBase<T>. Because ClientBase<T> has a number of constructor overloads, we'll need to (or should) implement all of the constructor overloads and simply forward to calls on to base().

In addition to the constructor overloads, we need to implement the methods our ICustomerService has. As you can see, from the code listing below, it's really not had and at all. It should be noted that the ChannelFactory does not support asynchronous calls. So if we need to make asynchronous calls we could either get the IDE to generate a proxy for us, provided our service supports generating metadata. Or we'll need to implement the proxy ourselves and also implement the asynchronous versions of the methods as well. See this earlier post on how to make existing synchronous calls asynchronous.

IAsyncResult–Making Existing methods Asnchronous

using System.Collections.Generic;
using System.ServiceModel;
using WcfServer;
 
namespace WCFTcpClient
{
  class CustomerClient : ClientBase<ICustomerService>
  {
 
    public CustomerClient()
    {
    }
 
    public CustomerClient(string endpointConfigurationName) :
      base(endpointConfigurationName)
    {
    }
 
    public CustomerClient(string endpointConfigurationName, string remoteAddress) :
      base(endpointConfigurationName, remoteAddress)
    {
    }
 
    public CustomerClient(string endpointConfigurationName, System.ServiceModel.EndpointAddress remoteAddress) :
      base(endpointConfigurationName, remoteAddress)
    {
    }
 
    public CustomerClient(System.ServiceModel.Channels.Binding binding, System.ServiceModel.EndpointAddress remoteAddress) :
      base(binding, remoteAddress)
    {
    }
 
    public IEnumerable<Customer> GetCustomers()
    {
      return base.Channel.GetCustomers();
    }
 
    public Customer GetCustomer(int id)
    {
      return base.Channel.GetCustomer(id);
    }
 
    public int InsertCustomer(Customer customer)
    {
      return base.Channel.InsertCustomer(customer);
    }
 
    public void UpdateCustomer(Customer customer)
    {
      base.Channel.UpdateCustomer(customer);
    }
 
    public void DeleteCustomer(int customerId)
    {
      base.Channel.DeleteCustomer(customerId);
    }
  }
}

Once we have this class in place, the way we'd create an instance of the proxy and use it is shown in the code listing below.

var customerService = new CustomerClient(new NetTcpBinding(),
  new EndpointAddress("net.tcp://localhost:8081/CustomerService"));
  try
  {
    foreach (var customer in customerService.GetCustomers())
      Console.WriteLine(customer);
  }
  finally
  {
    customerService.Close();
  }

In the code above, the proxy is using the NetTcpBinding. You can make the required changes as shown earlier if you wanted to use the BaseHttpBinding or the NetNamedPipesBinding.

WCF Service supporting metadata publishing

Next, well look at programmatically configuring our WCF service to support metadata publishing. There two ways for a service in WCF to support metadata publishing.

Before we start adding this capability to our service, let's confirm that our service does not support this capability yet. So start up the service application and then using a browser browser to the base address of the http binding, which is http://localhost:8080 . What you should see is a page that looks like this:

As you can see in the image above, metadata publishing is not currently enabled. There are two styles in which a WCF service may published metadata. Both are industry standards. One is by way of a WSDL document and it always works over HTTP-GET. Let's look at enabling this first. The code below shows that we've added a behavior to our service description in order to enabled metadata publishing as WSDL document. This code MUST appear before you Open the host.

var serviceMetadataBehavior = new ServiceMetadataBehavior();
serviceMetadataBehavior.HttpGetEnabled = true;
host.Description.Behaviors.Add(serviceMetadataBehavior);

Now, run the service and browse to the same url again. What you should see is something similar to what is shown in the image blow.

Navigating to the first link we see will bring up the WSDL document that clients can use to build a proxy for our service. The other way to publish metadata is to use the Metadata Exchange protocol. This protocol works via endpoints (and not behaviors). And since they are endpoints they also include (or indicate) the transport protocol (the binding). In other words, if your service only supports the NetTcpBinding, then you can publish the metadata using another NetTcpbinding endpoint (rather than only HTTP-GET) or you may decide to publish the metadata over HTTP. It's up to you.

Binding mexBinding = MetadataExchangeBindings.CreateMexHttpBinding();
host.AddServiceEndpoint(typeof(IMetadataExchange), mexBinding, "mex");

Notice in the code above that we're using the MexHttpBinding (CreateMexHttpBinding()). The other methods available are:

1. CreateMexHttpsBinding
2. CreateMexTcpBinding
3. CreateMexNamedPipeBinding

At the time of adding the endpoint to our service host, we've included a relative Url ("mex"). So once again, the complete Url will be the BaseAddress (of that Binding http://localhost:8080) plus the relative Url. So the complete Url for our MEX metadata endpoint is http://localhost:8008/mex.

It should be noted that even if we wanted to only publish our MEX endpoint, we'd still need to add the ServiceMetadataBehavior like we did to enabled WSDL publishing. We won't have to set the HttpGetEnabled property to true (we could if we wanted to) however.

This is a short post and I'm writing this because I've seen a lot of people using Response.Redirect(url) or Response.RedirectPremanent(url) and not using the overloads:

• Response.Redirect(url, false)
• Response.RedirectPermanent(url, false)

So rather than repeat the same thing over and over, I've decided to write a blog post about it in the hopes that others may find it and learn from it. There are two overloads to these methods (I'm only showing the Response.Redirect method but the same applies to Response.RedirectPermanent):

Response.Redirect(string url)
Response.Redirect(string url, bool endResponse)

You typically use Response.Redirect when we want to redirect a request to another page in our application or another website or a send an Http status 301 Permanently moved or something similar. In our code the way that this works is that the line right after the call to redirect does not execute. Indeed, most times (if not all the time) this is exactly the behavior we want/expect. It's like (in C#) saying return and the execution jumps right out of the current method. It makes structuring your flow of logic that much easier.

Here's the problem. Think about how, when you call Response.Redirect(url) the next line in our code is not executed (without explicitly saying return)? Yup, that's right, an exception was thrown. That's the only way to boot right out of a method without explicitly saying return. When we call Response.Redirect(url), it calls the second overload passing in true as the parameter to endResponse. What this does is, it calls Response.End(), and Response.End() throws a ThreadAbortException (that's the only way it can truly end the response).

So what we should do is use the second overload (always) and pass it false as the second parameter (always). But….but…but, yes, I know, in that case our code continues to execute and that's not what we want!

There are two solutions to this predicament.

1. Structure our code such that the condition that warrants a Response.Redirect follows a path such that nothing else is done in our Page/Handler/Controller and our method ends gracefully.
2. Call CompleteRequest() right after calling Response.Redirect(url, false). This is a method of the ApplicationInstance object, which we can get from the current context, so something like HttpContext.ApplicationInstance.CompleteRequest();. Note that the execution of code will continue as normal (and not exit/abort). But what will happen is that as soon as the Page/Handler/Controller completes the response the ASP.NET pipeline will jump straight to the EndRequest event. So the Request does "complete" (without any ThreadAbortException), but your code also executes. This may not be something you want to happen. But I thought I'd mention it anyway.

As regards option 1 above, here is a code listing that show you what I mean. Think of the BuildPage method shown below as the ProcessRequest method of a handler. That is once this method finishes we're done processing the request (this is code I've pulled out from Orion by blog engine). There is only one case in which we want to process the request (the if condition). All other cases warrant either a permanent redirect or redirect. All of the code expected to be executed under the normal case is inside the if condition.

    protected override void BuildPage(HttpContextBase httpContext, object model)
    {
      var postDisplayDataDto = (PostDisplayDataDto)model;
      if (model != null && postDisplayDataDto.PostsDataTable.Rows.Count == 1)
      {
        //Ttile of the Post + On + Blog name
        TitleView.Text = postDisplayDataDto.PostsDataTable.Rows[0][4].ToString() + " On " + BlogInfo.BlogInfoName;
        //Title of the Post plus the Post Text
        MetaDescriptionView.Description = postDisplayDataDto.PostsDataTable.Rows[0][4].ToString() + " " + postDisplayDataDto.PostsDataTable.Rows[0][5].ToString();
 
        RegisterView("Body", new PostView(httpContext, postDisplayDataDto,
          Configuration["CategoryCaption"],
          Configuration["TagCaption"]
          ));
      }
      else if (model == null)
        HttpContext.Response.RedirectPermanent(Url.Post(AppDomainAppVirtualPath, PostSlug), false);
      else
        PermanentlyRedirectToAppropriateLocation();
    }

If the workload is compute bound (rather than I/O bound) then it makes complete sense to use multiple threads (provided you have multiple cores and the work is fairly long running and so justifies the overhead of spawning and managing multiple threads), as discussed in this post Data Parallel – Parallel Programming in C#/.NET

Using the APM is not the easiest thing to do primarily because it works with callbacks so you start in one method in your code and you get the result in another method. So ideally you should re-think you design so as to work in an asynchronous workflow pattern rather than a synchronous pattern.

In this post I’m going to show you a few ways in which you can use the HttpWebRequest class to make multiple http requests while getting really good performance/throughput. I strongly advice against using threads for I/O bound work when you're after performance and throughput. Of course in some cases, you could be using the synchronous option or the synchronous option with threads if I/O throughput is not a big concern. Or you could use a combination of threads and asynchronous work. It all depends.

When working with Http we invariably have different needs in different projects. Sometimes we:

1. Need to make multiple calls to some endpoint and wait till each one is done before we can process all of the responses.
2. Need to make multiple calls but have to wait for each one to complete because the response of one feeds into the request of the next.
3. Need to make multiple calls to some endpoint and can continue processing the responses as soon as we receive them (the ideal situation for an Asynchronous work flow).
4. All of the above could be using either the Http GET or POST methods. In the case of the Http POST method there is an additional asynchronous call to be made so it adds a bit more complexity.
   

In this post I’ll show you various options to accomplish these kinds of things using the APM as well as Tasks (System.Threading.Tasks.Task).

The HttpWebRequest makes it particularly difficult to use the APM because if you want to use it (correctly) for POST methods in an asynchronous manner it involves calling two methods asynchronously back to back and thus also involves 2 separate callback methods.

First let’s take a look at how we’d use the HttpWebRequest in an asynchronous manner, with the assumption that your program logic (or workflow) is designed to be asynchronous as well. I’m intentionally not using lambdas here for those of you who either don’t like to use lambdas or get confused by them.

In the code listing below, I present the HttpSocket class that essentially makes it really simple to make asynchronous Http calls. Note that the reason I've presented a class here is because there are some common methods that the primary methods (listed below) use and so I thought it best to present 4 of these methods in the form of a class.

The class is a static class and essentially a wrapper that provides some helper methods as well. This class has 4 public methods of interest to this discussion, namely:

1. PostAsync
2. GetAsync
3. PostAsyncTask
4. GetAsyncTask

The PostAsync and GetAsync methods use the Begin/End pairs of methods available in the HttpWebRequest class. While the PostAsyncTask and GetAsyncTask methods use Tasks (new in .NET 4.0) to get the job done. In particular, they use the Task.Factory.FromAsync method.

From a performance perspective, both styles perform the same, but the PostAsync and GetAsync methods use less resources and this could be beneficial in large production systems where you make tons of http calls out to some other http service.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Collections.Specialized;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using System.Runtime.Serialization.Json;

namespace ConsoleApplication3
{
  public static class HttpSocket
  {
    static HttpWebRequest CreateHttpWebRequest(string url, string httpMethod, string contentType)
    {
      var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
      httpWebRequest.ContentType = contentType;
      httpWebRequest.Method = httpMethod;
      return httpWebRequest;
    }

    static byte[] GetRequestBytes(NameValueCollection postParameters)
    {
      if (postParameters == null || postParameters.Count == 0)
        return new byte[0];
      var sb = new StringBuilder();
      foreach (var key in postParameters.AllKeys)
        sb.Append(key + "=" + postParameters[key] + "&");
      sb.Length = sb.Length - 1;
      return Encoding.UTF8.GetBytes(sb.ToString());
    }

    static void BeginGetRequestStreamCallback(IAsyncResult asyncResult)
    {
      Stream requestStream = null;
      HttpWebRequestAsyncState asyncState = null;
      try
      {
        asyncState = (HttpWebRequestAsyncState)asyncResult.AsyncState;
        requestStream = asyncState.HttpWebRequest.EndGetRequestStream(asyncResult);
        requestStream.Write(asyncState.RequestBytes, 0, asyncState.RequestBytes.Length);
        requestStream.Close();
        asyncState.HttpWebRequest.BeginGetResponse(BeginGetResponseCallback,
          new HttpWebRequestAsyncState
          {
            HttpWebRequest = asyncState.HttpWebRequest,
            ResponseCallback = asyncState.ResponseCallback,
            State = asyncState.State
          });
      }
      catch (Exception ex)
      {
        if (asyncState != null)
          asyncState.ResponseCallback(new HttpWebRequestCallbackState(ex));
        else
          throw;
      }
      finally
      {
        if (requestStream != null)
          requestStream.Close();
      }
    }

    static void BeginGetResponseCallback(IAsyncResult asyncResult)
    {
      WebResponse webResponse = null;
      Stream responseStream = null;
      HttpWebRequestAsyncState asyncState = null;
      try
      {
        asyncState = (HttpWebRequestAsyncState)asyncResult.AsyncState;
        webResponse = asyncState.HttpWebRequest.EndGetResponse(asyncResult);
        responseStream = webResponse.GetResponseStream();
        var webRequestCallbackState = new HttpWebRequestCallbackState(responseStream, asyncState.State);
        asyncState.ResponseCallback(webRequestCallbackState);
        responseStream.Close();
        responseStream = null;
        webResponse.Close();
        webResponse = null;
      }
      catch (Exception ex)
      {
        if (asyncState != null)
          asyncState.ResponseCallback(new HttpWebRequestCallbackState(ex));
        else
          throw;
      }
      finally
      {
        if (responseStream != null)
          responseStream.Close();
        if (webResponse != null)
          webResponse.Close();
      }
    }

    /// <summary>
    /// If the response from a remote server is in text form
    /// you can use this method to get the text from the ResponseStream
    /// This method Disposes the stream before it returns
    /// </summary>
    /// <param name="responseStream">The responseStream that was provided in the callback delegate's HttpWebRequestCallbackState parameter</param>
    /// <returns></returns>
    public static string GetResponseText(Stream responseStream)
    {
      using (var reader = new StreamReader(responseStream))
      {
        return reader.ReadToEnd();
      }
    }

    /// <summary>
    /// This method uses the DataContractJsonSerializer to
    /// Deserialize the contents of a stream to an instance
    /// of an object of type T.
    /// This method disposes the stream before returning
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="stream">A Stream. Typically the ResponseStream</param>
    /// <returns>An instance of an object of type T</returns>
    static T DeSerializeToJson<T>(Stream stream)
    {
      using (stream)
      {
        var deserializer = new DataContractJsonSerializer(typeof(T));
        return (T)deserializer.ReadObject(stream);
      }
    }

    /// <summary>
    /// This method does an Http POST sending any post parameters to the url provided
    /// </summary>
    /// <param name="url">The url to make an Http POST to</param>
    /// <param name="postParameters">The form parameters if any that need to be POSTed</param>
    /// <param name="responseCallback">The callback delegate that should be called when the response returns from the remote server</param>
    /// <param name="state">Any state information you need to pass along to be available in the callback method when it is called</param>
    /// <param name="contentType">The Content-Type of the Http request</param>
    public static void PostAsync(string url, NameValueCollection postParameters,
      Action<HttpWebRequestCallbackState> responseCallback, object state = null,
      string contentType = "application/x-www-form-urlencoded")
    {
      var httpWebRequest = CreateHttpWebRequest(url, "POST", contentType);
      var requestBytes = GetRequestBytes(postParameters);
      httpWebRequest.ContentLength = requestBytes.Length;

      httpWebRequest.BeginGetRequestStream(BeginGetRequestStreamCallback,
        new HttpWebRequestAsyncState()
        {
          RequestBytes = requestBytes,
          HttpWebRequest = httpWebRequest,
          ResponseCallback = responseCallback,  
          State = state
        });
    }

    /// <summary>
    /// This method does an Http GET to the provided url and calls the responseCallback delegate
    /// providing it with the response returned from the remote server.
    /// </summary>
    /// <param name="url">The url to make an Http GET to</param>
    /// <param name="responseCallback">The callback delegate that should be called when the response returns from the remote server</param>
    /// <param name="state">Any state information you need to pass along to be available in the callback method when it is called</param>
    /// <param name="contentType">The Content-Type of the Http request</param>
    public static void GetAsync(string url, Action<HttpWebRequestCallbackState> responseCallback,
      object state = null, string contentType = "application/x-www-form-urlencoded")
    {
      var httpWebRequest = CreateHttpWebRequest(url, "GET", contentType);

      httpWebRequest.BeginGetResponse(BeginGetResponseCallback,
        new HttpWebRequestAsyncState()
        {
          HttpWebRequest = httpWebRequest,
          ResponseCallback = responseCallback,
          State = state
        });
    }

    public static void PostAsyncTask(string url, NameValueCollection postParameters,
      Action<HttpWebRequestCallbackState> responseCallback, object state = null,
      string contentType = "application/x-www-form-urlencoded")
    {
      var httpWebRequest = CreateHttpWebRequest(url, "POST", contentType);
      var requestBytes = GetRequestBytes(postParameters);
      httpWebRequest.ContentLength = requestBytes.Length;

      var asyncState = new HttpWebRequestAsyncState()
      {
        RequestBytes = requestBytes,
        HttpWebRequest = httpWebRequest,
        ResponseCallback = responseCallback,
        State = state
      };

      Task.Factory.FromAsync<Stream>(httpWebRequest.BeginGetRequestStream,
        httpWebRequest.EndGetRequestStream, asyncState, TaskCreationOptions.None)
        .ContinueWith<HttpWebRequestAsyncState>(task =>
        {
          var asyncState2 = (HttpWebRequestAsyncState)task.AsyncState;
          using (var requestStream = task.Result)
          {
            requestStream.Write(asyncState2.RequestBytes, 0, asyncState2.RequestBytes.Length);
          }          
          return asyncState2;
        })
        .ContinueWith(task =>
        {
          var httpWebRequestAsyncState2 = (HttpWebRequestAsyncState)task.Result;
          var hwr2 = httpWebRequestAsyncState2.HttpWebRequest;
          Task.Factory.FromAsync<WebResponse>(hwr2.BeginGetResponse,
            hwr2.EndGetResponse, httpWebRequestAsyncState2, TaskCreationOptions.None)
            .ContinueWith(task2 =>
            {
              WebResponse webResponse = null;
              Stream responseStream = null;
              try
              {
                var asyncState3 = (HttpWebRequestAsyncState)task2.AsyncState;
                webResponse = task2.Result;
                responseStream = webResponse.GetResponseStream();
                responseCallback(new HttpWebRequestCallbackState(responseStream, asyncState3));
              }
              finally
              {
                if (responseStream != null)
                  responseStream.Close();
                if (webResponse != null)
                  webResponse.Close();
              }
            });
        });
    }

    public static void GetAsyncTask(string url, Action<HttpWebRequestCallbackState> responseCallback,
      object state = null, string contentType = "application/x-www-form-urlencoded")
    {
      var httpWebRequest = CreateHttpWebRequest(url, "GET", contentType);
      Task.Factory.FromAsync<WebResponse>(httpWebRequest.BeginGetResponse,
        httpWebRequest.EndGetResponse, null).ContinueWith(task =>
          {
            var webResponse = task.Result;
            var responseStream = webResponse.GetResponseStream();
            responseCallback(new HttpWebRequestCallbackState(webResponse.GetResponseStream(), state));
            responseStream.Close();
            webResponse.Close();
          });
    }
  }

  /// <summary>
  /// This class is used to pass on "state" between each Begin/End call
  /// It also carries the user supplied "state" object all the way till
  /// the end where is then hands off the state object to the
  /// HttpWebRequestCallbackState object.
  /// </summary>
  class HttpWebRequestAsyncState
  {
    public byte[] RequestBytes { get; set; }
    public HttpWebRequest HttpWebRequest { get; set; }
    public Action<HttpWebRequestCallbackState> ResponseCallback { get; set; }
    public Object State { get; set; }
  }

  /// <summary>
  /// This class is passed on to the user supplied callback method
  /// as a parameter. If there was an exception during the process
  /// then the Exception property will not be null and will hold
  /// a reference to the Exception that was raised.
  /// The ResponseStream property will be not null in the case of
  /// a sucessful request/response cycle. Use this stream to
  /// exctract the response.
  /// </summary>
  public class HttpWebRequestCallbackState
  {
    public Stream ResponseStream { get; private set; }
    public Exception Exception { get; private set; }
    public Object State { get; set; }

    public HttpWebRequestCallbackState(Stream responseStream, object state)
    {
      ResponseStream = responseStream;
      State = state;
    }

    public HttpWebRequestCallbackState(Exception exception)
    {
      Exception = exception;
    }
  }
}

Using HttpWebRequests in Asynchronous Workflows

Using any of the methods in the HttpSocket class for asynchronous workflows is really simple. You call one of the Postxxx or Getxxx methods passing it some parameters and the callback delegate as shown in the Code Listing 2 below.

In this case we're using the PostAsyn method. We send in the url and since it is a POST (rather than a GET) we send in the postParameters as well. Each of these methods also requires a callback delegate that is called back with the response of the http call. In the example below we've used a lambda as the callback. The response call back is an

Action<HttpWebRequestCallbackState>

The HttpWebRequestCallbackState is a custom class that holds a bunch of state information including the Response Stream (that contains the response). You can find that class in Code Listing 1 above.

What's important to realize here is that the loop finishes almost instantly. We don't block the main thread and we're not spawning additional threads. The callback will be called for each response, as and when a response is received and we simply continue on with processing the response as we see fit. In the example, we're simply writing out the response to the console. Another thing to keep in mind is that the responses will not arrive in the order in which requests were made. This is quite normal and if our workflow is truly asynchronous then this shouldn't be a problem.

      ServicePointManager.DefaultConnectionLimit = 50;
      var url = "http://localhost/HttpTaskServer/default.aspx";

      var iterations = 1000;
      for (int i = 0; i < iterations; i++)
      {
        var postParameters = new NameValueCollection();
        postParameters.Add("data", i.ToString());
        HttpSocket.PostAsync(url, postParameters, callbackState =>
          {
            if (callbackState.Exception != null)
              throw callbackState.Exception;
            Console.WriteLine(HttpSocket.GetResponseText(callbackState.ResponseStream));
          });
      }

Making Multiple Concurrent Requests and Waiting on All

Sometimes, we can't use an asynchronous workflow but we'd like to fire multiple requests as fast as possible and then wait till all jobs have complete before we continue on to process the responses. There are many ways in which to tackle this kind of problem. In this post we'll be looking at 5 different ways in which we can accomplish this kind of thing.

Each one has it's merits (ease of use versus complexity) and performance and resource utilization metrics.

1. Use Tasks to make synchronous Http requests and wait on all tasks to complete before proceeding. The method CallHttpWebRequestTaskAndWaitOnAll in Code Listing 4 below demonstrates this.
2. We could spawn n number of threads and in each thread make synchronous Http requests and as soon as a thread completes, spawn another thread and keep doing this till all work items have finished. In other word, we put a cap on the number of threads we spawn and therefore the number of threads executing at any given point in time but ensure that at all times there are n threads busy doing work rather than sitting idle. The method CallHttpWebRequestSyncAndWaitOnAll in Code Listing 5 below demonstrates this.
3. We could do the actual Http requests asynchronously and somehow collect the results of each response as they complete while blocking the main thread till all asynchronous jobs have completed. This technique is demonstrated in the method CallHttpWebRequestAsyncAndWaitOnAll in Code Listing 6 below.
4. A variation on the previous theme is that we could use a combination of a parallel programming technique called Data Parallel, where the work items are partitioned in chunks and then each chunk is processed by a thread. Within each thread we make asynchronous Http requests collecting the responses as they come in and wait for all threads to complete before returning. The method CallHttpWebRequestASyncDataParallelAndWaitOnAll shown in Code Listing 7 below demonstrates this.
5. We could use another construct available to us as part of the .NET 4.0 framework, the Parallel, class. This class essentially uses the Data Parallel technique as well but the it has the advantage of being able to use new features of the .NET 4.0 thread pool. Using the Parallel class is really simple as you you'll see. Within the Parallel.ForEach loop, we'll make synchronous Http calls and store the responses for each request and wait till all work items have been processed. The Parallel class makes it simple to "wait on all" because it finishes when all work items have finished. The method CallHttpWebRequestSyncParallelForEachAndWaitOnAll in Code Listing 8 below shows such an implementation.

Some explanation as to how some of this works (in the way these methods are being used in these tests) is in order. In Code Listing 3 below you can see We initialize the DefaultConnectionLimit to 100. We then initialize an array of Work object instances (100 elements in the array) that we'll be passing to each of these methods as the "work load". The Work class is also shown in Code Listing 3 below.

    static void Main(string[] args)
    {
      ServicePointManager.DefaultConnectionLimit = 100;
      var url = "http://localhost/HttpTaskServer/default.aspx";
      var iterations = 10;

      var workItems = (from w in Enumerable.Range(0, 100)
                       let postParameters = new NameValueCollection { { "data", w.ToString() } }
                       select new Work() { Id = w, PostParameters = postParameters }).ToArray();

      Thread.Sleep(1000);
      Benchmarker.MeasureExecutionTime("ParallelForEach              ", iterations, CallHttpWebRequestASyncParallelForEachAndWaitOnAll, url, workItems);
      Benchmarker.MeasureExecutionTime("AsyncAndWaitOnAll            ", iterations, CallHttpWebRequestAsyncAndWaitOnAll, url, workItems);
      Benchmarker.MeasureExecutionTime("ASyncDataParallelAndWaitOnAll", iterations, CallHttpWebRequestASyncDataParallelAndWaitOnAll, url, workItems);
      Benchmarker.MeasureExecutionTime("TaskAndWaitOnAll             ", iterations, CallHttpWebRequestTaskAndWaitOnAll, url, workItems);
      Benchmarker.MeasureExecutionTime("SyncAndWaitOnAll             ", iterations, CallHttpWebRequestSyncAndWaitOnAll, url, workItems);
      Console.WriteLine("All work Done.");      
      Console.ReadLine();
    }

  class Work
  {
    public int Id { get; set; }
    public NameValueCollection PostParameters { get; set; }
    public string ResponseData { get; set; }
    public Exception Exception { get; set; }
  }

While initializing the work object instances we initialize the Id property of each instance to the "index" of the array element. So the first element's Id property is set to 1, the second's to 2 and so on. We also initialize the PostParameters property to an instance containing the name and value of "data" and "1" for the first element and "data" and "2" for the second element and so on. This is essentially the data that is POSTed to the remote server. In other words the Http content for each http request will be

data=x

Where "x" is different for each request and will be the same as the Id of the work item. Of course you'll change this to suit your particular needs.

Next, you should notice that the Work class has a ResonseData property. This is the property that holds the response of each http request we make. Again, in this case the response is a string but you can change this to be any other type to suite your response (like a JSON object for example).

The key point to notice here is that each work item contains the data required to make the request and then the response that comes back for each request. So after each of these methods returns, iterating over the array of work items and accessing the ResponseData property of each element will yield the response that each work item got back from the remote server.

The Benchmarker class simply calls a method n times and profiles it. In this case there are 100 items on our workItems array and we call each method 10 times. So effectively, each method makes a total 1,000 Http Requests but in batches of 10. We could increase the number of items in our workItems array to 1000 and do only one iteration as well.

    /// <summary>
    /// This method makes a bunch (workItems.Count()) of HttpRequests using Tasks
    /// The work each task performs is a synchronous Http request. Essentially each
    /// Task is performed on a different thread and when all threads have completed
    /// this method returns
    /// </summary>
    /// <param name="url"></param>
    /// <param name="workItems"></param>
    static void CallHttpWebRequestTaskAndWaitOnAll(string url, IEnumerable<Work> workItems)
    {      
      var tasks = new List<Task>();
      foreach (var workItem in workItems)
      {
        tasks.Add(Task.Factory.StartNew(wk =>
        {
          var wrkItem = (Work)wk;
          wrkItem.ResponseData = GetWebResponse(url, wrkItem.PostParameters);
        }, workItem));
      }
      Task.WaitAll(tasks.ToArray());
    }

    /// <summary>
    /// This method makes a bunch (workItems.Count()) of HttpRequests synchronously
    /// using a pool of threads with a certain cap on the number of threads.
    /// As soon as a thread finishes a now job is started till no more work items are left.
    /// After all http requests are complete, this method returns
    /// </summary>
    /// <param name="url"></param>
    /// <param name="workItems"></param>
    static void CallHttpWebRequestSyncAndWaitOnAll(string url, IEnumerable<Work> workItems)
    {
      //Since the threads will be blocked for the most part (not using the CPU)
      //We're using 4 times as many threads as there are cores on the machine.
      //Play with this number to ensure you get the performance you keeping an
      //eye on resource utlization as well.
      int maxThreadCount = Environment.ProcessorCount * 4;
      int executingThreads = 0;
      //This variable is used to throttle the number of threads
      //created to the maxThreadCount
      object lockObj = new object();

      foreach (var workItem in workItems)
      {
        ThreadPool.QueueUserWorkItem((state) =>
          {
            var work = (Work)state;
            try
            {
              work.ResponseData = GetWebResponse(url, work.PostParameters);
              Interlocked.Decrement(ref executingThreads);
            }
            catch (Exception ex)
            {
              work.Exception = ex;
              Interlocked.Decrement(ref executingThreads);
            }

          }, workItem);

        //If maxThreadCount threads have been spawned
        //then wait for any of them to finish before
        //spawning additional threads. Therby limiting
        //the number of executing (and spawned)
        //threads to maxThreadCount
        lock (lockObj)
        {
          executingThreads++;
          while (executingThreads == maxThreadCount)
            Thread.Sleep(1);
        }
      }

      //Wait on all executing threads to complete
      while (executingThreads != 0)
        Thread.Sleep(1);
    }

    /// <summary>
    /// This method makes a bunch (workItems.Count()) of HttpRequests asynchronously
    /// The main thread is blocked until all asynchronous jobs are complete.
    /// After all http requests are complete, this method returns
    /// </summary>
    /// <param name="url"></param>
    /// <param name="workItems"></param>
    private static void CallHttpWebRequestAsyncAndWaitOnAll(string url, IEnumerable<Work> workItems)
    {
      var pending = workItems.Count();
      using (var mre = new ManualResetEvent(false))
      {
        foreach (var workItem in workItems)
        {
          HttpSocket.PostAsync(url, workItem.PostParameters, callbackState =>
          {
            var hwrcbs = (HttpWebRequestCallbackState)callbackState;
            using (var responseStream = hwrcbs.ResponseStream)
            {
              var reader = new StreamReader(responseStream);
              ((Work)hwrcbs.State).ResponseData = reader.ReadToEnd();
              if (Interlocked.Decrement(ref pending) == 0)
                mre.Set();
            }
          }, workItem);
        }
        mre.WaitOne();
      }
    }

    /// <summary>
    /// This method makes a bunch (workItems.Count()) of HttpRequests asynchronously
    /// but partitions the workItem into chunks. The number of chunks is determined
    /// by the number of cores in the machine. The main thread is blocked
    /// until all asynchronous jobs are complete
    /// After all http requests are complete, this method returns
    /// </summary>
    /// <param name="url"></param>
    /// <param name="workItems"></param>
    private static void CallHttpWebRequestASyncDataParallelAndWaitOnAll(string url, IEnumerable<Work> workItems)
    {
      var coreCount = Environment.ProcessorCount;
      var itemCount = workItems.Count();
      var batchSize = itemCount / coreCount;

      var pending = itemCount;
      using (var mre = new ManualResetEvent(false))
      {
        for (int batchCount = 0; batchCount < coreCount; batchCount++)
        {
          var lower = batchCount * batchSize;
          var upper = (batchCount == coreCount - 1) ? itemCount : lower + batchSize;
          var workItemsChunk = workItems.Skip(lower).Take(upper).ToArray();

          foreach (var workItem in workItemsChunk)
          {
            HttpSocket.PostAsync(url, workItem.PostParameters, callbackState =>
              {
                var hwrcbs = (HttpWebRequestCallbackState)callbackState;
                using (var responseStream = hwrcbs.ResponseStream)
                {
                  var reader = new StreamReader(responseStream);
                  ((Work)hwrcbs.State).ResponseData = reader.ReadToEnd();
                  if (Interlocked.Decrement(ref pending) == 0)
                    mre.Set();
                }
              }, workItem);
          }
        }
        mre.WaitOne();
      }
    }

    /// <summary>
    /// This method makes a bunch (workItems.Count()) of HttpRequests synchronously
    /// using Parallel.ForEach. Behind the scenes Parallel.ForEach partitions the
    /// workItem into chunks (Data Parallel). The main thread is blocked
    /// until all workItems have been processed.
    /// After all http requests are complete, this method returns
    /// </summary>
    /// <param name="url"></param>
    /// <param name="workItems"></param>
    private static void CallHttpWebRequestSyncParallelForEachAndWaitOnAll(string url, IEnumerable<Work> workItems)
    {
      Parallel.ForEach(workItems, work =>
      {
        try
        {
          work.ResponseData = GetWebResponse(url, work.PostParameters);
        }
        catch (Exception ex)
        {
          work.Exception = ex;
        }
      });
    }

The table below shows the timing results of each of the methods listed above. The results are shown in the order for fastest first. As you can see the method that does Not use threads is the fastest. The next is the method that uses as many threads as there are CPU cores on the machine (The machine used to perform these tests has 4 cores) but all work is done asynchronously (on 4 threads).

The hand crafted method SyncAndWaitOnAll comes in, in 3rd place and very close in 4th place is the Parralel.ForEach and last is the method that uses Tasks.

I should note that I performed each test by itself multiple times, and during these test the SyncAndWaitOnAll and the Parallel.ForEach methods were extremely close, frequently swapping places and you can see from their timings that they were in fact very close. The TaskAndWaitOnAll method wasn't too far behind most times either.

The reason the results turned out the way they did is because the workload is an I/O bound workload and I/O bound workloads don't benefit by spawning threads. In fact spawning threads for I/O bound work only hurts performance due to the thread management and context switching overheads. Keep in mind also that the results here are for making 1000 http requests and even then the results are not that different.

In the real world it may not be possible to use an asynchronous workflow, especially if you are trying to back-fit it into already existing synchronous workflows. So any of the methods resented here will work just as well.

All times are in milliseconds

The method shown in Code Listing 9 below is the method some of the methods listed above call. It makes synchronous Http requests and returns after receiving the response.

    static string GetWebResponse(string url, NameValueCollection parameters)
    {
      var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
      httpWebRequest.ContentType = "application/x-www-form-urlencoded";
      httpWebRequest.Method = "POST";

      var sb = new StringBuilder();
      foreach (var key in parameters.AllKeys)
        sb.Append(key + "=" + parameters[key] + "&");
      sb.Length = sb.Length - 1;

      byte[] requestBytes = Encoding.UTF8.GetBytes(sb.ToString());
      httpWebRequest.ContentLength = requestBytes.Length;

      using (var requestStream = httpWebRequest.GetRequestStream())
      {
        requestStream.Write(requestBytes, 0, requestBytes.Length);
      }

      Task<WebResponse> responseTask = Task.Factory.FromAsync<WebResponse>(httpWebRequest.BeginGetResponse, httpWebRequest.EndGetResponse, null);
      using (var responseStream = responseTask.Result.GetResponseStream())
      {
        var reader = new StreamReader(responseStream);
        return reader.ReadToEnd();
      }
    }

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;

namespace Diagnostics.Performance
{
  public static class Benchmarker
  {
    private static void WarmUp(Action action)
    {
      action();
      GC.Collect();
    }

    private static void DisplayResults(string benchmarkName, TimeSpan totalTime, TimeSpan averageTime, TimeSpan minTime, TimeSpan maxTime)
    {
      Console.WriteLine("---------------------------------");
      Console.WriteLine(benchmarkName);
      Console.WriteLine("\tTotal time  : " + totalTime.TotalMilliseconds + "ms");
      Console.WriteLine("\tAverage time: " + averageTime.TotalMilliseconds + "ms");
      Console.WriteLine("\tMin time    : " + minTime.TotalMilliseconds + "ms");
      Console.WriteLine("\tMax time    : " + maxTime.TotalMilliseconds + "ms");
      Console.WriteLine("---------------------------------");
      Console.WriteLine();
    }

    public static void MeasureExecutionTime(string benchmarkName, int noOfIterations, Action action)
    {
      var totalTime = new TimeSpan(0);
      var averageTime = new TimeSpan(0);
      var minTime = TimeSpan.MaxValue;
      var maxTime = TimeSpan.MinValue;

      WarmUp(action);
      GC.Collect();
      GC.WaitForPendingFinalizers();
      GC.Collect();

      var total = new TimeSpan(0);
      var sw = Stopwatch.StartNew(); 
      for (int i = 0; i < noOfIterations; i++)
      {
        sw.Restart();
        action();
        sw.Stop();

        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();

        var thisIteration = sw.Elapsed;
        total += thisIteration;

        if (thisIteration > maxTime)
          maxTime = thisIteration;
        if (thisIteration < minTime)
          minTime = thisIteration;
      }

      totalTime = total;
      averageTime = new TimeSpan(total.Ticks / noOfIterations);

      DisplayResults(benchmarkName, totalTime, averageTime, minTime, maxTime);
    }
  }
}

I hope you've found this post and the information and data I've presented here useful.

Projects each element of a sequence to an IEnumerable<T> and flattens the resulting sequences into one sequence.

I don't know about you, but I had to re-read this statement multiple times just to figure out what it was saying and even then, couldn't figure out where or why I'd use it. And then I thought well, maybe it's like doing a join, where for each element of sequence 1, I could produce many elements from sequence 2 and then get a flatten result (similar to doing a join in a database across a one-to-many relationship). Another way to look it is a nested foreach, where where you iterate over sequence 1 and for each element of sequence 1 you "find" or get all matching elements in sequence 2.

Once I had that picture in mind I set about trying to write a linq query to see how I could achieve this. Before we take a look at that, lets take a look at the classes and data, we'll be using in this post. Code Listing 1 below, shows the two classes we'll be working with, and the initial data these classes have.

  public class Category
  {
    public int Id { get; set; }
    public string Name { get; set; }
  }

  class Post
  {
    public int Id { get; set; }
    public int CategoryId { get; set; }
    public string Title { get; set; }
  }

In Code Listing 2, below we can see the initial data we'll be working with. You'll notice that there are 3 Categories

1. Programming
2. Html 5
3. jQuery

And that there are multiple posts in each of these categories. For instance there are 2 posts categorized as Html 5 (Category id = 2).

1. Html 5 Video Element Poster
2. Html5 File Upload with Progress

var categories = new Category[]
{
  new Category() { Id=1, Name ="Programming"},
  new Category() { Id=2, Name ="Html"},
  new Category() { Id=3, Name ="jQuery"}
};

var posts = new Post[]
{
  new Post { Id=1, CategoryId=1, Title="Data Parallel – Parallel Programming in C#/.NET"},
  new Post { Id=2, CategoryId=1, Title="Getting the No Of CPUs and Cores"},
  new Post { Id=3, CategoryId=1, Title="Orion - A Blog Engine"},
  new Post { Id=4, CategoryId=1, Title="Quartz for ASP.NET"},
  new Post { Id=5, CategoryId=2, Title="Html 5 Video Element Poster"},
  new Post { Id=6, CategoryId=2, Title="Html5 File Upload with Progress"},
  new Post { Id=7, CategoryId=3, Title="Expanding Code listings"},
  new Post { Id=8, CategoryId=3, Title="jQuery working with select option"},
  new Post { Id=9, CategoryId=3, Title="jQuery working with checkboxes and radio button"}
};

The "foreign key" in the Post class in this case is CategoryId. No that's not how the database is modeled in Quartz for ASP.NET (the blogging engine I use for this blog), I'm using this structure to keep things simple. Moving on…So if we were to do a "inner join" between these two sequences, the result should look like the output below.

Programming :  Data Parallel - Parallel Programming in C#/.NET
Programming :  Getting the No Of CPUs and Cores
Programming :  Orion - A Blog Engine
Programming :  Quartz for ASP.NET
Html        :  Html 5 Video Element Poster
Html        :  Html5 File Upload with Progress
jQuery      :  Expanding Code listings
jQuery      :  jQuery working with select option
jQuery      :  jQuery working with checkboxes and radio button

The linq query syntax for this is actually quite simple if you've done joins using linq before. It looks like that shown in Code Listing 3 below. Both versions produce the same result. It's almost like the old join syntax versus the ASNI SQL syntax for a join in SQL.

var categoryPosts = from c in categories                          
                    join p in posts on c.Id equals p.CategoryId
                    select new { CategoryName = c.Name, PostTitle = p.Title };

OR

var categoryPosts = from c in categories
                    from p in posts                           
                    where c.Id == p.CategoryId
                    select new { CategoryName = c.Name, PostTitle = p.Title };

What's important to note here is that the first version shown in Code Listing 3 above is actually translated by the compiler to use a SelectMany. The code thus generated would look similar to what have in Code Listing 4 below.

So far so good. Now lets go back to the SelectMany method. Let's also re-examine the statement of what SelectMany does.

Projects each element of a sequence to an IEnumerable<T>.

So the elements in our sequence are the various Categories and we have the ability to project each Category to an IEnumerable<Post>. That is for each Category, project the Posts that pertain to the Category. Effectively, a one-to-many. But we're not done yet.

Flattens the resulting sequence into one sequence.

So essentially producing one resulting sequence that is the flattened version of multiple, one-to-manys, thus giving us a "join". You could also think of SelectMany that does a nested foreach (as explained earlier), where the outer foreach is our categories sequence and the inner foreach is the sequence of posts.

Well, even after digesting the above, attempting to write the linq statement using the method syntax (versus the query expression syntax) turned out to be quite confusing indeed.

Here is the equivalent linq method syntax for using one of the many overloads of the SelectMany method:

var categoryPosts2 = categories.SelectMany(
            c => posts.Where(p => p.CategoryId == c.Id),
            (c, p) => new { CategoryName = c.Name, PostTitle = p.Title });

Let's break this down so we can better understand what is going on. The overload we're using in this case is:

public static IEnumerable<TResult> SelectMany<TSource, TCollection, TResult>(
    this IEnumerable<TSource> source,
    Func<TSource, IEnumerable<TCollection>> collectionSelector,
    Func<TSource, TCollection, TResult> resultSelector
)

which in our case translates to

Enumerable<Category>.SelectMany(Func<Category, IEnumerable<Post>>, collectionSelector, Func<Category, Post, `a> resultSelector

So the first parameter is a Func that takes a Category and returns an IEnumerable<Post>. Of course the posts it returns are those that match the category (see  code listing 4 above). In other words, the first parameter maps each category to a sequence of matching posts.

The second parameter transforms each matched pair:

{ (c1, p1), (c1, p2), (c1, p3), (c1, p4), (c2, p5), (c2, p6), (c3, p7), (c3, p8), (c3, p9)}

into a new anonymous type that has a CategoryName property and a PostTitle property.

The implementation of SelectMany would look like the following:

foreach (TSource item in source)
  foreach (TCollection subItem in collectionSelector(item))
    yield return resultSelector(item, subItem);

Where the collectionSelector delegate and resultSelector delegates have the signature described above.

Now it so happens that the second parameter of the SelectMany method we're using is optional, in that there is another overload that requires only the first parameter. So what would happen if we used this overload instead?

In order to better understand what we'll get, we need to modify our data a bit. So let's remove the Programming Category from our sequence of categories (just comment out the first item).

What we'll get is simply a sequence of Posts (IEnumerable<Post>) that have associated categories (an "inner join" if you will) but without the projection (our anonymous type). So the first 4 posts in our data won't be part of this result a shown below. The number is the Id of the Post (and not the CategoryId).

5 : Html 5 Video Element Poster
6 : Html5 File Upload with Progress
7 : Expanding Code listings
8 : jQuery working with select option
9 : jQuery working with checkboxes and radio button

There was a time when multiple CPUs were only available on servers and server side application platforms such as IIS are inherently multi-threaded so if we were building an ISAPI or ASP.NET application we automatically got the benefit of multiple threads in our applications without the need to know how to write parallel programs in order to make the best use of server hardware.

On the client side, we typically used multiple threads to perform some background work so as to not freeze the UI. But very rarely did we spawn threads in client application to perform some compute-bound work, because for the most part, client machines had only one CPU and one core. But that has changed. Most desktop computers have at least 2 cores and in the future even normal desktop machines will have 4-8 cores and we need to learn how to utilize these cores in order to speed up our applications and to make good use of the available hardware.

Even in server side applications, it is important to know how to best utilize the hardware available in order to speed up our applications. If the CPU cores on your server are not busy and your application is not running as fast as it should then it is likely we could benefit from parallelizing some of the work we do in our application.

Parallel programming is extremely difficult to get right. I don’t mean getting it to work, but rather to get it right. It requires an in-depth knowledge of various hardware components such as CPU and CPU Architecture, Memory and memory architecture (UMA/NUMA), and the Disk I/O subsystem.

In this article, we’re not going to go into the details of hardware, though if you’re serious about parallel programming you should dig real deep into hardware architecture of modern computers, especially as it relates to Memory access, including L1, L2, L3 cache and the Disk I/O subsystem.

Parallelization support in .NET 4.0

In .NET 4.0 we have access to a few specialized classes/libraries that can help us very easily achieve parallelization. In this post we'll be looking at the following:

1. PLINQ (Parallel LINQ)
2. The Parallel class. Namely the Parallel.For method

Both of these classes make it extremely simple to parallelize new and even existing code. We'll look at these two later in the post, but I wanted to introduce them and to assure you that stuff is simple. But like with most things, one needs to have a really good understanding of the background or foundations in order to best utilize the tools available. So in this post I attempt to introduce you to some of the foundational concepts with examples.

A big change in .NET 4.0 is the CLR thread pool. There have been some major overhauls to the thread pool in .NET 4.0 to support parallel workloads and of course PLINQ and the Parallel class (as well as the new Task class that's part of the TPL – Task Parallel Library) make good use of these changes. A key change is the introduction on Work Stealing. We'll talk about work stealing later in this post as it makes a huge difference to the performance of some parallel workloads.

Data Parallelism

Data Parallelism is a parallelization pattern. This is, it is commonly used and is therefore considered a pattern. It is also called SIMD (Single Instruction Multiple Data). What this means is that given a set of data you want to perform a certain function on it. A very simple example of is a typical for-loop where within the loop you perform some action on this data.

Task parallelism is another parallelization pattern that is commonly used. You use Task parallelization when you have a bunch of tasks that need to be performed on a piece of data and typically (or ideally) the order in which these tasks are performed is not important.  I plan to write a post on Task Parallelism soon.

In this post I’d like to present Data Parallelism and how we can very easily start to use this pattern in our .NET applications for compute-bound work. A lot of what we do in application programming is a good candidate for data parallelism. In Code Listing 1 below we see a simple for-loop, that iterates over an array of integers. Within the loop we determine if the array element is a prime number. If it is, we increment a count. When the loop finishes, we have our answer, which is, the number of prime numbers in a given array of integers. The IsPrime() method is a compute-bound method.

    private static void GetNumberOfPrimes(int[] numbers)
    {
      var count = 0;
      for (int i = 0; i < numbers.Length; i++)
      {
        if (IsPrime(numbers[i]))
          count++;
      }
      //Console.WriteLine(count);
    }

    public static bool IsPrime(int number)
    {
      if ((number & 1) == 0)
        return (number != 2) ? false : true;

      for (int i = 3; i * i <= number; i += 2)
      {
        if (number % i == 0)
          return false;
      }
      return number != 1;
    }

For the moment let's stick with the non-parallelized method GetNumberOfPrimes()shown in Code Listing 1 above and the initialization code in the Main method shown in Code Listing 2 below.

static void Main(string[] args)
{
  var numbers = Enumerable.Range(1, 500000).ToArray();
  var iterations = 500;

  Benchmarker.MeasureExecutionTime("Non-Parallel", iterations, () => GetNumberOfPrimes(numbers));
  Benchmarker.MeasureExecutionTime("Plinq", iterations, () => GetNumberOfPrimesUsingPLinq(numbers));
  Benchmarker.MeasureExecutionTime("QueueUserWorkItem", iterations, () => GetNumberOfPrimesUsingThreadPool(numbers));
  Benchmarker.MeasureExecutionTime("Parallel.For", iterations, () => GetNumberOfPrimesUsingParallelFor(numbers));
      
  Console.ReadLine();
}

At the very beginning, you'll notice that the numbers int[] is being initialized to integers from 1-500,000. That is we want to see how many numbers between 1 and 500,000 are prime numbers. We run each method 500 times because we're doing some performance benchmarking and so the average time across the 500 runs will give us a good reference for how long each method takes to do the job.

What we see below is the result of running the GetNumberOfPrimes() (single threaded) method.

---------------------------------
Non-Parallel
        Total time  : 62920.7195ms
        Average time: 125.8414ms
        Min time    : 125.2776ms
        Max time    : 130.5586ms
---------------------------------

The average time was 125.3 milliseconds. What's more interesting is what you see in Image 1 below. I'm running this on a quad core machine. so 25% utilization on a quad core CPU equates to 1 core being used 100%. It is a tight loop so 100% utilization is expected. However, we're only using 1 of the 4 cores available to us.

Typically, when we parallelize, we want to see all CPU cores saturated. That is all cores are utilized 100% for the duration of processing. That indicates 2 things:

1. The work we're doing is being done as fast as possible on the current hardware

2. More importantly, if we were to throw more cores at the machine the work will be done sooner.

This is the ideal. But very rarely do we achieve this ideal and in production environments we have to take into consideration other software that's running on the hardware. Nonetheless if the job needs to be done, it will be done faster if the cores being utilized are saturated. So you may decide that it's ok to saturate all cores for a shorter period of time, than to saturate fewer cores for a longer period of time.

Image 1: Showing that the non-parallelized method uses only 1 core

Now let's talk about what Data Parallelization is and how we'd use it in this context. Basically, the "data" is partitioned. That is the data is split into multiple partitions and each partition of data is executed on a different thread. Within a loop inside the thread we call our method (in this case IsPrime()) passing it the "data. Remember SIMD? Single Instruction multiple data. Does that make sense now? The single instruction is the IsPrime() method. The multiple data is the partitioned data.

Image 2: Data Parallelization – Partitioning data and processing each partition in a thread

Since we're working with an array the easiest way to partition our data is to break it up into smaller partitions. How many partitions? Well, since we have 4 cores (in this case) we could partition the data into 4 equal parts and then spawn 4 threads and pass each one a partition. This technique is called Range Partitioning. Range partitioning is by far the simplest algorithm one can use to partition data. As you'll see later in this post the partitioning algorithm is a key component of data parallelization and the best algorithm to use depends largely on the data and the function you perform on this data.

When using PLINQ or the Parallel class we don't have to do the partitioning ourselves (Whew!). More about PLINQ and Parallel class partitioning algorithms later.

I mentioned 4 partitions because we have 4 cores. Without going into details, remember that normally you may double up on the partitions and threads. That is 8 partitions and 8 threads. but it all depends on the data, the partitioning logic, hardware in use and the existing workload (other applications that share the hardware etc.).

We'll be looking at 3 parallelized implementations that do the same job (count the number of prime numbers in our int[] numbers) using different techniques.

1. PLINQ
2. Parallel.For
3. QueueUserWorkItem

The QueueUserWorkItem implementation does not require .NET 4.0 and in fact is something you can do using earlier versions of .NET including .NET 2.0. It's implementation is fairly complex so what I hope to impart in this post is that with the support for parallelization introduced in .NET 4.0 (especially Data Parallelization and Task Parallelization) it behooves us to use these capabilities in our software so we make good use of the hardware our software is running on.

PLINQ

As I mentioned before, with PLINQ we don't have to concern ourselves with partitioning or threads, PLINQ handles it all for us. All you have to do (for the most part) is call the .AsParallel() method on the IEnumerable! That is to say that the only difference between the non-parallel LINQ version and the parallel PLINQ version is the .AsParallel() method call. Couldn't be any simpler!

This gives us a lot of opportunities to parallelize our code where possible. Just don't go about blindly adding the .AsParallel() call to all LINQ to object queries! They won't all work and not all LINQ queries will benefit from being parallelized.

Behind the scenes PLINQ will use one of many partitioning algorithms depending on various criteria. (We can also provide our own partitioning algorithm implementation but for the most part that's not required and not covered in this post). I won't go into the details of how PLINQ makes these decisions in this post but the following are the algorithms that could be used

1. Range Partitioning
2. Chunk Partitioning
3. Stripe Partitioning
4. Hash Partitioning

With simple queries PLINQ will spawn multiple threads (normally equaling the number of cores on the system) to get the job done. A lot of the algorithms PLINQ uses in order to get the results of queries require cross thread communication. This could slow things down a bit (depending on the query and data). In some cases the algorithm could require PLINQ to wait for all threads to finish while in other cases it may not. Any way this is not stuff you need to be concerned about. And indeed, you don't need to know the inner workings of PLINQ in order to use it.

private static void GetNumberOfPrimesUsingPLinq(int[] numbers)
{
  var query = from n in numbers.AsParallel()
              where IsPrime(n)
              select n;
  var count = query.Count();
  //Console.WriteLine(count);
}

Parallel.For

Using Parallel.For we don't have to concern ourselves with data partitioning or threads either and link with PLINQ we could provide our own partitioning algorithm if we wanted to as well.

Parallel.For uses combinations of partitioning algorithms (dynamic partitioning) depending on the data and other workloads on the system. It could start with Range Partitioning and change to range/work stealing (explained later). Again, we don't need to know the inner workings in order to use this class.

The Parallel.For method has a number of overloads. For most tasks, you can use the simplest (to use) overload that essentially works like a regular for-loop. The method signature for this method looks like this:

public static ParallelLoopResult For(int fromInclusive, int toExclusive,
  Action<int> body);

To implement the functionality we require in this case the code would look like this:

    private static void GetNumberOfPrimesUsingParallelForSharedState(int[] numbers)
    {
      var total = 0;
      Parallel.For(0, numbers.Length,(j, loopState) =>
      {
        if (IsPrime(numbers[j]))
          Interlocked.Add(ref total, 1);
      });
      //Console.WriteLine("Parallel.For - # Of Primes: " + total);
    }

This code works and is really simple to understand. But it is not as performant as another alternative. So let's understand the problem with this code and see what other alternatives the Parallel class has to offer.

Notice (in the code listing above) that we have a single variable called total. Next, in the parallel loop, if the number in question is a prime number we need to increment the value of total. But because the code within the loop is running in parallel (on multiple threads), we need to serialize access to the total variable because it is shared by multiple threads. That is we need to ensure that only one thread writes to (or reads from) this variable at any time. And so, we use the Interlocked class to increment the value of the total variable. The Interlocked class provides us with many methods that are atomic (meaning that they are performed in their entirety or not at all) and thread safe. Worst case scenario is that during each iteration, when a thread needs to increment the count of the total variable, all other threads are blocked. This could hurt performance terribly in other situations but no so in our situation because not all numbers are prime.

But what if the tasks you want to perform in the real world is such that the result of each task has to be written to a variable that is shared by all threads (shared state)? That's the worst case scenario for the above code. So in this post we will concentrate on another overload of the Parallel.For method how we would implement this method. It is a bit complex and unfortunately not the most intuitive either but I'll do my best to explain the parameters about the implementation.

Parallel.For Thread-local state

So we know we don't want to use shared state. So we could use thread-local state (that is a variable that is local to each thread and somehow at the end of each loop if we could add up all of the results, we'd have our answer. The .NET team has given us this capability in the Parallel class as another overload of the For method. Let's take a look at the signature of this overload and try and understand what each of the parameters mean and how we'd use them.

public static ParallelLoopResult For<TLocal>(int fromInclusive, int toExclusive,
  Func<TLocal> localInit, Func<int, ParallelLoopState, TLocal, TLocal> body,
  Action<TLocal> localFinally);

The first two parameters are the same, so no problem there. The third parameter

localInit: Is a Func delegate that returns the initial state of the local data for each thread. In other words a method that returns the initial value of the thread local data (or local state). "Local state" in this context means a variable whose lifetime extends from just prior to the first iteration of the loop on the current thread, to just after the last iteration. In our case our local state is a variable of type int and it's initial state needs to be zero (for all threads).

body: This is a Func delegate for the body of our loop that is invoked once per iteration. The first parameter to this method is our loop increment counter (same as the other method overload), the second parameter is a type ParallelLoopState (let's not worry about this parameter since we're not using it here. The third parameter is the local state. So in other words, we're passed in this parameter (so we have the current state it is in, we can do what we need to do with it (in this case increment it is the number is a prime number). The last parameter (as per the typicaly Func delegate convention is the return type. So we return the local state at the end of each iteration. so it can be given back to us in the next iteration.

localFinally: The Action delegate that performs a final action on the local state of each thread. This is where you define the method that will be called once, after all the iterations on this thread have completed. The parameter this method provides us is none other than our thread-local state, so basically, this is where we get the opportunity to add up all of the thread-local variables into a variable that holds our final result.

Make sense? If this doesn't make sense now, it probably will when you see our implementation using this overload (you'll be able to join the dots). Our implementation of this overload is shown in Code Listing 4 below.

private static void GetNumberOfPrimesUsingParallelFor(int[] numbers)
{
  var total = 0;
  Parallel.For(0, numbers.Length, () => 0, (j, loopState, subtotal) =>
    {
      if (IsPrime(numbers[j]))
        subtotal++;
      return subtotal;
    },
      (x) => Interlocked.Add(ref total, x)
    );
  //Console.WriteLine(total);
}

Looking at Code Listing 4 above, our 3rd parameter (localInit) is a lamba function that gets no parameters and returns the initial state of our local state (which is zero). The body parameter (the 4th parameter) is a Func delegate that passes in 3 parameters and expects a return parameter of type int. j is our loop counter, we don't use loopState in this example and subTotal is the loop state variable that holds the count of prime numbers of each thread. It is passed in as a parameter, we increment it in the body of our loop and return it in each iteration (so it can be passed in to us in the next iteration). The 5th parameter (localFinally) is an Action delegate that passes in one parameter, our local state. It is implemented as a lambda here, where x is the parameter that is our local state and we need to increment our total variable with this value. But here, because we're now dealing with shared state, we use the Interlocked class again. But keep in mind that in this case the localFinally action delegate is called once at the end of each thread's execution. So we're only blocking other threads once, at the end of each thread's execution rather than once during each iteration.

So in other words, using thread-local data, we can avoid the overhead of serializing a large number of accesses to shared state. Instead of writing to a shared resource on each iteration, we compute and store the value until all iterations for the task are complete.

Performance Results

Let's look at some performance numbers now. Here are the timing results of the various implementations.

---------------------------------
Plinq
        Total time  : 20860.5636ms
        Average time: 41.7211ms
        Min time    : 41.5165ms
        Max time    : 62.3814ms
---------------------------------

---------------------------------
QueueUserWorkItem
        Total time  : 20107.1865ms
        Average time: 40.2143ms
        Min time    : 39.9199ms
        Max time    : 58.2553ms
---------------------------------

---------------------------------
Parallel.For
        Total time  : 16517.1344ms
        Average time: 33.0342ms
        Min time    : 32.5172ms
        Max time    : 37.0311ms
---------------------------------

From the performance numbers we see above, the method that uses Parallel.For performs the best. The method that uses QueueUserWorkItem comes in at 2nd place, while the Plinq method comes in at 3rd place. These numbers could change depending on the workload, data as well as the hardware architecture on which you run this code. We'll examine another case later in this post where the data (or the arrangement of data) makes a difference to the outcome.

However, let's not loose sight of the point of this post and that is to make use of and to  benefit from parallelization of either existing or new code. Any one of these methods is almost 4 times faster than the original method. 4 times faster because I'm running this on a 4 core machine. If I was to run this on an 8 core machine they would all be almost 8 times faster.

The Plinq implementation does not saturate all cores as you can see in the CPU utilization screen shot below. There is a reason for this and we'll talk about this later in the post. Over the duration of processing the CPU utilization fluctuated between 74% and 76% but stayed mostly at 76%. So I've shown you 76% here. The same goes with the other CPU utilization screen shots you see later.

Image 3: Plinq CPU Utilization – all cores, but not 100% saturation

In the next image we see the CPU utilization of the method that uses QueueUserWorkItem to queue multiple worker threads on the thread pool, each with it's own partition of data. As you can see here, the CPU utilization is slightly higher (consistently) showing that the parallelization implementation makes better use of the hardware available to us.

Image 4: QueueUserWorkItem CPU Utilization almost 100% saturation

The next image shows the CPU utilization of the Parallel.For method. Here we see that all cores are 100% saturated (almost all the time). So it looks like the Parallel.For method does the best job of utilizing all of the available hardware.

That's the nature of parallel computing. There are a lot of variations possible and way too many factors that will impact the outcome. So what I want you to keep in mind is that there is no one method that will always outperform any other. It depends largely on the data and and kind of processing (the prime number algorithm used in case) and how well each of the methods manage threads, memory I/O and minimizes the cost of overheads as well as the hardware architecture of the system (as mentioned earlier).

Image 5: Parallel.For CPU Utilization showing 100% saturation

Data Partitioning is Important

So why is the data partitioning algorithm important? In this case it is extremely important and it has to do with the way the IsPrime() method is implemented. You see in order to determine if a given number is a prime number or not, we iterate from 3 up to the square of the number to see if the number is wholly divisible by any of those numbers. As the number gets larger we have to iterate over more numbers and check against each of those numbers to determine if the number is a prime number or not. So effectively, larger numbers will take longer to determine and since our initial int[] is an ordered sequence of numbers from 1-500,000 the last partition has the largest numbers and processing the last partition will take the longest to process and processing the second to last partition will take a little less time and so on.

If that part is clear, then you'll realize that the way we're partitioning the data is kind of unfair, in that the thread that gets the first partition will finish before the thread that gets the last partition since the data in the last partition contains larger numbers. So how we partition our data plays an important role in how performant our parallelization is.

But it's not so much about being unfair but rather about underutilization of hardware. You see when the first thread finishes, it sits there doing nothing until the other threads complete their workload. Then the second thread finishes and waits for the other two to finish. As a result, the cores are not 100% saturated. That's not good.

Work Stealing And Dynamic Data Partitioning

So then how come the method that uses Parallel.For seems to not be affected by the unbalanced data partitioning? What's not quite apparent is that because of the new features built into the CLR Thread Pool, namely work stealing and dynamic data partitioning, the method that uses Parallel.For outperforms all the others even though the data partitioning is not ideal, because when thread 1 finishes it's work load, it starts to steal work from the other threads, when thread 2 finishes it's workload it starts to steal from the other threads as well. So effectively all threads are busy all of the time giving us 100% or close to 100% saturation of cores.

So the clear lesson here is that the .NET implementation of Parallel.For is highly optimized and behind the scenes the CLR thread pool is able to employ the new work stealing feature (using the Hill Climbing heuristic where the aim is to keep all cores busy). The Parallel.For method (a few of the other overloads) allows you to control or define the data partitioning that should be used as well. But here we're just letting it do whatever it does by default.

It should be noted that the new features of the thread pool are used by the Parallel class, Plinq as well as Tasks (all introduced in .NET 4.0). ThreadPool.QueueUserWorkItem does not utilize these new features because the way these features are implemented would likely break legacy code and since QueueUserWorkItem has been around since .NET 1.1 the team at Microsoft decided to make it an "opt-in" feature and you opt-in by simply using the new features in .NET 4.0.

In order to fix our data partitioning all we need to do is order the integers in the array randomly. Of course bear in mind that working with real world data is very different and balancing your data partitions may not be that simple or possible or may not be required at all. It just depends on what you're doing with your data.

The code below shows a simple way to randomly sort our int[] of numbers between 1-500,000.

  var rnd = new Random();
  var numbers = Enumerable.Range(1, 500000)
                          .OrderBy(r => rnd.Next())
                          .ToArray();

Let's see if this change has any impact on our benchmarks.

---------------------------------
Plinq
        Total time  : 16712.8642ms
        Average time: 33.4257ms
        Min time    : 33.1166ms
        Max time    : 37.4965ms
---------------------------------

---------------------------------
QueueUserWorkItem
        Total time  : 16080.1191ms
        Average time: 32.1602ms
        Min time    : 31.7991ms
        Max time    : 48.0894ms
---------------------------------

---------------------------------
Parallel.For
        Total time  : 16497.6999ms
        Average time: 32.9953ms
        Min time    : 32.7595ms
        Max time    : 35.3307ms
---------------------------------

Compared to the earlier results each of the algorithms has improved by about 3-4%. The method that uses QueUserWorkItem comes in at the 1st place and next is the Parallel.For and in 3rd place is Plinq.

The Plinq and QueueUserWorkItem method both showed CPU utilizations of about 99% as shown below, while the Parallel.For method remained close to 100% through the duration (probably due to thread management and other overhead reasons).

Please take note that this example (and the data) is very simplistic and in the real world, with real data the Parallel.For method will almost always outperform the other approaches. The thread pool with its work stealing, hill climbing heuristic, dynamic data partitioning and awareness of what other applications/processes are currently running on the hardware has a lot more information and capabilities than our simplistic (almost naïve) implementation of the method that uses QueueUserWorkItem.

If you take a look at the implementation of method that uses QueueUserWorkItem in Code Listing 5 below, you'll see that there is a lot going on here. But also notice that we determine the number of threads to spawn based on the number of Cores available (but we don't take into account the workload already present on the hardware). So this method should scale just as well as the other methods presented in this post. But is the complexity of the implementation worth the gains? And remember every workload is different and there might be more complexity involved that will make your implementations more complex as well. So I highly recommend using Plinq and/or the Parallel class' Parallel.For or Parallel.ForEach methods and let the underlying framework do the optimizations for you while you keep you code simple and easy to understand and maintain.

    private static void GetNumberOfPrimesUsingThreadPool(int[] numbers)
    {
      var noOfPrimes = 0;
      var coreCount = Environment.ProcessorCount;
      var batchSize = numbers.Length / coreCount;

      var pending = coreCount;
      using (var mre = new ManualResetEvent(false))
      {
        for (int batchCount = 0; batchCount < coreCount; batchCount++)
        {
          var lower = batchCount * batchSize;
          var upper = (batchCount == coreCount - 1) ? numbers.Length : lower + batchSize;
          ThreadPool.QueueUserWorkItem(st =>
          {
            var count = 0;
            for (int i = lower; i < upper; i++)
              if (IsPrime(numbers[i]))
                count++;
            Interlocked.Add(ref noOfPrimes, count);
            if (Interlocked.Decrement(ref pending) == 0)
              mre.Set();
          });
        }
        mre.WaitOne();
      }
      //Console.WriteLine(noOfPrimes);
    }

8 Core Machine

I ran the application on an 8 core machine. That's 2 CPUs with 4 cores each. Keep in mind that a single CPU with 8 cores will perform better than 2 CPUs with 4 cores each. That's because memory access is extremely slow as compared to CPU speed and if all cores can get the data they need (including the work stealing) from the same chip (L1, L2 cache) rather than having to make trips to main memory or in the case of work stealing, having to steal from work that's on a different chip altogether, things will slow down a bit. I should add that the CLR Thread pool's work stealing algorithms do account for this and optimize accordingly.

The benchmark timings and CPU utilization is shown below

---------------------------------
Plinq
        Total time  : 4023.2581ms
        Average time: 8.0465ms
        Min time    : 6.8322ms
        Max time    : 10.0536ms
---------------------------------

---------------------------------
QueueUserWorkItem
        Total time  : 3658.2514ms
        Average time: 7.3165ms
        Min time    : 6.3065ms
        Max time    : 9.6705ms
---------------------------------

---------------------------------
Parallel.For
        Total time  : 3508.3063ms
        Average time: 7.0166ms
        Min time    : 6.0755ms
        Max time    : 9.5767ms
---------------------------------

• No of Cores
• No of Logical Processors
• No of Physical Processors
• Hardware Bitness (32/64 bit etc.)
• CPU Architecture (x86, x64, Itanium etc.)

    private static void GetCpuInformation()
    {
      var noOfCores = 0;
      foreach (var item in new System.Management.ManagementObjectSearcher("Select * from Win32_Processor").Get())
      {
        noOfCores += int.Parse(item["NumberOfCores"].ToString());
        Console.WriteLine("Bitness: {0}", item["AddressWidth"]);
        Console.WriteLine("Architecture: {0}", GetArchitectureDiscription(int.Parse(item["Architecture"].ToString())));
      }

      Console.WriteLine("Number Of Cores: {0}", noOfCores);

      foreach (var item in new System.Management.ManagementObjectSearcher("Select * from Win32_ComputerSystem").Get())
      {
        Console.WriteLine("Number Of Logical Processors: {0} ", item["NumberOfLogicalProcessors"]);
        Console.WriteLine("Number Of Physical Processors: {0} ", item["NumberOfProcessors"]);
      }      
    }

    private static string GetArchitectureDiscription(int architectureNo)
    {
      switch (architectureNo)
      {
        case 0: return "x86";
        case 1: return "MIPS";
        case 2: return "Alpha";
        case 3: return "PowerPC";
        case 6: return "Itanium-based systems";
        case 9: return "x64";
        default:
          return "Unkown";
      }
    }

There is quite a bit more information available from both the Win32_Processor WMI class as well as the Win32_ComputerSystem, so you may want to check out the other properties/information these object provide.

I use the number of cores information to (naïvely) determine how many threads I can/should spawn when I'm doing some multi-threaded work. Of course nowadays with availability of Tasks in .NET and Parallel.For etc. one should probably use facilities built into the .NET framework where possible.

You can optimize the queries if you need to. For example, we're doing a select * from for each of queries you see in the code listing above. You could change that to return only the properties you're interested in. For example the seconds query above could be rewritten as

"Select NumberOfLogicalProcessors, NumberOfProcessors from Win32_ComputerSystem"

The rest of the code will remain the same, including the way in which the properties are extracted from the result.

Before we go down that route, let me say that I assume you're somewhat familiar with using asynchronous methods. The intent of this post is to show you how you would convert one or more of your own synchronous methods into asynchronous method. So I'm not going to be explaining the basics of asynchronous programming in .NET.

The APM design pattern in .NET uses some naming conventions for the method names as well as the basic method signatures. You'll have BeginXXX and EndXXX method, where XXX is the name of the synchronous method. In other words asynchronous operations that use the APM (IAsyncResult) design pattern are implemented as a pair of methods. The basic method signature for the BeginXXX method is

IAsyncResult BeginXXX(AsyncCallback callback, object state)

The basic method signature for the EndXXX method is

void EndXXX(IAsyncResult result)

Now if we want to make one of our own methods asynchronous then we'll have to follow the basic signature. If our method required additional parameters or has a return type then we can include these parameters, while keeping the basic signature. The idea is to re-use our existing method (and implementation) but as an asynchronous operation.

So lets say we have a method in our business layer class that does some compute-bound work. For the sake of an example let's say the method requires an int[]as an input parameter and it returns the number of prime numbers in the array as an int. The synchronous version of the method in shown in the code listing below.

    /// <summary>
    /// Synchronous Method to Get the number of prime numbers in an array of numbers
    /// </summary>
    /// <param name="numbers"></param>
    /// <returns></returns>
    public int GetNumberOfPrimeNumbersInArray(int[] numbers)
    {
      int count = 0;
      for (int i = 0; i < numbers.Length; i++)
      {
        if (IsPrime(numbers[i]))
          count++;
      }
      return count;
    }

    private bool IsPrime(int number)
    {
      if (number == 4)
        return false;
      for (int i = 2; i < number / 2; i++)
        if (number % i == 0)
          return false;
      return true;
    }
  }

Implementing Asynchronous Methods

The BeginXXX method takes any parameters declared in the signature of the synchronous version of the method that are passed by value or by reference (in this case the int[]). Any out parameters are not part of the BeginXXX method signature. The BeginXXX method signature also includes two additional parameters which are part of the basic method signature as shown above. Further, the BeginXXX method needs to also return an instance of IAsyncResult which is also part of the basic method signature. So effectively, our BeginGetNumberOfPrimeNumbersInArray method signature will look like the following

 public IAsyncResult BeginGetNumberOfPrimeNumbersInArray(int[] numbers,
   AsyncCallback callback, object state)
 {
 }

Similarly, our EndGetNumberOfPrimeNumbersInArray method will follow the basic signature but because our synchronous version returns an int, so our End method will return an int. Following the basic method signature and making appropriate modifications to the signature, the method signature for our EndGetNumberOfPrimeNumbersInArray will look like the following:

public int EndGetNumberOfPrimeNumbersInArray(IAsyncResult result)
{
}

So now, all we're left with is the actual implementation of these methods. Keep in mind that we're going to use the existing synchronous method in here so we don't have to re-implement the existing method but rather re-use it asynchronously.

Looking at the signature of our BeginXXX method a question that's going to come up (if it hasn't already) is that we'll need an implementation of the IAsyncResult interface. That's true. Implementing the IAsyncResult is not a trivial task. However all delegates provide us with an instance of an IAsyncResult upon calling their BeginInvoke method. So we could look at using this approach. The general consensus on the .NET community is that using the delegate's BeginInvoke/EndInvoke is slow. Slow is a relative term. If the method itself is going to take 2-3 seconds or more to complete then any performance gains you might get by implementing your own IAsyncResult quickly diminishes. Remember that you only call BeginInvoke once. Now in the case where you're calling your asynchronous method thousands of times or more then improving on this slowness could make a difference.

The code listing below shows the completed implementations of the BeginGetNumberOfPrimeNumbersInArray and the EndGetNumberOfPrimeNumbersInArray method. We also use an AsyncState class that is shown towards the end of the code listing. It's just a simple class that is used to pass state information between the Begin and End methods.

In the Begin method, we define a delegate that has the same signature as our synchronous method. That's because we will be calling our synchronous method asynchronously and we use a delegate to help with calling it asynchronously. A reference to this delegate is passed along to the End method via the AsyncState object.

  public class BusinessModule
  {
    public IAsyncResult BeginGetNumberOfPrimeNumbersInArray(int[] numbers, AsyncCallback callback, object state)
    {
      Func<int[], int> del = GetNumberOfPrimeNumbersInArray;
      var asyncState = new AsyncState(del, state);
      return del.BeginInvoke(numbers, callback, asyncState);
    }

    public int EndGetNumberOfPrimeNumbersInArray(IAsyncResult result)
    {
      var asyncState = (AsyncState)result.AsyncState;
      var del = (Func<int[], int>)asyncState.Del;
      return del.EndInvoke(result);
    }

    /// <summary>
    /// Synchronous Method to Get the number of prime numbers in an array of numbers
    /// </summary>
    /// <param name="numbers"></param>
    /// <returns></returns>
    public int GetNumberOfPrimeNumbersInArray(int[] numbers)
    {
      int count = 0;
      for (int i = 0; i < numbers.Length; i++)
      {
        if (IsPrime(numbers[i]))
          count++;
      }
      return count;
    }

    private bool IsPrime(int number)
    {
      if (number == 4)
        return false;
      for (int i = 2; i < number / 2; i++)
        if (number % i == 0)
          return false;
      return true;
    }
  }

  public class AsyncState
  {
    public Delegate Del { get; private set; }
    public object State { get; private set; }

    public AsyncState(Delegate del, object state)
    {
      Del = del;
      State = state;
    }
  }

Using Asynchronous Methods

The code listing below is the entire code you'll need to run this application. Using our new asynchronous method is similar to using any asynchronous method you would encounter in the .NET BCL. You can see the call to our BeginGetNumberOfPrimeNumbersInArray method in the main method shown below.

  class Program
  {
    static void Main(string[] args)
    {
      var bm = new BusinessModule();
      var numbers = Enumerable.Range(0, 10000).ToArray();

      bm.BeginGetNumberOfPrimeNumbersInArray(numbers, GetNumberOfPrimeNumbersInArrayCallback, bm);
      Console.WriteLine("You should see the number of primes after this message");      
      Console.ReadLine();
    }

    static void GetNumberOfPrimeNumbersInArrayCallback(IAsyncResult result)
    {
      var asyncState = (AsyncState)result.AsyncState;
      var del = (Func<int[], int>)asyncState.Del;
      var numberOfPrimes = del.EndInvoke(result);
      Console.WriteLine(numberOfPrimes);
    }
  }

  public class BusinessModule
  {
    public IAsyncResult BeginGetNumberOfPrimeNumbersInArray(int[] numbers, AsyncCallback callback, object state)
    {
      Func<int[], int> del = GetNumberOfPrimeNumbersInArray;
      var asyncState = new AsyncState(del, state);
      return del.BeginInvoke(numbers, callback, asyncState);
    }

    public int EndGetNumberOfPrimeNumbersInArray(IAsyncResult result)
    {
      var asyncState = (AsyncState)result.AsyncState;
      var del = (Func<int[], int>)asyncState.Del;
      return del.EndInvoke(result);
    }

    /// <summary>
    /// Synchronous Method to Get the number of prime numbers in an array of numbers
    /// </summary>
    /// <param name="numbers"></param>
    /// <returns></returns>
    public int GetNumberOfPrimeNumbersInArray(int[] numbers)
    {
      int count = 0;
      for (int i = 0; i < numbers.Length; i++)
      {
        if (IsPrime(numbers[i]))
          count++;
      }
      return count;
    }

    private bool IsPrime(int number)
    {
      if (number == 4)
        return false;
      for (int i = 2; i < number / 2; i++)
        if (number % i == 0)
          return false;
      return true;
    }
  }

  public class AsyncState
  {
    public Delegate Del { get; private set; }
    public object State { get; private set; }

    public AsyncState(Delegate del, object state)
    {
      Del = del;
      State = state;
    }
  }

Because the method is asynchronous and we're calling in asynchronously rather than making a blocking call (which is also an option when using the APM), you'll see the message:

You should see the number of primes after this message

Notice the UrlEncode() method in the code listing below, that's the method I present in this post. The outputs of the HttpUtility.UrlEncode() method any my UrlEncode() method are listed below such that you can compare the two outputs.

  class Program
  {
    private readonly static string unreservedCharacters = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_.~";

    static void Main(string[] args)
    {
      var url = "!*'();:@&=+$,/?#[] <>~.\"{}|\\-`_^%";

      Console.WriteLine("Using HttpUtility: " + HttpUtility.UrlEncode(url));
      Console.WriteLine("Using UrlEncode:   " + UrlEncode(url));
      Console.WriteLine(HttpUtility.UrlDecode(UrlEncode(url)));

      Console.ReadLine();
    }

    /// <summary>
    /// This method is the correct implementation of UrlEncode.
    /// Given a string, it returns a UrlEncoded version on the string.
    /// </summary>
    /// <param name="value">the string/url to be UrlEncoded</param>
    /// <returns>The UrlEncoded string</returns>
    public static string UrlEncode(string value)
    {
      if (String.IsNullOrEmpty(value))
        return String.Empty;

      var sb = new StringBuilder();

      foreach (char @char in value)
      {
        if (unreservedCharacters.IndexOf(@char) != -1)
          sb.Append(@char);
        else
          sb.AppendFormat("%{0:x2}", (int)@char);
      }
      return sb.ToString();
    }
  }

Nothing spectacular really, but the purpose of me posting these code snippets is so its out there for those who are trying to figure this out and for me if/when I need to do this kind of thing again.

Some features of the code presented below are:

1. Re-Directs the Standard output stream using RedirectStandardOutput = true
2. Re-Directs the Standard Error stream using RedirectStandardError = true
3. Waits for the process to complete. In this case it waits for two processes to complete. The process that stops and then the process that starts.
4. Asynchronously read from the Standard output and Standard Error streams

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using System.Runtime.Serialization;

namespace ResetService
{
  enum NetOperation { STOP, START };

  public static class ServiceBounce
  {
    private static StringBuilder errorMessage = new StringBuilder();
    private static StringBuilder outputMessage = new StringBuilder();

    public static int TIMEOUT = 60000;

    private static int ExecuteNetProcess(NetOperation operation, string instanceName)
    {
      using (var process = new Process())
      {
        process.StartInfo = new ProcessStartInfo("net", operation.ToString() + " " + instanceName);
        process.StartInfo.UseShellExecute = false;
        process.StartInfo.RedirectStandardOutput = true;
        process.StartInfo.RedirectStandardError = true;
        process.OutputDataReceived += new DataReceivedEventHandler(OnOutputDataReceived);
        process.ErrorDataReceived += new DataReceivedEventHandler(OnErrorDataReceived);
        process.Start();
        process.BeginOutputReadLine();
        process.BeginErrorReadLine();
        process.WaitForExit(TIMEOUT);
        int exitCode = process.ExitCode;
        process.Close();
        return exitCode;
      }
    }

    private static void ClearMessages()
    {
      outputMessage.Length = 0;
      errorMessage.Length = 0;    
    }

    public static void Restart(string serviceName)
    {
      Stop(serviceName);
      Start(serviceName);
    }

    public static void Stop(string serviceName)
    {
      ClearMessages();
      string errMessage = null;

      var exitCodeStop = ExecuteNetProcess(NetOperation.STOP, serviceName);
      if (exitCodeStop != 0)
      {
        errMessage = errorMessage.ToString();
        if (!errMessage.Contains("service is not started"))
          throw new MSSQLServiceBounceException(errorMessage.ToString());
      }
    }

    public static void Start(string serviceName)
    {
      ClearMessages();
      string errMessage = null;

      var exitCodeStart = ExecuteNetProcess(NetOperation.START, serviceName);
      Console.ForegroundColor = ConsoleColor.White;
      if (exitCodeStart != 0)
      {
        errMessage = errorMessage.ToString();
        if (!errMessage.Contains("The requested service has already been started"))
          throw new MSSQLServiceBounceException(errMessage);
      }
    }

    private static void OnOutputDataReceived(object sender, DataReceivedEventArgs e)
    {
      outputMessage.Append(e.Data);
      Console.ForegroundColor = ConsoleColor.DarkGreen;
      Console.WriteLine(e.Data);
    }

    private static void OnErrorDataReceived(object sender, DataReceivedEventArgs e)
    {
      errorMessage.Append(e.Data);
      Console.ForegroundColor = ConsoleColor.DarkRed;
      Console.WriteLine(e.Data);
    }
  }

  public class MSSQLServiceBounceException : Exception
  {
    public MSSQLServiceBounceException(string message)
      :base(message)
    {
    }

    public MSSQLServiceBounceException(string message, Exception innerException)
      : base(message, innerException)
    {
    }

    public MSSQLServiceBounceException(SerializationInfo serializationInfo, StreamingContext context)
      : base(serializationInfo, context)
    {
    }
  }
}

The class shown in the code listing below uses the ServiceController class from the System.ServiceProcess assembly. This is a pure .NET solution to working with Windows Services. The ServiceController class makes it simple to make "synchronous" calls to stop and start services using the WaitForStatus method as you can see in the code listing below. The next code listing (the one below this one) shows you sample code of how you might use this class.

The features of the class presented below are:

The methods provide synchronous calls. That is the calls don't return until the service is either stopped or stated or stop/started as the case may be. If you do required asynchronous behavior, it's easy enough to modify the code since the normal behavior of the ServiceController class is async.

  public static class WindowsServiceBounce
  {
    public static IEnumerable<ServiceController> FindService(Func<ServiceController, bool> predicate)
    {
      ServiceController[] services = ServiceController.GetServices();
      var servicesFound = from s in services
                          where predicate(s)
                          select s;
      return servicesFound;
    }

    public static void Restart(ServiceController service)
    {
      Restart(service, new TimeSpan(0, 1, 0));
    }

    public static void Restart(ServiceController service, TimeSpan waitTimeout)
    {
      Stop(service, waitTimeout);
      Start(service, waitTimeout);
    }

    public static void Stop(ServiceController service, TimeSpan waitTimeout)
    {
      if (service.Status == ServiceControllerStatus.StartPending)
        service.WaitForStatus(ServiceControllerStatus.StartPending, waitTimeout);
      
      if (service.Status == ServiceControllerStatus.Running)
      {
        service.Stop();
        service.WaitForStatus(ServiceControllerStatus.Stopped, waitTimeout);
      }
    }

    public static void Start(ServiceController service, TimeSpan waitTimeout)
    {
      if (service.Status == ServiceControllerStatus.StopPending)
        service.WaitForStatus(ServiceControllerStatus.StopPending, waitTimeout);

      if (service.Status == ServiceControllerStatus.Stopped)
      {
        service.Start();
        service.WaitForStatus(ServiceControllerStatus.Running, waitTimeout);
      }
    }
  }

The code listing below shows and sample of how you might use the class presented in the previous code listing.

      var services = WindowsServiceBounce.FindService(s => String.Compare(s.ServiceName, "MSSQL$SQLEXPRESS", StringComparison.OrdinalIgnoreCase) == 0);
      
      foreach (var service in services)
        WindowsServiceBounce.Restart(service, new TimeSpan(0, 1, 0));
      foreach (var service in services)
        service.Dispose();

DBCC CHECKIDENT('sometable', RESEED, 0);

Change 'sometable' to the name of your table. If you need to set the seed value to something other than zero, change the last parameter in the call from 0 to what you need. The way that last parameter works is this:

Let's say you have a table (with an identity column) with 20 records. You've deleted a few records (because you've inserted records during test and you've now deleted them) and you know if you were to insert another record it is going to get an identity value of 42 say, but you want it to be 21. In order for this to work as you'd expect, set the last parameter to 20. Note, that's 20 and not 21. Of course make sure there are no records with an identity value greater than the value you set.

In order to solve this we use Group By and we group by the AddedDate property and if any group has a count greater than 1, it is a duplicate and we need to get those. For test purposes, I've defined a variable called testDateAndTime that is assigned to the current DateTime. And then at the time of initializing the Lesson instances, I've assigned two lessons the same date and time using this variable. The two lessons (as shown in the code listing below) are:

1. Preposition / Phrasal Verb
2. German Verbs in the Present Tense

  class Program
  {
    static void Main(string[] args)
    {
      var testDateAndTime = DateTime.Now;

      var lessons = new Lesson[] {
        new Lesson { Title="Verb Tense", Subject="English", AddedDate = DateTime.Now.AddMinutes(1)},
        new Lesson { Title="Conditionals", Subject="English", AddedDate = DateTime.Now.AddDays(2)},
        new Lesson { Title="Gerunds and Infinitives", Subject="English", AddedDate = DateTime.Now.AddDays(3)},
        new Lesson { Title="Vocabulary", Subject="English", AddedDate = DateTime.Now.AddDays(4)},
        new Lesson { Title="Preposition / Phrasal Verb", Subject="English", AddedDate = testDateAndTime},
        new Lesson { Title="Greetings", Subject="German", AddedDate = DateTime.Now.AddMinutes(5)},
        new Lesson { Title="Personal pronouns", Subject="German", AddedDate = DateTime.Now.AddDays(6)},
        new Lesson { Title="Introduction to nouns and gender", Subject="German", AddedDate = DateTime.Now.AddDays(7)},
        new Lesson { Title="Two important verbs", Subject="German", AddedDate = DateTime.Now.AddDays(8)},
        new Lesson { Title="German Verbs in the Present Tense", Subject="German", AddedDate = testDateAndTime}
      };

      var groupedLessons = from l in lessons
                           group l by l.AddedDate into g
                           where g.Count() > 1
                           select new { AddedDate = g.Key, Lessons = g };

      foreach (var k in groupedLessons)
      {
        Console.WriteLine("Added Date: " + k.AddedDate);
        foreach (var l in k.Lessons)
        {
          Console.WriteLine("\t Lesson Title: " + l.Title);
        }
      }
    }
  }

  public class Lesson
  {
    public string Title { get; set; }
    public string Subject { get; set; }
    public DateTime AddedDate { get; set; }
  }

If you run this program you'll see the following output, which is what you'd expect.

Added Date: 2/2/2011 8:08:59 AM
         Lesson Title: Preposition / Phrasal Verb
         Lesson Title: German Verbs in the Present Tense

There are some nice features this class offers in contrast to JavaScriptSerializer like the ability to control how property names get serialized using the DataMember attribute. One problem with this class is that it does not support serializing IEnumerable<T>. It supports generic collections such as List<T> but that's not what I need since there is a reason I'm using an IEnumerable<T> to serialize to JSON.

In this post I'll present a JsonSerlizer class that supports IEnumerable<T>. Some of the features of this class are as follows:

1. Supports serializing IEnumberable<T> in addition to other collections types as well as non-collection types
2. Serializes directly to System.IO.Stream (in contrast to JavaScriptSerializer)
3. Supports JSONP, so you can publish your JSON feeds to the public

First let me begin by showing how I get the data and why I need to serialize and IEnumerable<T> in the first place.

In code listing 1 below, you can see the class SubCategoryDrw that will be serialized. It looks a bit complex but it's really just a POCO. It's what I call a DataReader Wrapper. Anyway, I've decorated the class with the required DataContract attribute and its properties with the DataMember attribute while also modifying the property names and/or case that I'd like each property to be serialized as. This feature (the ability to change the name and /or case of property name) is the reason why I prefer using the DataContractSerializer class over the JavaScriptSerializer class.

  [DataContract]
  public sealed class SubCategoryDrw : BaseDbDataReaderWrapper
  {
    #region Properties

    private Int32 categoryId;
    [DataMember(Name="categoryId")]
    public Int32 CategoryId { get { switch (Mode) { case WrapperMode.Wrapper: return (Int32)DbDataReader[0]; case WrapperMode.Dto: return categoryId; default: return default(Int32); } } set { categoryId = value; } }
    private Int32 subCategoryId;
    [DataMember(Name = "subCategoryId")]
    public Int32 SubCategoryId { get { switch (Mode) { case WrapperMode.Wrapper: return (Int32)DbDataReader[1]; case WrapperMode.Dto: return subCategoryId; default: return default(Int32); } } set { subCategoryId = value; } }
    private String categoryDescription;
    [DataMember(Name = "categoryDesc")]
    public String CategoryDescription { get { switch (Mode) { case WrapperMode.Wrapper: return (String)DbDataReader[2]; case WrapperMode.Dto: return categoryDescription; default: return default(String); } } set { categoryDescription = value; } }
    private String subCategoryDescription;
    [DataMember(Name = "subCategoryDesc")]
    public String SubCategoryDescription { get { switch (Mode) { case WrapperMode.Wrapper: return (String)DbDataReader[3]; case WrapperMode.Dto: return subCategoryDescription; default: return default(String); } } set { subCategoryDescription = value; } }

    #endregion Properties

    public SubCategoryDrw(DbDataReader dbDataReader, WrapperMode mode)
      : base(dbDataReader, mode)
    {
      switch (Mode)
      {
        case WrapperMode.Wrapper:
          break;
        case WrapperMode.Dto:
          categoryId = (Int32)DbDataReader[0];
          subCategoryId = (Int32)DbDataReader[1];
          categoryDescription = (String)DbDataReader[2];
          subCategoryDescription = (String)DbDataReader[3];
          break;
      }
    }
  }

In code listing 2, you see a method I have in my business layer, that returns an IEnumerable<SubCategoryDrw>. Because the intent is to go from data stream, straight out to the Http Response Stream in order to get the best performance possible, not only do I use a DbDataReader directly (along with a DataReader wrapper class so I get type safety) and return an IEnumerable<T>, which basically means the data that comes out of the database is streamed directly to the Http Response stream in one single iteration.

    public IEnumerable<SubCategoryDrw> GetSubCategories()
    {
      using (var dr = DataModule.GetSubCategories())
      {
        var subCategoryDrw = new SubCategoryDrw(dr, WrapperMode.Wrapper);
        while (dr.Read())
          yield return subCategoryDrw;
      }
    }

In code listing 3 below you can see a WCF method wherein I use the JsonSerializer presented in this post to serialize a sequence of SubCategories. Note that I'm showing you a WCF method implementation, but you'll most likely use the JsonSerializer presented in this post in non-WCF applications since WCF 4.0 does handle serializing IEnumerable<T>.

    [WebGet(UriTemplate = "/subcategories/")]
    public void GetSubCategories()
    {
      var categories = BusinessModule.GetSubCategories();
      JsonSerializer.Serialize(categories, Response.OutputStream, Request.QueryString["callback"]);
    }

    [WebGet(UriTemplate = "/subcategories/")]
    public Message GetSubCategories()
    {
      var categories = BusinessModule.GetSubCategories();
      return WebOperationContext.Current
        .CreateJsonResponse<IEnumerable<SubCategoryDrw>>(categories);
    }

JsonSerializer

As I mentioned earlier, some of the features of this class are as follows:

1. Supports serializing IEnumberable<T> in addition to other collections types as well as non-collection types
2. Serializes directly to System.IO.Stream (in contrast to JavaScriptSerializer)
3. Supports JSONP, so you can publish your JSON feeds to the public

There are just 2 public methods in this class. One that expects and IEnumerable<T> as the object to serialize and another where the object is not an IEnumerable<T> either a List<T> or a non collection type object.

public static class JsonSerializer
{
  private static byte[] lBracketBytes = Encoding.UTF8.GetBytes(new char[] { '[' });
  private static byte[] rBracketBytes = Encoding.UTF8.GetBytes(new char[] { ']' });
  private static byte[] commaBytes = Encoding.UTF8.GetBytes(new char[] { ',' });
  private static byte[] rparenBytes = Encoding.UTF8.GetBytes(new char[] { ')' });

  public static void Serialize<T>(T obj, Stream stream, string jsonpCallback = null)
  {
    if (jsonpCallback == null)
      WriteObject(obj, stream);
    else
    {
      WriteJsonPCallbackStart(stream, jsonpCallback);
      WriteObject(obj, stream);
      WriteJsonPCallbackEnd(stream);
    }
  }

  public static void Serialize<T>(IEnumerable<T> sequence, Stream stream, string jsonpCallback = null)
  {
    if (jsonpCallback == null)
      WriteSequence(sequence, stream);
    else
    {
      WriteJsonPCallbackStart(stream, jsonpCallback);
      WriteSequence(sequence, stream);
      WriteJsonPCallbackEnd(stream);
    }
  }

  private static void WriteJsonPCallbackStart(Stream stream, string jsonpCallback)
  {
    byte[] callbackStartBytes = UTF8Encoding.UTF8.GetBytes(jsonpCallback + "(");
    stream.Write(callbackStartBytes, 0, callbackStartBytes.Length);
  }

  private static void WriteJsonPCallbackEnd(Stream stream)
  {
    stream.Write(rparenBytes, 0, rparenBytes.Length);
  }

  private static void WriteObject<T>(T obj, Stream stream)
  {
    var serializer = new DataContractJsonSerializer(typeof(T));
    serializer.WriteObject(stream, obj);
  }

  private static void WriteSequence<T>(IEnumerable<T> sequence, Stream stream)
  {
    var serializer = new DataContractJsonSerializer(typeof(T));
    stream.Write(lBracketBytes, 0, lBracketBytes.Length);
    int i = 0;
    foreach (var item in sequence)
    {
      if (i != 0)
        stream.Write(commaBytes, 0, commaBytes.Length);
      serializer.WriteObject(stream, item);
      i++;
    }
    stream.Write(rBracketBytes, 0, rBracketBytes.Length);
  }
}

That brings us to the end of this post.

Once I figured out all of this the error happened to be really simple (the class I was trying to serialize was not marked with the DataContract attribute! If I were doing the same thing in ASP.NET or Quartz I would have seen the exception the first time and fixed it in the next second.

The error I was seeing in Fiddler and Chrome was:

Replying to an operation threw a exception

Where is scvTraceViewer?

On my machine I found it in the following folders:

• C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\
• C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\NETFX 4.0 Tools
• C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin\x64
• C:\Program Files\Microsoft SDKs\Windows\v6.0A\Bin

The first one seemed to be the latest version so I used that.

Modifying Web.config for tracing

  <system.diagnostics>
    <trace autoflush="true" />
    <sources>
        <source name="System.ServiceModel" switchValue="Information, ActivityTracing, Error, Critical" propagateActivity="true">
          <listeners>
            <add name="sdt" type="System.Diagnostics.XmlWriterTraceListener" initializeData= "wcfTrace.svclog" />
          </listeners>
        </source>
    </sources>
  </system.diagnostics>

If you don't have a <system.diagnostics> section in your web.config file already, this section is a top level section, that is, it is a sibling of <system.web> or <system.webServer> or <appSettings> or a child of the root node <configuration>

Once you've set up your diagnostics section as shown above, exercise your service (and the method that's causing the problem) and you'll see the file wcfTrace.svclog created in the folder of your website application/website.

It so happens that files with the extension *.svclog are associated with svcTraceViewer! Of course I didn't know that when I first configured tracing and so I spent all that time trying to find scvTraceViewer. When I went to open the trace log file using svcTraceViewer and noticed that the default file extension it expects is .svclog I changed it.

Once you open the viewer you'll see log entries with errors highlighted in red (as shown in the image below), click on one of those entries and you'll see the cause for your error on the right hand side pane.

I hope this post helps others who have a similar problem. It's just not fun when you're trying to debug a problem and you can't figure out how to configure tracing and finding the tool that's going to get you out of the predicament.

I should add that Quartz for ASP.NET does not sacrifice maintainability for performance however, there are many aspects of the design of the framework that lend themselves towards mind-bending performance. And of course code constructs.

Anyway, this post is about the fastest website on the planet and you should know, you're on it! Yes, Matlus is currently the fastest website ranked by http://www.showslow.com/. Matlus has the following score:

YSlow – Grade A

PageSpeed – 99/100

If it weren't for the Google analytics JavaScript file not being cached, PageSpeed would have given it a 100/100!

In Google Webmaster tools under "Labs" there is a Site Performance link and this is what I see there

Under the "Diagnostics" section, the "Crawl stats" shows me this. It so happens that I moved my blog from using WordPress as the blog engine to using Orion. So if you look at the January portion of the graph that takes a big dip towards the end, that's when the switch happened.

I was initially using WordPress for my blog (this blog) and eventually just got tired of having to deal with some of the nuances of WordPress and more specifically the theme I was using. However I had gotten used to using Windows Live Writer to write my posts and so the first thing I did, even before starting the Orion project was to implement and have a functional MetaWeblog API implementation that I could use in Orion.

I’ve posted source code for the Orion Project (as is). This is a VS.NET 2010 solution that also requires (references) the Quartz engine project (and not just the assemblies). If you are not too comfortable with VS.NET solutions and/or are not able to recover from problems such as missing projects due to missing paths etc. this download may not be for you. The only part I expect people might have trouble with is that this solution includes the Quartz project rather than referencing the assemblies and so the solution will not find the Quartz project when you first open it.

Orion VS.NET 2010 Solution

Download the Quartz for ASP.NET project and source code for the link provided on that project’s page.

Windows Live Writer

If you've got a blog, you should look at using Windows Live Writer and your blog writer/editor, there is nothing like it out there. It's very much like Microsoft Word and it is complete tailor made for blog writing, and for the kind of blog writing I do, which includes a lot of images and code snippets, it's perfect since there is a plug-in available for converting code from your VS.NET editor into html including all of the color highlighting etc.

Windows Live Writer supports a number of blog writing/editing APIs and so I decided to implement one of those APIs for ThoughtStream so I didn't have to write and screen for the same (at least not right off the bat). And that's where the MataWeblog API comes in.

MetaWeblog API

The MetaWeblog API is a well known API for writing and editing blog posts. There are many blog writers that support this API. It's not the best (read that as most complete) API for blog writing/editing but I've chose it after a lot of thought and debate. None of the APIs have good documentation so I went with the one API that almost all other APIs are based on.

Personally I'm surprised that in this day and age there is no "real" blogging API. I've looked at the WordPress blogging API and even though it handles a few more things as compared to the MetaWeblog API (and Live Writer supports the WordPress API as well) it's really not a well thought out API. In fact none of the API are well thought out.

Since documentation is sparse and I've seen a lot of people struggling with implementing this API, I've decided to make the library I've developed available to the public. I've use this library for my blog along with Windows Live Writer so I know it is tests and works as expected.

MetaWeblog C# Library

The first thing you want to look at are all the methods that are available in the API and their signatures. Take a look at code listing 1 below. The IMetaWeblogProvider interface has a bunch of methods as you can see.

This is the interface you will need to implement in blog your blogging engine in order to make it compliant with the MetaWeblog API, so you can use a blog writer such as Window Live Writer to write your blog posts.

You'll notice that the interface is completely decoupled from any http or xml nuance that is part of the MetaWeblog API. This was by design so as to make testing your implementation of this interface a lot simpler.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;

namespace Matlus.MetaWeblog
{
  public interface IMetaWeblogProvider
  {
    BlogInfo[] GetUserBlogs(string appkey, string userName, string password);
    Post[] GetRecentPosts(string userName, string password, string blogid, int noOfPosts);
    CategoryInfo[] GetCategories(string userName, string password);
    int NewPost(string userName, string password, string blogid, Post post);
    void EditPost(string userName, string password, Post post);
    Post GetPost(string userName, string password, int postid);
    void DeletePost(string userName, string password, int postid);
    MediaObjectInfo NewMediaObject(string userName, string password, MediaObject mediaObject);
  }
}

Once you have the implementation of this interface functioning as you expect it to, getting it to work with Windows Live Writer or any other blog writer that supports the MetaWeblog API is quite simple. I do strongly suggest implementing this interface and testing it thoroughly before using the MetaWeblogManager class.

Now lets just take a brief look at the various other classes that are part of the library. Below is the class diagram of the complete MetaWeblog API library that's available for download. All of the heavy lifting is done by the MetaWeblogManager class. This class talks the MetaWeblog API language (which is essentially xml over http) and translates all of the communication into simple C# method calls. So you don't really have to mess with anything but your own implementation of the IMetaWeblogProvider interface. All of the other classes are supporting classes. The classes you see going down the left hand side are classes that will shuttle data between the MetaWeblogManager and your implementation of the IMetaWeblogProvider. They are simple DTOs and have no methods, just properties.

MetaWeblogManager

Now, let's take a look at the MetaWeblogManager class in code listing 2 below. Once again you'll notice that this class really only works with a System.IO.Stream. This design decouples this class from any "web" stuff per se. Of course the Stream must contain the correct content (the xml content as per the API specification - you don't really have to worry about the xml aspect since the library handles that for you) but you're free to decide how exactly you'll provide it with this stream. The simplest way is to use this class from within an IHttpHandler, but you're not confined to this. It could be a WCF service or a regular ASP.NET web page.

I'm only listing the ProcessMetaWeblogApiCall method, which is the entry point of this class. This method calls into other methods within the class that do the translation from xml to regular C# (and back) and then call the appropriate methods in the IMetaWeblogProvider interface.

public void ProcessMetaWeblogApiCall(Stream inputStream, Stream outputStream)
{
  XmlDocument = new XmlDocument();
  XmlDocument.Load(inputStream);
  OutputStream = outputStream;

  var methodName = GetMethodName();

  try
  {        
    switch (methodName)
    {
      case "blogger.getUsersBlogs":
        GetUsersBlogs();
        break;
      case "metaWeblog.getCategories":
      case "wp.getCategories":
        GetCategories();
        break;
      case "metaWeblog.getRecentPosts":
        GetRecentPosts();
        break;
      case "metaWeblog.newPost":
        NewPost();
        break;
      case "metaWeblog.editPost":
        EditPost();
        break;
      case "metaWeblog.getPost":
        GetPost();
        break;
      case "blogger.deletePost":
        DeletePost();
        break;
      case "metaWeblog.newMediaObject":
        NewMediaObject();
        break;
      case "blogger.getUserInfo":
        GetUserInfo();
        break;
    }

  }
  catch (Exception e)
  {
    WriteXmlRpcFault(e, outputStream);
  }      
}

The Complete Setup

Of course in order to get Window Live Writer or any other blog writer to work against your blogging engine, you'll need to go all the way out to Http and that translates to an IHttpHandler or something similar. I've use a Builder (part of the Quartz for ASP.NET framework) as my point of entry into the Orion blog engine, but I'll show an implementation of an IHttpHandler. The handler is really very simple as you can see in code listing 3 below.

The MetaWeblogManager class expects an instance of a class that implements the IMetaWeblogProvider interface. In my case, my BusinessModule class is the class that implements this interface.

You'll need to register your handler like you would any other handler in ASP.NET.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.IO;
using System.Xml.Linq;
using System.Xml;
using Matlus.MetaWeblog;

namespace Orion.WebLib
{
  public class XmlRpcHandler : HandlerBase
  {
    public override bool IsReusable
    {
      get { return true; }
    }

    public override void ProcessRequest(HttpContext context)
    {
      var metaWeblogManager = new MetaWeblogManager(BusinessModule);
      metaWeblogManager.ProcessMetaWeblogApiCall(HttpContext.Current.Request.InputStream, HttpContext.Current.Response.OutputStream);
    }
  }
}

This brings us to the end of this post. You can download the library from the link below.

Matlus.MetaWeblog

The download contains the complete source to the library. I'm using .NET 4.0 and I don't believe I'm using any .NET 4.0 specific features so you should be able to compile the library for .NET 3.5 if need be.

The ability to use a DbDataReader as an IEnumerable<T>, giving is a performance advantage over other options because there is only ever one instance of the wrapper class created no matter how many records you've got.

In this post I'll be presenting a tool that generates the Data Access Layer code using ADO.NET Core as the method of Data Access. ADO.NET core is the fastest way to access any database, bar none. According to Microsoft ADO.NET Core is about 3 times faster than any other Data Access method including Entity Framework and Linq to SQL.

In addition, this tool also produces the DbDataReader wrapper classes and code you'll require.

Features of the Generated Code

For every "Get_XXX" stored procedure in your database the tool generates the appropriate data access code, including the DbDataReader wrapper class that matches the schema of the result set.

If your stored procedure returns multiple result sets then the code generator will generate the appropriate DbDataReader wrapper classes for each of the results sets.

For the Insert, Update, Delete stored procedures, the code generates the code for the command parameter assignment part of the code. You'll need to write the code that makes that actual call to the stored procedure. Mind you, the bulk of the code is generated for you (creating instances of the command parameters that match your stored procedure's parameters and types and assigning them to a command). You could easily modify the code generation to create that code for you as well.

The generated code also includes a method that creates a DataTable or DataSet (as the case may be) for each of the "Get_XXX" stored procedures.

If you prefer to use POCO classes instead of a DataTable or DbDataReader, you can easily add a code generator that does that for you. The tool is extensible and it's really a matter of subscribing to an event that provides all the meta data information about your stored procedure making it really easy to generate the code required for a POCO class definition.

The code that available for download also includes a class that generates DataTable wrappers. This allows you to use a DataTable in a strongly typed fashion. It is not "active" (that is by default this class is not in use) but it's a simple matter of hooking it in.

I've used the tool to generate all of the Data Access layer code including the DbDataReader wrapper classes and a few other classes (explained later) for over 980 stored procedures I have in one of the projects I've worked on. It takes about 30 seconds to generate close to 100,000 lines of code.

Code listing 1 below, show the code it generated for one of my stored procedures called usp_GET_MEMBER_BLOGS. The namespace and class name of the partial class for the Data Access layer it generates are configurable. Your main Data Access layer class will be a partial class that has the same name and is in the same namespace.

The two internal methods:

GetMemberBlogs (this returns a DbDataReader)

GetMemberBlogsDataSet (this returns a DataSet in this particular case since my stored procedure returns multiple result sets)

are the ones you'll call from your Business Layer. Notice that the method names are derived from the name of the stored procedure and are in Pascal case as is the convention in C# for method names.

//################################################################################
//                 This code file is an Auto Generated file
//      Do NOT Modify this file. All changes to this file WILL BE LOST
//################################################################################

using System;
using System.Data;
using System.Data.Common;

namespace MyApp.DataAccessLayer
{
    internal partial class DataAccessModule
    {
        #region usp_GET_MEMBER_BLOGS

        internal DbDataReader GetMemberBlogs(long memberId)
        {
            using (var command = CreateCommandForGetBlogItem(memberId))
            {
                DbConnection.Open();
                return command.ExecuteReader(CommandBehavior.CloseConnection);
            }
        }

        internal DataSet GetMemberBlogsDataSet(long memberId)
        {
            var adapter = DbProviderFactory.CreateDataAdapter();
            adapter.SelectCommand = CreateCommandForGetBlogItem(memberId);
            var ds = new DataSet();
            adapter.Fill(ds);
            return ds;
        }

        private DbCommand CreateCommandForGetBlogItem(long memberId)
        {
            var command = DbConnection.CreateCommand();
            command.CommandType = CommandType.StoredProcedure;
            command.CommandText = "usp_GET_BLOG_ITEM";
            var parameter = DbProviderFactory.CreateParameter();
            parameter.DbType = DbType.Int32;
            parameter.Direction = ParameterDirection.ReturnValue;
            parameter.ParameterName = "@RETURN";
            command.Parameters.Add(parameter);
            parameter = DbProviderFactory.CreateParameter();
            parameter.DbType = DbType.Int64;
            parameter.Direction = ParameterDirection.Input;
            parameter.ParameterName = "@MEMBER_ID";
            parameter.Value = memberId;
            command.Parameters.Add(parameter);
            return command;
        }

        #endregion usp_GET_MEMBER_BLOGS

    }
}

Code Listing 1 – Showing the basic code generated for the Data Access Layer

Code listing 2 below shows the DbDataReader wrapper code that is generated. The code listing also includes a base class that was generated. The wrapper class that has been generated is called MemberBlog0 in this case. The stored procedure I've used here returns 8 result sets and so the real code that was generated included 8 class definitions numbered from 0-7. In cases where the stored procedure returns only one result set, the name of the generated class would be MemberBlogs.

//################################################################################
//                 This code file is an Auto Generated file
//      Do NOT Modify this file. All changes to this file WILL BE LOST
//################################################################################

using System;
using System.Data;
using System.Data.Common;

namespace MyApp.BusinessLayer
{
    /// <summary>
    /// The WrapperMode enum determines the behavior of
    /// the Wrapper class. That is the class is either
    /// a "wrapper" around the DbDataReader or
    /// it initializes its properties using the DbDataReader
    /// as a "data source"
    /// </summary>
    public enum WrapperMode { Wrapper, Dto }

    /// <summary>
    ///This is the Base Class for all DbDataReader Wrappers
    /// </summary>
    public class BaseDbDataReaderWrapper
    {
        protected DbDataReader DbDataReader { get; private set; }
        protected WrapperMode Mode { get; private set; }

        public BaseDbDataReaderWrapper(DbDataReader dbDataReader, WrapperMode mode)
        {
            DbDataReader = dbDataReader;
            Mode = mode;
        }
    }

    #region [usp_GET_MEMBER_BLOGS]

    /// <summary>
    ///This class is a wrapper around a DbDataReader,
    ///Associated with the stored procedure - usp_GET_MEMBER_BLOGS
    ///This class provides a strongly typed interface to access data from the DbDataReader.
    ///containing the result of the given stored procedure.
    /// </summary>
    public sealed class MemberBlog0 : BaseDbDataReaderWrapper
    {
        #region Properties

        private Int64 categoryId;
        public Int64 CategoryId { get { switch(Mode){ case WrapperMode.Wrapper: return (Int64)DbDataReader[0]; case WrapperMode.Dto: return categoryId; default: return default(Int64); } } set { categoryId = value; } }
        private String categoryName;
        public String CategoryName { get { switch(Mode){ case WrapperMode.Wrapper: return (String)DbDataReader[1]; case WrapperMode.Dto: return categoryName; default: return default(String); } } set { categoryName = value; } }
        private Int64 itemId;
        public Int64 ItemId { get { switch(Mode){ case WrapperMode.Wrapper: return (DbDataReader[2] != DBNull.Value) ? (Int64)DbDataReader[2] : default(Int64); case WrapperMode.Dto: return itemId; default: return default(Int64); } } set { itemId = value; } }
        private Int64 memberId;
        public Int64 MemberId { get { switch(Mode){ case WrapperMode.Wrapper: return (DbDataReader[3] != DBNull.Value) ? (Int64)DbDataReader[3] : default(Int64); case WrapperMode.Dto: return memberId; default: return default(Int64); } } set { memberId = value; } }

        #endregion Properties

        public BMemberlog0(DbDataReader dbDataReader, WrapperMode mode)
            :base(dbDataReader, mode)
        {
            switch(Mode)
            {
                case WrapperMode.Wrapper:
                    break;
                case WrapperMode.Dto:
                    categoryId = (Int64)DbDataReader[0];
                    categoryName = (String)DbDataReader[1];
                    itemId = DbDataReader[2] != DBNull.Value ? (Int64)DbDataReader[2] : default(Int64);
                    memberId = DbDataReader[3] != DBNull.Value ? (Int64)DbDataReader[3] : default(Int64);
                    break;
            }
        }
    }

    #endregion [usp_GET_MEMBER_BLOGS]
}

Code Listing 2 – The DbDataReader Wrapper generated code

Using the Generated DbDataReader wrapper class

Code listing 3 shows 2 methods from the Business Layer that call into the Data Access layer. One of the methods returns IEnumerable<MemberBlogs> and the other return a IList<MemberBlogs>.

These methods are not code generated, however it would be simple enough to generate them as well if need be. The key thing to remember is that the classes that are part of the tool provide all the metadata you'll need to generate any code that works off of this information.

There are times when you'd use the first method and there are times when you'd use the second method. If you're simply iterating over the result (as fast as possible) to generate html in your ASP.NET MVC or ASP.NET WebForms application or streaming the result as a JSON stream or as a collection in a WCF application or WebService, then you should use the method that returns an IEnumerable<T>.

On the other hand, if you intend to hang on to the result (in your UI layer – ASP.NET WebForms/MVC application for example) or you intend to manipulate the result in some way (using Linq for example) then you should use the method that returns an IList<T>.

The reason is that when you're using a DbDataReader you want to consume the result as fast as possible, since the connection to your database is "open" till you do. In the same token, for the cases cited above you *want* to use a DbDataReader because there is no need to create a collection of some POCO objects only to stream them (serialize them) across the internet as a WebService message or JSON message for example). Going from a DbDataReader straight out as a JSON/WebService message is much faster and taxes your server far less (by way of less CPU utilization as well as less memory pressure) allowing you to process many more Request/Response cycles per second.

internal IEnumerable<BlogItem> GetMemberBlogs(long memberId, long itemId)
{
  using (var dr = DataAccessModule.GetMemberBlogs(memberId, itemId))
  {
    var blogItem = new BlogItem(dr, WrapperMode.Wrapper);
    while (dr.Read())
      yield return blogItem;
  }
}

internal IList<BlogItem> GetMemberBlogsList(long memberId, long itemId)
{
  using (var dr = DataAccessModule.GetMemberBlogs(memberId, itemId))
  {
    var list = new List<BlogItem>();
    while (dr.Read())
      list.Add(new BlogItem(dr, WrapperMode.Dto));
    return list;
  }
}

Code Listing 3 – Showing how you'd use the Wrapper classes from your Business Layer

The Case for using DbDataReaders

There are few reasons for using a DbDataReader in general. I know ma lot of people in the .NET community shun the use of a DbDataReader or DataTable/DataSet. They think that using these classes ties them to a database. Well, that's not true and all. You can create a DataTable or DataSet simply by new-ing up an instance of a DataTable or DataSet and pumping data into it manually or reading in data from an Xml file. That is they don't have to be connected to a database at all.

Similarly, from an instance of a DataTable you can get an instance of a DbDataReader (which is an in memory DbDataReader effectively). No matter how you dice it, these classes do not tie you to a database and nor do they imply you're using a database of any kind.

There are some issues with using DbDataReader or DataTable/DataSet from within a Business layer, and I talked it earlier (DataReader Wrappers – Type Safe) and the wrapper classes are the solution to those issues.

In cases where you're extracting data from your database and simply iterating over this data and producing Html (via an ASP.NET MVC or WebForms app, or JSON or any other "serialized" format, using a DbDataReader makes total sense. There is absolutely no need to go from a DbDataReader to a collection of POCO objects to Html or other serialized formats.

Creating a collection of POCO objects means you've iterated over your result once. Then producing html or JSON or any other serialized format means you're iterating over your result again. Not only that, creating these objects only to throw them away is adding a lot of memory pressure on the GC and it takes a lot of time (CPU cycles) to create instances and assign their properties with values from a DbDataReader. Keep in mind that no matter what data access technology you use, including any ORMs as well as Linq to SQL, they all use DbDataReaders under the hood.

In some performance tests I've conducted, wherein I go from a DbDataReader, eventually out to a Web Page using Html, XML and JSON via one of the following:

1. POCO objects

2. DataTable/DataSet

3. Directly to the output format

I found that going from DbDataReader to POCO objects and then serializing them was by far the slowest. Going from DbDataReader to DataTable to the output format was way faster than POCO but quite a bit slower than going directly from DbDataReader to the desired output format. The difference becomes even more noticeable as the number of records in your result set increases.

And as I mentioned in the beginning of this post, according to Microsoft, ADO.NET core is about 3 times faster than any other Data Access technology. So if speed and scalability are a concern then DbDataReaders are the way to go.

Nonetheless, the code generation tool presented here doesn't stop you from creating POCO classes and using those instead. You're still using ADO.NET core (which is 3 times faster) and you get to use POCO objects in your business layer code.

OAuth 1.0 was based largely on two existing proprietary protocols:

1. Google’s AuthSub
2. Flickr’s API Auth

At the time (due the combined experiences) it was considered the best implementation of a security protocol for the purposes of authenticating third party applications and granting access to limited secured information of the user.

OAuth is a complex protocol with a lot of to and fro between the relying party (the consumer or third party application) and the provider. As a result there are only a few implementation of the protocol that actually work across multiple providers.

In this post, I'll be presenting you with a C# library that not only works across multiple providers but is also very simple to use and not over engineered like one or two of the other implementations and not so simple that it only works against one provider.

The library presented has been tested against the following providers

1. Google – OAuth Reference
2. Twitter – OAuth Reference
3. Yahoo! – OAuth Reference
4. Vimeo – OAuth Reference

The links above point to documentation on each of those sites so I won't repeat that information here as I believe each one presents the process flow pretty well and there is nothing I can really add to that information except to say that you do need to understand it pretty clearly in order to use any library that simplifies the process for you. Because there is a process involved and you have to code against that process.

I found that working against Google's OAuth provider really cleaned up the library I present here (and where most other libraries fail) because of the tokens and keys Google sends back (which invariably need to be encoded before that can be used). I also believe that Google's design is the worst of the lot (unnecessarily complicated).

For those of you just wanting dive right in and get the code, I've started an open source project on Google code, that houses the source code and a very simple ASP.NET application that will help you hit the ground running in no time flat. You can find the source code and sample project here

I've intentionally kept the code in the sample project really simple so it is easy to grasp how to use the library as well as the sequences of steps involved in the OAuth protocol. Once you understand the process you'll most likely want to re-engineer your application's code for your production systems. The library can be used as is.

The Sample Project

The screen shot below shows the default page of the application. If you've signed up with Google for example, then fill in the consumer key and consumer secret in the appropriate places in the code (there are two places you'll need to do this in – if you don't you'll get an exception, letting you know that that's what you need to do).

Clicking on the Google icon will take you to Google. And Google will present the log in page where you can enter your Google account credentials. The page on Google will also present you with other information as shown below.

Once you grant access you will have completed the process and you'll see the screen shot shown below

OAuth Process

Essentially there are two kinds of application that may need to use OAuth for authentication purposes.

• Web Applications
• Installed Application

The library presented here is for Web Application, however the core code should be usable in an installed application as well since the core code handles all of the encryption/decryption (or hashing really) and other nuances of the protocol itself.

In Web Application (for the purposes of OAuth) we employ what is frequently known as the 3-legged scenario. In word words there are essentially 3 steps to the entire process. It's all explained in the links above.

Getting Started

The very first step is getting a consumer key and consumer secret (also known as the shared secret). You'll need to sign up with a provider and register your domain and application in order to get these. Each provider will provide you with a consumer key and consumer secret that you'll have to use when communicating with their service.

Once you have a consumer key and consumer secret, you'll also you're ready to begin. The sample project has code and the urls you'll need for the 4 providers I listed earlier. If you're working with another provider you'll also need the 3 endpoints (the urls) for each of the steps.

/// <summary>
/// Step 1 of the oAuth protocol process
/// Make a request with the provider for a <see cref="RequestToken"/>
/// </summary>
/// <param name="requestTokenEndpoint">The url (as per the provider) to use for making a requet for a token</param>
/// <param name="realm">Typically the url of Your "application" or website</param>
/// <param name="consumerKey">The Consumer Key given to you by the provider</param>
/// <param name="consumerSecret">The Consumer Secret given to you by the provider</param>
/// <param name="callback">The url you'd like the provider to call you back on</param>
/// <param name="signatureMethod">defaults to HMAC-SHA1 - the only signature method currently supported</param>
/// <returns>An instance of a <see cref="RequestToken" /> class</returns>
public RequestToken GetOAuthRequestToken(string requestTokenEndpoint, string realm, string consumerKey, string consumerSecret, string callback, SignatureMethod signatureMethod = SignatureMethod.HMACSHA1)
{
  var oAuthUtils = new OAuthUtils();
  var authorizationHeader = oAuthUtils.GetRequestTokenAuthorizationHeader(requestTokenEndpoint, realm, consumerKey, consumerSecret, callback, signatureMethod);
  return MakeRequest<RequestToken>(requestTokenEndpoint, authorizationHeader);
}

/// <summary>
/// Step 3 of the oAuth protocol process
/// Make a request on the provider to Exchange a <see cref="RequesToken"/> for an <see cref="AccessToken"/>
/// (Step 2 is a simple redirect and so there is no method for it in this class)
/// </summary>
/// <param name="accessTokenEndpoint">The url (as per the provider) to use for making a requet to Exchange a request token for an access token</param>
/// <param name="realm">Typically the url of Your "application" or website</param>
/// <param name="consumerKey">The Consumer Key given to you by the provider</param>
/// <param name="consumerSecret">The Consumer Secret given to you by the provider</param>
/// <param name="token">The token you got at the end of Step 1 or Step 2</param>
/// <param name="verifier">The verifier you got at the end of step 2</param>
/// <param name="tokenSecret">The tokenSecret you got at the end of step 1</param>
/// <param name="signatureMethod">defaults to HMAC-SHA1 - the only signature method currently supported</param>
/// <returns>An instance of a <see cref="AccessToken"/> class</returns>
public AccessToken GetOAuthAccessToken(string accessTokenEndpoint, string realm, string consumerKey, string consumerSecret, string token, string verifier, string tokenSecret, SignatureMethod signatureMethod = SignatureMethod.HMACSHA1)
{
  var oAuthUtils = new OAuthUtils();
  var authorizationHeader = oAuthUtils.GetAccessTokenAuthorizationHeader(accessTokenEndpoint, realm, consumerKey, consumerSecret, token, verifier, tokenSecret, signatureMethod);
  return MakeRequest<AccessToken>(accessTokenEndpoint, authorizationHeader);
}

Code Listing 1: The Two Methods you'll use

The two methods you see in the code listing above are the only two methods you'll use during the process of authenticating the user. The seconds step in the process is a simple re-direct and so there is no method for it in the library per-se. Besides the sample project if you need an explanation of the various parameters, please read the comments provided.

These two methods will get you as far as authenticating the user using an external OAuth service provider. In other words, she is who she says she is. But you don't necessarily have any other information about her. In other words, executing and completing the Authentication process does not inherently provide you with any information about the authenticated user.

To get additional information you'll need to use the AccessToken (given to you by the provider in the 3rd and final step of the OAuth process), and make additional calls against the provider. The core library has the methods and functionality you'll need (at the core level) to make these additional calls.

Each provider has a different endpoint you'll need to use and not all providers have the same API and not all providers will provide the information you'll need about your user either.

Retrieving Additional Information

In the sample application there is a line that has been commented out at the end of the method that will retrieve the user's email address from Google. Of course you can use that only if you're using Google as the provider.

var responseText = oAuthConsumer.GetUserInfo("https://www.googleapis.com/userinfo/email", realm, consumerKey, consumerSecret, accessToken.Token, accessToken.TokenSecret);

Retrieving an Authenticated user's Email Address (Google)

You should be able to write any additional methods you'll need using the code in OAuthConsumer.cs (partial listing in the code listing above) file that pertains to your needs and the specific provider(s) you're using.

That brings us to the end of this post. I truly hope the library and sample project will help you get started and in no time you'll have an OAuth implementation in your production applications.

1. Decouples the client code (your code that uses these classes) from knowing how to create instances of concrete types
2. Decouples the client code from knowledge of the concrete classes themselves. That is the client code knows only of the abstract base class or an interface but not the derived or concrete class

An important facet of system design is how objects are created. Most frameworks (including the .NET framework) make it extremely simple to create instances of classes and rightly so. However, what is not apparent is that you introduce coupling between the class that has been created and the class that uses the object. In small systems this coupling is not a problem but in large systems coupling between classes makes the system inflexible or rigid making changes over time more difficult to accommodate. The Factory design pattern helps alleviate this coupling.

There are many ways in which to implement a factory. Ranging from simple to exotic. But essentially their implementations differ in the following area:

1. How the decision is made as to which class to instantiate
2. How an instance of a class is actually created

Understanding the use of a Factory

enum EmployeeType { Salaried, Commission, Hourly }

class Employee
{
  public int Id { get; private set; }
  public string Name { get; private set; }
  public EmployeeType EmployeeType { get; private set; }

  public Employee(int id, string name, EmployeeType employeeType)
  {
    Id = id;
    Name = name;
    EmployeeType = employeeType;
  }
}

var employees = new List<Employee>
  {
    new Employee(1, "Employee One", EmployeeType.Commission),
    new Employee(2, "Employee Two", EmployeeType.Hourly),
    new Employee(3, "Employee Three", EmployeeType.Salaried),
    new Employee(4, "Employee Four", EmployeeType.Salaried)
  };

  foreach (var employee in employees)
  {
    switch (employee.EmployeeType)
    {
      case EmployeeType.Salaried:
        //Process payment for salaried employee
        break;
      case EmployeeType.Commission:
        //Process payment for commision employee
        break;
      case EmployeeType.Hourly:
        //Process payment for hourly employee
        break;
    }
  }

abstract class PaymentProcessorBase
{
  public abstract void ProcessPayment(Employee employee);
}

class SalariedPaymentProcessor : PaymentProcessorBase
{
  public override void ProcessPayment(Employee employee)
  {
    Console.WriteLine(employee.Name + "'s Payment was processed using Salaried Payment Processor");
  }
}

class CommissionPaymentProcessor : PaymentProcessorBase
{
  public override void ProcessPayment(Employee employee)
  {
    Console.WriteLine(employee.Name + "'s Payment was processed using Commission Payment Processor");
  }
}

class HourlyPaymentProcessor : PaymentProcessorBase
{
  public override void ProcessPayment(Employee employee)
  {
    Console.WriteLine(employee.Name + "'s Payment was processed using Hourly Payment Processor");
  }
}

PaymentProcessorBase processor = null;

foreach (var employee in employees)
{
  switch (employee.EmployeeType)
  {
    case EmployeeType.Salaried:
      processor = new SalariedPaymentProcessor();
      break;
    case EmployeeType.Commission:
      processor = new CommissionPaymentProcessor();
      break;
    case EmployeeType.Hourly:
      processor = new HourlyPaymentProcessor();
      break;
  }
  processor.ProcessPayment(employee);
}

1. The client code is tightly coupled to the various concrete classes and has intimate knowledge of how to create them
2. If you add a new EmployeeType you would have to alter this code to account for the new EmployeeType.

Factory - Simple implementation

class SimplePaymentProcessorFactory
{
  public PaymentProcessorBase CreateFactory(string identifier)
  {
    if (identifier == "SalariedPaymentProcessor")
      return new SalariedPaymentProcessor();
    else if (identifier == "CommissionPaymentProcessor")
      return new CommissionPaymentProcessor();
    else if (identifier == "HourlyPaymentProcessor")
      return new HourlyPaymentProcessor();
    throw new ArgumentException("The identifier: " + identifier + ", is not a valid identifier", identifier);
  }
}

1. Adding new classes means the code in the factory now need to be modified. So you shifted "the problem" from the client code to the factory.
2. If you're dealing with a lot of classes then this code could get pretty unwieldy
3. This design won't work in situations where you don't know the classes up front. Say in a plug-in scenario or similar, where all descendants current and future won't be known at the time of compiling this code.

Activator.CreateInstance()

High Performance Factory

Factory class - Using the default constructor of your classes

using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Reflection;
using System.Reflection.Emit;

namespace Factory
{
  static class PaymentProcessorFactory
  {
    private static Type classType = typeof(PaymentProcessorBase);
    private static Type[] constructorArgs = new Type[] { };

    private static readonly ConcurrentDictionary<string, Type> classRegistry = new ConcurrentDictionary<string, Type>();
    private static readonly ConcurrentDictionary<string, ConstructorDelegate> classConstructors = new ConcurrentDictionary<string, ConstructorDelegate>();

    delegate PaymentProcessorBase ConstructorDelegate();

    static PaymentProcessorFactory()
    {
      var sw = System.Diagnostics.Stopwatch.StartNew();
      
      var paymentProcessors = from b in Assembly.GetEntryAssembly().GetTypes()
                             where b.IsSubclassOf(classType)
                             select b;

      foreach (var type in paymentProcessors)
        classRegistry.TryAdd(type.Name, type);
    }


    public static PaymentProcessorBase Create(string identifier)
    {
      if (String.IsNullOrEmpty(identifier))
        throw new ArgumentException("identifier can not be null or empty", identifier);
      if (!classRegistry.ContainsKey(identifier))
        throw new ArgumentException("No PaymentProcessor has been registered with the identifier: " + identifier);

      return Create(classRegistry[identifier]);
    }

    private static PaymentProcessorBase Create(Type type)
    {
      ConstructorDelegate del;

      if (classConstructors.TryGetValue(type.Name, out del))
        return del();

      DynamicMethod dynamicMethod = new DynamicMethod("CreateInstance", type, constructorArgs, classType);
      ILGenerator ilGenerator = dynamicMethod.GetILGenerator();

      ilGenerator.Emit(OpCodes.Newobj, type.GetConstructor(constructorArgs));
      ilGenerator.Emit(OpCodes.Ret);

      del = (ConstructorDelegate)dynamicMethod.CreateDelegate(typeof(ConstructorDelegate));
      classConstructors.TryAdd(type.Name, del);
      return del();
    }
  }
}

foreach (var employee in employees)
{
  var paymentProcessor = PaymentProcessorFactory.Create(employee.EmployeeType.ToString() + "PaymentProcessor");
  paymentProcessor.ProcessPayment(employee);
}

Factory class - Handling two parameters in the constructor of your classes

using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Reflection;
using System.Reflection.Emit;

namespace Factory
{
  static class PaymentProcessorFactory
  {
    private static Type classType = typeof(PaymentProcessorBase);
    private static Type[] constructorArgs = new Type[] { typeof(Employee), typeof(int) };

    private static readonly ConcurrentDictionary<string, Type> classRegistry = new ConcurrentDictionary<string, Type>();
    private static readonly ConcurrentDictionary<string, ConstructorDelegate> classConstructors = new ConcurrentDictionary<string, ConstructorDelegate>();

    delegate PaymentProcessorBase ConstructorDelegate(Employee employee, int intParam);

    static PaymentProcessorFactory()
    {
      var sw = System.Diagnostics.Stopwatch.StartNew();
      
      var paymentProcessors = from b in Assembly.GetEntryAssembly().GetTypes()
                             where b.IsSubclassOf(classType)
                             select b;

      foreach (var type in paymentProcessors)
        classRegistry.TryAdd(type.Name, type);
    }


    public static PaymentProcessorBase Create(string identifier, Employee employee, int intParam)
    {
      if (String.IsNullOrEmpty(identifier))
        throw new ArgumentException("identifier can not be null or empty", identifier);
      if (!classRegistry.ContainsKey(identifier))
        throw new ArgumentException("No PaymentProcessor has been registered with the identifier: " + identifier);

      return Create(classRegistry[identifier], employee, intParam);
    }

    private static PaymentProcessorBase Create(Type type, Employee employee, int intParam)
    {
      ConstructorDelegate del;

      if (classConstructors.TryGetValue(type.Name, out del))
        return del(employee, intParam);

      DynamicMethod dynamicMethod = new DynamicMethod("CreateInstance", type, constructorArgs, classType);
      ILGenerator ilGenerator = dynamicMethod.GetILGenerator();

      ilGenerator.Emit(OpCodes.Ldarg_0);
      ilGenerator.Emit(OpCodes.Ldarg_1);
      ilGenerator.Emit(OpCodes.Newobj, type.GetConstructor(constructorArgs));
      ilGenerator.Emit(OpCodes.Ret);

      del = (ConstructorDelegate)dynamicMethod.CreateDelegate(typeof(ConstructorDelegate));
      classConstructors.TryAdd(type.Name, del);
      return del(employee, intParam);
    }
  }
}

ilGenerator.Emit(OpCodes.Ldarg_1);

1. Abstract the actual database engine or other data store, such that your applications can switch from using say Oracle to using MS SQL server
2. Abstract the logical data model such that your Business Layer is decoupled from this knowledge and is agnostic of it. Giving you the ability to modify the logical data model without impacting the business layer

The Business Layer

Design of a Data Access Layer

GetCustomers();
or 
GetCustomerOrders(42);
or
GetAllOrdersForThisMonth();

1. DbDataReaders that contain one or more result sets
2. DataTables or DataSets
3. Instances of some POCOs (Plain Old CLR Objects)

CreateCustomerOrder(42, order);

Figure1

Business Layers and Global

1. A Request-Response is handled by one and only one (IIS) thread
2. This thread has it's own instance of Global
3. By default there there may be up to 10 instances of Global created
4. These instances are cached

If we create an instance of our Business Layer within the Global class the instance is not shared with other threads and so it's all thread-safe. Further, instances of our Business Layer will be in tandem with instances of Global so there will be only 10 instances created and these will be cached, so we won't be creating and disposing instances for every request-response cycle in our application. This gives us a huge performance benefit as well.

public class Global : HttpApplication
{
  public BusinessModule BusinessModule { get; private set; }

  public override void Init()
  {
    base.Init();
    BusinessModule = new BusinessModule();      
  }
}

protected BusinessModule BusinessModule { get { return ((Global)HttpContext.Current.ApplicationInstance).BusinessModule; } }

1. No Type Checking
2. If field ordering or field name changes in your stored procedures or queries will likely break your code
3. Code that uses DataReaders tends to be not so clear to read

1. ItemId
2. MemberId
3. ItemTitle
4. ItemDesc
5. ItemPubdate
6. ItemCommentCnt
7. ItemAllowComment

using (DbDataReader dr = GetBlog(42))
{
  var blogs = new List<BlogItem>();
  while (dr.Read())
  {
    var blogItem = new BlogItem();

    blogItem.ItemId = (long)dr["ItemId"];
    blogItem.MemberId = (long)dr["MemberId"];
    blogItem.ItemTitle = (string)dr["Itemtitle"];
    blogItem.ItemDesc = dr["ItemDesc"] != DBNull.Value ? (string)dr["ItemDesc"] : null;
    blogItem.ItemPubdate = (DateTime)dr["ItemPubdate"];
    blogItem.ItemCommentCnt = (int)dr["ItemCommentCnt"];
    blogItem.ItemAllowComment = (bool)dr["ItemAllowComment"];

    blogs.Add(blogItem);

  }
  return blogs;
}

using (DbDataReader dr = GetBlog(42))
{
  var blogs = new List<BlogItem>();
  while (dr.Read())
  {
    var blogItem = new BlogItem();

    blogItem.ItemId = (long)dr[0];
    blogItem.MemberId = (long)dr[1];
    blogItem.ItemTitle = (string)dr[2];
    blogItem.ItemDesc = dr[3] != DBNull.Value ? (string)dr[3] : null;
    blogItem.ItemPubdate = (DateTime)dr[4];
    blogItem.ItemCommentCnt = (int)dr[5];
    blogItem.ItemAllowComment = (bool)dr[6];

    blogs.Add(blogItem);

  }
  return blogs;
}

1. No compile type type checking
2. Readability isn't too good
3. If a field is null-able (like ItemDesc above) then the code to access the field's value is even more cumbersome

/// <summary>
///This is the Base Class for all DbDataReader Wrappers
/// </summary>
public class BaseDbDataReaderWrapper
{
    public DbDataReader DbDataReader { get; set; }

    public BaseDbDataReaderWrapper()
    {
    }

    public BaseDbDataReaderWrapper(DbDataReader dbDataReader)
      :this()
    {
        DbDataReader = dbDataReader;
    }
}

/// <summary>
///This class is a wrapper around a DbDataReader,
///Associated with the stored procedure - usp_GET_BLOG_ITEM
///This class provides a strongly typed interface to access data from the DbDataReader.
///containing the result of the given stored procedure.
/// </summary>
public sealed class BlogItemDrw : BaseDbDataReaderWrapper
{
    public Int64 ItemId { get { return (Int64)DbDataReader[0]; } }
    public Int64 MemberId { get { return (Int64)DbDataReader[1]; } }
    public String ItemTitle { get { return (String)DbDataReader[2]; } }
    public String ItemDesc { get { if (DbDataReader[3] != DBNull.Value) return (String)DbDataReader[3]; else return default(String); } }
    public DateTime ItemPubdate { get { return (DateTime)DbDataReader[4]; } }
    public Int32 ItemCommentCnt { get { return (Int32)DbDataReader[5]; } }
    public Boolean ItemAllowComment { get { return (Boolean)DbDataReader[6]; } }
    public BlogItemDrw()
      :base()
    {
    }

    public BlogItemDrw(DbDataReader dbDataReader)
      :base(dbDataReader)
    {
    }
}

IEnumerable<T> - DbDataReader

public IEnumerable<BlogItem> GetBlogs()
{
  using (DbDataReader dr = GetBlogsDataReader())
  {
    var blogItem = new BlogItem(dr);
    while (dr.Read())
    {
      yield return blogItem;
    }
  }
}

return command.ExecuteReader(CommandBehavior.CloseConnection);

List<T> - DbDataReader

  public sealed class BlogItem
  {
    public Int64 ItemId { get; private set; }
    public Int64 MemberId { get; private set; }
    public String ItemTitle { get; private set; }
    public String ItemDesc { get; private set; }
    public DateTime ItemPubdate { get; private set; }
    public DateTime ItemPubtime { get; private set; }
    public Decimal ItemRatingAvg { get; private set; }
    public Int32 ItemViewCnt { get; private set; }
    public Int32 ItemCommentCnt { get; private set; }
    public Int32 ItemRatingCnt { get; private set; }
    public Int32 ItemFavoriteCnt { get; private set; }
    public Boolean ItemIsPrivate { get; private set; }
    public Boolean ItemAllowComment { get; private set; }
    public Boolean ItemIsPublished { get; private set; }

    public BlogItem(DbDataReader dbDataReader)
      : base()
    {
      ItemId = (Int64)dbDataReader[0];
      MemberId = (Int64)dbDataReader[1];
      ItemTitle = (String)dbDataReader[2];
      ItemDesc = dbDataReader[3] != DBNull.Value ? (String)dbDataReader[3] : default(String);
      ItemPubdate = (DateTime)dbDataReader[4];
      ItemRatingAvg = (Decimal)dbDataReader[5];
      ItemViewCnt = (Int32)dbDataReader[6];
      ItemCommentCnt = (Int32)dbDataReader[7];
      ItemRatingCnt = (Int32)dbDataReader[8];
      ItemFavoriteCnt = (Int32)dbDataReader[9];
      ItemIsPrivate = (Boolean)dbDataReader[10];
      ItemAllowComment = (Boolean)dbDataReader[11];
      ItemIsPublished = (Boolean)dbDataReader[12];
    }
  }

     internal  List <BlogItem > GetMemberBlogsList(long  memberId)

     {

       using  (var  dr = DataAccessModule.GetMemberBlogs(memberId))

       {

         var  blogItems = new  List <BlogItem >();

         while  (dr.Read())

           blogItems.Add(new  BlogItem (dr));

         return  blogItems;

       }

     }

Code Generation

Quartz for ASP.NET is the most intuitive way to build dynamic web pages, using a powerful combination of class Composition and Inheritance. Today’s dynamic pages are rarely simple, so you need something that gives you more control over how your pages are composed. With Quartz for ASP.NET you continue to get the loose coupling and separation of concerns that the MVC design pattern provides and in addition you also get

• Loose coupling between Views and Template
• Templates are Html. That is there is no html+code OR code+html
• Views can be treated as real classes - inheritance/polymorphism and you write code in a.cs file
• Strong typing and F12 navigation
• Plug-able Websites/applications

However, the primary driver for Quartz for ASP.NET was the way in which you build pages in ASP.NET MVC and the problems with that design. Figure 1 below, shows the relationship and flow of control in the ASP.NET MVC design, between a Controller and a View, Partial View and Master Page/Layout Page.

Figure 1: ASP.NET MVC - relationship between a Controller and "views"

You'll notice that the process of building a page is akin to a chain reaction. That wouldn't be so bad if it weren't for the fact that the controller is not really controlling anything besides starting off a specific chain reaction (everything to the left of the Controller on the right in Figure1).

How Quartz for ASP.NET Works

Figure 2 shows the relationship and flow of control between the Builder and the Views and Master Page Template.

Figure 2: Quartz for ASP.NET - relationship between a Builder and Views

1. The Builder picks a Master Template (an Html file). This template usually has spots that need to be filled in, as shown in Figure 2 above, the "Page"
2. It then picks the appropriate views for each of these spots (a base class (of the Builder) could enlist some views to do the "common" parts and the descendant Builder could choose to override these choices)
3. Since the Builder has access to instances of each of the views enlisted to render the page, it can talk directly to these views to hand them the data each of them will need to do their part.
4. Again, since the Builder has access to views as objects instances it can treat views polymorphic-ally if required or use inheritance or both
5. Once it has done the above, each of the views go off and do their part (generate html) and a page is produced.
6. Since views are regular .NET classes you have the full might of the .NET framework at your disposal. And since their only job is to produce html in conjunction with the Request Context (of the current request) and the data they’ve been handed by the Builder, it’s totally up to you, how you do that. The Quartz framework does provide some options out of the box, and they are:
   • Simply generate html using just code and data, writing directly to the output
   • Use html templates (with placeholder tags – parser included with the framework) to render html directly to the output. This method has the added advantage that you have no html in code and templates can be re-used across multiple views and/or a view can use multiple templates or decide to use a different template conditionally
   • Use any other mechanism to generate html, such as T4 Templates.

The code listing below is a sample of the code you'd write in a Builder's BuildPage() method.

protected override void BuildPage(HttpContextBase httpContext, object model)
{
    var featuresAllPageViewModel = (FeaturesAllPageViewModel)model;

    TitleView.Title = "Quartz for ASP.NET - Features";
    AdBoxTopView.BoxTitle = "Highlights";
    AdBoxTopView.Model = featuresAllPageViewModel.AdBoxModel;

    RegisterView("Body", new FeaturesAllView(httpContext, this, featuresAllPageViewModel.Features));
  }
}

Composing Pages with Quartz for ASP.NET

Traditionally, web application frameworks work in a inside-out fashion. That is, when you field a request for a page, you’re really starting with the inner most part of the page and working your way out (by calling RenderAction or RenderPartial in MVC parlance). In Quartz for ASP.NET you start with a template and compose your page using multiple views. It’s really a lot more intuitive to think and work this way, than inside-out.

Figure 3: Quartz for ASP.NET - Composing Pages in an intuitive manner

More Features of Quartz for ASP.NET

• Since all templates are html, your designer can build templates using the html design tool of her choice, and both you and the designer can live an uncluttered life (no more html+code or code+html)
• Views are completely unaware of the Builder that enlisted them or any other Builders for that matter. In Quartz for ASP.NET, Views are the "leaf nodes". There is only one loosely coupled linkage, that between the Builder and the views it enlists.
• All the code you write is strongly typed so you get all the benefits of strong typing and the IDE can help you navigate your classes and methods (F12).
• Blazingly FAST!

Quartz for ASP.NET - Quick-Dive Video

Quartz for ASP.NET - Data Binding

Downloads

1. The Complete source code (.Net 4.0 compatible only at the moment)
2. A Sample Project
3. A New Quartz Project Item - for VS 2010
4. A New Quartz Builder Item - for VS 2010
5. A New Quartz View Item - for VS 2010

Installing Project and Item Templates in VS.NET

If you've not installed project and item templates in Visual Studio before, please follow these steps:

Extract the files from the downloaded zip file to a temporary folder.

First we'll installed the Project Item Template.

1. Locate the file called – Quartz Web Application.zip
2. Navigate to the Visual Studio Project Templates (C#) folder. On my machine this folder is located at:

3. C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ProjectTemplates\CSharp\

4. Create a folder called Quartz under the CSharp folder as shown in Figure 4 below
5. Copy the Quartz Web Application.zip into this folder
6. Follow the steps listed in the Rebuilding the Visual Studio Cache section below in order to rebuild the VS.NET cache.
7. Start Visual Studio (rebuild the cache first!) and create a new project.
8. You'll notice a Quartz folder in Installed Templates ree view on the left (as seen in Figure 5 below) and when you select it, you'll see the Quartz Web Application project item template and shown in Figure 5 below

Figure 4: Creating the Quartz Project Template folder

Figure 5: Quartz Web Application Project Item

The steps involved in installing the Item Templates in Visual Studio are similar to the steps we followed earlier for installing the Project Item Template. This time you need to create a folder under a different hive.

1. You'll need to create a folder under the ItemTemplates\CSharp folder, which on my machine is:

2. C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ItemTemplates\CSharp

3. Like before, create a folder called Quartz under this folder as shown in Figure 6 below.
4. Copy the QuartzBuilder.zip and QuartzView.zip files into this new folder
5. Rebuild the Visual Studio cache again using the steps listed in the Rebuilding the Visual Studio Cache section below.
6. After you've done that, you'll see item templates for a Quartz Builder and Quartz View when you add an item to your Quartz Web Application project, as shown in Figure 7 below

Figure 6: Showing the Quartz folder under the ItemTemplates\CSharp folder

Figure 7: Showing the Quartz Builder and View items in the Add new Item dialog

Rebuilding the Visual Studio Cache

Once installed the item templates into VS.NET, you'll need to get Visual Studio to rebuild its cache. You do that by doing the following:

1. Close all instances of Visual Studio
2. Open a Visual Studio Command Prompt (i.e. Start | All Programs | Microsoft Visual Studio 2010 | Visual Studio Tools | Visual Studio Command Prompt)
3. Execute the following command: devenv /installvstemplates

• Using the Convert static class
• Using the Convert.ChangeType method if your types implement theIConvertible interface
• Using the Type Descriptor Architecture that's part of the .NET framework (typically used by visual controls at design time)

Converting from String to ValueTypes

A Solution to the ChangeType problem

1. 4236
2. $4236
3. 4,236
4. $4,236

public T ChangeType<T>(string parameterName, T defaultValue = default(T))

public T ChangeType<T>(string parameterName, T defaultValue = default(T))
{
  Type typeOft = typeof(T);
  var value = Request.Form[parameterName];

  if (typeOft == typeof(Int32))
    return (T)Int32.Parse(value);
  else if (typeOft == typeof(Int64))
    return (T)Int64.Parse(value);
}

A Simple Solution

public T ChangeType<T>(string parameterName, T defaultValue = default(T))
{
  Type typeOft = typeof(T);
  var value = Request.Form[parameterName];

  if (String.IsNullOrEmpty(value.Trim()))
    return default(T);
  if (typeOft == typeof(Int32))
    return (T)ConvertToInt32(value);
  else if (typeOft == typeof(Int64))
    return (T)ConvertToInt64(value);
  else if (typeOft == typeof(Double))
    return (T)ConvertToDouble(value);
  else if (typeOft == typeof(Decimal))
    return (T)ConvertToDecimal(value);
  return default(T);
}

private object ConvertToInt32(string value)
{
  return Int32.Parse(value,
    NumberStyles.Currency ^ NumberStyles.AllowDecimalPoint);
}

private object ConvertToInt64(string value)
{
  return Int64.Parse(value,
    NumberStyles.Currency ^ NumberStyles.AllowDecimalPoint);
}

private object ConvertToDouble(string value)
{
  return Double.Parse(value, NumberStyles.Currency);
}

private object ConvertToDecimal(string value)
{
  return Decimal.Parse(value, NumberStyles.Currency);
}

An Extensible solution for ChangeType<T>

int amount = ChangeType<int>("Amount");

int amount = ChangeType<int>("Amount", -1);

string[] tags = ChangeType<string[]>("tags");

1. And Interface called IRequestBinder. Your custom binders will implement this interface (just one method)
2. The RequestBinder class that maintains a list of registered binders and issues the appropriate binder
3. A Generic method ChangeType<T> that does all the hard work of abstracting all of this and allows you to write code like that shown above.

public interface IRequestBinder
{
  object Bind(HttpContextBase httpContext, Type convertToType,
    string parameterName = null, object defaultValue = null);
}

public static class RequestBinder
{
  private static readonly Dictionary<string, RuntimeTypeHandle>
    registeredBinders = new Dictionary<string, RuntimeTypeHandle>();
  private static readonly Dictionary<string, IRequestBinder>
    instantiatedBinders = new Dictionary<string, IRequestBinder>();

  static RequestBinder()
  {
    Add(typeof(String), typeof(StringRequestBinder));
    Add(typeof(Int32), typeof(Int32RequestBinder));
    Add(typeof(Int64), typeof(Int64RequestBinder));
    Add(typeof(Double), typeof(DoubleRequestBinder));
    Add(typeof(DateTime), typeof(DateTimeRequestBinder));
    Add(typeof(Boolean), typeof(BooleanRequestBinder));
    Add(typeof(Decimal), typeof(DecimalRequestBinder));
    Add(typeof(Int32[]), typeof(Int32ArrayRequestBinder));
    Add(typeof(String[]), typeof(StringArrayRequestBinder));
    Add(typeof(Double[]), typeof(DoubleArrayRequestBinder));
    Add(typeof(Enum), typeof(EnumRequestBinder));
  }

  public static void Add(Type binderForType, Type requestBinderType)
  {
    Type interfaceType = requestBinderType.GetInterface("IRequestBinder");
    if (interfaceType != null)
    {
      var fullName = binderForType.FullName;
      if (registeredBinders.ContainsKey(fullName))
        registeredBinders.Remove(fullName);
      registeredBinders.Add(fullName, requestBinderType.TypeHandle);
    }
  }

  public static bool Contains(Type type)
  {
    return registeredBinders.ContainsKey(type.FullName);
  }

  public static IRequestBinder GetBinder(Type type)
  {
    var fullName = type.FullName;
    if (!registeredBinders.ContainsKey(fullName))
      throw new RequestBinderException(
        "No RequestBinder found for Type: " + fullName);
    if (instantiatedBinders.ContainsKey(fullName))
      return instantiatedBinders[fullName];
    else
    {
      var typeHandle = registeredBinders[fullName];
      IRequestBinder binder = 
        (IRequestBinder)Activator.CreateInstance(
        Type.GetTypeFromHandle(typeHandle));
      instantiatedBinders.Add(fullName, binder);
      return binder;
    }
  }
}

var binder = RequestBinder.GetBinder(typeof(Int32));
Int32 value = (Int32)binder.Bind(HttpContext, type, parameterName, defaultValue);

protected T ChangeType<T>(string parameterName,
  T defaultValue = default(T))
{
  Type type = typeof(T);
  if (RequestBinder.Contains(type))
    return (T)RequestBinder
      .GetBinder(type)
      .Bind(HttpContext, type, parameterName, defaultValue);
  else if (type.IsEnum)
    return (T)RequestBinder
      .GetBinder(typeof(Enum))
      .Bind(HttpContext, type, parameterName, defaultValue);
  else
    return defaultValue;        
}

Handling Custom Types

protected T ChangeType<T>() where T : class, new()
{
  Type type = typeof(T);
  if (RequestBinder.Contains(type))
    return (T)RequestBinder.GetBinder(type).Bind(HttpContext, type);
  else
    return default(T);
}

the Complete Solution

using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Globalization;
using System.Web;
using Matlus.Web.Builder.Exceptions;

namespace Matlus.Web.Builder
{
  public interface IRequestBinder
  {
    object Bind(HttpContextBase httpContext, Type convertToType,
      string parameterName = null, object defaultValue = null);
  }

  public static class RequestBinder
  {
    private static readonly Dictionary<string, RuntimeTypeHandle>
      registeredBinders = new Dictionary<string, RuntimeTypeHandle>();
    private static readonly Dictionary<string, IRequestBinder>
      instantiatedBinders = new Dictionary<string, IRequestBinder>();

    static RequestBinder()
    {
      Add(typeof(String), typeof(StringRequestBinder));
      Add(typeof(Int32), typeof(Int32RequestBinder));
      Add(typeof(Int64), typeof(Int64RequestBinder));
      Add(typeof(Double), typeof(DoubleRequestBinder));
      Add(typeof(DateTime), typeof(DateTimeRequestBinder));
      Add(typeof(Boolean), typeof(BooleanRequestBinder));
      Add(typeof(Decimal), typeof(DecimalRequestBinder));
      Add(typeof(Int32[]), typeof(Int32ArrayRequestBinder));
      Add(typeof(String[]), typeof(StringArrayRequestBinder));
      Add(typeof(Double[]), typeof(DoubleArrayRequestBinder));
      Add(typeof(Enum), typeof(EnumRequestBinder));
    }

    public static void Add(Type binderForType, Type requestBinderType)
    {
      Type interfaceType = requestBinderType.GetInterface("IRequestBinder");
      if (interfaceType != null)
      {
        var fullName = binderForType.FullName;
        if (registeredBinders.ContainsKey(fullName))
          registeredBinders.Remove(fullName);
        registeredBinders.Add(fullName, requestBinderType.TypeHandle);
      }
    }

    public static bool Contains(Type type)
    {
      return registeredBinders.ContainsKey(type.FullName);
    }

    public static IRequestBinder GetBinder(Type type)
    {
      var fullName = type.FullName;
      if (!registeredBinders.ContainsKey(fullName))
        throw new RequestBinderException(
          "No RequestBinder found for Type: " + fullName);
      if (instantiatedBinders.ContainsKey(fullName))
        return instantiatedBinders[fullName];
      else
      {
        var typeHandle = registeredBinders[fullName];
        IRequestBinder binder =
          (IRequestBinder)Activator.CreateInstance(
          Type.GetTypeFromHandle(typeHandle));
        instantiatedBinders.Add(fullName, binder);
        return binder;
      }
    }
  }

  public abstract class RequestBinderBase : IRequestBinder
  {
    protected static NameValueCollection GetRequestNameValueCollection(HttpContextBase httpContext)
    {
      var combinedNameValueCollection = new NameValueCollection();
      foreach (string key in httpContext.Request.Form.Keys)
      {
        var value = httpContext.Request.Form[key];
        if (value != null)
          combinedNameValueCollection.Add(key, value);
      }

      foreach (string key in httpContext.Request.QueryString.Keys)
      {
        var value = httpContext.Request.QueryString[key];
        if (value != null)
          combinedNameValueCollection.Add(key, value);
      }
      return combinedNameValueCollection;
    }

    #region IRequestBinder Members

    public abstract object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null);

    #endregion
  }

  public sealed class Int32RequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        return Int32.Parse(value, NumberStyles.Currency ^ NumberStyles.AllowDecimalPoint);
      }
      else
        return defaultValue;
    }
  }

  public sealed class StringRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
        return value;
      else
        return defaultValue;
    }
  }

  public sealed class Int64RequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        return Int64.Parse(value, NumberStyles.Currency ^ NumberStyles.AllowDecimalPoint);
      }
      else
        return defaultValue;
    }
  }

  public sealed class DoubleRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        return Double.Parse(value, NumberStyles.Currency);
      }
      else
        return defaultValue;
    }
  }

  public sealed class DateTimeRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        return DateTime.Parse(value);
      }
      else
        return defaultValue;
    }
  }

  public sealed class DecimalRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        return Decimal.Parse(value, NumberStyles.Currency);
      }
      else
        return defaultValue;
    }
  }

  public sealed class BooleanRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        switch (value.ToString().Trim())
        {
          case "False":
          case "false":
          case "0":
          case "off":
          case "":
            return false;
          case "True":
          case "true":
          case "1":
          case "on":
            return true;
          default:
            return false;
        }
      }
      else
        return defaultValue;
    }
  }

  public sealed class StringArrayRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        value = value.Trim();
        if (value.Length == 0)
          return Activator.CreateInstance(convertToType, new object[] { 0 });
        else
        {
          var values = value.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
          var strArray = new String[values.Length];
          for (int i = 0; i < values.Length; i++)
            strArray[i] = values[i];
          return strArray;
        }
      }
      return defaultValue;
    }
  }

  public sealed class Int32ArrayRequestBinder : RequestBinderBase
  {
    #region IRequestBinder Members

    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        value = value.Trim();
        if (value.Length == 0)
          return Activator.CreateInstance(convertToType, new object[] { 0 });
        else
        {
          var values = value.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
          var intArray = new Int32[values.Length];
          for (int i = 0; i < values.Length; i++)
            intArray[i] = Int32.Parse(values[i]);
          return intArray;
        }
      }
      return defaultValue;
    }

    #endregion
  }

  public sealed class DoubleArrayRequestBinder : RequestBinderBase
  {
    #region IRequestBinder Members

    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        value = value.Trim();
        if (value.Length == 0)
          return Activator.CreateInstance(convertToType, new object[] { 0 });
        else
        {
          var values = value.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
          var dblArray = new Double[values.Length];
          for (int i = 0; i < values.Length; i++)
            dblArray[i] = Double.Parse(values[i]);
          return dblArray;
        }
      }
      return defaultValue;
    }

    #endregion
  }

  public sealed class EnumRequestBinder : RequestBinderBase
  {
    public override object Bind(HttpContextBase httpContext, Type convertToType, string parameterName = null, object defaultValue = null)
    {
      var value = GetRequestNameValueCollection(httpContext)[parameterName];
      if (!String.IsNullOrEmpty(value))
      {
        return Enum.Parse(convertToType, value);
      }
      else
        return defaultValue;
    }
  }

}

Handling Custom classes (DTOs)

using System;
using System.Collections.Generic;
using System.Globalization;
using System.Reflection;
using System.Web;

namespace Matlus.Web.Builder
{
  public static class DtoBinder
  {
    private static readonly BindingFlags PROP_BINDING_FLAGS = BindingFlags.Instance | BindingFlags.Public | BindingFlags.IgnoreCase;
    private static readonly BindingFlags PROP_SET_BINDING_FLAGS = BindingFlags.SetProperty | BindingFlags.SetField | BindingFlags.IgnoreCase;

    public static object GetPropertyValue(object dtoObject, string propertyName)
    {
      PropertyInfo propInfo = dtoObject.GetType().GetProperty(propertyName, PROP_BINDING_FLAGS);
      if (propInfo != null)
        return propInfo.GetValue(dtoObject, null);
      else
        return null;
    }

    public static Dictionary<string, PropertyInfo> GetPropertyInfos(object dtoObject)
    {
      var properties = new Dictionary<string, PropertyInfo>();
      foreach (var propInfo in dtoObject.GetType().GetProperties(PROP_BINDING_FLAGS))
        properties.Add(propInfo.Name, propInfo);
      return properties;
    }

    public static T CreateInstanceFromRequest<T>(HttpRequestBase request) where T : new()
    {      
      Type t = typeof(T);
      var propertyInfos = t.GetProperties(PROP_BINDING_FLAGS);
      var instance = new T();
      
      foreach (var propInfo in propertyInfos)
      {
        var value = request.Form[propInfo.Name];
        if (value == null)
          value = request.QueryString[propInfo.Name];
        if (value != null)
          SetPropertyValue(instance, propInfo, value);
      }
      return instance;
    }

    public static void SetPropertyValue(object dtoObject, string propertyName, object propertyValue)
    {
      var propInfo = dtoObject.GetType().GetProperty(propertyName, PROP_BINDING_FLAGS);
      SetPropertyValue(dtoObject, propInfo, propertyValue);
    }

    private static void CleanUpPropertyValue<TResult>(ref object propertyValue) where TResult : struct
    {
      var value = propertyValue as String;
      if (value != null)
      {
        var trimmed = value.Trim();
        if (trimmed.Length == 0)
          propertyValue = default(TResult);
        else
        {
          Type resultType = typeof(TResult);

          if (resultType == typeof(Int32))
            propertyValue = ConvertToInt32(trimmed);
          else if (resultType == typeof(Int64))
            propertyValue = ConvertToInt64(trimmed);
          else if (resultType == typeof(Double))
            propertyValue = ConvertToDouble(trimmed);
          else if (resultType == typeof(Decimal))
            propertyValue = ConvertToDecimal(trimmed);
          else
            propertyValue = trimmed;
        }
      }
    }

    public static void SetPropertyValue(object dtoObject, PropertyInfo propInfo, object propertyValue)
    {
      Type propType = propInfo.PropertyType;
      var typeCode = Type.GetTypeCode(propertyValue.GetType());

      if (propType == typeof(String))
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      else if (propType == typeof(Int32))
      {
        CleanUpPropertyValue<Int32>(ref propertyValue);        
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
      else if (propType == typeof(Int64))
      {
        CleanUpPropertyValue<Int64>(ref propertyValue);
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
      else if (propType == typeof(Boolean))
        propInfo.SetValue(dtoObject, ConvertToBool(propertyValue), PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      else if (propType == typeof(DateTime))
      {
        CleanUpPropertyValue<DateTime>(ref propertyValue);
        propInfo.SetValue(dtoObject, Convert.ToDateTime(propertyValue), PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
      else if (propType == typeof(Double))
      {
        CleanUpPropertyValue<Double>(ref propertyValue);
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
      else if (propType == typeof(Decimal))
      {
        CleanUpPropertyValue<Decimal>(ref propertyValue);
        propInfo.SetValue(dtoObject, Convert.ToDecimal(propertyValue), PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
      else if (propType.IsEnum)
        propInfo.SetValue(dtoObject, Enum.Parse(propType, Convert.ToString(propertyValue), true), PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      else if (propType == typeof(Array))
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      else if (propType == typeof(String[]))
      {
        if (propertyValue is String)
        {
          var propertyValueAsString = propertyValue.ToString().Trim();
          if (propertyValueAsString.Length == 0)
            propertyValue = new string[0];
          else
            propertyValue = propertyValueAsString.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
        }
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
      else if (propType == typeof(Int32[]) || propType == typeof(Int64[]))
      {
        if (propertyValue is String)
        {
          var propertyValueAsString = propertyValue.ToString().Trim();
          if (propertyValueAsString.Length == 0)
            propertyValue = Activator.CreateInstance(propType, new object[] { 0 });
          else
          {
            var values = propertyValueAsString.Split(new char[] { ',' }, StringSplitOptions.RemoveEmptyEntries);
            if (propType == typeof(Int32[]))
            {
              var intArray = new Int32[values.Length];
              for (int i = 0; i < values.Length; i++)
                intArray[i] = Int32.Parse(values[i]);
              propertyValue = intArray;
            }
            else
            {
              var intArray = new Int64[values.Length];
              for (int i = 0; i < values.Length; i++)
                intArray[i] = Int64.Parse(values[i]);
              propertyValue = intArray;
            }
          }
        }
        propInfo.SetValue(dtoObject, propertyValue, PROP_SET_BINDING_FLAGS, null, null, CultureInfo.CurrentCulture);
      }
    }

    private static Int32 ConvertToInt32(string value)
    {
      return Int32.Parse(value, NumberStyles.Currency ^ NumberStyles.AllowDecimalPoint);
    }

    private static Int64 ConvertToInt64(string value)
    {
      return Int64.Parse(value, NumberStyles.Currency ^ NumberStyles.AllowDecimalPoint);
    }

    private static Double ConvertToDouble(string value)
    {
      return Double.Parse(value, NumberStyles.Currency);
    }

    private static Decimal ConvertToDecimal(string value)
    {
      return Decimal.Parse(value, NumberStyles.Currency);
    }

    private static bool ConvertToBool(object objBool)
    {
      switch (objBool.ToString().Trim())
      {
        case "False":
        case "false":
        case "0":
        case "off":
        case "":
          return false;
        case "True":
        case "true":
        case "1":
        case "on":
          return true;
        default:
          return false;
      }
    }
  }
}

protected T ChangeType<T>() where T : class, new()
{
  Type type = typeof(T);
  if (RequestBinder.Contains(type))
    return (T)RequestBinder.GetBinder(type).Bind(HttpContext, type);
  else if (!type.IsValueType)
    return DtoBinder.CreateInstanceFromRequest<T>(Request);
  else
    return default(T);
}

protected T ChangeType<T>(string parameterName, T defaultValue = default(T))
{
  Type type = typeof(T);
  if (RequestBinder.Contains(type))
    return (T)RequestBinder.GetBinder(type).Bind(HttpContext, type, parameterName, defaultValue);
  else if (type.IsEnum)
    return (T)RequestBinder.GetBinder(typeof(Enum)).Bind(HttpContext, type, parameterName, defaultValue);
  else
    return defaultValue;        
}

Razor Engine Host Wrapper

using System;
using System.CodeDom.Compiler;
using System.IO;
using System.Reflection;
using System.Text;
using System.Web.Razor;
using Microsoft.CSharp;

namespace RazorHost
{
  /// <summary>
  /// This class is a wrapper around the Razor Engine and Razor Engine Host
  /// </summary>
  public class RazorEngineHostWrapper
  {
    private RazorTemplateEngine InitializeRazorEngine(Type baseClassType, string namespaceOfGeneratedClass, string generatedClassName)
    {
      RazorEngineHost host = new RazorEngineHost(new CSharpRazorCodeLanguage());
      host.DefaultBaseClass = baseClassType.FullName;
      host.DefaultClassName = generatedClassName;
      host.DefaultNamespace = namespaceOfGeneratedClass;
      host.NamespaceImports.Add("System");
      host.NamespaceImports.Add("System.Collections.Generic");
      host.NamespaceImports.Add("System.Linq");
      return new RazorTemplateEngine(host);
    }

    /// <summary>
    /// This method Parses and compiles the source code into an Assembly and returns it
    /// </summary>
    /// <param name="baseClassType">The Type of the Base class the generated class descends from</param>
    /// <param name="namespaceOfGeneratedClass">The Namespace of the generated class</param>
    /// <param name="generatedClassName">The Class Name of the generated class</param>
    /// <param name="ReferencedAssemblies">Assembly refereces that may be required in order to compile the generated class</param>
    /// <param name="sourceCodeReader">A Text reader that is a warpper around the "Template" that is to be parsed and compiled</param>
    /// <returns>An instance of a generated assembly that contains the generated class</returns>
    public Assembly ParseAndCompileTemplate(Type baseClassType, string namespaceOfGeneratedClass, string generatedClassName, string[] ReferencedAssemblies, TextReader sourceCodeReader)
    {
      RazorTemplateEngine engine = InitializeRazorEngine(baseClassType, namespaceOfGeneratedClass, generatedClassName);
      GeneratorResults razorResults = engine.GenerateCode(sourceCodeReader);

      CSharpCodeProvider codeProvider = new CSharpCodeProvider();
      CodeGeneratorOptions options = new CodeGeneratorOptions();

      string generatedCode = null;
      using (StringWriter writer = new StringWriter())
      {
        codeProvider.GenerateCodeFromCompileUnit(razorResults.GeneratedCode, writer, options);
        generatedCode = writer.GetStringBuilder().ToString();
      }

      var outputAssemblyName = Path.GetTempPath() + Guid.NewGuid().ToString("N") + ".dll";
      CompilerParameters compilerParameters = new CompilerParameters(ReferencedAssemblies, outputAssemblyName);
      compilerParameters.ReferencedAssemblies.Add("System.dll");
      compilerParameters.ReferencedAssemblies.Add("System.Core.dll");
      compilerParameters.ReferencedAssemblies.Add(Assembly.GetExecutingAssembly().CodeBase.Substring(8));
      compilerParameters.GenerateInMemory = false;

      CompilerResults compilerResults = codeProvider.CompileAssemblyFromDom(compilerParameters, razorResults.GeneratedCode);
      if (compilerResults.Errors.Count > 0)
      {
        var compileErrors = new StringBuilder();
        foreach (System.CodeDom.Compiler.CompilerError compileError in compilerResults.Errors)
          compileErrors.Append(String.Format("Line: {0}\t Col: {1}\t Error: {2}\r\n", compileError.Line, compileError.Column, compileError.ErrorText));

        throw new Exception(compileErrors.ToString() + generatedCode);
      }

      return compilerResults.CompiledAssembly;
    }
  }
}

Code Listing 1: RazorEngineHostWrapper.cs

Using the Razor Engine Wrapper

• public virtual void Write(object value)
• public virtual void WriteLiteral(object value)

    private void button1_Click(object sender, EventArgs e)
    {
      TextReader sourceCodeReader = new StringReader(textBox1.Text);
      string[] referencesAssemblies = new String[]
      {
        @"System.dll",
        @"System.Core.dll"
      };
      RazorEngineHostWrapper razorEngineHostWrapper = new RazorHost.RazorEngineHostWrapper();
      var generatedAssembly = razorEngineHostWrapper.ParseAndCompileTemplate(typeof(TemplateBase), "RazorHost", "MyTemplate", referencesAssemblies, sourceCodeReader);

      Type type = generatedAssembly.GetType("RazorHost.MyTemplate");
      var instance = (TemplateBase)Activator.CreateInstance(type);
      object result = type.InvokeMember("Execute", BindingFlags.InvokeMethod, null, instance, null);
      textBox2.Text = instance.Buffer.ToString();
    }

  public abstract class TemplateBase
  {
    public StringBuilder Buffer { get; private set; }
    private TextWriter writer;
    private TextWriter Writer { get { return writer; } }

    public List<Customer> Customers { get; set; }

    public TemplateBase()
    {
      Buffer = new StringBuilder();
      writer = new StringWriter(Buffer);
      Customers = new List<Customer>()
      {
        new Customer() { FirstName="Bill", LastName="Gates" },
        new Customer() { FirstName="Steve", LastName="Jobs" },
        new Customer() { FirstName="Larry", LastName=" Ellison" },
        new Customer() { FirstName="Samuel", LastName="Palmisano" }
      };
    }

    public virtual void Write(object value)
    {
      Writer.Write(value);
    }

    public virtual void WriteLiteral(object value)
    {
      Writer.Write(value);
    }

    public abstract void Execute();
  }

  public class Customer
  {
    public string FirstName { get; set; }
    public string LastName { get; set; }
  }

Code Listing 3: The TemplateBase class

A Sample Template

@functions {
  string GetCustomerFullName(Customer customer)
  {
    return customer.FirstName + " " + customer.LastName;
  }
}
<ul>
@foreach(var customer in Customers)
{
    <li>@customer.FirstName @customer.LastName</li>
    <li>Full Name: @GetCustomerFullName(customer)</li>
}
</ul>

Code Listing 4: A Sample Template using the Razor Syntax

<ul>
    <li>Bill Gates</li>
    <li>Full Name: Bill Gates</li>
    <li>Steve Jobs</li>
    <li>Full Name: Steve Jobs</li>
    <li>Larry  Ellison</li>
    <li>Full Name: Larry  Ellison</li>
    <li>Samuel Palmisano</li>
    <li>Full Name: Samuel Palmisano</li>
</ul>

Code Listing 5: The Generated Output

Figure 1: ASP.NET MVC Design and control flow

List of Problems with ASP.NET MVC

1. No access to view or partial view instances
2. ViewData is your loosely typed information carrier
3. Controller is not really in control
4. Child Actions have no sense of the Request Context
5. Views are coupled to controllers
6. Referencing Layout Pages using virtual paths
7. Routing
8. Html interspersed with code and code interspersed with html nightmare
9. Naming conventions

No Access to View or Partial View instances

public ActionResult Index()
{
  /* If a specific features is selected */
  if (FeatureId != 0)
  {
    var featurePageModel = BusinessModule.GetFeaturePageModel(FeatureId);

    ViewData["Title"] = featurePageModel.CurrentFeature.Title;
    ViewData["AdBoxTopViewBoxTitle"] = "More Features";
    ViewData["AdBoxTopViewModel"] = featurePageModel.AdBoxModel;
    ViewData.Model = featurePageModel.CurrentFeature;
    return View("FeatureDisplay");
  }
  else
  {
    var featuresAllPageModel = BusinessModule.GetFeaturesAllPageModel();

    ViewData["Title"] = featurePageModel.CurrentFeature.Title;
    ViewData["AdBoxTopViewBoxTitle"] = "Highlights";
    ViewData["AdBoxTopViewModel"] = featuresAllPageModel.AdBoxModel;
    ViewData.Model = featuresAllPageModel.Features;
    return View("FeaturesAllView");
  }
}

Code Listing 1: Showing ASP.NET MVC code in a Controller Action method

protected override void BuildPage(HttpContextBase httpContext, object model)
{
  /* If a specific features is selected */
  if (FeatureId != 0)
  {
    var featurePageModel = (FeaturePageModel)model;

    TitleView.Title = featurePageModel.CurrentFeature.Title;
    AdBoxTopView.BoxTitle = "More Features";
    AdBoxTopView.Model = featurePageModel.AdBoxModel;
    RegisterView("Body", new FeatureDisplayView(httpContext,
      this, featurePageModel.CurrentFeature));
  }
  else
  {
    var featuresAllPageModel = (FeaturesAllPageModel)model;

    TitleView.Title = "Builder for ASP.NET - Features";
    AdBoxTopView.BoxTitle = "Highlights";
    AdBoxTopView.Model = featuresAllPageModel.AdBoxModel;
    RegisterView("Body", new FeaturesAllView(httpContext, this,
      featuresAllPageModel.Features));
  }
}

Code Listing 2: Showing Builder for ASP.NET code in a Builder's BuildPage() method

1. The controller can’t ensure that the view and all partial views are looking for (and getting) the data it stuffed in ViewData using the same string keys.
2. By looking at the code in the controller's action method, it is impossible to tell what is really going on, because all you see is a bunch of things added to ViewData, but you can’t tell who is using what (if at all). That is, you can't tell which View and or partial view is using what thing in the dictionary and if this thing (the item in the dictionary) is even the correct type for the other thing (view and/or partial view) using it
3. No strong typing, and so no type checking, no compile time checks and no IDE navigation help between controller and view's methods and properties, sub classes/super classes
4. If the same partial view is used by other controllers then these controllers also need to use the same string keys to stuff their data into ViewData. But there is no way to ensure that at the time of writing or reading code
5. If the API of the partial view changes or the names of the keys a partial view relies upon changes there is no way to tell during development time what pages are going to break. In a strongly typed system like .NET a simple Ctrl + Shift + B would highlight all such breaking changes. But this is not the case with ASP.NET MVC due to the loosely typed ViewData

1. The controller has to know about all the requirements of the view, partial views, master views as well as any child actions
2. Either the view further breaks down your ViewModel into sub viewmodels and hands these sub viewmodels to each of the partial views. Or, every partial view and master view are privy to some other view's data. If you do the former, then the view needs to be aware of what the partial views need (just like the controller needs to know), which means both the controller and view need to know about the partial views, master views and child actions. So that's not any better. Doing the later is not correct either.

The Controller is not really in control

  public class HomeController : BaseController
  {
    public ActionResult Index()
    {
      ViewData["Title"] = "Movies In Theaters";
      ViewData["AdvertBoxTopName"] = "Popular Channels";
      ViewData.Model = BusinessModule.GetHomePageThumbnails();

      ViewEngineResult result = new ViewEngineResult(
        new RazorViewExt(ControllerContext,
        "~/Views/Home/Index.cshtml",
        "~/Views/Shared/LayoutSite.cshtml"),
        new RazorViewEngine());      
      
      return View(result.View);
    }
  }

  public class RazorViewExt : RazorView
  {
    public RazorViewExt(ControllerContext controllerContext, string viewPath,
      string layoutPath)
      :base(controllerContext, viewPath, layoutPath, false, null) 
    {

    }

    protected override