Thursday, October 4, 2012

Backup Options for Your SQL Azure Database

SQL Azure guarantees “Monthly Availability” of 99.9% during a calendar month, but it currently does not back up your database automatically.  Without a proper DB backup plan, your business is literally one-click away from disaster.

There are a few options you can back up your database:

1. Copy one Azure database to another
2. Use BCP command line utility to backup and restore your data on SQL Azure to/from your local disk
3. SQL Data Sync

Friday, September 7, 2012

Display Interactive Twitter Timelines on Your Website

Twitter just launched a new tool called "embedded timelines" that can syndicate any public Twitter timeline to your website with one line of code.

A couple of cool things about this widget are:
  • No coding is needed -- all you need to do is to configure then copy & paste HTML code
  • You can interact with the tweets (show photos/media, reply, retweet, favorite, show more tweets) within the widget
  • You can display other Twitter user's public timelines instead of your own if you choose so
I played with it and was able to get it integrated with this Blogger site rather quickly. Following is the steps to light this widget up:
  1. Go to after signing in to your Twitter account
  2. Click "Create new" button to create a new widget
  3. Configure your widget such as Height, Theme, Link color etc then click "Create widget" button (see pic below)
  4. Copy the HTML code at the bottom right of the screen 
  5. Log on to your Blogger site, click Layout menu item on the left
  6. Click Add a Gadget, select Basics gadget category, scroll down to select HTML/JavaScript
  7. Type your Title, paste the HTML code from step #4, hit Save
  8. After click Save arrangement, your timeline is live on your Blogger site! (the live demo is available at the bottom of this page's sidebar)
Twitter Embedded Timeline Widget Configuration

If you need to integrate your timelines with other non-Blogger websites, Step 1-4 are the same.  You just need to replace step 5-8 with pasting generated HTML into your own webpage.

Monday, August 20, 2012

Build a Reliable and Scalable Twitter Streaming Worker Role in Windows Azure

In our early prototype, we used a single worker role instance that connects to Twitter public streams endpoint, parses the tweets and persist them to a SQL Azure database.  There were two issues with this approach:

1. Reliability:
Windows Azure requires at least two instances for each role to achieve 99.5% uptime.  Yet Twitter public streams only allow one standing connection to the public endpoints.  Connecting to a public stream more than once with the same account credentials will cause the oldest connection to be disconnected.  Creating multiple accounts to skirt that limitation is in borderline violation of Twitter policy.  Because we had only one worker role instance, we lose 15-30min streaming data every time Windows Azure upgrades or redeploys role instances.  This is unacceptable as users are expecting and relying on the low-latency data.

2. Scalability:
Reading and parsing tweets is order of magnitude faster than saving the tweets to SQL Azure.  Saving tweets to SQL Azure wouldn't catch up with the incoming tweets when put in stress.

The Old Architecture

To address both issues, we made the following architectural changes:

First, decouple the worker role into Streamer and Importer.  Streamer reads tweets from Twitter public streams and put them in an Azure queue; Importer reads tweets off the Azure queue and parses them before importing into the database. Now we can scale out the streaming and importing independently based on their own loads.

Second, we instantiate two Streamer role instances and use heartbeats to coordinate which instance should own the streaming connection to Twitter.

To be more specific,
1) the role instance that currently maintains the Twitter streaming session writes a heartbeat (its instance ID and a time stamp) to an Azure blob in a fixed interval.
2) each Streamer role instance checks that heartbeat. If missing heartbeat is detected, current role instance takes over the streaming session, writes out its own heartbeat and spins off a thread that backfills any potential missing tweets by calling Twitter REST APIs.  Using REST APIs in conjunction with the Streaming API for backfilling is one of the best practices recommended by Twitter.
3) if the original owner of the streaming session who missed heartbeats ever wakes up, it detects no missing heartbeats and disposes streaming resources.

The New Architecture
The new architecture was deployed a couple of months and has been running smoothly so far.

Tuesday, August 14, 2012

Wireframing with Balsamiq

After playing with and comparing a few wireframing tools, we settled with Balsamiq.  We've been using it for the past few months and are quite happy with it so far.  Following is a screenshot of one of the sample mockups.

What we like about Balsamiq are:

  • It produces unreal-looking yet professional mockups
  • Ability to link pages and click through them
  • Extensive set of UI controls (out-of-the-box, from the Balsamiq community and upload your own) 
  • Create reusable common elements across different mockups with symbols (aka template, master page) 

We use Balsamiq desktop version during intensive wireframing sessions and web version for sharing and collaboration.  One thing Balsamiq could do better is integration between the desktop version and web version. Currently you have to export all mockup as separate BMML files and upload them one-by-one to the web.  It is a tedious and time-consuming process. It'll be nice if we can pull mockups directly from the web into desktop and publish them right back when it's done.

Disclaimer: my company and I are not associated or affilicated with Balsamiq in any way other than we are a paying customer.  

Saturday, July 7, 2012

Handling JSON in .Net/C#

XML is used as the native data representation in the .Net world.  Yet JSON, "The Fat-Free Alternative to XML" (quote from, is winning hearts and minds of many web developers.  If you're relying on some public web services like we do on Twitter APIs, chances are you'd need to use JSON as the data exchange format (Twitter dropped its support for XML not long ago).  This post explores how you can handle JSON effectively in your .Net/C# code.

1. Pick a Json library:
It is tedious and unnecessary to parse Json string yourself if you're just building an app or a website. You can leverage some of the existing Json libraries. Sagi did some perf test and published the following perf comparison chart at his blog. We chose Json.Net because of its being open-source, performance and popularity.

 Parsing Twitter JSON: Comparing C# libraries performance
2. Deserialize Json string to C# class
Before you can do something interesting with Json, you need to deserialize it into a C# object.

First step of serialization is to define your C# class. If you are dealing with complex Json structure, can help you generate C# class signature automatically.

Once you have the class, you may
1) Deserialize the entire Json string by using the ReadObject method of the DataContractJsonSerializer. MSDN provides a couple of quick examples. 

2) If you prefer to customize the deserialization (e.g. you need more granular control at field level), StackOverflow has an good example showing how to do this using Json.Net:

    public class User {
        public User(string json) {
            JObject jObject = JObject.Parse(json);
            JToken jUser = jObject["user"];
            name = (string) jUser["name"];
            teamname = (string) jUser["teamname"];
            email = (string) jUser["email"];
            players = jUser["players"].ToArray();

        public string name { get; set; }
        public string teamname { get; set; }
        public string email { get; set; }
        public Array players { get; set; }

In either case, there is one caveat -- the DateTime field often needs special handling.  Json represents DateTime as string, which may or may not be in the format that can be recognized by .Net's DateTime parser.  When it is not, it may fail the entire deserialization. 

For example, date/time fields in Twitter's Json responses are in the following format:
    Sat Jul 07 22:21:50 +0000 2012

To convert that into a proper C# DateTime object, you need to explicitly specify the format for the .Net parser:

public const string Const_TwitterDateTemplate = "ddd MMM dd HH:mm:ss +ffff yyyy";

DateTime createdAt = DateTime.ParseExact((string)jo["created_at"], Const_TwitterDateTemplate, new System.Globalization.CultureInfo("en-US"));

3. Return Json from an Asp.Net MVC method
One big benefit of returning a Json object from your Asp.Net application is that it can be consumed natively by jQuery Ajax. For example,

    url: '/Customer/GetCustomerInJson',
    dataType: "json",
    success: function (data) {

If you want to return a native C# object in Json format, you can use Controller.Json method in your controller.
public class Customer
    public string name { getset; }
    public string country { getset; }

public ActionResult GetCustomerJson()
    Customer newCustomer = new Customer() { name = "John Doe", country = "USA" };
    return Json(newCustomer);

If you already have a Json string that you obtain from somewhere else (function/API call,  database etc.), you can just return a Content result with application/json content type.

        public ActionResult GetCustomerJson()
            string jsonString = string.Empty;

            // TODO: Get json string somewhere

            return Content(jsonString, "application/json");

Wednesday, June 13, 2012

ASP.Net MVC Error Handling at Method, Controller and Global Levels

Traditional Asp.Net supports
  • Custom error redirect in web.config 
  • Application_Error in Global.asax
On top of that MVC3 introduced 
How do you put these pieces together to build a more robust MVC site?

1. Handle controller/method level MVC errors with HandleErrorAttribute
You can decorate your controllers or action methods with HandleErrorAttribute.

    public class MyController : Controller
        public ActionResult Index()
            return View();
    public class MyController : Controller
        public ActionResult Index()
            return View();

When an action method in the controller with HandleError attribute decoration throws an exception and it is not handled by your code, MVC displays the Error view that is located in the ~/Views/Shared folder. It's worth to note that HandleErrorAttribute renders the view without passing through any error handling controller you may have.  Yes this is somewhat confusing.  I first learned this from Tom Dupont's blog and later confirmed with simulating an exception and stepping through the code.

You may override HandleErrorAttribute's default behavior by specifying the exception types that should be handled, the name of the view to render, the order in which the filters are applied if you have more than one HandleErrorAttribute for your controller/method.  See MSDN for more details.

If you want to enable error handling for all of your controllers yet don't like the tedious work of decorating your controllers one-by-one, you can register global HandleErrorAttribute in Global.asax.  This is enabled by default in MVC3.

        public static void RegisterGlobalFilters(GlobalFilterCollection filters)
            filters.Add(new HandleErrorAttribute());
        protected void Application_Start()


2. Log errors by overriding HandleErrorAttribute.OnException or Controller.OnException
By now you may wonder how/where I can log errors.  HandleErrorAttribute renders an error view to users, but doesn't log errors for you automatically.  This is where OnException method comes into play.  Both HandleErrorAttribute and Controller classes have OnException method that can be overridden for error logging.  

In Controller's case, first you override the OnException method and add your own logic for error logging or notification.

    public class BaseController : Controller
        protected override void OnException(ExceptionContext filterContext)
            // Do additional things like logging or emailing here.


Then you inherit your controllers from the new BaseController instead of the default Controller class.

    public class HomeController : BaseController
        public ActionResult Index()
            // Do something here

            return View();

From that point on, your OnException will be called whenever an unhandled exception is thrown in your controller methods.
3. Handle non-MVC errors at global level with Application_Error
HandleErrorAttribute and Controller.OnException deal with application or server errors, but not errors outside of MVC pipeline such as 404 Not Found error.  Those non-MVC errors should be handled/logged in your Application_Error method in Global.asax.

If you neither register global HandleErrorAttribute nor decorate your controller/method with HandleErrorAttribute, exceptions will bubble up to Application_Error.

4. Display custom errors to the end users.
If you already registered global HandleErrorAttribute or decorated your methods/controllers with the same attribute but still saw the Yellow Screen of Death (YOSD) with potentially security-compromising error details like stack trace, you should check customErrors element in your web.config to make sure the mode attribute is set to either RemoteOnly (by default) or On.

    <customErrors mode="RemoteOnly" defaultRedirect="/Error/HttpError">
      <error statusCode="404" redirect="/Error/Http404"/>

Other Resources

Friday, June 1, 2012

Twitter Stream APIs Comparison

Twitter REST API is polling-based.  On contrast, Twitter Stream API pushes the tweet data to client as they become available thereby eliminating the polling overhead and reducing the latency.

There're three types of Stream APIs: public streams, User streams and Site streams.  Our company has been using public stream for the past year or so and now are moving to UserStreams/SiteStreams. Following is a side-by-side comparison of these three APIs.

Streaming Endpoint
Use case
Public streams

Streams of the public data flowing through Twitter.
One user app for data analysis.
.         Each Twitter account may create only one standing connection.
         No support for Direct Message.

User streams
Single-user streams, containing roughly all of the data corresponding with a single user's view of Twitter.
Single-user, client-side desktop app/mobile app that provides equivalent view of their home timeline.
         Each Twitter account is limited to only a few simultaneous connections per OAuth application, regardless of IP.
.       The number of simultaneous connections per IP address is still limited, regardless of application.  
Direct message is supported in the context of the authenticated user under which the connection is established.
Site streams

Multi-user version of user streams.
Multi-tenant websites or mobile push services that provide home timeline on behalf of many users at a time.
          Max 100 users per stream while connecting; can add up to 1,000 users per connection through API.
         Max 25 new connections per second.
Deliver both public and private data in a single stream.  Must be very meticulous about what data they display to whom.

Workaround for SELECT INTO in SQL Azure

If you try to do select insert in SQL Azure like the statement below

INSERT INTO dbo.MyTarget

You will get error message:

Statement 'SELECT INTO' is not supported in this version of SQL Server.

There's a quick workaround for that -- use INSERT INTO SELECT statement instead.  The caveat is that you need to create the table beforehand.


INSERT INTO dbo.MyTarget (ID)

Tuesday, May 22, 2012

From elevator speech to one-sentence pitch

Most of us have heard of elevator speech -- pitch your idea to a VC or angel investor in the time span of an elevator ride.  When I was in Founder's Institute program last year, I first heard of the concept of one-sentence pitch from Adeo Ressi --  distill your startup idea into a single sentence.  It was quite refreshing.  Here's the template:

My company, __(insert name of company)__, 

is developing __(a defined offering)__

to help __(a defined audience)____

(solve a problem)__

with __(secret sauce)__

You may wonder, elevator speech is challenging enough, why make it even shorter and tougher?   I guess people in general have less and less attention span and VCs and angel investors in particular have too many startup ideas to review. :)

Anyway my partner Aseem and I just drafted our first version of one-sentence pitch a couple of days ago:

Our company, Uptown Treehouse,

is developing a social CRM tool on Twitter 

to help businesses build relationship with their audience at scale and improve their ROI on social media.

For sure this will continue to evolve in the coming months. If you're interested to try out our product, you may sign up to our beta notification list at