In-Depth

Application Analytics: Why Is the Developer Always the Last to Know?

A look at how developers can use the integrated application analytics capabilities of Visual Studio 2010 to improve software quality.

• By Sebastian Holst
• 07/01/2011

No relationship can survive without effective and honest communication, and the special relationship that exists between you and your users is no exception. Yet development organizations too rarely exploit the most immediate communication tool at their disposal: their application.

The Agile Manifesto states: "Working software is the primary measure of progress," and that development's "highest priority is to satisfy the customer through early and continuous delivery of valuable software." In that context, measuring development success can be found "at the point of work/play," where users and their applications meet.

Two things must happen to achieve that measurement. First, you need to get your hands on the raw data for analysis. You must have the means to collect, transport and manage application runtime data. And second, the resulting analysis has to yield actionable intelligence. The development framework within which projects are defined and managed should enable organizations to take timely and effective action based on runtime intelligence extracted from runtime data.

Today, with Visual Studio 2010, you have the tools and the technology to do both.

Development Is the Customer
The objective of application analytics is to help you succeed by helping to increase application value and customer satisfaction. As a category, this is what distinguishes application analytics from Web analytics, network monitoring and other neighboring technologies. In Agile parlance, with application analytics, development is the customer.

In good Agile form, one can model application analytics requirements and stories where the developer plays the customer role. Let's take a closer look into some examples where application analytics fits inside common development scenarios.

A "typical" Agile project is planned as a series of iterations. A good practice is to time (or schedule) iterations to help ensure that feedback from early iterations can be used to refine the scope and plans for the later ones. Application analytics shortens the time to gather feedback (potentially reducing the number and duration of required iterations) and improves the quality and precision of that feedback (which ultimately should lead to improvements in the quality of the resulting work).

Of course, applications themselves typically go through their own major "iterations" (versions) and the same pattern holds; application versions should also be spaced to ensure that feedback from earlier versions can be effectively incorporated into future planning.

What might this look like when application analytics are a part of the mix? See "Task: Updating Existing Code," which extends some common Microsoft recommendations for "modifying existing code" to show specifically how application analytics can help.

When real-world usage statistics are placed in the right hands, inside the right tools and at the right times inside the development process, they can enable effective requirements development and user-experience profiling. But if application analytics are so great, why aren't they used everywhere?

The answer is, in part, in the "how" and not the "why." As the cliché says, "if there's no solution, then there's no problem -- it's just life." Runtime analytics have most often been seen as a luxury that most development organizations believe they can't afford (or certainly one they can afford to live without). But the convergence of Web, cloud and mobile computing is challenging the status quo.

Application Analytics Implementation
There's no magic formula for guaranteeing the value of an analytics implementation; value is almost entirely dependent on the specific project and the care taken when deciding which indicators will be most impactful and what responses they'll trigger. But it's possible to focus on the nuts and bolts of implementation, with an eye toward minimizing complexity and managing project risk. When the "cost of knowing" becomes simple, secure and cheap enough, then application analytics move naturally from a development luxury to a development commodity.

The following application instrumentation steps typically take no more than an hour or two, and include support for feature and custom data monitoring, opt-in mechanisms and data security. However, given the particulars around "value," I carefully skirt the tricky bits of prescribing what precisely to collect and what to do about it -- these are questions for the product owners, Scrum masters and architects to settle.

Scope and planning: You need to map objectives and data requirements, as well as identify the specific data fields and how much code may need to be written to meet these requirements. The answers to these questions will be found in the combination of the data requirements and the instrumentation and analytics technologies selected. Table 1 includes common examples.

Privacy, identity and security: Based on application functionality, user demographics and defined requirements, identify appropriate privacy policies, communication obligations, opt-in processes and enforcement mechanisms. Again, you must determine coding activities needed to meet these requirements. And, once again, the answers to these questions will be found in the combination of the requirements and the instrumentation and analytics technologies selected.

Instrumentation: What technologies and patterns will be used to modify an application to meet the requirements outlined here? I'm going to restrict that to the instrumentation and application analytics technologies surfaced within Visual Studio 2010 and/or as part of the Windows Phone 7 Developer Tools download (in other words, if you have Visual Studio or are developing Windows Phone applications, all of these tools are included).

Specifically, the remaining implementation steps outlined here use Visual Studio 2010 Dotfuscator Software Services Community Edition or Dotfuscator for Windows Phone and Runtime Intelligence Service for Windows Phone (provided with Windows Phone 7 Developer Tools).

Before I get started, one pattern worth noting is the Dotfuscator injection approach to instrumentation. Rather than rely upon code changes, Dotfuscator injects instrumentation into already compiled code. This is done in two parts. First, using custom attributes or an external XML resource, you specify the following:

• Organizational identity owning the application.
• Application identity (a GUID, a name and version number).
• Opt-in, security and privacy settings used to communicate and enforce privacy settings and, potentially, to secure a user's consent to opt-in.
• Runtime data to be collected, including session, system, feature, custom data, exception reporting and feature-level tracking.
• The target endpoint where the runtime data is sent.
• Additional configuration options including offline caching, SSL transmission, exception reporting and so on.

Second, Dotfuscator injects the code required to implement all of these items into the assemblies -- obviously, all post-compile.

This technique is often effective when analytics need to be implemented inside existing applications or across components and applications developed by multiple teams. It's also useful for targeted user groups that don't typically get a special build (for instance, internal users or other well-defined user segments), or when there's no easy access to the source code or the developers who wrote the code.

Using injection for instrumentation results in the application analytics implementation workflow shown in Figure 1.

[Click on image for larger view.]

Figure 1. High-level application analytics implementation workflow.

Once the configuration is complete, Dotfuscator will consume assemblies/XAP files and inject the appropriate instrumentation logic. At this point, an application can be optionally obfuscated, signed and packaged with an installation package.

Figure 2 illustrates the next three steps of the application analytics workflow, which include:

• Ingestion of the runtime data into an endpoint.
• Aggregation, analytics and publishing capabilities of a standalone endpoint.
• Additional analytics potential stemming from the integration of runtime intelligence with related systems, including Visual Studio, Dynamics CRM and Operations Manager.

[Click on image for larger view.]

Figure 2. Workflow of steps required to ingest, aggregate, publish, distribute and integrate runtime data with the Runtime Intelligence Service.

Application runtime data streams to an endpoint where it's ingested, managed and published to individuals. It's also surfaced through a RESTful API to facilitate data consumption by other systems, including application lifecycle management (ALM), customer relationship management and IT operations management.

Dashboards, Reports and Extracts
For application analytics to be worthwhile, you must transform runtime data into runtime intelligence. The default Runtime Intelligence Service portal provides summary and detail dashboards, addressing relative application usage by user, by application and version, by location, and across features. Filters make it easier to drill into specific behaviors of application instances and individual users. Custom data fields can reflect anything from the most common search terms or the average time it took for a remote service call to return, to delivering exception data including unhandled, handled and thrown exceptions.