Symblr

Overview

What Is Symblr?

Symblr is a symbol and source server that I’m writing. It’s designed to make symbol maintenance as painless as possible for people who care (or should care) about having reliable symbols around.

Uh, What Are Symbols?

Something that you are oblivious about until you need them. When you have them and you don’t need them you are going to be in a very sore place.

Symbols, also known as “debug symbols,” are used to instruct a debugger on how it should interpret the memory that your program has manipulated at runtime. In other words, if you have ever used an interactive debugger (such as Visual Studio) symbols have made the following features possible:

  • Determine what the method names in the call stack are, instead of function addresses.
  • Inspect data structures and view fields, instead of raw memory.
  • Set breakpoints on individual lines by interacting with your code editor, instead of having to know the method address.
  • Determine exactly which statement/expression is executing, instead of only having the instruction pointer.
  • All of the above when inspecting a memory dump.

And a Source Server?

This is a feature that Microsoft symbols (PDB) can optionally implement. In addition to providing all the services that symbols provide, Microsoft symbols can also contain information on how to acquire the original source code that was used to build the binary that is being debugged. One example of a source server is the “.Net Framework Source Stepping” feature. When this feature is enabled Visual Studio will use a special symbol server to acquire symbols for .Net assemblies. These symbols contain source information and Visual Studio transparently downloads the source code if you attempt to step into .Net Framework methods.

Should I Care?

Why Are Symbols Important?

Working at an ISV I’ve been forced to appreciate how important a good symbol server is. My original lesson was a customer who was having some issues with our software. You can’t just ask a customer if you can install Visual Studio on their production server, remote in and start stepping through code. They’ll laugh at you and rightfully so. You’re going to have to rely on memory dumps.

So what you do is ask for a memory dump, open up WinDBG and think your going to start finding out what the issue is. What you are instead going to find out is how important reliable symbols are, as you are greeted by method addresses and raw memory. You’re going to have to get really creative with hunting down the issue.

Why Is a Source Server Important?

It nearly always isn’t. It is a nice-to-have the majority of the time.

One situation that I’ve found it important is when working with QA’s machines. Every night our CI server builds an installer/setup package that gets installed on QA’s environment the next day. Our installer doesn’t have our symbols because symbols can be quite big, often significantly larger than their associated binary.

Sometimes you run up against one of those environmental issues that you simply can’t reproduce on your development environment and you’re forced to ask QA if you can debug on their environment. In the absence of a source server you are going to have to get the code you want to debug and build it on the machine. QA is very likely to laugh you off. Building code on a QA environment is an incredibly bad idea because it can result in false negatives (bugs that don’t exist in release code) or false positives (bugs that don’t exist in debug code). It can also potentially outright break the environment, resulting in lost QA time.

With a source server all you need to do is point Visual Studio at it and you will be stepping into code in moments.

Does This Apply to Me?

No matter what type of developer you are, without exception: yes.

  • I’m my own customer, my code runs on environments that I control.
    • Examples: Saas, an API
    • Do you have the symbols for the code that is running in production, right now?
  • My customers are always running my latest stable code.
    • Examples: consultant, freelancer, automatic/required updates
    • Do you have the symbols for the binaries that your customers have, right now?
    • You do? What if you patched last night you are looking at a memory dump from yesterday?
  • I maintain different versions of my product.
    • Examples: ISV
    • You need a symbol server, yesterday. For now set up a UNC share and script it.
  • I use Linux/platform X.
    • I don’t know how your platform deals with symbols but I hope Symblr will support it at some point in the future.
    • You should find out how to archive symbols on your platform right now.

A Solution

What I’m looking for is something that:

  • Is compatible with the SymbolSource publishing model.
  • Has a scripted publishing model (e.g. for your CI server).
  • Has retention policies:
    • e.g. Daily builds don’t need to be kept around forever.
    • e.g. Product releases and hotfixes do.
  • Can be deployed behind a firewall.
  • Can act as a caching symbol and source proxy.
  • Can be hosted on any platform where .Net exists, not just Windows.
    • Completely trivial to set up, if the above condition holds.

Which is what you can expect from Symblr.

Show Me The Source

It’s up on GitHub. At the time of writing this post it’s not yet ready for contributors, nor is it even complete. What I have done is research and correctly implement PDB reading/writing (which I will document in another post).

Reflections on Trusting Virtue

There's a very good paper called ["Reflections on Trusting Trust"][1] by Ken Thompson - within that paper he goes on to explain the poten...… Continue reading

.Net Sockets and You

Published on June 01, 2015