There’s a very good paper called “Reflections on Trusting Trust” by Ken Thompson - within that paper he goes on to explain the potential pitfalls of trusting something you consider trustworthy. As you follow him down the rabbit hole he modifies a C++ compiler to inject a backdoor when compiling UNIX “login”, as well as inject the backdoor injection routine when compiling itself. This results in a compiler that can perpetuate this backdoor without being caught and provokes an important thought: can you actually trust something that behaves as though it is trustworthy?

Recently we’ve been rather rudely reminded that maybe we should have paid more attention Ken’s words of warning. The story of SourceForge isn’t too different from Ken’s C++ compiler.

SourceForge

SourceForge was once the place to set up your open source project. Source control hosting, mailing lists, download mirrors, website hosting - if your project needed something you could most likely get it from SourceForge. “Find, create, and publish open source software for free” currently heralds all new users to the site. Everything about the website screams “trustworthy” by virtue of open source.

The problem is that the website can no longer be trusted: they have turned to adding malware and adware to project binaries. We’re seeing the start of projects moving away from SourceForge to more trustworthy competitors - clearly a responsible thing to do, given the circumstances. The question is, does moving away actually achieve anything?

Your Tombstone

A lot of these projects are leaving tombstones over at SourceForge. The existence of the gimp-win tombstone SourceForge’s justification for hijacking the page: “if you don’t care about the project we will continue to ‘maintain’ it for you.” The most obvious solution here is to simply update all your hyperlinks so that they now point at the new host, again, does this actually achieve anything?

For really big and successful projects, probably. They probably have their own web server and deal with their own binary distribution - they haven’t been tied to SourceForge for a very long time.

In the case of SourceForge-hosted binary, updating links is likely to achieve very little. The primary reason is that there is a good chance that the internet is littered with hotlinks directly to your SourceForge download page, instead of your project domain. How does this happen? Let’s take ruby-lang.org as an example:

  1. Click the big “Download Ruby” button.
  2. Presented with a wall of text. Hmmm… “Current stable: Ruby x.x.x” seems about right.
  3. A .tar.gz starts downloading, not what I’m looking for.
  4. Let me try that “Installation” page.
  5. See the “RubyInstaller (Windows)” link and click it.
  6. Oh, looks like I need to click another “RubyInstaller” link.
  7. Click the big “Download” button.
  8. Finally, “ruby.x.x.x.exe” is a clickable thing.

The happy path is 7 clicks. A non-technical user would fail at step 3 and likely YouTube for a tutorial on how to install Ruby. If it’s hard to get to your binaries chances are that 16-year old who made the YouTube video linked directly to your SourceForge download page for the convenience of his viewers. Your bad landing page has earned a kid some advertising revenue and has earned you a hotlink to SourceForge.

That’s just one example of how hotlinks happen, even if your landing page is good (e.g. mozilla.org) chances someone saw the need to hotlink for one reason or another - possibly copying a hotlink that someone else posted. The easiest way to deal with the hotlinks is to simply invalidate them by deleting your project, so does this work?

SourceForge Controls SourceForge

Even if you delete your project, who’s to stop SourceForge from simply re-creating it? They most likely have a backup of the project and can simply re-host it with their adware binaries. Unless you have a legal way to prevent this (which you probably don’t) there’s nothing you can do except waste obscene amounts of time hunting down links and asking webmasters to change them.

Oh, It’s Just SourceForge

You only need to do this once, right?

  • The year is 2007, what is your opinion of SourceForge?
  • The year is 2015, what is your opinion of Google Code?
  • The year is 2020, what is your opinion of CodeHub?

That’s not to say the the SourceForge story is the end of all stories, simply be aware that it is one possible outcome. You’re placing an immense amount of trust in these people. In many cases there is nothing you can do, you have to place to trust in them.

What Can I Do?

Unless you have the means to host your own binaries from the get-go, you can only mitigate this risk:

  • Become authoritative: have a project domain - even if it points at your project host.
  • Reduce hotlinking: user agent sniffing:
    • No more than one click to get the installer for their platform - or -
    • Up-front CLI one-liner to install your thing - or -
    • Up-front instructions on how to install your thing - always -
    • Clearly communicate how to get your official binaries
  • Reduce hotlinking: bring in platform maintainers. If someone is building binaries for a platform that you don’t support yourselves ask them if you can mirror those binaries on the official project website.
  • Reduce tombstones: delete project pages if you leave a host.
  • Maintain: even if all the bugs have been fixed and all the features are in, maintaining a project is never over. Be prepared to make logistical changes, such as moving project hosts.

Writing a socket server in .Net is one of the most rewarding things you can do in .Net. Unfortunately at the same time there is a lot of pre-requisite knowledge – and the knowledge isn’t easy to find online (in fact it is very easy to examples and tutorials that are blatantly wrong).

After reading this you should know enough to write a pretty good socket server; and hopefully where to start looking when bad things happen. This post is written progressively, meaning I may use certain bad practices in the first few sections. Don’t just go and copy and paste – make sure you read the whole thing.

IOCP (I/O Completion Ports)

IOCP is probably the most important part of a socket server. If you have ever heard of the Berkely socket pattern; this is the exact opposite. Berkely relies on a tight loop that queries each socket asking it if it has any data ready; it doesn’t scale. Another often-used (and incorrect) pattern is the multi-threaded Berkely socket pattern. Instead of a single tight loop querying each socket, the server assigns a thread-per-client. While in theory this is a good idea; realise that spinning up 1000 tight loops is going to bring your server to a screeching halt very quickly.

So what is IOCP? Put simply; it’s a way for a program to say to the kernel “Hey, I am in charge of these streams. When data arrives on ANY of them please resume this thread I have created for you.” The .Net BCL team took IOCP a step further and shield developers from having to manage that thread. Using IOCP in .Net is as easy as implementing your server using the Begin/End async pattern. Using this pattern is quite simple:

  1. Create your own Begin* method (as a bad example, BeginReceive).
  2. Create a stub for the End* method. This method should accept a single IAsyncResult as a parameter.
  3. In your Begin* method, call socket.Begin* passing your End* as a delegate parameter.
  4. In your End* method, call socket.End* passing the IAsyncResult you got as a parameter.
  5. Then call your Begin* method.

This sets up an async loop; there is a shorter form of this loop (designed for DRY error handling) – which looks like this:

private Socket _socket;
private ArraySegment<byte> _buffer;
public void StartReceive()
{
    ReceiveAsyncLoop(null);
}

// Note that this method is not guaranteed (in fact
// unlikely) to remain on a single thread across
// async invocations.
private void ReceiveAsyncLoop(IAsyncResult result)
{
    try
    {
        if (result != null)
        {
            int numberOfBytesRead = _socket.EndReceive(result);
            if(numberOfBytesRead == 0)
            {
                OnDisconnected(null); // 'null' being the exception. The client disconnected normally in this case.
                return;
            }

            var newSegment = new ArraySegment<byte>(_buffer.Array, _buffer.Offset, numberOfBytesRead);
            OnDataReceived(newSegment);
        }
        _socket.BeginReceive(_buffer.Array, _buffer.Offset, _buffer.Count, SocketFlags.None, ReceiveAsyncLoop, null);
    }
    catch (Exception ex)
    {
        // Socket error handling here.
    }
}

Note that you should use the async pattern for everything: Accept, Receive, Send, Connect. As a side-note – using BeginSend with a null callback is fine (fire and forget); but it will play havoc with the socket performance counters for your process. You may also want to set UseOnlyOverlappedIO to true. I also suggest you look at Reactive Extensions; as one of the things it deals with is the async pattern.

Use Streams

Don’t use the built-in Receive and Send methods, construct a NetworkStream over the socket. This is important because it allows you wrap your socket in a SslStream or DeflateStream (side note: always wrap the data in compression BEFORE encryption).

DeflateStream was implemented in a really strange way (the existence of the CompressionMode parameter). You are typically going to write a Stream that uses one DeflateStream (in CompressionMode.Compress) to write and another (in CompressionMode.Decompress) to read. Also note that in prior to .Net 4.0 DeflateStream should only be used to compress English/markup as it has a hard-coded dictionary.

Async/Await

Contradictory to the advice above, async and await allow you to use sockets in Berkley style without jeapordizing the scalability of your server. The only caveat is that Socket does not provide native TPL methods and therefore Task.Factory.FromAsync needs to be used.

class Server
{
    // ...
    Socket _socket;
    bool _listening;

    private async void Listen()
    {
        // ...
        while (_listening)
        {
            try
            {
                var client = await Task.Factory.FromAsync<Socket>(_socket.BeginAccept, _socket.EndAccept, null);
                AcceptClient(client);
            }
            catch (Exception e)
            {
                // Log etc.
            }
        }
    }

    private async void AcceptClient(Socket client)
    {
        var buffer = BufferPool.Instance.Checkout();
        try
        {
            using (var ns = new NetworkStream(client, true))
            {
                while (_listening && client.Connected)
                {
                    try
                    {
                        var count = await ns.ReadAsync(buffer.Array, buffer.Offset, buffer.Count);
                        if (count == 0)
                        {
                            // Client disconnected normally.
                            break;
                        }
                        else
                        {
                            OnDataRead(new ArraySegment<byte>(buffer.Array, buffer.Offset, count));
                        }
                    }
                    catch (Exception e)
                    {

                    }
                }
            }
        }
        finally
        {
            BufferPool.Instance.CheckIn(buffer);
        }
    }
}

If possible, I personally use async/await in preference to the async pattern as it typically improves maintainability. The performance difference should be negligible.

The Evils of Pinning

The Microsoft CLR uses P/Invoke to provision sockets. A side-effect of P/Invoke is that any reference types (and a byte array a.k.a buffer is a reference type) that you pass as parameters are pinned for the duration of the call. Pinning is a process whereby the garbage collector is informed that it should not move an object around (the garbage collector moves objects around to ensure that large continuous areas of free space are available). What this means is that if you pass a new buffer to each Send/Receive call you will wind up with a lot of pinned objects, which can lead to fragmentation. There have been stories of processes getting so fragmented that a process using 800MB of memory and 1.2GB of free memory throwing an OutOfMemoryException (because it could not find a large enough space to allocate an object). Remember that an OutOfMemoryException kills your .Net process without ever calling any catch blocks – not good.

The way to get around this is to pre-allocate large blocks of memory for buffers. The way to get around this is to pre-allocate large blocks of memory for buffers. ArraySegment is a good way to get handles on ‘sub-arrays’. Here is an effective buffer pool to get you started:

/// <summary>
/// Represents a buffer pool.
/// </summary>
public class BufferPool
{
    private readonly int _segmentsPerChunk;
    private readonly int _segmentSize;
    private readonly ConcurrentStack<ArraySegment<byte>> _buffers;

    /// <summary>
    /// Gets the default instance of the buffer pool.
    /// </summary>
    public static readonly BufferPool Instance = new BufferPool(
        64,
        4096, /* Page size on Windows NT */
        64
        );

    /// <summary>
    /// Gets the segment size of the buffer pool.
    /// </summary>
    public int SegmentSize
    {
        get
        {
            return _segmentSize;
        }
    }

    /// <summary>
    /// Gets the amount of segments per chunk.
    /// </summary>
    public int SegmentsPerChunk
    {
        get
        {
            return _segmentsPerChunk;
        }
    }

    /// <summary>
    /// Creates a new chunk, makes buffers available.
    /// </summary>
    private void CreateNewChunk()
    {
        var val = _segmentsPerChunk * _segmentSize;

        byte[] bytes = new byte[val];
        for (int i = 0; i < _segmentsPerChunk; i++)
        {
            ArraySegment<byte> chunk = new
                ArraySegment<byte>(bytes, i * _segmentSize, _segmentSize);
            _buffers.Push(chunk);
        }
    }

    /// <summary>
    /// Creates a new chunk, makes buffers available.
    /// </summary>
    private void CompleteChunk(byte[] bytes)
    {
        for (int i = 1; i < _segmentsPerChunk; i++)
        {
            ArraySegment<byte> chunk = new
                ArraySegment<byte>(bytes, i * _segmentSize, _segmentSize);
            _buffers.Push(chunk);
        }
    }

    /// <summary>
    /// Checks out a buffer from the manager.
    /// </summary>
    /// <remarks>
    /// It is the client's responsibility to return the buffer to the manger by
    /// calling <see cref="CheckIn" /> on the buffer.
    /// </remarks>
    /// <returns>A <see cref="ArraySegment{Byte}" /> that can be used as a buffer.</returns>
    public ArraySegment<byte> Checkout()
    {
        ArraySegment<byte> seg = default(ArraySegment<byte>);
        if(!_buffers.TryPop(out seg))
        {
            // Allow the caller to continue as soon as possible.
            var chunk = new byte[_segmentsPerChunk * _segmentSize];
            var action = new Action<byte[]>(CompleteChunk);
            action.BeginInvoke(chunk, x => action.EndInvoke(x));
            // We have the buffer at the start of the chunk.
            seg = new ArraySegment<byte>(chunk, 0, _segmentsPerChunk);
        }

        return seg;
    }

    /// <summary>
    /// Returns a buffer to the control of the manager.
    /// </summary>
    /// <param name="buffer">The <see cref="ArraySegment{Byte}"></see> to return to the cache.</param>
    public void CheckIn(ArraySegment<byte> buffer)
    {
        if (buffer.Array == null)
            throw new ArgumentNullException("buffer.Array");
        _buffers.Push(buffer);
    }

    /// <summary>
    /// Constructs a new <see cref="BufferPool" /> object
    /// </summary>
    /// <param name="segmentChunks">The number of chunks to create per segment</param>
    /// <param name="chunkSize">The size of a chunk in bytes</param>
    public BufferPool(int segmentChunks, int chunkSize) :
        this(segmentChunks, chunkSize, 1)
    {

    }

    /// <summary>
    /// Constructs a new <see cref="BufferPool"></see> object
    /// </summary>
    /// <param name="segmentsPerChunk">The number of segments per chunk.</param>
    /// <param name="segmentSize">The size of each segment.</param>
    /// <param name="initialChunks">The initial number of chunks to create.</param>
    public BufferPool(int segmentsPerChunk, int segmentSize, int initialChunks)
    {
        _segmentsPerChunk = segmentsPerChunk;
        _segmentSize = segmentSize;
        _buffers = new ConcurrentStack<ArraySegment<byte>>();
        for (int i = 0; i < initialChunks; i++)
        {
            CreateNewChunk();
        }
    }
}

Note that you can also use BufferManager (which doesn’t support ArraySegment).

Unfortunately this one is actually really hard to get 100% right in the scenario where you do use SslStream and DeflateStream. These two streams allocate their own buffers; which means even if you do write to the targetStream using pooled buffers, unpooled buffers will still be sent to the underlying socket. If you do run into this scenario you are going to land up writing quite a bit of code (subclassing Stream; placed directly above your NetworkStream) that swaps out the buffers passed to it for pooled buffers (keeping in mind you need to do this async).

There are also two ways to manage these pooled buffers. If your server behaves in a streaming fashion (which is probably the case) you will want to experiment with allocating a buffer to each client; instead of a buffer to each Read/Write call. Whether this will work out better or not depends on the nature of your server.

DOS

Denial-Of-Service (not that I didn’t mention DDOS; pretty-much the only way to deal with that is horizontal scaling) is one you need to look out for. DOS is really simple to pull off if you don’t protect yourself against it. It merely comes down to asking “why is this client connected to me?”, “can I handle this much data?” and for the love of God staying away from byte arrays (except for, obviously, at the socket level). For instance, what is a client doing connected to you if it has not authenticated itself in 30 seconds? Why is this client sending you 5GB of data? Can the data be streamed to disk instead?

PUSH Parsing

Make sure that your server can PUSH parse. One form of DOS is to send a really big (but valid) packet. Naïve servers will build up a large buffer in memory to attempt to store this packet before parsing it - potentially overallocating and crashing the process. Don’t do this, instead parse the bytes as you get them. For instance instead of using a StreamReader look at the following:

private Encoding _encoding;
private Decoder _decoder;
private char[] _charData = new char[4];

public PushTextReader(Encoding encoding)
{
    _encoding = encoding;
    _decoder = _encoding.GetDecoder();
}

// A single connection requires its own decoder
// and charData. That connection should never
// call this method from multiple threads
// simultaneously.
// If you are using the ReadAsyncLoop you
// don't need to worry about it.
public void ReceiveData(ArraySegment<byte> data)
{
    // The two false parameters cause the decoder
    // to accept 'partial' characters.
    var charCount = _decoder.GetCharCount(data.Array, data.Offset, data.Count, false);
    charCount = _decoder.GetChars(data.Array, data.Offset, data.Count, _charData, 0, false);
    OnCharacterData(new ArraySegment<char>(_charData, 0, charCount));
}

If you need to build up the packet before parsing it (say it’s XML and you don’t want to write/test your own PUSH parser) rather write it to a temporary file.

Hope This Helps

If you can think of anything I should add or correct please let me know, or fork this post and send a pull request.

Overview

What Is Symblr?

Symblr is a symbol and source server that I’m writing. It’s designed to make symbol maintenance as painless as possible for people who care (or should care) about having reliable symbols around.

Uh, What Are Symbols?

Something that you are oblivious about until you need them. When you have them and you don’t need them you are going to be in a very sore place.

Symbols, also known as “debug symbols,” are used to instruct a debugger on how it should interpret the memory that your program has manipulated at runtime. In other words, if you have ever used an interactive debugger (such as Visual Studio) symbols have made the following features possible:

  • Determine what the method names in the call stack are, instead of function addresses.
  • Inspect data structures and view fields, instead of raw memory.
  • Set breakpoints on individual lines by interacting with your code editor, instead of having to know the method address.
  • Determine exactly which statement/expression is executing, instead of only having the instruction pointer.
  • All of the above when inspecting a memory dump.

And a Source Server?

This is a feature that Microsoft symbols (PDB) can optionally implement. In addition to providing all the services that symbols provide, Microsoft symbols can also contain information on how to acquire the original source code that was used to build the binary that is being debugged. One example of a source server is the “.Net Framework Source Stepping” feature. When this feature is enabled Visual Studio will use a special symbol server to acquire symbols for .Net assemblies. These symbols contain source information and Visual Studio transparently downloads the source code if you attempt to step into .Net Framework methods.

Should I Care?

Why Are Symbols Important?

Working at an ISV I’ve been forced to appreciate how important a good symbol server is. My original lesson was a customer who was having some issues with our software. You can’t just ask a customer if you can install Visual Studio on their production server, remote in and start stepping through code. They’ll laugh at you and rightfully so. You’re going to have to rely on memory dumps.

So what you do is ask for a memory dump, open up WinDBG and think your going to start finding out what the issue is. What you are instead going to find out is how important reliable symbols are, as you are greeted by method addresses and raw memory. You’re going to have to get really creative with hunting down the issue.

Why Is a Source Server Important?

It nearly always isn’t. It is a nice-to-have the majority of the time.

One situation that I’ve found it important is when working with QA’s machines. Every night our CI server builds an installer/setup package that gets installed on QA’s environment the next day. Our installer doesn’t have our symbols because symbols can be quite big, often significantly larger than their associated binary.

Sometimes you run up against one of those environmental issues that you simply can’t reproduce on your development environment and you’re forced to ask QA if you can debug on their environment. In the absence of a source server you are going to have to get the code you want to debug and build it on the machine. QA is very likely to laugh you off. Building code on a QA environment is an incredibly bad idea because it can result in false negatives (bugs that don’t exist in release code) or false positives (bugs that don’t exist in debug code). It can also potentially outright break the environment, resulting in lost QA time.

With a source server all you need to do is point Visual Studio at it and you will be stepping into code in moments.

Does This Apply to Me?

No matter what type of developer you are, without exception: yes.

  • I’m my own customer, my code runs on environments that I control.
    • Examples: Saas, an API
    • Do you have the symbols for the code that is running in production, right now?
  • My customers are always running my latest stable code.
    • Examples: consultant, freelancer, automatic/required updates
    • Do you have the symbols for the binaries that your customers have, right now?
    • You do? What if you patched last night you are looking at a memory dump from yesterday?
  • I maintain different versions of my product.
    • Examples: ISV
    • You need a symbol server, yesterday. For now set up a UNC share and script it.
  • I use Linux/platform X.
    • I don’t know how your platform deals with symbols but I hope Symblr will support it at some point in the future.
    • You should find out how to archive symbols on your platform right now.

A Solution

What I’m looking for is something that:

  • Is compatible with the SymbolSource publishing model.
  • Has a scripted publishing model (e.g. for your CI server).
  • Has retention policies:
    • e.g. Daily builds don’t need to be kept around forever.
    • e.g. Product releases and hotfixes do.
  • Can be deployed behind a firewall.
  • Can act as a caching symbol and source proxy.
  • Can be hosted on any platform where .Net exists, not just Windows.
    • Completely trivial to set up, if the above condition holds.

Which is what you can expect from Symblr.

Show Me The Source

It’s up on GitHub. At the time of writing this post it’s not yet ready for contributors, nor is it even complete. What I have done is research and correctly implement PDB reading/writing (which I will document in another post).