So, I’ve been programming a lot in D lately, which means systems programming, which means pointers! Pointers are amazing. You can do all sorts of crazy and awesome hacks with them. C# and the .NET Framework try to abstract pointers away from us. But fear not, for there are ways around this safety nonsense!

It turns out that on Mono, you can pin an object even if it contains managed data. This makes it safe to modify an object’s internal layout even in the presence of a moving GC (well, kinda, sorta). You can’t take the address of the object, however, which means we must resort to some IL hackery.

Thus, the code:

using System;
using System.Reflection.Emit;
using System.Runtime.InteropServices;

public unsafe delegate void PointerAction(byte* pointer);

public static class ObjectExtensions
{
    private static object _lock = new object();
    private static Func _addrOf;

    private static IntPtr AddressOf(object obj)
    {
        lock (_lock)
        {
            if (_addrOf == null)
            {
                var dm = new DynamicMethod(string.Empty, typeof(IntPtr), new[] { typeof(object) },
                                           typeof(ObjectExtensions), true);

                var il = dm.GetILGenerator();

                il.Emit(OpCodes.Ldarg_0);
                il.Emit(OpCodes.Conv_I);
                il.Emit(OpCodes.Ret);

                _addrOf = (Func)dm.CreateDelegate(typeof(Func));
            }
        }

        return _addrOf(obj);
    }

    public static unsafe byte* ToPointer(this object obj)
    {
        if (obj == null)
            throw new ArgumentNullException("obj");

        GCHandle handle;

        try
        {
            handle = GCHandle.Alloc(obj, GCHandleType.Pinned);

            return (byte*)AddressOf(obj).ToPointer();
        }
        finally
        {
            if (handle.IsAllocated)
                handle.Free();
        }
    }

    public static unsafe void WithPointer(this object obj, PointerAction action)
    {
        if (obj == null)
            throw new ArgumentNullException("obj");

        if (action == null)
            throw new ArgumentNullException("action");

        GCHandle handle;

        try
        {
            handle = GCHandle.Alloc(obj, GCHandleType.Pinned);

            action((byte*)AddressOf(obj).ToPointer());
        }
        finally
        {
            if (handle.IsAllocated)
                handle.Free();
        }
    }
}

(Kudos to ki9a on #mono for coming up with the dynamic method IL!)

Now, don’t try to run that on Microsoft .NET. It will blow up.

Now that we can get at the internals of objects, let’s try it out:

static class Program
{
    static unsafe void Main()
    {
        var str = "foo!";
        object obj = new { foo = 0, bar = (object)null }; // create an object with an int32 field and a reference field

        void* vtable = null;
        void* sync = null;
        int length = 0;
        void* chars = null;

        str.WithPointer(ptr =>
        {
            vtable = *(void**)ptr;
            sync = *(void**)(ptr + sizeof(void*));
            length = *(int*)(ptr + sizeof(void*) * 2);
            chars = *(void**)(ptr + sizeof(void*) * 2 + sizeof(int));
        });

        obj.WithPointer(ptr =>
        {
            *(void**)ptr = vtable;
            *(void**)(ptr + sizeof(void*)) = sync;
            *(int*)(ptr + sizeof(void*) * 2) = length;
            *(void**)(ptr + sizeof(void*) * 2 + sizeof(int)) = chars;
        });

        Console.WriteLine((string)obj); // prints "foo!"
    }
}

Oh boy. Some explanation is probably in order.

First of all, let me explain the layout of objects in Mono. Every object has a header consisting of two machine words. The first word points to the object’s VTable, which describes its type, methods, etc. The second word holds the so-called synchronization block, which is used to implement monitors (C# lock). Now, strings have two extra fields: One 32-bit field which holds the string’s length, and one machine word field which points to the raw characters backing the string.

Now look at this line:

object obj = new { foo = 0, bar = (object)null }; // create an object with an int32 field and a reference field

What we’re doing here is creating an anonymous object with a physical structure similar to that of a string. Needless to say, whether it actually is similar is completely implementation-defined. The runtime may decide to reorder fields as it sees fit, and may even make the foo field a 64-bit integer on a 64-bit system. But, what’s important here is that the resulting object has enough space to hold the contents of a string object.

Now we get to the fun stuff:

str.WithPointer(ptr =>
{
    vtable = *(void**)ptr;
    sync = *(void**)(ptr + sizeof(void*));
    length = *(int*)(ptr + sizeof(void*) * 2);
    chars = *(void**)(ptr + sizeof(void*) * 2 + sizeof(int));
});

Here, we read out the VTable, synchronization block, length, and character array fields of the “foo!” string object. It is (more or less) safe, since the WithPointer method ensures that the object is pinned while we do our stuff.

Next:

obj.WithPointer(ptr =>
{
    *(void**)ptr = vtable;
    *(void**)(ptr + sizeof(void*)) = sync;
    *(int*)(ptr + sizeof(void*) * 2) = length;
    *(void**)(ptr + sizeof(void*) * 2 + sizeof(int)) = chars;
});

Here we assign the values we read out from the string object earlier to the object we created with a similar physical layout to that of string objects. Our anonymously-typed object is now effectively a string.

Let’s prove that:

Console.WriteLine((string)obj); // prints "foo!"

Note two things here: First, the cast succeeded. It would normally have resulted in an InvalidCastException. So far so good. Second, the call actually prints what we expect. This means that we got the physical layout of the string object right.

Note that all of this was done on an x86 machine running 64-bit Linux with Mono 2.10.8. I have no idea whether it will work with any other Mono version, architecture, bitness, or operating system.

Will this break the runtime? Possibly! Will the GC explode? Likely! Will it leak memory? Definitely!

Don’t do this in production code. Like, seriously, don’t.

Update: As lupus points out, I actually got the string layout wrong! The characters of a string are embedded directly in the object. This code just turned out to work because of word size on 64-bit x86. I’ll leave it as an exercise to the reader to adapt the code to arbitrary amounts of embedded characters. ;)

Note that our branch is for version 2.0 of the GitHub API; the author of GithubSharp, Erik Zaadi, is now working on GithubSharp for GitHub’s 3.0 API, using ServiceStack.Text.

First, I’ve just become an intern at Xamarin. I’ll be working on the documentation build system and integration into the website and the MonoDevelop IDE. I was hired after GSoC was over, following a recommendation from Miguel. I’m really looking forward to working with Xamarin; it’s an awesome team of skilled hackers.

Second, I’ve been accepted into university, starting February 1 (at the UCN). The Danish name for the course I’m taking is “datamatiker”. They call it computer science in English, but it’s actually much more practically oriented than that (i.e. actual software development rather than theory). I was accepted based on my ability in the software development field, as I don’t fulfill the formal requirements (yet). This means I’ll probably quit high school, as I’d rather pursue something software-related than natural sciences.

In other (less life-related) news, I’ve also been putting some time into writing a compiler infrastructure library. I’ll be posting more about the goals and directions of that later.

First of all, the project is not completely done. I had anticipated this early on, and discussed with JB which parts I should focus on for the deadline. I did get a fair bit done for the deadline, but I wouldn’t quite say it’s ready for production use yet. I did, however, pass the final evaluation, as JB agreed to let me finish the project outside of GSoC (which I fully intend to do).

The project actually ended up being more involved than originally planned. The idea was to simply swap the PEAPI back end with Cecil, but I quickly realized that this was not quite as trivial as it seemed back then. I ended up ripping out the old code generation completely, emptying the code of every last parser production, updating the parser to .NET 4.0, and finally, adding the Cecil code generation back end in. You might think that all of this would have taken most of the entire GSoC period, but a lot of time was spent on spec reading too, not to mention figuring out weird quirks and corner cases in the Microsoft implementation of ILAsm. It’s scary just how much of the ILAsm language is poorly documented or not documented at all.

I’d like to extend my thanks to everyone in #cecil @ irc.gnome.org for helping me understand ILAsm, CIL, and the CLI in general.

I’ve also made some progress on the ILDasm front. It’s still missing a lot of features, but it does disassemble types and methods decently at this point. I’ll work more on it later; ILAsm comes first.

I recently switched ILAsm to a deferred type resolution model. This was JB’s idea, as a way to solve the generic parameter issue I blogged about earlier, and it’s worked nicely so far. This was probably the last major issue in the new ILAsm, so I expect to be doing more ILDasm work soon (there are only a couple of known ILAsm issues left).

During GSoC I also wrote a command line interface to the Mono soft debugger (SDB). It’s available here. I wrote it primarily because using MonoDevelop (which was the only existing interface to SDB at the time) as a debugger for a command line application was not very convenient. This debugger also has a few cool features such as decompilation of code with no source code (through ICSharpCode.Decompiler).

Overall, it’s been a very fun and educational program for me. I now know a lot of things about the CLI that I didn’t have the slightest clue about before, and I finally managed to contribute something really significant to Mono. Working with JB and the Mono community has been an awesome experience, as everyone’s very helpful and easily approachable. I would definitely do this again if I get the chance.

I’ll probably make a final blog post whenever I get this entire project merged into mainline Mono.

First of all, we have generic parameters. In MS.NET’s ILAsm, !0 and !!0 are simply emitted as VAR 0 and MVAR 0 in the metadata, meaning that you could do something like:

.method public static !!0 Foo(!!1 bar)
{
    ldarg bar
    ret
}

Even though !!1 refers to a generic parameter that does not exist, MS.NET’s ILAsm will gladly emit it, and not even warn you. Due to an API “limitation” in Cecil, Mono’s ILAsm cannot do this. All GenericParameter objects must have an owner (i.e. the type or method definition/reference) and must be contained in that owner’s GenericParameters collection at the correct index. Therefore, Mono’s ILAsm would throw an error for the above code, since !!0 is out of bounds. This is a slightly annoying incompatibility, but there’s not much I can do about it other than patching Cecil. That’s only going to happen, though, if it won’t cause API breaks for version 1.0.

Second, properties like .file alignment, .imagebase, and .stackreserve aren’t fully supported. Cecil currently has no way of setting these on a module. ILAsm will check the values and error if they’re invalid, though.

Third, several native and variant types are not supported in marshal signatures. These include variant, void, syschar, decimal, date, objectref, nested struct and null, void, int64, uint64, unsigned int64, lpstr, lpwstr, safearray, hresult, carray, userdefined, record, filetime, blob, stream, storage streamed_object, stored_object, blob_object, cf, clsid, as well as pointers/references/vectors of these. The reason is that Cecil doesn’t expose any way to set these types. Either way, most of these are deprecated nowadays. All other native/variant types are supported.

Fourth, support for .data declarations is very limited (we actually only do a weak attempt at emulating them). Cecil has no way of emitting data constants, and they’re implementation-defined anyway. Currently, the only thing we do with them is copy them to the InitialValue property of field definitions where appropriate. This is far from correct, but it’s the best we can do. Either way, you should avoid using these declarations. They don’t make a whole lot of sense in managed land.

Fifth, we have no support for parsing System.Reflection-notation strings (yet). This means that things like custom marshalers aren’t supported (the syntax will simply be ignored). A type parsing API will supposedly be exposed in Cecil 1.0.

Sixth, support for declarative security syntax is limited. While we support the full syntax specified by the standard, we don’t support all of the syntax MS.NET’s ILAsm does (specifically, the syntax resembling verbal custom attribute initialization).

Seventh, exported types are unfinished because Cecil lacks a way to manipulate the File table directly, and because of the lack of a type parser.

Eighth, we can’t emit custom attributes on manifest resources and assembly references. This is a Cecil limitation which should be relatively easy to fix.

Lastly, the .vtable/.vtfixup/.vtentry directives are unsupported. Again, these will be implemented once Cecil has support for them.

There are some other incompatibilities, but they’re very subtle and you’re unlikely to ever encounter them. And even if you do, ILAsm will warn you.

Generally, the differences between the two implementations will only be encountered in obscure features that most users are highly unlikely to be using, or when attempting to use nonstandard syntax. Rule of thumb: Stay away from these things and you’re safe.

A good while back, I blogged about how bytecode is evil. This post will explain what I intend to do with the AST idea.

The project I’m working on consists of five major components:

• A new executable format consisting of binary-form ASTs
• A managed JIT compiler
• A new assembly-like intermediate language
• An abstract bare-bones type system
• An AST/assembly optimization layer

The new executable format will be similar to the CIL format, but without all the PE bloat, and with one major difference: Code is stored as binary-serialized ASTs rather than bytecode. This allows easy introspection of compiled code, opening the doors for many interesting scenarios, such as translation from one language to another at runtime. I’ve chosen to call the format CAST (Compiled Abstract Syntax Trees).

To make the AST format actually interesting, a new JIT compiler needs to be implemented. This JIT compiler will understand the AST and operate on it for things such as optimization. For compatibility reasons, the AST will have a sort of RawCodeNode that contains plain assembly language (something similar to CIL but with registers). This is useful when one wants to emit non-AST code such as CIL/ILAsm, Java bytecode, Erlang BEAM, etc. I’ve chosen to call this JIT compiler Mono.Jit.

The new assembly-like intermediate language will be similar in nature to CIL. However, unlike CIL, this language will be based on registers and a stack, rather than locals and a stack. While the CIL approach is generally much prettier here, it’s less abstract and in some cases it is impossible to translate code to CIL (such as x86 machine code). I’m not sure what to call this yet, so for now, I’ll refer to it as IAL (Intermediate Assembly Language).

For CAST, I’ll eventually need to implement a compiler/decompiler, but for now, plans are to create a fluent API to create CAST code from .NET land. For IAL, I’ll just make a simple assembler/disassembler.

Now, in order for the AST to have any sort of meaning, a type system is needed. The type system in Mono.Jit is extremely simple; it supports only fields and methods on types. The JIT doesn’t need to know anything else in order to emit code. I do, however, plan to make an interface that allows giving the JIT optimization hints (such as “this type is completely immutable”, “this method makes no modification to members”, etc.). The JIT’s type system currently supports generics (types and methods), but I’m strongly contemplating delegating this to code loaders unless there is a compelling reason for the JIT to be aware of them.

Lastly, an optimization layer needs to be written. This layer will primarily concern itself with the AST input to the JIT, because that’s the entire point of this project. As mentioned, IAL is just a fallback for when you don’t have an AST to feed to the JIT. I don’t have any particular plans for optimization of IAL other than simple stuff like constant folding/propagation.

That’s it for now. More details will come as I write more code.

This post will be about the ILAsm standard (specified in ECMA 335) and the Microsoft implementation. There are several annoying differences and incompatibilities between the two. As part of figuring them out, I’ve been reading Expert .NET 2.0 IL Assembler and the Microsoft-annotated ECMA 335 standard.

The first difference is that MS.NET’s ILAsm does not respect the unsigned modifier on integer types. This means that an unsigned int32 will in effect be treated as an int32 (signed). I’ve chosen to break compatibility with Microsoft and correctly emit an unsigned integer, primarily because I think this is an outright stupid bug. Note that you can still get an unsigned integer by using uint32 with MS.NET. The latter form is preferred, anyway.

Second, MS.NET does not have the platformapi keyword. Instead, you have to use the winapi keyword, which makes your ILAsm source code unportable. In Mono’s ILAsm, we support both, in order to aid portability.

Third, MS.NET allows #line instead of .line for specifying source line information. This seems to be for compatibility reasons. Mono’s ILAsm supports both notations.

Fourth, MS.NET has a bunch of directives that just don’t exist in the standard. Specifically .file alignment, .imagebase, .language, and .namespace. We support all of them, though the first two currently have no actual effect. Note that use of .namespace is considered bad practice.

Fifth, MS.NET’s ILAsm doesn’t use .culture for specifying culture information, but rather .locale. This is directly against the standard, and the assembler won’t even recognize .culture. Mono’s ILAsm supports both notations.

Sixth, the MS.NET ILAsm doesn’t require the .hash directive; if it’s not specified, the hash will automatically be computed.

Seventh, the MS.NET ILAsm allows using “value class” in place of “valuetype” for indicating value types. This is relatively easy to handle, and Mono’s ILAsm does so. It is, however, considered bad notation.

Lastly, the MS.NET ILAsm allows specifying things like calling convention, type attributes, method attributes, field attributes, parameter attributes, and so on using flags(int32) notation. I’m sure this was done for a reason, but it seems like a great way to give developers the opportunity to make their programs unportable or impossible to run at all.

There are a lot of other incompatibilities, but they’re less annoying and easier to work around/implement.

That’s it for the this post. Next time: Implementation details of Mono’s ILAsm.

My friend, Wolfgang Steffens, who’s the primary developer on SL#, also had a GSoC project accepted, which seeks to incorporate SL# in the Axiom game engine. This would be a great way for us to draw more attention to the library, and also makes shader development for Axiom users much easier.

There are a lot of other interesting projects going on with Mono this year – check them out here.

Another bug we encountered (which is only relevant for SL# master) was invalid IL generation, which has also been fixed. Curiously, Microsoft’s CLR actually executed the IL correctly, even though the local variable had an incompatible type… Mono was much more helpful here, in that it threw an InvalidProgramException. As said, the latter issue is only relevant for master, not 1.4.

We might release a 1.4.1 package of SL# in a few days that addresses the first issue. If you don’t want to wait, grab this commit and apply the CancellationToken workaround.

Edit: Oh, and this seems to be my 100th blog post! Yay!

This is probably the release with the most changes that we’ve done so far:

• Full support for all GLSL data types (including matrices)
• Support for in and inout through C#’s ref and out keywords
• Switched to ICSharpCode.Decompiler instead of our homegrown one
• Support for many more language constructs due to switching decompiler engine
• Support for hundreds of GLSL functions that we didn’t have before
• Shader.DebugMode now actually works as expected
• Better type checking/verification when translating
• We now expose a sane standalone API (GlslTransform) for GLSL translation, should you want to use it
• We now only ship one assembly, IIS.SLSharp.dll
• Lots of cleanups to the public API; we no longer expose any internals
• Lots and lots of bug fixes

That’s a fairly impressive list of changes if you compare to our previous ones! We’ve been incredibly productive with SL# lately, and GitHub certainly has been a booster, with its new intuitive Issues 2.0.

As usual, the Git repository sits here (note that master is quite ahead of this release). Downloads can be found here, and the updated NuGet package has been published.

If you run into any issues, please report them on the issue tracker as always.

Lastly, the reason we chose to release 1.4 so quickly is because we’re doing heavy work on making SL# engine-agnostic. This means that 1.4 will be the last release that depends on OpenTK. Future versions (2.0 and on) will support various engines (OpenTK, Axiom, XNA, SlimDX) as well as several languages (GLSL, HLSL, Cg) through bindings. This new architecture should also make it easier to add more native-feeling support for F# (through function quotations). This of course means that we’ll be doing massive (breaking) changes and refactoring to our API and possibly even our syntax, hence the major version bump.

With that said, enjoy!