Tag Archives: XPCOM

My previous post already explained how to embed and control a VirtualBox Virtual Machine in a C#/.Net application on Windows. That was too easy, and it’s really less fun if it does not work cross platform, so the next challenge was to make it work on Linux, with Mono as the .Net runtime. A few people have been working on this before, but that was a long time ago and I bet they never got this far. Nevertheless, they were the ones to implement XPCOM interop in mono, making my experiment a lot easier.

On Linux, VirtualBox uses XPCOM as it’s component model, to replace (MS)COM. The two appear to be essentially the same; they both use vtables, interfaces and IUnknown. The two are *almost* (binary) compatible. An important difference is in the object creation or ‘activation’. (MS)COM relies on a system wide database (somewhere deep in the Windows registry) to track all known Objects, and from there provides a platform API function to create an object instance. XPCOM, the Cross Platform Component Object Model, does not provide such a feature. Combined with some missing features and a lack of proper documentation, this resulted in slightly different implementations being in use today. Other, perhaps even more important differences are in how some data types are represented (especially arrays and strings), and the lack of IDispatch support in (VirtualBox) XPCOM. The last one is a real problem, as it changes the layout of *every* interface definition, making the Interop assembly generated by Visual Studio .Net useless on Linux, and making it very difficult if not impossible to support both COM and XPCOM with one codebase.

The object activation in VirtualBox is quite simple. The VBoxXPCOMC.so library provides a function named VBoxGetXPCOMCFunctions, which simply returns a structure with a bunch of function pointers (which looks much like a COM vtable). By calling the appropriate functions, one can easily activate the needed XPCOM objects.

To work around the missing IDispatch support in the XPCOM interfaces, the Interop assembly needs to be changed. I ended up writing some C# code which can automatically generate appropriate interface definitions based on the VirtualBox.xidl file (available in the SDK package and in the SVN repository, note that it is important to use the xidl file that corresponds to the VirtualBox version you want to use, as the interface definitions do change over time!). This also provides an opportunity to work around the differences in how strings are represented. Arrays are still troublesome, as VirtualBox internally represents one Array parameter as two parameters (size and a pointer to the first element). Such an array can be ‘marshaled’ as an UnmanagedType.LPArray, but having two parameters instead of one makes the definition incompatible with the one generated by VS.Net (and used on Windows/MSCOM).

So what’s the deal if it is not compatible? Well, if the interface definitions define the same managed interface members, one interop assembly can serve as a drop in replacement for the other. Note that only the functions we use need to match, and the order in which they are defined or their attributes are not relevant. One can write code in VS.Net while using the COM interop assembly, and run the same code on Linux with the XPCOM interop assembly. It’s even better: the framework only loads an assembly when it *needs* a type defined in it, and only if the type is not already available. So if we detect a Linux system, we can simply load the XPCOM interop assembly, before the framework loads the COM interop assembly, and the frontend code won’t know the difference! This is also the reason for using multiple stages in the program initialization (ProgramLoader.Main, ProgramLoader.Main2, ProgramRun.Run, and the Object type of ProgramRun.WaitingForProgress) – to prevent the runtime from loading the assembly before we have had a chance to load the alternative one.

So, arrays are still unsupported in the XPCOM interop code. We don’t really need them much, yet. Currently, only the VBoxEventListener uses an array parameter, and uses an ugly hack for this purpose: it defines an alternative version of the interface it wants to use, where the function it wants to call has two parameters for the array (size and pointer). It then decides which version of the interface to use based on the OS platform. Perhaps a way to properly marshal the arrays already exists, or could be created using custom marshaling. Alternatively, wrapper classes could be created for all COM interfaces, to convert the function parameters appropriately, essentially re-implementing the COM interop/proxying. The use of custom wrapper classes is probably the most flexible one as it gives full control over the marshaling.

Another near-showstopper is what I think is a bug in the Mono runtime: Mono releases COM interface pointers, which it receives as a result of a COM function call, twice. Once in the Interop calling code, and once while marshaling the interface pointer to a .Net interface. This results in crashes while accessing methods and properties which return an interface type. The fix for this is very simple, and will hopefully find it’s way to the Mono distribution soon. Alternatively, the custom COM wrapper classes can be used to work around this issue and avoid the Mono (XP)COM interop code altogether.

On Linux, most users do normally not have ‘root’ privileges, but root privileges are needed to access the VirtualBox kernel driver. VirtualBox handles this by making it’s executables ‘setuid root‘, meaning that any user can run the executables with root privileges. By itself, this would be a gaping security hole, so normally VirtualBox is compiled with ‘hardening’: a wrapper application is started as root, opens the kernel driver, drops all privileges (by calling setuid and setgid) and then loads a dynamic library containing the actual application.

This approach works very well as long as all executables and libraries involved are known, but we can’t easily add a new frontend application without recompiling all of the VirtualBox code. One hackish way is to replace one of the existing shared libraries, for example VBoxBFE, with our own version which will fire up mono. This approach works, but still requires modifications to the VirtualBox installation. Note that the mono engine has to be loaded as a library, as the kernel driver will be closed on exec(ution) of another executable.

Another approach is to start mono as root, with the VBOX_USER_HOME variable set correctly. Then, after starting the VM, the application can drop it’s privileges. This works, but will result in files owned by root appearing in your VirtualBox data directory, possibly resulting in trouble when you want to use the standard frontends later on. Also, VirtualBox may not be able to clean up some unix sockets and pipes which it has created as root. And last, the GTK GUI library, which is used by the Windows.Forms implementation, does not like to be started as root, so the GUI can only be created after the VM has been started. To make it all a bit more friendly, it’s possible to create a ‘setuid root’ wrapper application, which simply executes mono (or loads mono as a library).

Luckily, VBoxXPCOMC.so also exports the hidden RTR3InitAndSUPLib function. This function can open the kernel driver, without starting a VM. This function can be called from the C# code, before dropping privileges, and the application can then safely create the GUI and start the VM. One problem here is that mono and the .Net code still needs to be started as root, which might be dangerous if we don’t properly lock down what code mono will run. An easy way to get around this, is to create a wrapper application in C, which initializes the kernel driver, drops privileges and then runs the .Net code. According to the VirtualBox team, it should be very difficult to abuse the kernel driver access for anything other than hanging the host system, so this should be pretty safe, although you should still be careful.

And again, the sourcecode for the complete project, including Linux support, is available in my Mercurial repository at http://oss.ucis.nl/hg/vboxdotnet/.