OpenGL Picking with the Tao Framework

Disclaimer: This is an old post, which I am migrating from my old blog, for the sake of preserving it.  It originally was posted on June, 30, 2008.

I’ve been playing around with the Tao Framework quite a bit, and I found that I had serious issues trying to figure out how to do “picking” or “selecting.” My goal was to create a 3D isometric tile map-editor (in short, a tilted-perspective map editor for a game).

There’s always the option of “drawing” under selection rendering mode — but this is ugly and complex, and didn’t work very good. The real trick that did the job, was to actually use GLUT (OpenGL’s Utility Library) to find a pixel and read the depth buffer (which had been set after the prior scene was rendered) and determine the spatial coordinates. In C++, this is an easy task, but in C#, this is a little more complex, since the Tao developers lurk in the dark at night and seldom come out into the light of day to write a tutorial or two. So, I’ve managed to figure out (after a month, tearing out of hair, and nearly a bit of seppuku tossed in the mix) how to properly do “Picking” in GL. I’m very pleased with the results, as should the Gods of GL (who roam the interweb and all GPUs).

//Variable Declaration
double Output_X, Output_Y, Output_Z;
double[] ModelviewMatrix = new double[16];
double[] ProjectionMatrix = new double[16];
int[] Viewport = new int[4];
float[] Pixels = new float[1];

//Used for some of the messy pointer work.
IntPtr PixelPtr = System.Runtime.InteropServices.Marshal.AllocHGlobal( sizeof( float ) );

//Grab Information about the Scene in OpenGL
Gl.glGetDoublev( Gl.GL_MODELVIEW_MATRIX, ModelviewMatrix );
Gl.glGetDoublev( Gl.GL_PROJECTION_MATRIX, ProjectionMatrix );
Gl.glGetIntegerv( Gl.GL_VIEWPORT, Viewport );

//Find the Depth and store it in a conventional manner with C# in mind
Gl.glReadPixels( e.X, Viewport[3] - e.Y, 1, 1, Gl.GL_DEPTH_COMPONENT, Gl.GL_FLOAT, PixelPtr );
System.Runtime.InteropServices.Marshal.Copy( PixelPtr, Pixels, 0, 1 );
System.Runtime.InteropServices.Marshal.FreeHGlobal( PixelPtr ); //yes, free the memory

//Finally grab the actual X, Y, and Z from all the data we have
Glu.gluUnProject( (double) e.X, (double) ( Viewport[3] - e.Y ), (double) Pixels[0], ModelviewMatrix, ProjectionMatrix, Viewport, out Output_X, out Output_Y, out Output_Z );

Let’s dissect this a little.

Each matrix in GL is a 4×4 matrix, so I initialize the storage for the matrices as a 16-length double array. The viewport is quite simply, just four variables, but we want to store them all in one continuous array, so that’s why the Viewport is stored as a 4-length integer array. And finally, the Pixels array is necessary for C# and it’s weirdness with typecasting code. This code is all technically “safe” because of how I use the Marshal InterOp code (also note the fact that I built a PixelPtr to associate Pixels to a pointer).

The next 3 methods grab all of the data necessary to the matrices for projection, etc, and the display settings. Pretty simple and straight-forward.

Now, we need to figure out what the “depth” is at the current location. Notice that the code has literally been pulled out of a MouseMove event, so e.Y and e.X refer to the mouse’s X and Y coordinates. The depth is stored in PixelPtr. Note that to get the Y coordinate in correct terms, I subtract the Mouse Y from the size of the Viewport.

Now, I use the Marshal to copy data back over to the Pixels pointer. Very simple, but a pain-in-the-butt to figure out on your own with lack of documentation.

Finally, we make use of gluUnproject to un-project (normally we project the display) based on the data we have and determine the location of the mouse in X, Y, and Z coordinates. This is why we have that Output_X, Output_Y, and Output_Z declaration at the top.