Research

Body Tracking

           I started my research with the different methods of body tracking. Because I have used Wii remotes in multiple pieces, I knew the possibilities and limitations of the device, but I had extremely limited experience with both Sony’s Playstation Move and Microsoft’s Kinect. I had only used each device once prior during last summer (2010) at the Electronic Entertainment Expo trade show in Los Angeles. During the show, I managed to spend a few minutes with each device to explore their potential. Unfortunately, the amount of time allowed was not nearly enough to make an informed decision.
Luckily after the Playstation Move came out, a friend of mine bought one, and I was able to more extensively experiment with it. As I suspected, the peripheral seemed to be more accurate than the Wii remote in many regards; however, it had nearly identical limitations. For example, both devices work by combining light tracking by a camera and information from internal gyroscopes. If the device is moved behind something, obscuring the light from reaching the camera, the only information available comes from the internal gyroscopes. Alone the gyroscopes can become very inaccurate over time due to what is known as “drifting.” The gyroscopes work by measuring gravity on three different axes of movement. However, only two of the axes can depend on gravity at once while the other must be measured by other (far less accurate) means. Imagine a ball in a bowl. If you tilt the bowl by moving your left hand up and right hand down (or vice versa), the ball will remain in the center of the bottom of the bowl; the same thing occurs if you tilt the bowl back or forth. The device is able to detect that the “bowl” has moved, but the ball is still in the middle; therefore, it is very accurately able to detect the current rotation of the device as a whole. However, if you were to rotate the bowl left or right (as if it were on a Lazy Susan), the ball would still be in the center, but the device would not be able to tell that the bowl was moving. At least, not as easily as the other two axes. Over time, the estimation of this less accurate axis is thrown off. In addition to this, if one were to flail the device around quickly enough, the measurements for all axes would likely be thrown off relatively quickly. So again, while it seemed the Move was, for the most part, more accurate than the Wii remote, it still had the same problems. Plus it is twice as expensive and far less durable than the Wii remote. So far, the Wii remote still seemed to be the best bet.
            Soon after, Microsoft released the Kinect. One day I decided to go out and buy one to test it out. On top of this, people around the world had been hacking the living daylights out of the device and were doing some really unique things with it. I figured that with so many people working on it and with all of the shared code out there, it was a very real possibility that I could use a Kinect for the piece. After playing around with it for a bit, I realized that this was indeed a very powerful device though it too had its limitations. The device works by combining information from an RGB camera and an infrared camera. With the combined information, the Kinect is able to basically create a virtual 3D representation of whatever is in front of it. Programmed software takes this information and creates a bone structure for any people detected within the frame. With the virtual skeleton, the device can accurately track the person’s movement. However, this 3D environment is only based on the front faces of whatever it sees. For example, it can create an extremely accurate 3D representation of the face of someone facing the device, but it has no information about the back of that person’s head or anything that may be behind the person’s head. For all the device knows, there isn’t anything behind the person’s face. So if one were to turn to the side (as would be very possible in my piece), the Kinect would likely only see one shoulder, one arm, one hand, one leg, and the head; the entire other half of the person would be obscured by the side facing the device. If this were to happen, the Kinect would not be able to accurately track the person’s motions. In addition to this, the device is so new that, despite the large amount of people working to hack it, there is really very little usable code to work with it; everyone is still developing their own methods and very little shared code has been released.
            Another considered method was inspired by both the Wii remote and Playstation Move as well as actual motion capture – in other words, tracking light. I have previously made a motion capture suit and used it in a project, and I found that lighting is extremely important in such a situation. With all of the projectors I planned to use, there would be light passing through the sphere/cube onto the user, and this would most likely make it incredibly difficult to accurately track both reflective balls and small LED lights attached to the suit. In addition, it would be very hard to create a suit that fit everyone, particularly considering the importance of the lights’ locations on the body.
            With all of this in mind, it seemed that the Wii remotes were still the best way to go. Because of this, I began to do some research into a nifty little program that I have used before – OSCulator. This program, among other things, allows Open Sound Control (OSC) messages to be sent to different computers/applications based upon some connected device. OSC messages are generally used for music creation/synthesizing, but the creator of this program has worked Wii remote functionality into the program. This makes it very easy to connect and send/receive data from connected Wii remotes. While I had used the program before, I had never used it with Wii MotionPluses connected. Some research went into seeing how to send/receive data from the MotionPluses, but due to the program’s ease of use, it wasn’t long before I had that all figured out.
            The bigger problem came from connecting the number of Wii remotes I wanted to use. OSCualtor has the ability to store up to eight Wii remotes in its memory at a time. This allows for quicker connection for those eight registered controllers. However, only four remotes can be connected at any one time. I needed far more than that for my piece. Many methods were tried, and most of them failed. The program has the ability to duplicate messages so that each one message can be sent to multiple targets. However, due to the way the program receives data from the Wii remote, this did not work. I found that OSC messages could be routed to a router, and thereby received from any OSC-ready computer on the network, but after many attempts, this too failed. Eventually, I found out that more than one instance of OSCulator could be opened on one computer. This had not occurred to me because this is not possible in many Mac applications. But with two (or more) instances open, I could essentially send different messages to and from the different instances. By doing this, I could connect four Wii remotes to a “host” computer as well as four remotes to one or more “client” computers. The client computers could send the data to the different OSCulator instances of the host computer. Because the host computer had two (or more) instances of OSCulator running, each sending and receiving data from four different Wii remotes, I successfully had the sufficient number of connected Wii remotes.

Visual Immersion

After considering my options, I decided that a cube would be the easiest way to display the piece. The question was: how do I project onto the six surfaces of the cube? It seemed I would need six projectors to do so. Easy enough. But how would I run my application simultaneously on six projectors? For years I have run four monitors off of my desktop computer, so this lead to my first option: I could run the application from one computer and have the six projectors hooked up to the one computer. My application was going to be created in Unity, so I needed to find out how I cold manage doing something like this within Unity. After perusing the forums and scripting reference, I came across an idea. It was possible to set the application’s window size as well as use multiple cameras at once. In addition, the cameras could all be resized on within the application’s window. So I created six cameras, arranged them as an unfolded cube (a sideways cross), and made the window 4,096x2,304 pixels (four screens wide and three screens high). However, my desktop is very old and was unable to run the application. I then sought other methods of running the application from one computer. I asked around to see if any of my friends had any computers with the ability to display six outputs. Unfortunately no one did. Many did offer to let me use their computers though, but I would have to buy many new graphics cards in order to accomplish my goal, and sadly I could not use the graphics cards from my desktop because they are too old to be used in newer computers. Using one computer seemed to be out of the question.
So now I considered using multiple computers to run the application. Two methods crossed my mind in considering this. I could network the computers together to create one very powerful “super computer,” or I could run the application on six different computers and find a way to synchronize the actions on each instance of the application to make it appear as if it were just one application. After a bit of research, I chose to pursue the second option. I have used networking in Unity before through a third-party plug-in called SmartFoxServer, though I have always had difficulty using it, and it seemed to be a little overkill for what I needed to do. Luckily, I found that Unity has built in networking features that were pretty perfect for what I needed to do (send rotation information between the different instances of my program). After testing this out a bit, I decided to stick with it and use multiple instances of my program with multiple different computers networked together. Each computer would be connected to one projector that would project onto one side of the cube.

Movement

To achieve my goal I needed a treadmill-like device that could move in multiple directions. To my surprise, I very quickly found material describing such a device. Such machines are known as omnidirectional treadmills. I was very unhappy to find this information as this was an idea I have had for many years; an idea that I thought was unique. Not only did such devices exist, but they are also used for the same purpose as I was planning to use mine – virtual immersion. My disappointment quickly vanished as my research lead me to diagrams and descriptions on how the machines are made. To my pleased surprise, my original ideas were actually quite close. The most modern omnidirectional treadmills are basically a treadmill of treadmills. Again imagine a square treadmill. Just like any other treadmill, this one can move backwards and forwards. Now imagine this “main” treadmill is divided length-wise into many thinner treadmills that can move left and right. With the combination of the smaller treadmills moving backwards and forwards in a group and the individual smaller treadmills moving from side to side, any direction of movement is possible. Regardless of how pleased I was to find such information, I soon realized that building such a device was far outside my spending range. So while I would have much rather used a device that allowed for actual walking, I decided to stick with the Wii Balance Board. It would still communicate my intent, it was a technology available to me, and I had prior experience using it with Unity.