In today’s work session have 2 research questions to try out: 1) how well does using OSC as a protocol for sensors speeds up development of smartphone-sensors-based games? 2) how well can magnetic field based input be interpreted as playful input?

OSC as prototyping protocol / tool

My idea is to use a flow like this:

  1. The sensor information is read from the Android device and sent via OSC using the free app called Sensor2Osc
  2. On the Unity side, I’ll add the OscJack library, to receive these messages and parse them as input.

With this setup, I’d be able to bypass two steps that took quite a long time and fiddling with when creating the shellphone experiment: 1) running the game with the Unity Remote app, which has quite poor performance and limited access to sensors and 2) having to re-build the phone app every time you want to run a test or getting the current values of a set of sensors. Skipping these two steps will make it much quicker to figure out the numbers and sensitivity of the sensors, and how they could be used expressively.

Ok, in about 5 minutes I have a sample running that shows the light sensor value on screen, with minimal setup. I’ll try now to make something that allows me to visualize the magnetic field values. I implemented a simplified 3-bar visualization, which I then remixed to be able to work with negative values. After that, I used the value of one of the axis to create a proximity detector for a houseplant vase with a magnet on its vase.

Test with OSC relay of sensor values

I believe this does show that working with the OSC based flow works well. I was able to try out different ideas on how to use the magnetic field values in quick iterations. This is definitely a working system to be considered further.

Using magnetic fields

The plant example is pretty cool, but so far I’m having mixed success in getting any output that is more expressive than just a 1D float value based on proximity. I believe some changes might helpo with that:

  • Use a stronger and easier to handle magnet. I’m currently using a cheap and weak fridge magnet.
  • Work on better algorithms for filtering the input values. I already implemented a running average algorithm, and it did improve jittering considerably.

I went back to my references, specifically the MagGetz paper [1]. I’m currently trying to understand and implement a similar approach to how they handled the input values, more specifically the interpolation and “traces”.

I tried interpolating the position based on 2 reference values (start / end of a slider). It worked well, as long as it did not involve large changes in relative position to the sensor. I guess this is why the MagGetz physical widgets were attached to the surface and had to keep such a stable position in relation to the sensor. I believe that with continuing with this approach could get me to similar results as the MagGetz project, including their “recording” approach for curved movements. However, I’ll try something different first.

Gesture recognition

One of the challenges of dealing with magnetic data is that the values are non-linear and change is ways that are hard for me (with my limited geometry and descriptive math knowledge) to model. One alternative I believe can work is to use gesture recognition libraries and algorithms to record specific movements and then detect them. While this might not be very good for real-time or reflex based inputs, I believe it could be useful to create signs or other more “coded” actions.

For this, I’ll experiment with a free and open source gesture recognition library available on the Asset Store. It is a part of the $ family of gesture recognition algorithms for 2-D gestures. I’ll basically try to feed the magnetic field points as the input and see if I can record and recognize gestures with it.

It worked! I was able to record and then recognize different gesture patterns, as long as the relative distance from the sensor was somewhat similar, but there is some flexibility there. I believe that using a better magnet and some form of casing to reduce the variation in each gesture I’ll get even better results.

There are quite a few limitations in terms of precision, but it might be useful for playing slower-paced games. What I did was to use the $P dollar algorithm, which is optimized for multistroke gestures. However, the magnet never “turns off” so separating the flow of inputs into strokes would need another layer of inputs. The next step would be to use the $1 algorithm, which can recognize directionality in one-stroke patterns, which seems to me like a more interesting capability.

I guess that I could also try and check if this algorithm could alos identidy some multi-magnet or multi-object input. It might be more buggy, but maybe could work.

[1] Sungjae Hwang, Myungwook Ahn, and Kwang-yun Wohn. 2013. MagGetz:Customizable Passive Tangible Controllers on and Around Conventional Mobile Devices. Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, ACM, 411–416.