An Opinion: The virtual keyboard and the beginning of the end for physical user PC controls

Technological developments always demand renewal of the old system, whether by developing it or by replacing it completely with the new system. We've seen a lot, about this. However, should everything be so?

Let's look at an opinion on conventional keyboard evolution leading to a virtual keyboard - should it be eliminated?

The idea of a virtual keyboard isn’t new, we’ve had projection keyboards in market on and off for years, but they never sold well.  But with the advent of Mixed Reality and, thanks to Smartphones, a far greater acceptance for buttons that don’t move, this concept of dumping the hardware for something else it is gaining momentum again.  Microsoft, [Disclosure: Microsoft is a client of the author] just got a patent for a new type of virtual keyboard to be used in Mixed Reality that not only has the potential to end physical keyboards, but all physical input methods including Stylus and mice as well. 

Illustration of Virtual Keyboard

This is extremely important because we are moving to use VR for more and more things like luxury shopping for instance. Let’s talk about what this could mean (accepting it is around 5 years out). 

Acceptance

For some time, buyers have been reticent to use keyboards that didn’t have physical keys.  We are creatures of habit and since we were used to keys moving the feel of unmoving keys was disconcerting and related products largely didn’t sell well.  But with the success of the smartphone and screen keyboards acceptance has been growing and most tablets and Smartphones today use screen keyboards with the attach rate of physical keyboards generally down well below 10%.  

The market is close to ready for an alternative way of inputting information.

Hololens

One of the big problems for Hololens users has been keyboard use.  You can use a physical keyboard but Hololens is designed to be a mobile device and either having to return to a physical keyboard or carrying one has been problematic. As has been using a representation of a keyboard because the cameras in the product aren’t accurate enough to handle the individual variances with typing in the air.   

Yet when you are engaged in a project, or game, and need to do text entry having something that works is still a critical part of the solution. 

Virtual keyboard

Microsoft’s now patented solution is to use a virtual hand plane in the holographic environment.  This is kind of like a virtual table that the keyboard can be rendered on which massively reduces the potential variances both with a single user and across groups of users.  They ten use precise hand tracking, with depth perception, to emulate both visually and electronically the users’ actions as if they were typing on a real keyboard.  It won’t feel like a keyboard, the user will be typing in the air or on a convenient surface but it will look like one and similar movements will be involved suggesting a relatively short, but still necessary, learning curve. 

Mice, pens and more

Once you fully represent a complex keyboard creating similar virtual representations that work to replace pointing devices (which might just shift to your finger anyway in a Mixed Reality Word) and a stylus should be relatively easy, though users might find it easier if they used some kid of stylus proxy.   But once you virtualize a Keyboard virtualizing other controls, game pads, flight or driving controls for games, should be equally easy though here too I think users would appreciate a proxy.  But the proxy wouldn’t need to be instrumented, just scanned, and then represented more realistically in the utility or game. 

For instance, for a driving simulation, you could use a real cars’ steering wheel and controls for feel but the tie in to the application would be virtual and you wouldn’t need to wire the things into your PC.   This could make for some very affordable but still realistic simulator controls.   You would need a depth sensing camera focused on the activity but that would be a ton cheaper than building a simulator with real automotive gear hardwired into your PC. 

Wrapping up

As a new technology comes to market the typical first implementations are by treating it like whatever came before.  As it matures companies start to understand what makes it unique and change their approach and designs.  For instance, the first cars looked a lot like buggies without horses.  

With Mixed Reality, we are now moving from that phase and this patent from Microsoft showcases this, they are beginning to think differently.  Eventually they’ll realize they don’t need something that looks like a keyboard at all and then things are likely to get a tad crazy for a while.  

This is often called out of the box thinking and if there was any environment that will promote that kind of thinking, Mixed Reality, where you can even change the physical laws of nature, is likely it.  These moves will be critical as we step away from reality and into a far more virtual world. 

Written by: Rob Enderle, Computerworld.