Almost ten months ago,
Nintendo filed a patent application for a game controller design that would remove traditional buttons entirely in favor of a free-form touch screen. At the time, few people thought much of it—Nintendo files tons of patent ideas all the time, and the company’s always been insistent that buttons will always be preferable to touch screens for traditional console games. A couple weeks ago, however, the idea suddenly resurfaced in the form of a series of faked images.
Of course, because the images are fake, it may seem at first glance that there’s nothing to see here—it’d just be the latest in a long line of scam leaks from people who claim to have uncles and coworkers who work at or with Nintendo. But that’s not the end of the story.
Game Informer editor-in-chief Andy McNamara chimed in, saying he’d actually heard from his own sources that Nintendo was working on a controller without buttons.
Could that outlandish controller concept from the patent actually come to fruition? Why would Nintendo want to make a controller that eschews traditional physical buttons?
Examining the Buttonless Controller Concept
Before we answer those questions, let’s take a second look at the details of the patent. Here’s the general idea, captured in the abstract:
A non-limiting example information processing apparatus comprises a housing, and a first portion of the housing is formed in an elliptical form when viewing from the front. A display panel and a touch panel constitute one main surface of the first portion. Holes are formed in left and right end portions of the display panel and the touch panel, and two operation sticks are provided through the two holes. When viewing the first portion from the front, an area except key tops of the operation sticks becomes a display area.
In layman’s terms: it’s a controller with an elliptical shape, whose entire front surface is a touch display, with holes in the display to make room for two analog sticks. There will also be two shoulder buttons.
The two analog sticks, in addition to giving the player 360 degrees of directional control, can also be pressed like buttons. And, like the Wii U GamePad, it’ll also be capable of detecting motion, which likely means this hypothetical controller would come packed with a gyroscope and accelerometer.
Because the screen is an oblong shape, it can contain both a display image that’s comparable to what you’d see on a 16:9 display like a TV, as well as display images that lie “outside of the effective visual field.” Since a player’s thumbs will naturally rest on the two sticks on the front face, and since those sticks are built directly into the display—instead of set off to the side, as with the Wii U GamePad—it’ll be easier for the player to interact with the touch screen surface—especially the portions of the screen “outside of the effective visual field,” but also the center of the screen—thereby increasing the level of immersion in games that truly take advantage of touch.
If you’re not convinced, just think about the experience of using the Wii U GamePad touch screen. It’s simply not possible to make use of the screen in its entirety while holding the GamePad in both hands the way you do a traditional controller. Your thumbs simply can’t reach most of the display. With a controller where the entire face is a screen, the entire screen will be within the reach of your thumbs at any given moment. There won’t be any “reaching” for the screen because your thumbs are always situated inside the screen.
Because the entire face of the controller is a touch screen,
there are no physical buttons. Instead of physical buttons, the player can either press the sticks to use them as buttons, or tap “button images” on the touch screen. Those virtual buttons can be displayed in an array around either stick, or displayed contextually based on the scenario. In some games, the player may also be able to customize the position and layout of the virtual buttons. Another suggestion involves pushing the right stick in different directions and depressing the stick to use it in place of the traditional ABXY inputs.
In other words, the idea is to create an interface that doesn’t need to rely on lots of different
fixed inputs. The inputs are instead dynamic and can be designed differently depending on the game being played.
There are some interesting ways in which such a controller could be used in Nintendo’s traditional franchises.
For example, side-scrolling
Mario games have traditionally used three main inputs: the D-pad for controlling character movement, the A button for jumping, and the B button for running and using power-ups. A controller with only sticks and no traditional buttons on its face could achieve similar controls using just the two clickable sticks. The left stick could control Mario’s movement, while pressing the right stick like a button could control his jump. Power-ups could be controlled either with a shoulder button or even by pressing in the left stick.
The result? A game that’s much simpler to control. Your thumbs would always rest on the same two inputs, and you wouldn’t need to constantly hold down the B button to run while also pressing the A button to jump. That may seem like second nature to many
Mario fans, but with the advent of analog sticks it’s no longer necessary or especially intuitive. Analog sticks work perfectly well at controlling a character’s speed; you don’t need a dedicated button for that.
Zelda game, you could control Link’s movement using the left stick and the camera with the right stick—just like a traditional game. You could also replace the traditional A button with one of the two clickable sticks, except now you wouldn’t have to move your thumb between the stick and the button. The other stick could be pressed to control Link’s sword. The left shoulder button would still be used for targeting, and the right one could still be used to bring up other items like his shield or bow. Meanwhile, the touch screen could be used to manage and equip Link’s items as well as view and interact with a game map—in other words, it could emulate the functionality we’ve already seen in
Zelda games on Wii U.
Just as in the
Mario example, you’d still be able to use many of the actions available in past Zelda games, except without having to constantly assign different items to different buttons. Meanwhile, there’d be a very accessible touch screen for managing your inventory—instead of having to free up a hand to make use of the screen, as is the case today with the bulky Wii U GamePad, that screen would always be within easy reach of one of your thumbs.
Because the touch screen is now easier to use thanks to the new form factor, more creative games that combine traditional controls with unique and intuitive touch-based controls would also be possible. For example, it’d be pretty natural to tap the space at the left or right side of the screen in a
Metroid Prime game to switch visors, or quickly manipulate an object in a puzzle game using the space in the center of the screen. It’d also be much more comfortable to use multi-touch controls with a screen that covers most of the controller face, making it easier to port over and build mobile-style games for the less traditional gamers in the family to try out and enjoy.
Combining the Buttonless Controller With Other Patents
Nintendo’s also floated ideas for some other interesting input concepts over the past year or so. While these ideas aren’t mentioned in this particular patent, they mesh pretty nicely with their buttonless controller idea.
One idea involves
shoulder buttons that also function as scroll wheels, like those you see on a modern computer mouse. You can press them just like buttons, but also rotate the wheels to gain even more input options for your index fingers, which typically are just used for pressing shoulder buttons.
Zelda control scheme I devised earlier, it’s really easy to see the applications of a scroll wheel: it’d give you a really fast, intuitive way to cycle through your inventory. So in my hypothetical control scheme, where you use your equipped item by pressing the R button, you could also rotate the R scroll wheel to cycle through the items that can be assigned to that button. The L scroll wheel could also be used for consumable items like potions: as you rotate the wheel, an icon on the touch screen cycles between items, and then you tap the icon when you’ve scrolled to the item you want to use. You could get a lot of mileage out of relatively few buttons that way.
Another idea would introduce analog sticks that provide force feedback in certain situations. For example, when a player runs up against a wall, the stick might provide resistance. I’ve imagined another application of my own: using force feedback to simulate the functionality of a D-pad. For example, in a 2D
Mario game, force feedback could create an artificial octagonal gate so the player can more easily discern which direction he or she is pressing the stick. This would allow the controller to be accommodating to 2D and 3D games without having to include both a D-pad and an analog stick—there’d only be one primary left-thumb input.
And, lastly, Nintendo filed a patent for
touch screens that use haptic feedback to simulate the sensation of pressing a physical button. The applications for this are pretty straightforward: in cases were the limited number of fixed inputs isn’t going to cut it, the player can always use virtual buttons on the touch screen to mimic the buttons on a traditional controller. The obvious risk: that the technology won’t quite be up to snuff, and players won’t find the touch buttons as intuitive and usable as traditional ones. Of course, I’d imagine Nintendo wouldn’t even bother to put this technology in a controller before they’ve overcome this hurdle.
The Million-Dollar “Why”
At the end of the day, it isn’t the number of wildcard ideas you can pack into a controller that matters—it’s all about whether your controls actually make your experiences better than they would have been with another kind of controller.
While most existing gamers probably can’t see why a buttonless controller would be an improvement for anyone, Nintendo’s made it their mission to satisfy different players with different needs, often delivering innovations that don’t follow the expected linear path of progress.
One of the big ideas that drove Nintendo’s big successes with both DS and Wii was that Nintendo decided to take a different course on controller inputs. The conventional wisdom was that, as games became more complex alongside the pace of technology, more modern controllers needed to pack in more inputs to give experienced gamers as many potential actions as possible. Nintendo saw things differently; they saw the increasing complexity of controllers as a
barrier, not a benefit.
Nintendo’s frequently spoken at length about how this new paradigm completely changed their trajectory when designing DS and Wii.
Kotaku reported in 2010 that one of the driving forces behind the addition of a touch screen to Nintendo DS was the desire to eliminate the hurdles that newer players had to overcome when approaching a game machine for the first time. The result was a totally different kind of game input.
“What’s keeping people from touching game machines? What’s making them run away?”
Their discussions started there.
Recent game systems had button-encrusted controllers that were too complicated. Software that used complicated technology was becoming more common, and the gap between experienced players and beginners was growing wider all the time. It scared people off—or worse, made them actively dislike videogames. Their discussions grew to encompass the themes of games.
While adventure games like Mario were fine, was that really enough? What if games included themes that related to the lives of ordinary people? Would people who considered gaming a waste of time embrace it then? As they converged on the company’s new direction, the idea came to Miyamoto: dedicate one of the displays to touch control.
One screen would be used for intuitive, approachable controls, and the other would be the main display. The system would be easy for anyone to control, and it would allow the development of new kinds of games.
From there, Nintendo moved on to trying to make a game console that would be just as accessible. The result, as we all know, was the immensely popular Wii.
The DS, with its dual screens and stylus controls, lowered the barrier to videogames. The Revolution needed to do the same thing.
The controller would be at the core of its interface, and it could not be less than perfect. Certain aspects of the controller were decided early in its development: It had to be wireless, and it could not be intimidating.
When Iwata was talking to Miyamoto and company about the circumstances that were leading to gaming’s decline, the first thing that came to the president’s mind was a TV remote control—a piece of technology the entire family used.
People who didn’t play videogames never touched game controllers. The wires that snaked out from the console were nothing but a nuisance to them, and if controllers dared to be left about, they were put away. But the TV remote never bothered anybody. As Iwata considered the difference between the two, he realized the new controller would have to be wireless.
Then he wondered if people found controllers intimidating because of the way they looked.
Game controllers were constantly getting more complicated; in addition to the standard direction pad and buttons, they were now encrusted with all manner of analog control sticks and triggers, placed seemingly everywhere. Was that alone enough, perhaps, to drive people away from a videogame? The new controller had to be simple and approachable. Iwata also felt that, like the DS’s touch screen, it needed to facilitate direct, intuitive controls-and Miyamoto and Takeda agreed.
In the end, Nintendo designed a controller that wasn’t just as easy to understand and control as a TV remote—the Wii Remote
literally was born out of common TV remotes, from its shape to its use of IR functionality all the way down to the way it’s designed to be held by the user. To drive the sense of familiarity home, Nintendo even designed their home console operating system interface around the idea of TV channels, offering an array of programming including News, Weather, and Shopping apps and eventually even Internet TV in the form of Netflix, Hulu, YouTube, and other video streaming services.
In other words, Wii’s interface was as accessible as it was partly because it also embraced ideas that originated outside of gaming—ideas that its users were already familiar with, regardless of whether or not they’d played a video game console before.
Shigeru Miyamoto, then one of Nintendo’s general directors, thought it would be important for the controller so anyone could pick up and use it. As he said in an Iwata Asks interview in 2006:
Iwata: Miyamoto-san, what was the key concept for you when you started making the [Wii] controller?
It was the idea of accessibility. Rather than make something that would make people wonder if they could use it or not, I wanted to make something that would make people want to pick it up and try using it. Of course, I also had to keep my own experience of making video games in mind. It was absolutely essential to make something that would also work with older games. I also spent a lot of time thinking about what makes a design accessible. It was in this context that we started to question everything about conventional controllers, including the idea that a controller had to be held with both hands.
Mr Miyamoto would bring out his mobile phone during meetings and say enthusiastically “Can’t we make something like this?” (laughs)
I don’t think that Mr. Miyamoto realized how prescient his comments about mobile phones would turn out to be. Today, the mobile games market represents a large and growing segment of the industry at large. And today’s kids—the new lifeblood entering the gaming world—
actually prefer playing games on mobile devices instead of consoles. Kids aren’t the only ones playing games on mobile, either. Statista estimates that 164.9 million people in the US are mobile phone gamers. That means that most people who play games today are doing so on devices that use touch as the primary means of input.
As a result, Mr. Miyamoto’s remarks are exceptionally potent today. If Nintendo’s going to design a controller that’s as approachable as the Wii Remote, they’re going to need a controller that’s perceived to be just as usable as the mobile devices people are currently using. Meanwhile, they’re going to need to offer a library of games that people are attracted to but can’t get on their phones and tablets.
There’s a big hurdle to meeting those usability expectations, of course, and it’s the classic problem Nintendo’s already worked to solve in the past: standard controllers just have too many buttons, while smartphones have a touch screen with a user interface that can be designed in the way that’s most appropriate for the app experience the developer wants to make. If Nintendo’s going to make a controller that bridges the gap, it’s going to need to have a touch screen that’s just as intuitive as the ones on smartphones.
Some saw the Wii U GamePad as Nintendo’s answer to this problem. But there’s a snag in the plan: the Wii U GamePad isn’t just as usable as a smartphone or tablet. It was bigger and bulkier, and had just as many buttons and triggers as conventional controllers. Its touch screen was relegated to the middle of the controller, in between all the bulky plastic at either end, which made it uncomfortable to use while holding the GamePad with both hands. In the end, the GamePad was barely any simpler than a traditional controller—the controls were actually even more complicated at first glance.
This isn’t too surprising, since the GamePad wasn’t
really designed so anyone could pick it up and play. It was designed to give hardcore gamers and developers the “traditional buttons” they said they wanted, while also adding a touch screen that Nintendo hoped would appeal to other players. The result was a controller that was only marginally better for hardcore gamers (and only if they saw the second screen as a value-add) and wasn’t really very appealing to non-traditional players at all since it was much less intuitive than their iPhones, what with all those buttons and sticks.
To make a controller that’s really just as usable as a phone, but that can actually function as a game controller and isn’t just a carbon copy of a phone, Nintendo needs to go one step further: they need to get rid of as many of the fixed inputs—the buttons—as possible. Then they need to figure out how to use this minimal-buttons controller to actually improve their video games.
Looking back over the
Mario ideas I discussed earlier, the most obvious benefit for removing traditional buttons would be that players would no longer need to hold down a button to run. While us Mario vets may have no problem keeping the run button held at basically all times, watching new players has shown me that those controls are far from intuitive—and therefore far from ideal.
A new 2D
Mario game could eliminate this quirk and take full advantage of the analog stick to control movement speed instead. No more awkwardly pressing both the A and B buttons at the same time. One thumb for movement and power-ups, the other for jumps. Simple and straightforward.
As with Mario, a Zelda game that doesn’t rely on tons of different face buttons could dramatically lower the barriers to play. You’d keep your thumbs on the sticks the vast majority of the time, controlling all your basic actions either by tilting or pressing the sticks. And cycling between items with the shoulder wheel would let players quickly and seamlessly equip the right weapons for the occasion.
Zelda, removing buttons could mean steamlining the way players interact with the world. There isn’t really a need to have a bunch of different buttons available for talking to people, swinging your sword, assigning up to three items at once, and so on. You could interact with NPCs by simply touching them. You could equip your bow using the scroll wheel and then tap an enemy in your field of view to shoot. Or you could ready a bomb and tap the spot where you’d like to toss it. The few buttons and sticks that are present could be implemented in ways that complement the touch screen elements.
But such a controller wouldn’t have to alienate traditional players by forcing them to use strange new control schemes, either. If Nintendo can really get haptic feedback for touch buttons
just right, they can finally have a controller that offers the best of both worlds—a simplified layout, appearance, and control interface that doesn’t overwhelm people with buttons, alongside options to use a traditional control setup by overlaying simulated buttons on the touch screen.
It’d be an ambitious project, to be sure, that would require Nintendo to totally revisit their assumptions about how traditional inputs can and should work together with touch. But if Nintendo can pull it off—if they can get new players to want to try out their franchises using a controller that’s just as usable as their phones but that offers richer levels of control for deeper game experiences—I think a buttonless controller might not be such a bad idea after all.