Back in February, John Underkoffler guided TED conference attendees through a brief tour of several g-speak applications. The TED staff did a great job filming and editing the talk, which you can now watch on TED.com and on YouTube.

All of us at Oblong enjoyed this opportunity to show our code off to a theater full of technology enthusiasts. John demonstrates g-speak applications that our clients use every day. But several of those capabilities -- the scalable image navigation environment, the SQL data-diving interface, the gestural wand -- haven't previously been part of our public presentations.

As usual, technology development outpaces marketing and the future is here, just lumpy in distribution.

John also talks about why we think it's time to transition to a new way of thinking about and using computers. And he tells the story of g-speak, describing some of his precursor work at the MIT Media Lab and for Minority Report.

At the very end of the video you can see a bunch of us running around in the background taking everything offstage. We had set up a portable version of our typical four-screen, multi-user g-speak system, as well as a separate workstation and various cables patching us into the TED video and network infrastructure. We got everything out of the way and into the wings in just over three minutes. Parallelism of a different sort than we are usually concerned with!

Reporter Cris Valerio and producer Elizabeth Gould spent several days with us in LA in October. You've likely seen Cris hosting Venture, as well as her coverage of the technology beat. Elizabeth was responsible for the ground-breaking "Green Business" series.

Cris and Elizabeth shot footage for a mini-series called Innovators. The series has aired, and can be watched on the web, too, along with a bunch of great supplementary, online-only content.

Oblong is the topic of the first episode, "Turning Point."

Carlton Sparrell, Oblong's VP of Product Development, spoke at the MIT Media Laboratory's Fall 2009 Sponsor Week meeting.

Over the course of the twenty minute talk (viewable via the video above) Carlton furnishes the first public overview of the technical capabilities of plasma, Oblong's rewindably networked framework for data self-description, encapsulation, and interprocess transport.

Carlton also provides a glimpse of a 'common operating environment' built atop plasma and g-speak. This new environment enables collaborative work at multiple levels: (1) application sharing and control; (2) input-output coordination; and (3) data synchronization. The video that he shows -- a video-within-video, as you view it overhead -- involves an air-traffic analysis scenario.

For the record, Carlton's work in gestural interface is the earliest (at Oblong): as far back as the first few years of the 1990s, he was building systems like ICONIC at MIT. Here's some documentation of that work; more is available by searching the world wide web.

Amber Frid-Jimenez and Kelly Dobson are teaching a g-speak lab at Rhode Island School of Design again this semester. We're excited to see what another class of designers and artists do with g-speak. We count ourselves really lucky to have the opportunity work with so many great people at RISD, MIT and USC.

Last semester saw experiments in sculpture, textile pattern design, projection on furniture and fabric, and new glove (and not-so-glove) designs.

So far this year there are experiments in story telling, gestural etiquette, dance, spatial form making, and digital pottery. The images below link through to videos on youtube.

... today pixels are cheap and computers are fast -- after 40 years we are almost ready to move past the mouse. With new gestural interfaces like g-speak [3] we can create digital environments that are nearly seamless with architectural space. We can create hybrid spaces where groups of people can interact with digital objects, physical objects, and each other using a shared physical language that is both legible and scalable.

One early instantiation of these ideas is the g-stalt interface:

Our Sundance guests -- who already number in the thousands -- find the experience exhilarating. A few grim cinephiles have supplementally raised an eyebrow (one per cinephile) at the filmic heresy that TAMPER provides: a fluid new ability to isolate, manipulate, and juxtapose (rudely, say the grim) disparate elements (ripped from some of the greatest works of cinema, continue the grim). For us, what's important is the style of work: real-time manipulation of media elements at a finer granularity than has previously been customary or, for the most part, possible; and a distinctly visceral, dynamic, and geometric mode of interaction that's hugely intuitive because the incorporeal suddenly now reacts just like bits of the corporeal world always have. Also, it's glasses-foggingly fun.

Subsequent posts will likely detail the installation's software architecture, but for the present consult TAMPER's youtube channel for video documentation of our Sundance engagement.

John and I met at MIT in 1984. I was a sophomore and John was a freshman. Everyone at MIT is "interesting" -- from the moment I met John I knew he was "uniquely interesting." We became very close friends and did a lot of fun things together. When I moved from Boston to Boulder in 1995, I stopped seeing John on a regular basis, but I periodically heard about some of the amazing things he was doing at the MIT Media Lab with new approaches to user interface.

John resurfaced in my life when Minority Report came out in 2002. I found out that he was the movie's science and technology advisor and had created all of the computer / user interactions. I was stunned by the UI having spent the previous 15 years stuck in the world of keyboards and mice being tethered to a computer screen. With one very public demo, John had created a clear vision of the future of how people would interact with computers that built on all the work he had been doing at MIT over the years. We talked a little over the next few years, but we didn't get together until November 2006. I wrote about this meeting in my post I've Seen The Future. During that evening, I had one of those special moments where I knew I was not just seeing the future, but interacting with it.

Through my venture capital fund, Foundry Group, we made a significant investment in Oblong. One of our investing themes is "human computer interaction" -- we strongly believe that the current way we interact with computers -- primarily with a keyboard and a mouse -- is obsolete and will feel as quaint in 20 years as the idea of using punch cards does today. John and his colleagues at Oblong have created what we believe will be the next major UI paradigm -- something we are calling the spatial operating environment (SOE).

What Oblong has created definitely falls in the "you've got to see it to believe it" category. As an avid reader of science fiction, I'm always amazed to look backwards and think how the past of science fiction actually predicted the current of science fact. Oblong is creating a deep, fundamental base of software that in 2002 first appeared as science fiction in the movie Minority Report. While the UI is the first thing that you'll see, this is merely the surface of Oblong's spatial operating environment, which understands space, geometry, your hands and what they are doing, and the huge amount of real time data required to make an environment like this work effectively in the networked world.

Over the past two years, Oblong has very quietly put together an incredibly gifted team of software developers to create what we believe is the future of how humans will interact with computers. We hope you will join us on this amazing journey.

• the batch processing age.
• development of interactive systems: teletypes; time-sharing; terminals. the rise of the command line.
• transition to graphical user interfaces. character-mode and bitmapped applications. the personal computer. the spread and standardization of mouse-driven, two-dimensional window systems.
• networks. tcp/ip and udp in use almost everywhere.

That's a history from the perspective of adoption, organized according to the broad availability of a technology (rather than its first appearance in a lab). Each of the developments on the list is enabled by two things: new capabilities of computing hardware; and large amounts of software written to take advantage of those capabilities.

Both the spread of new hardware and the construction of a software ecosystem take time. Technological transitions are moderated by the installed base of hardware and software and mediated by users' expectations and programmers' assumptions.

The Macintosh, in 1984, introduced to the mass market a windows-and-mouse interface standard. This interface has evolved incrementally, but it is fair to say that a version of it continues to ship on every personal computer sold today. Internet connectivity has vastly expanded the capabilities of programs running within the GUI environment, but the modes of interaction are pretty much the same as those provided by the original Macintosh.

Today's computers, however, are capable of a much broader interactivity.

We've constructed a platform -- hardware and software -- for building and using applications that combine

• finely calibrated, free-hand gestural input and output
• multi-core, multi-screen and multi-machine application programming libraries and patterns
• real-space representation of all input objects and on-screen constructs

We call our platform g-speak. It is, in many ways, a new thing. Reasoning about new things is often easier if they are categorized, and if the category is named. So we call g-speak a spatial operating environment.

A spatial operating environment provides abstractions that help programmers to write applications that use gestural input, that function well on large screens and for simultaneous users, that work across multiple computers and screens, and that can be built from loosely-coupled small programs.

motivations

We built g-speak initially for ourselves. The ideas and various partial implementations date back a decade and a half. Early work took place at the MIT Media Lab. We formed Oblong Industries in 2006, with the goal of building g-speak out as a broadly useful platform.

Some of the big itches we have tried to scratch, over the years and up to the present, include

• Multi-user applications are hard to build. GUI toolkits assume a single pointing device.

• Similarly, interfaces that include more than one input device, tangible objects, gestural input, network control streams and other non-pointer input are difficult to build.

• Navigation through three-dimensional data spaces is always a kludge. People who play a lot of first-person video games get good at it. So do CAD afficionados. But even the most practiced users of mice and game-pads are always working against the indirection of their input devices.

• The mouse is a fairly constrained (and constraining) physical object. It has to sit on a surface (or be a surface, in the case of a trackpad). Heavy mouse use often leads to repetitive strain injuries. The mapping between the two axes of mouse motion and the two axes of on-screen pointer motion is indirect. It would be nice to bypass the mouse entirely and control pointer motion and position directly.

• On-screen layout in a traditional GUI environment is impoverished; the more pixels you have available, the more you feel this. Windows occlude other windows and there are almost no idioms for dynamic resize, rearrangement, decoration and redisplay of windows. GUI toolkits don't help you write multi-screen applications.

• On-screen feedback in a traditional GUI environment is sparse, too. As graphics cards have become more powerful, platforms have layered on animation and window decorations. But there are few provisions for consistent dynamic feedback about input state and options. The mouse mostly doesn't need this (cursor glyph changes and "tool tips" suffice). But more complicated input devices do.

• GUI applications aren't very network friendly. Even the venerable X Windows protocol has become less useful as toolkits have become more complicated. It's not easy to share input and control streams between computers in the same room. Protocols for coordinating fine-grained application state across multiple machines are not very mature (and no GUI toolkit addresses multi-machine operation).

• GUI applications aren't very composable. Toolkits support only monolithic styles of development: an application is a single process with one or a small number of graphics windows. Only one process can draw into a graphics window. It's almost impossible to write a set of small graphical applications that work together to perform complex tasks.

At a glance, this might seem like a grab-bag of complaints. But we've come to believe that a common thread runs through all of them.

We're dissatisfied with the the contemporary application development stack because the tools and toolkits we use today fail to take advantage of the graphical capabilities, horsepower and interconnectedness of modern machines. Addressing any of our complaints fully will require addressing all of them.

Perhaps the best argument for this is to look for an analogy in the last major interface shift, the move from character-mode computing to the mouse-driven GUI.

Making the most of the mouse, more affordable RAM, and improved displays of the early 1980s required rebuilding the personal computer from the ground up. The Macintosh team designed new hardware, knowing from the beginning that they were building a graphical operating system. The hardware work, os-level development, application toolkits and user interface design all happened as part of a single, unified effort. That effort produced the GUI that defined the next 25 years of computing. Other projects (at both Apple and other companies) took a more piecemeal approach and produced less impressive systems.

It's time for another redesign. We probably don't have to build our own motherboards this time around. And the unix-descended operating system kernels we use are pretty good. But there's a lot of work to do in the software stack that sits on top of the kernel.

We want to redress the conversational imbalance that we have with our computers -- great graphical output but very limited user input. We want all our applications to make use of all our computers. We want a common interface for all the screens we use every day: laptop and desktop computers, televisions, the nav systems in our car dashboards.

design

We started out building a framework for gestural I/O. (Though started, here, is a slippery idea. Our work on all of this goes back fifteen years or so.)

From the beginning we knew that brand new input and output languages would have to be designed at the same time; a richer control interface requires a new on-screen graphical vocabulary.

Again, this is most obvious when looking back at previous work. The PARC and Apple folks building new software around the mouse developed scrollbars, overlapping windows, pull-down menus, title bars, desktop icons and most of the other UI features we take for granted today.

To an applications programmer, the shift to gestural input is as big as the shift to the mouse was twenty-five years ago. It's both exciting and a little daunting. The g-speak input framework allows direct, either-handed, multi-person manipulation of any pixel on any screen you can see. Pointing is millimeter-accurate from across the room. Hand pose, angle, and position in space are all available at 100 hertz, with no perceptible latency and to sub-millimeter precision.

With all of this articulation available, it makes little sense to use GUI components that were built for the two-axis, three-button mouse.

The g-speak platform relies on a new system of on-screen primitives. There's more work to do, of course: the breadth of possible gestural interactions exceeds what we could do with earlier input devices.

A fully gestural inteface also requires a new geometric grounding. Contemporary GUI toolkits all bottom out in pixels. But gestural input is spatial. Pointing is literal and three dimensional. Programs draw on multiple screens.

So we don't think in pixels when writing g-speak applications. We think in millimeters, and in z as well as x and y. This takes some getting used to, but it's liberating. Even two-dimensional program interfaces -- and most of the interfaces we write are still fundamentally two-dimensional -- live in real-space with their users.

Gesture, graphics and spatial representation form a tight triad. And we pulled in two more building blocks from our earlier work.

First, we needed fine-grained and pervasive coordination between machines.

One of the really nifty things about gestural input is that from the user's perspective there is no input device at all. Hands work more or less as expected when pointing at pixels, both proprioceptively and socially. In particular, it's possible to point at and interact with many screens at once, which implies interaction with many computers at once.

There are practical limitations to how many screens can be driven by a single computer. And some screens -- laptops, televisions -- come bundled with a computer of their own. So we rebuilt the g-speak input pipeline around a network layer that makes it easy for lots of computers to join up and use the gestural event stream.

This approach evolved into a style of programming that links processes together tightly but flexibly. An application, from a user's perspective, is just a collection of coherent programmatic interactions. There's no good reason to restrict the collection of interactions to a single computer. Computations -- at the very least the interactive components of computation -- need to move around the network.

To describe things using a venerable and useful vocabulary, we now routinely move models, views, and controllers around between machines. We are able to write applications that assume they will run on a collection of computers, rather than a single box.

The unix environment is a conceptual touchstone for us, here. Unix provides two fundamental abstractions that together offer a lot of leverage: every resource looks like a file and every process has standard input, output and error file descriptors. These two ideas enable programmers to design small, flexible utilities that combine to perform big, complicated tasks.

Contemporary GUI toolkits aren't built on any such fundamental abstractions and don't enable small pieces loosely joined (to borrow and repurpose David Weinberger's nice phrase). As unix-steeped hackers, we've always felt this to be a great loss. And we're keen to go back to the future, to create a new pipes-and-files foundation for the graphical world, to develop late-binding, duck-typing, defadvice-like idioms for applications programming.

We design on top of a clean and increasingly scalable multi-participant data interchange mechanism. And we can cooperatively render from multiple processes (including across the network). We'll write more about all of this in future posts.

It's worth noting, before moving on, that there are benefits to our network-soluble programming approach even when only one machine is used at a time. Many of our cross-machine parallelisms are equally useful in a multi-core context.

The final fundamental component we turned out to need was time. Of course, you get time for free from the real world (unless you're programming in a purely functional language), but not in a particularly useful form.

Most of today's GUI applications are relatively static. They respond to user input, which is an intermittent and low-frequency event stream. They probably set up a few loop or idle timers, too, and do some light processing when those fire.

Our g-speak applications, for a variety of reasons, tend to have more going on.

Making effective use of a lot of pixels -- pixels spread across multiple screens -- requires careful application of motion, transition, animation and scaling. And because gestural input opens up new possibilities for visualizing complex data sets, we tend to have a great deal of information on screen at a time, so we're always juggling graphics card and processor resources. The gestural input pipeline carries more data than, for example, mouse and keyboard event streams. Finally, because g-speak applications are often built out of multiple coordinated processes (possibly running on different machines), there's a lot of data flying around asynchronously.

We have two major mechanisms for dealing with all of this: soft values and buffered data stores.

The soft values framework provides a standard way to set up and use time-dependent member variables. We use softs extensively in graphical classes. Standard soft varieties wrap functions that converge in linear and asymptotic fashions, cyclical repeaters like sine functions, and softs that are built from or facilitate combinations of other softs. Floats, colors and vects can all be soft. New soft classes are easily derivable.

The network framework described above is constructed on top of a standard, flexibly sized buffer implementation. All data flowing through a normal g-speak application is rewindable. Most of the time we design applications so that every data pool is big enough to buffer a few seconds of throughput. It's also common to use much bigger pools. Video and audio, for example, are obvious candidates for more buffering.

These rewindable buffers take up disk space. And depending on access patterns and the quality of the kernel's virtual memory system they can eat RAM, too. But having temporally ordered, backwards-accessible data pervasively available across processes makes it possible to build new kinds of applications.

Programming is in large part defined by the exercise of identifying, extracting, polishing and creating clean abstractions for patterns that start to appear over and over. At Oblong, we found ourselves writing a lot of code that was really just book-keeping for on-screen graphics and for plumbing-level data caching. We've pushed much of that down into the platform layer.

Building g-speak is a design exercise at three levels. Most obviously, there is a new graphical computing environment -- a new look and feel, in our industry's argot. Those graphics are inseperable from an architecture that motivates and produces them. Finally, we design and use applications that run on top of this foundation.

You are Oblong. There are different ways to count, and depending how you do it you are two and a half years old; or you are four years old; or almost fifteen; or you're a quarter century; or you appear to have been born on 13 November 2008. You have many parts, and ideas have arrived from all over the place to build you, but there are some central strands too. Go back.

1.   It's about 1994. Part of you is pursuing a new line of research at the MIT Media Laboratory, trying to make information more literally spatial. Your feeling is that, ten years in, the GUI that's taken over the world's idea of interface isn't getting at everything there is. Substantial swaths of human brain are dedicated to understanding space, understanding geometry, understanding physical structure. A cartoon of a messy desk surface doesn't much tax these swaths. The swaths can work harder, ought to be made to. You propose that informaton -- and maybe especially the newly-blooming internet -- has a topology but not yet a topography.

You build a number of examples of how giving information a topography might work. You build something like a flight simulator for music, in which a listener-pilot swoops and zooms and dives above a vast active living musical score. The music plays in synchrony with dynamic elements that swarm atop the score to keep track of time, to keep track of instruments. It's true that one of the elements is a blue ball that bounces in time to the music and touches down on the notes that are playing just then. Also true: a ball like that can never be entirely serious, but it can be entirely effective. What works is assisted synaesthesia, making sound seem like sight and looking seem like hearing. The time in which music happens is turned into the space of the animated score.

What are the shortcomings of the flight simulator for music? One is that it appears on a screen, a big CRT. The musical topography is dimensional and beckons, but you're separated from it by glass. The second is that you must fly using a mouse. There are many ways to navigate above and around a self-playing musical score, many different vantages that reveal something useful or unique or desirable; and the transitions between vantages are equally illuminating. Transitions can show how things are connected. But the mouse only moves two ways. People call those ways X and Y.

2.   Excellent persons who work on VR solve the first shortcoming by putting the pilot behind the glass, right back there with the graphics. The goggles wipe out the regular world so that the graphics world can enfold the pilot. Yet here in the mid 1990s, it suddenly seems to you that perhaps something is backwards. For one thing, Jeff Bridges had a fairly rotten time when pulled into the graphics by the MCP in the movie TRON. For another, it's not the real world that is the problem. Many parts of the real world are perfectly good, and people spend a lot of time becoming experts at using the real world. Perhaps the problem is that the graphics aren't enough like the real world. This does not mean that the graphics should look more like the real world. Your brain does not actually care about that. It means that perhaps the graphics should behave more like the real world.

And the second shortcoming: when you need to describe to someone else how you want the music landscape to move, you move your two hands in midair. The someone else instantly sees the music landscape in your hands, even though the only thing between your hands is midair. This is because you and the someone else are experts at holding things and moving them around and seeing what things look like as they're held and moved. When you need to tell the flight simulator for music this same thing, one of your very capable hands descends to clutch the mouse and scrape it around. You scrape it in X and Y. Your fingers, clutching, can't do anything else. Your other hand is useless.

So it suddenly seems to you here in the mid 1990s that putting the graphics in the real world might possibly solve both problems right at the same time.

3.   Just then at MIT it turns out that there are a few amazing people who are already thinking about other aspects of that very same idea: the idea that the real world is very useful and computers ought to make an effort to do more there. These people call themselves the Tangible Media Group. They're smarter and have been thinking about this stuff longer. Because it's the most natural thing in the world, you join up with them.

4.   The first thing to figure out is how to get graphics into the real world. You mean for graphics to go actually everywhere. Graphics should be able to go on floors and walls, which stay put, but also on tables, which only mostly stay put, and on chairs and clipboards and pets and people and all else that stays put rarely. These surfaces can't display graphics on their own, and shouldn't have to. So you'll use something like a projector, or several. It occurs that the place to put projectors is where lightbulbs are. A lightbulb always has a privileged position. You suggest that maybe the lightbulb is already a projector, a projector that sends out a single giant pixel. So the world's lightbulbs just need to be granted higher resolution. Then the planetwide infrastructure for illumination can be an infrastructure for information.

One-way information isn't very good. But you're already putting a little projector in the new lightbulb. The bulb's glass lets light pass both ways, so something should be done with the light that moves from outside in. Put a video camera in there too, immediately next to the tiny projector.

This is the I/O Bulb: an input-output lightbulb.

5.   An I/O Bulb looks out at the world, and so any computation that uses an I/O Bulb has a way to figure out what's going on out there. Then it can paint a piece of the world with information, in response to what's going on. An I/O Bulb grips the real world on both sides, info coming and going. Is a single I/O Bulb enough? It is not. You are serious when you say that every surface should be a potential site for interaction. You don't mean everywhere all at once. There must be discretion, which is the same as saying design. But still: it might need to happen anywhere, so you have to make it work everywhere. Use more than one, then. Put enough I/O Bulbs around a room so that every corner, every cranny can be lit up with interaction. Coordinate their activities until computation that depends on them sees the room as one continuous input-output space.

This is the Luminous Room: architecture with computation respectfully overlaid.

6.   You build different I/O Bulbs, to test what it means, to show the idea working. You build one on a weighted base with a small projector and miniature camera on hinged rods. It looks like a Luxo lamp. This is an I/O Bulb for a desk, for producing a small adjustable spotlight of interaction. You build a larger one for the ceiling. A motor lets it rotate around a vertical axis. A mirror is on board, mounted at an angle on a second motor. Together the two motors can sweep the beam of interaction around to any part of the room. The Seiko Epson company shows unutterable generosity. As part of a collaboration, Seiko Epson builds a tiny I/O Bulb that is actually the size of a normal lightbulb. It is beautiful. You build other fixed I/O Bulb configurations, optimized to serve particular tables, specific walls. Always a projector and a camera. Always light going both ways.

This is the hardware for building Luminous Rooms.

7.   Now you need software. The software must manifest a purpose, some purpose for which anyone would want to use a Luminous Room. There ought to be no end of purposes like that. Choose a few.

You build a prototyping workbench for optical engineers. You build a fluid flow simulator, a kind of digital wind tunnel. You build an environment for testing urban planning scenarios. The optics workbench has little models of lasers, mirrors, lenses, beamsplitters. The I/O Bulb locates these for the benefit of an optics simulation. The I/O Bulb projects the calculated laser beam path down onto the table, aligned precisely with the physical models. When you move or rotate the optics models, the beam path reacts instantly.

In every way but one they're two wholly different things, the physical models and the projected simulation results. Perceptually, they are not different at all. Perceptually, they are part of the same causal system. This is all that matters. When you rotate a mirror, the beam reflected by it sweeps around accurately. The simulation happens in the real world.

The fluid flow system accounts for the presence of real-world objects. Real-world objects are valid obstacles in the simulated wind tunnel. You can see the projected fluid diverting around the obstacles -- your hand, a sandwich, a drafting tool -- as you know fluid has to. Physical models of buildings in the urban planning testbed cast computational shadows. Trying a different orientation means twisting the building with your hand. The shadows change accordingly. You understand the cause and effect with your hands and with your eyes.

This is how the world works.

Through 1998, 1999 you build many other things in the Luminous Room. You build a whole office whose walls know how to store digital objects in physical containers that you can hold. In this office unwanted objects, physical and digital alike, go in a real trash can. You install a second workbench across the room from the first. When you swivel a laser model and aim it toward the new workbench, the beam that disappears off the first table reappears on the second.

Of course it works this way. This is the way human minds know the world to work, so this is how things must. It is not optional.

8.   To make all this happen, you've designed the Luminous Room software to understand space. It acknowledges that people and objects and architecture all occupy one continuous space. But it also acknowledges that projected pixels end up in that same space.

So in the Luminous Room, every pixel has real-world coordinates.

9.   Toward the end of this time you also begin building applications that depend on no external physical objects. These applications concern themselves only with human hands. For many kinds of work, free hands enable highly effective interaction. Human hands, excused from having to grasp other objects, are incredibly dextrous, sophisticated, and expressive instruments for conveying intent.

Pointing, pushing, palping, poking: this is how hands convey geometric and spatial intent, superbly. Arranging fingers into specific poses or describing trajectories through the air: this is how hands describe semantic intent, gracefully.

In this mode, the interaction now involves hands and pixels solely. You find that the programmatic formalism of pixels existing in real space -- in the same space as the hands -- is of redoubled importance. Every screen, every display has a real-world location, size, orientation. These properties are imparted to the pixels housed by each display. Collections of pixels arranged to represent something coherent in turn inherit that geometry.

Everything that appears on a display is a real object in the world.

You begin to design a language of hands, a gestural language. The vocabulary and grammar of this language are very much about describing, manipulating, and navigating through space, but the language also prescribes additional elements that render it general. You intend this language to be broad and expressive enough, in fact, that it can drive a general purpose computing environment. Such an environment will not look much at all like existing general purpose computing environments.

It is a few months later that Minority Report visits your lab. The great production designer Alex McDowell and the legendary propmaster Jerry Moss have arrived to see MIT's emerging technologies. They intend to extrapolate forward from what they find, in order to depict the plausible future demanded by the film's director. They spend a lot of time with the Luminous Room systems.

10.   It's the new millenium, and you are three thousand miles farther west. You are the science & technology advisor to the film Minority Report, which is in preproduction. You have the broad charge of insuring that all the future technology seen in the movie's 2054 is plausible. But your major responsibility is to design the interface that will be used in several key scenes. In these scenes, characters must exercise an astonishing control over vast streams of image and video data.

You adapt the gestural language from the Luminous Room work. You train the actors to use this language. They become adept, even though it is partly an exercise in mime. The production will shoot the actors performing gestural tasks in front of an enormous transparent screen, but the screen will be blank, a prop. Graphics will be composited onto the screen in post-production. You understand that for a sense of causality to emerge the actors must be able to clearly visualize the effects of their gestural work. You assemble a training video showing this.

When the time comes to shoot, the director explains what sequence of analysis should occur in each scene. You translate this into the gestural language. You explain what graphical elements the actors would be seeing on different parts of the screen, how they are manipulating those elements. You make sure that a detailed account of all this is subsequently available to the editor and the visual effects people. Continuity of the original intent is critical. The cameras roll.

The movie appears in 2002. The scenes of gestural computation show something apparently real. You are a nascent part of Oblong, and it seems to you that the promise of a new kind of human machine interface has been vindicated a second time. The actors in the film are some of the very first g-speak users.

You are ready now to be Oblong.

Oblong Industries is the developer of the g-speak spatial operating environment.

The SOE's combination of gestural i/o, recombinant networking, and real-world pixels offers the first major step in computer interface since 1984.

Oblong helps clients deploy SOE-based solutions to real-world problems, including:

• analysis of large data sets
• operation of three-dimensional interfaces
• construction of efficient multi-user collaborative applications
• integration of large screens and multiple computers into room- and building-scale work environments
• development of large-scale applications that run interactively across enterprise networks

Architecture of the g-speak SOE

Oblong offers g-speak as a software platform suitable for licensing and embedding. Turnkey systems built on top of g-speak are also available, as is custom development support, both from Oblong directly and from integration partners.

The g-speak platform provides three core functional components:

• Gestural I/O
  Hi-fi input for HD output

  The g-speak platform is built around free-hand, three-space gestural input. Applications are controlled by hand poses, movement and pointing. Finger and hand motion are tracked to 0.1mm at 100 hz; pointing is pixel-accurate. Two-handed and multi-user input are fully supported.

  Gestural I/O removes the imbalance between high-definition graphical output available from modern computers and the narrow input channel of the mouse and keyboard. Gestural input is measurably more efficient at performing complex navigation, sorting and selection tasks. A full library of components providing UI building blocks makes it easy for application programmers to take advantage of gestural benefits.

  The g-speak input framework provides for mouse and keyboard use, too. Applications built to take advantage of gestural I/O also make use of legacy input devices when appropriate.

• Recombinant Networking
  Structured coordination and data interchange

  SOE applications process large data sets and support multi-person work-flows. The g-speak networking framework provides a collection of core library components that allow applications to scale transparently and dynamically across clusters of machines.

  This transparent scalability offers three major benefits: effective use of cpu power in a LAN environment; built-in support for applications that enable collaborative work across the network; and the ability to add functionality to applications at run-time by adding new code, new machines, new screens and new people to a work context.

  The recombinant networking framework also provides a set of tools for integrating legacy applications into a g-speak environment. Existing code bases automatically benefit from network transparency and limited gestural input. A low-overhead legacy extension architecture makes it possible to build support for full gestural I/O and spatial semantics with very little new code.

• Real-world Pixels
  Spatial semantics at the platform level

  Every graphical and input object in a g-speak environment has real-world spatial identity and position. Anything on-screen can be manipulated directly. For a g-speak user, "pointing" is literal.

  The g-speak implementation of spatial semantics provides application programmers with a single, ready-made solution to the interlocking problems of supporting multiple screens and multiple users. It also makes control of real-world objects (vehicles, robotic devices) trivial and allows tangible interfaces and customized physical tools to be used for input.

  The g-speak platform is display agnostic. Wall-sized projection screens co-exist with desktop monitors, table-top screens and hand-held devices. Every display can be used simultaneously and data moves selectively to the displays that are most appropriate. Three-dimensional displays can be used, too, without modification to application code.

Next-generation Computing

The g-speak platform is in use today at Fortune 50 companies, government agencies and universities. Application areas include:

• Financial services
• Telepresence
• Network operations centers
• Logistics and supply chain management
• Military and intelligence
• Automotive
• Natural resource exploration
• Data mining and analytics
• Medical imaging
• High-touch retail
• Trade shows and theatrical presentations
• Consumer electronics interfaces

Oblong delivers room-sized and single-user g-speak environments as turnkey products.

A software development kit that runs on both Linux and Mac OS X is available. Applications are source-compatible across both operating systems and can run on ordinary desktop and laptop computers in addition to gesturally-equipped g-speak machines and clusters.

Oblong also works closely with partners to develop and deliver applications and integrated solutions.

If you are interested in g-speak products, applications or partnerships, please contact us.