3D & Other Senses - Examining How Sound, Smell &.......

3D In Another Sense - Less Technical

Sound, Touch, Heat, Smell, and Taste

The term “3D” almost always means “visually 3D.” Our eyes use lenses and receptor cells to focus and record light from many directions at once. We can see in millions of shades and colors and discriminate details with exceptional resolution. The end result is that the sense of vision is extremely complex and really taxes our brains (you can tell, because the visual cortex is a relatively large part of the brain compared to the areas reserved for other senses).

All of our other senses combined probably don’t use as much brain power as vision. Still, they can tell us things that vision cannot. If you hear a sound behind you, smell a dangerous, but colorless gas, or sit on something hot, you’ll know more about what’s around you than just what you can see.

Smell and taste are both based on interpreting chemicals that reach your mouth and nose. Touch is the sense that helps us navigate around objects accurately and quickly (sight helps with this at a distance, but not as much for instant changes around us). The sense of heat tells us the average energy of the air and objects we contact. Sound tells us what’s moving or changing, and can even penetrate through objects and walls. Additional senses include balance, body position, and pain, each with their own unique function beyond the realms of the other senses.

After sight, the sense of sound is the most important guide we have in the 3D world. It’s also the cheapest and easiest sense to copy for the greatest relative quality. Gramophones, telephones, cd players, and speakers were all created to give us new and unique simulated environments. Even music is a part of analyzing our environment, because it describes a set of instruments with unique locations and vibrations.

Speakers are the tools we use for recreating sound. For a specific 3D space, the placement of speakers tells us exactly what we can and cannot hear. We can hear the location and magnitude of recreated sounds that were originally in the speakers’ locations, but we cannot hear the location of recreated sounds from other places. The number of things we cannot hear accurately can be reduced by adding more speakers or using audio tricks, but the technique is limited by practicality, and doing so doesn’t help us discover exactly what 3D sound is in an environment. We need to understand it completely before we attempt to recreate it.

Let’s think about sound in a different way than you may be used to. Imagine that you can hear as if you could ‘see’ sound. Pretend that the objects around you are blurry, wavy blobs centered on where you hear the most noise. Each of your ears is doing this already, you just don’t think of it as an ‘image’, but as a bunch of independent noises. These ‘images’ are the foundation for understanding sound as a 3D sense.

The idea of perspective can be applied to more than just what we see; it can also work with the patterns and angles of sounds we hear. Further sound sources seem to get ‘smaller’ and more localized, nearer sound sources seem ‘bigger’ and harder to isolate (like the distorted shape of a close object). Because we have earlobes and a strong sense of changes in tone, we can hear the angle at which each noise is hitting each our ears (plug one ear then turn your head towards and away from a noise and you will be able to hear the angular difference clearly). These concepts create a true ‘image’ for our ears to ‘see’, even if it is at a relatively low resolution and is sometimes unreliable.

The end result of the perspective from each of our ears is that any specific number of speakers cannot reproduce an environment perfectly. Our ears will ‘see’ those speakers as objects and determine that the sounds are coming from their positions, even though the original environment had sounds coming from different sound ‘objects’ with different ‘shapes’ and locations. Multiple speakers can work together to convince our ears they are ‘seeing’ objects other than speakers, but that trickery does not work equally well for listener’s that are in different positions, and it only works for imitating ‘objects’ directly between two speakers. Our ears will always know something is amiss. This means that truly recreating 3D sound means making the right ‘images’ for each of our ears (just like 3D displays need to create the right images for each of our eyes).

The senses of taste and smell both work by reacting to unique chemicals and producing different sensations. Smell is simpler than taste because it only works on gases (if there is a liquid or solid up your nose, you probably have more problems to think about than what smell it has). Smell also feels like it’s ‘all in one place’ up our noses. Taste is much more complex because not only can we register liquids, solids, and gases in different places, our tongues and mouths can also feel pressure and temperature (not to mention that they can move on their own).

The sense of taste exists in 3D as the moving top surface of a tongue. Although it’s a common myth that each part of tongue is reserved for each taste, the different places on our tongues can still receive different tastes simultaneously. This local sense of 3D goes all the way back into our throats, where we can feel and taste swallowing (when it comes to recreating 3D taste, it’s probably safest to avoid the back of the throat).

Smell has two different ways it can be thought of in 3D: as the local surfaces inside a nose, or as all the smells in a surrounding area. The local variety is pretty simple because we can’t really feel different smells on different parts of our noses (unlike with our tongues). This means that recreating 3D smell can be as simple as continuously spraying new smells into a person’s nose as they move to different places.

If the 3D environment of smell is defined as the space around a person, it becomes much more complicated. To really recreate the scents of an environment, we’d need to spray tons of different scents into the air so they were where they were supposed to be, even when wind and time are taken into account. I’m sure it’s possible (and it would be neat because you wouldn’t have to have something spraying directly into your nose) but it’s definitely much more complex than the local variety of 3D smell.

The senses of touch and heat are both present throughout our bodies, so a 3D version of either is really just a matter of pinpointing “where on the body” and “how much to apply.” Although the sense of heat can be made more complex (temperatures can be defined throughout a space, just like with scents and smell), it won’t really change how it’s perceived. Contrarily, the sense of touch has features that local pressure just can’t cover. We don’t just feel objects; they can also affect our motion and position.

To make an entire room feel right in 3D means to make a room that has changeable objects or things that feel like objects. The general idea is that each point in a space should have its own force that changes correctly when you interact with it (e.g. if you push a wall, the force it pushes back with always becomes greater to match). This is a pretty complex task for simulators, but it highlights the 3D nature of touch as something more than just the sensation it gives. Its additional aspects means that touch also has an indirect relationship to other senses, such as body position, balance, and even pain felt from a particular position and situation.

Although most people only think they can see and feel in 3D, each sense actually has its own unique interpretation of three dimensional space. Looking at these different spaces individually is an important part of understanding modern 3D technology and where it’s headed. Dual-image 3D may currently seem to be the ‘most 3D’ you can get, but it’s just a small step in the long journey of 3D’s ultimate goal “to recreate the world in every sense.”

4/19/11

Change is Silver

Author of “How to Make a Holodeck” (5Deck.com)
~A funny, colorful manual details an exclusive type of glasses free three dimensional display.
Creator of Unili arT (UniliarT.com)
~Sarcastic, ironic, random, and carefree graphic designs on a variety of fun products.

3D In Another Sense - More Technical

Audition, Tactition, Thermoception, Olfaction, and Gustation

The concept of three dimensions is at its strongest with the sense of vision. Cones and rods throughout our eyes interpret the combined wavelength, quantity, and direction of light focused through our corneas onto each point of each of our retinas. The density of cones and rods drops off rapidly at any distance from the center, and there are fewer cones at high angles. All of this gives a specific map for our brains to interpret, and it’s one that changes continuously. The end result is an extraordinarily complex sense with all the 3D results we know and love (or get confused by).

All the rest of our senses combined probably don’t come close to the amount of information we get from light. However, they can give us critical information that light cannot. Chemical senses like gustation (taste) and olfaction (smell) give us go or no-go signals for foods and places. Tactition (touch) is the only real way our bodies can quickly interact with the world (no other sense can even come close to recreating the full benefits of this “rapid guidance system”). Thermoception (heat) is found throughout the body in tow with tactition; it allows us to sense the average kinetic energy present in air and other substances. Finally, audition (hearing) gives us cues to happenings in any direction, including behind solid objects and walls. Additional internal senses include equilibrioception (balance), proprioception (posture), and nociception (pain).

Audition is our primary target for discovering and recreating the 3D of the real world when vision is removed from consideration. Not only is the easiest sense to duplicate in terms of variety and cost, it is also the only sense that can locate and identify distant objects when vision is limited (tactition and thermoception have this ability with nearby objects).

Sound is recreated using one or more speakers. The number of speakers and their specialties (low, high, or all frequencies) sets the limiting factors on a given 3D sound environment. In a perfect 3D sound system, every wavefront that was present in the original environment (or calculated from an artificial environment) should be present in its original configuration. The task sounds simple when you consider that a single speaker can add countless sounds together into a single output that will affect an entire room. It’s not that easy though.

In the article “3D Perspective (More Technical)”, ‘perspective’ is described as something that can apply to sound. This is uncanny for people who understand it only in terms of vision, but let’s try an experiment. Turn your head so that your best ear is facing one or more noise sources. Now close your eyes and plug the ear that is not facing those sources (of course, read through the rest before you close your eyes or you won’t know what to do). The most prominent sense you experience should now be the sounds you hear from one ear (unless the temperature in the room is unusual or you are in pain). Imagine that those sounds are images, like blurry, wavy lines that are centered around ‘objects’, each with a shape of their own. Even if you can’t visualize it, this is the 3D perspective your ears are ‘seeing’.

Each part of each of object in a room can emit, reflect, and absorb sound waves. Each resulting wavefront will reach your ear at a different angle and intensity. Although ears don’t have the perfect angular tolerance of the rods and cones and your eyes, your brain can compensate for waves that come in at an angle and automatically interpret their most likely origin. Each wavefront will bounce off of your ear lobes and into each ear canal with a different pattern, and highly angled sounds grow quieter (angled waves hit the ear canal with a smaller profile and have to bounce back and forth to reach the tympanic membrane, losing energy with each reflection). These distinct characteristics give each of your ears a relatively low resolution sense of ‘sight’.

So each ear has the individual ability to create an ‘image’ of sound from the environment it faces. The only reason it doesn’t ‘look’ like a picture in your mind is likely because sound does not present enough information to provide a reliable image (with vision, objects you perceive don’t just suddenly change positions because your eyes didn’t perceive the data correctly-well… actually they can sometimes with optical illusions, but it is extremely rare in normal life). This unreliability means that giving you too much information (for example, enough to generate an image in your visual cortex) would result in misinformed decisions rather than the skepticism and caution that keep you alive.

The end result of all this speculation is that sound cannot be arbitrarily and perfectly recreated by any discrete number of speakers. Everywhere you put a speaker in a room is an ‘object’ your ears will ‘see’, so any sounds it produces will have that artificial position. Multiple speakers can create false ‘imagery’ in parallel, but it will only be correct for a listeners intended position (if you go right up to one speaker, the combined sound will seem to be coming from two different places, or just the one you’re next to if it’s loud enough). For your average simulation this is almost always good enough, but truly recreating 3D sound means that the 3D perspective of every listener’s ears each have to ‘see the right image’. The “How to Make a Holodeck+++” article “3D Sound?” at 5Deck.com describes the full complexity a 3D sound system would need to recreate an environment without error.

The senses of gustation and olfaction both use a continuous palate of chemical receptors (as well as a sense of tactition and thermoception in the case of gustation) to register gases, liquids, and solids. Olfaction only processes gases, but liquids, solids, and gases are present for gustation. In the case of a 3D environment, gustation is not a major concern because it requires direct contact with a person’s mouth, which only takes up a tiny portion of an interactive environment. The greatest sense of 3D with gustation is the fact that each taste can be experienced on each different part of the tongue (note that each taste is not localized to a different part of the tongue-a common myth).

The tongue goes back into the throat quite a ways, and the sense of swallowing it produces is a unique sense in itself (chewing food but not swallowing it is not satisfying because your body can feel the pressure, taste, and temperature after the point of no return, and it doesn’t like it if you trick it because it knows it’s not getting any nutrition then); but trying to directly address the throat area is probably too dangerous. Any other sensations in the mouth are more a matter of tactition and thermoception, which both give food an additional 3D presence.

Olfaction is available for consideration in two different 3D systems within the same environment. The first system is the nostrils and nasal cavity, just like with gustation. The alternative system (it probably doesn’t make sense to use both at once) is a 3D field of chemical density. Each point in a 3D space can have each nasally registrable chemical at a certain density. Such a field could be considered a 4 dimensional space for each chemical, with three dimensions referencing a position and the 4th indicating chemical concentration. Moreover, every real space would have the chemical concentrations naturally diffusing and the motion of wind carrying them around. This means that each chemical concentration can be considered 5 dimensional with time.

If all of the chemical concentrations in a 3D space were added onto a single graph, it would end up being 4 dimensional (space and time) plus 1 dimension (concentration) for each unique chemical. The complexity of this 3D system is apparent in the sheer number of dimensions. Although you could try to approximately recreate a 3D olfaction environment with continuous, dynamic, multi angular chemical sprayers, it is probably a better approach to just calculate all of that in a computer program and address the 3D space in the nostrils and nasal cavity directly (noting that our noses can’t really detect the exact inner location of smells, but that it would still probably feel weird if they were all coming from one place-unless you wanted to simulate a clogged nose…). The rear of the nasal cavity is very sensitive, so the best approach may be to address only the nostril area, just like addressing gustation is safer in the mouth than in the throat.

The senses of tactition and thermoception each occur throughout the body, even deep under the tissue in certain areas. Addressing such a 3D system has two options, just like with olfaction. The first is to directly address the different parts of a person’s body and the second is to create a physical 3D environmental system (remembering that an environmental system will get rapidly out of control in terms of dimensional complexity). The direct addressing option is relatively simple, but it limits forces to just pressure and local temperature. In the case of temperature this is alright because it is merely a sense of heat transfer and not a total force that can cause motion. However, the environment addressing option for a physical 3D system has the bonus of providing forces that affect a person’s position and motion, not just the local pressure they feel.

To address an entire room for tactition requires a changeable, multi-dimensional ‘force field’. The idea of a force field is nothing special because it’s only a template for creating something real, not an actual object or mechanism. A force field would be “a 4D reference space plus three directional force vectors for each point as 3 extra dimensions.” The idea is very similar to how electric and magnetic fields are drawn in physics, except that in this case the forces can be made by anything, because their source is not yet specified. These forces could be made by actual moveable objects, like generic building block shapes, or some sort of fluid-like system, such as threads, plates, or ribbons (maybe each with their own magnetic fields or possibly even static electric fields in an isolated, highly charged, ungrounded room).

Creating a force field with actual objects or flexible stand-ins has more advantages than direct addressing. Objects in a room can not only create pressure sensations on the different parts of a body, they can limit a person’s motions, halt them altogether, or even move them against their will. Objects in a real environment can also be moved and an entire environment can suddenly change. These concepts are all critical to fully 3D tactition. A proper tactile environment can even account for proprioception and equilibrioception, because they are both related to body position via surrounding forces. See the “How to Make a Holodeck+++” article “Force Fields” at 5Deck.com to learn about a complete hybrid approach for creating a Force Field.

The only 3D sense this article did not cover is nociception. “How to Make a Holodeck” discusses it briefly along with a few additional senses (Introception, Extraception, Coception, and Mortiception). Although sight is often the only sense referred to during discussions of 3D technology, each of the other human senses has its own dimensional map that deserves its own consideration. After all, the unwritten goal of 3D technology is “to recreate the real world in every sense.”

Note: The term “3D” used in this article refers to three “spatial dimensions.” In “How to Make a Holodeck,” I use the term 3D with time as a possible dimension (2 spatial, 1 time). Because almost all situations where 3D is discussed involve time without explicitly adding it as a dimension, I prefer the term 4D for most instances where people use 3D, but I still use 3D for ease of understanding.

4/19/11

Change is Silver

Author of “How to Make a Holodeck” (5Deck.com)
~A funny, colorful manual details an exclusive type of glasses free three dimensional display.
Creator of Unili arT (UniliarT.com)
~Sarcastic, ironic, random, and carefree graphic designs on a variety of fun products.