When you program for a designer, you don't always have creative input to use the live compositing capabilities of a media server. With Ortel, you're expected go further, beyond conventional programming. For Swan Lake, Ortel, Tucker, and I had the challenge of producing romantic, reactive, non-interruptive projections. Fortunately, the musical tempo paces the projections intuitively.
Ortel utilizes ingredients of simplicity and complex elegance. Being able to tell the difference between swans flying across a saturated dusk panorama and a full transformation from maiden to swan, one might assume the latter is more complex to achieve. Both are actually quite intricate and require musical, mathematical, and mechanical precision. However, swans gracefully flying from stage-right to stage-left over a saturated cloudy night sky in silhouette, past the moon, and then peacefully traveling offstage brings with it a new programming paradigm: curve programming.
Ortel wanted swans flying across an HD panorama rear-projection of night clouds blended in a 16x9 aspect ratio. These swans intersect and are in sync with 4x3 aspect ratio swans across the center-stage moon projection, exit the 4x3 aspect ratio moon projection back into the 16x9 aspect ratio, and continue through the rest of the raster. At first, one might be tempted to use a moon-shaped mask in the 16x9 projections that would facilitate the primary set of swans travelling left to right. The key problem is scale and plane intersection, and the solution is implementing a curve.
Curve programming is defined by several component groups. The first is the living variable group, comprising environment, aperture, breathing, and blend. Second is the mechanical group, including sync, scale, media, server, control, and braking. Third is perceived sync, which creates an individual's reality, causing an emotion-laden response, as the swans exit, and the curtain falls. The response for Ortel, Tucker, the audience, and me is, “Yes, please pass me a tissue.”
For the living variable group, the environment had to be defined: Where are you transporting the audience, and can you do it without distraction? Our direction was to embody the romance of Swan Lake. For aperture, we had to determine the viewpoint. Where is the focal point? Is there a dominant, subdominant, and subordinate relationship of all elements inside the aperture? And can you maintain the illusion?
To counter any “breathing” effects of the building, San Francisco Ballet head electrician Dennis Hudson quite elegantly took some old fire hoses and filled them with sand, making the longest sandbag ever. By placing these on both sides of the projection surface, we eliminated most of the impact of the building inhaling and exhaling, which can push the blend out of focus. The best part of the fire hose sandbag was the incredibly low profile—smooth and unobtrusive. Thanks, Dennis.
To properly have an HD panorama, you blend two halves of an image together in the middle by utilizing overlap and soft-edge. Because of how the building breathes versus projector placement, both halves of the keystone do not have equal adjustments. This is one of the key factors in the perceived sync of the movement across the screen. The most dangerous place to try to sync media is within a compact area on a horizontal plane. The synergy of the elements in the living variable group places the swans “in” the scene, as opposed to “on” the scene.
In the mechanical group, synchronization is achieved two ways: one in the world of mechanization and the other in the realm of perception. Machines have a preferred codec, or a control setting, for syncing media. Humans have a perception of sync, consisting of speed, action, scale, and color. This mechanized output activates our human abilities of perception and intertwines them to achieve a believable illusion. This was where Ortel and Fensom became key factors in deciding the perceived sync. By being in total control of the sync, the programming choices of flight path, height, and scale could be completely based upon the mood and tempo, without mechanical handicaps.
In terms of scale, both the 16x9 and the 4x3 swans must be identical in size and play in the same X and Y axis, both off and on the raster. Therefore, you choose the dominant aspect ratio and make the sub-dominant aspect ratio follow suit.
This brings us to media. Will your media playback in sync, without any double images or skipping? The simple practice of confirming the servers' sync capabilities is very often overlooked and can destroy the illusion completely. Tucker rendered Ortel's swans in a clip, and, utilizing the grandMA console, I can control the flight path, keeping perfect sync across the two servers it requires to establish the flyby. For this production, the four Pandoras Box Media Server Pros were the best choice for smooth 3D movement, positioning, and scale via DMX control. To administer the servers, three Pandoras Box Media Manager software systems are placed on the system for designer, compositor, and programmer. This requires a custom Pandoras Box configuration that necessitated a consultation with Kevin P. Morris (aka, “Lord of the Servers”) of coolux USA. We found out that the servers could take advantage of their own networking capabilities for media management, which speeds up the flow considerably.
Fortunately, San Francisco Ballet general manager Debra Bernard and Hudson invented a new department just for servers and gave us Erik Docktor as the server tech, Maria Mendoza as our operator/department head, and projectionist (and rollercoaster enthusiast) John Coffield. Without Bernard's continued support throughout the project, quite simply, we would have never gotten off the ground.
For control, this system gobbles up around 16 universes of DMX. I used the grandMA to create many different control options, such as manual overrides for speed, which can be manipulated independently from the primary cue list by Mendoza. All of these controls were programmed on the grandMA to turn themselves on and off, as required, all while maintaining the default state of the projections onstage, and without destroying the illusion.
Braking is the rubber band that holds the illusion together and one of the main ingredients of curve programming. In order for the 16x9 swans to catch up and stay in sync with the 4x3 swans, the 4x3 clip has to start playing at the same time as the 16x9 clip and wait for the 16x9 to overtake it. The 16x9 overtakes the 4x3 as it is in motion, they break the plane of the moon together, and at the far side of the moon, the 4x3 swans exit the raster, allowing the 16x9 swans to finish. Without any double image, the illusion is held together as the swans fly, breaking the plane of the moon completely in sync and remaining in sync across the entire raster with playback, scale, and movement. Now this is curve programming.
The curve equation looks a little bit like this: Go > Wait > Go > Brake > Wait > Ramp > Brake > Release. Thus, the whole equation resembles a curve. For more illustration, watch a child draw a circle—poetically simple.
This brings us to the last element: perceived sync. What activates perceived sync is our ability to believe in the illusion. This is accomplished by immersing the audience members in the illusion, as opposed to beating them over the head with it. Once the projections are considered as the environment, we can then manipulate the perception. Because curve programming controls all elements of movement, perception can be altered by changing movement, music, pace, height, width, scale, or just because the director asks us to reroute the flight path of the swans. The programming follows the perception of the design instead of the programming dictating the design.
Peter Vincent Acken has been pioneering the art of media design and media server programming since 2005. He resides in Battery Park, Manhattan, and when he's not working as a mad scientist/video programmer/live compositor/ anti-sneakernet vigilante, he's riding his motorcycle with his wife, their last big trip a 10,000-mile circumnavigation of the US.