When the fiery clash between Broadway producers and musicians was resolved after a short strike in March, David B. Smith, chair of the program in Entertainment Technology at New York City College of Technology, was delighted. “All this rancor was not good for Broadway, and I think the agreement ended up in a fair place.”

	David Smith working with students

City Tech’s resident sound wizard had a keen professional interest in the Broadway cliffhanger, however. When Smith is not focusing on his campus duties, his thoughts are apt to float through the looking-glass into the astonishing sound world of the “virtual orchestra”— a terrain populated by MIDI files, DML’s, “patches,” and TAP edits. In especially dreamy moments, Smith—who earned a Doctor of Musical Arts in composition and electronic music from the University of Cincinnati’s College Conservatory of Music—will sometimes peer into the future and see million-point Fast Fourier Transform convolution and performance spaces boasting a thousand speakers.

Welcome to the leading edge of virtual orchestral sound design, to which Smith has devoted his professional attention for 15 years—virtually, as it were, from the birth of the field in the late 1980s. His interest in computer-generated music began in the heyday of Switched on Bach and the Moog synthesizer (that is, Professor Moog of Queens College), when he was well along in a 15-year stretch of violin studies.

	The Sinfonia “box”

In 1999, Smith and Fred Bianchi, a digital music colleague from his Conservatory years, decided to put their expertise to the commercial test and formed the Midtown Manhattan firm Realtime Music Solutions. (Bianchi is director of Music Technology at Worcester Polytechnic Institute.) Their company was involved in the producers’ Plan B, in case a lengthy musicians’ strike threatened to silence Broadway. “We rehearsed our virtual orchestra with two shows, Les Misérables and Thoroughly Modern Millie, with full casts,” Smith says. “Rehearsals went well, and the producers were happy.”

But Smith was well aware of the dangers of making a debut with his new technology—trademarked Sinfonia—in such a volatile situation. “Neither the Broadway producers nor I want to replace a whole live orchestra with a virtual one. Broad-way is Broadway, the top of the art form, and it will always require human musicians in the pit,” Smith says. He did not want to become a “screaming focal point” of anti-sound enhancement demonizers.

Many with sharp ears have considered Broadway sound a disaster for years. Indeed, Smith considers his new and always-being-refined Sinfonia as a form of disaster relief from various old-fashioned sound enhancers that are commonly used in Broadway pits. “Our main focus is not to replace live musicians but to replace enhancement solutions that straitjacket a musical performance.”

The unmusicianly villains—unmusicianly because unresponsive to a conductor’s control—that Smith has in mind are synthesizers, sequencers (computers that control synthesizers—think player piano), click tracks (prerecorded parts that unfold at a given metronomic pace—think canned music), not to mention behind-the-curve sound retrieval and amplification systems.

Unbeknownst to New Yorkers, Realtime’s hardware-filled “boxes,” about the size of a microwave oven, have been traveling around the nation for several years now with considerable success—and not a single serious mid-performance “crash.” Sinfonia units are currently augmenting small orchestras of between eight and twelve live players with virtual musicians in national tours of Seussical, The Music Man, Miss Saigon, and Cinderella. The Realtime library of shows is currently about 30, among them Annie, Ragtime, Guys and Dolls, Fiddler on the Roof, and Into the Woods.

How does it all work? Once a score is captured, the individual instrumental parts are edited with a variety of “patches” that indicate how any given note or passage of notes will sound, based upon recordings made in Realtime’s studio by live musicians who have recorded every note in their instrument’s range in every possible manner of articulation.

As Smith, a violinist with 20 years of orchestral experience, explains, the refinement of nuance is considerable. “We have programmed more than 50 different ways to articulate a note on a violin—legato, pizzicato, short staccato, détaché, sforzando, to name a few. Our software, trademarked Sinfonia, allows us to make an initial choice from these for every violin note in a score. If the conductor asks for a change during rehearsals, we can create and/or edit a patch on the spot.”

The Sinfonia player—called, naturally, a Sinfonist—Smith likens to an associate conductor. He also has much editorial flexibility. For any given production or performance, it is easy to transpose songs, cut or rearrange their order, alter the sound volume, and, of course, mute the instrumental lines which are being played by live musicians.

The essence of Realtime’s technology, however, lies in a key labeled TAP. With each hit of the TAP key, the music moves at the assigned value (quarter, eighth, half, etc.), following the tempo of the performance. Sinfonia is able to follow the expressive nuances of the conductor and the stage action just like any other element of a live performance. Other keys at the disposal of the Sinfonist—who must of course be a trained musician with experience following a conductor—are CRUISE TAP (same as cruise control on the highway), A TEMPO, PAUSE (good for long fermatas when a final note is being belted), and VAMP (handy when a prop or set gets “hung up” and the performance is delayed).

Louis Crocco, a Sinfonist who has been on the road with Miss Saigon, considers himself as “live” a musician as any of his colleagues: “It is a musical instrument… It’s about timekeeping, listening to the other players, and watching the conductor. It’s not just plugging it in and turning it on.”

In 1997, Smith arrived at City Tech, whose Entertainment Technology program is the only one of its kind in the Northeast (its current 100 majors specializing in such specialties as sound, lighting, and scene construction, projection, and—recently added—show control). In spring 2000 his student sound engineers and designers were mounting the College’s first application of virtual orchestration in a major musical production, Evita, by the College’s TheatreWorks company.

Realtime’s goal is not a modest one. Smith says it is “to develop a perfect musical instrument for the 21st century,” and he is not shy about making the most of a quickly advancing state of the acoustic art. Music history, he points out, is on his side: “The 16th century was a ground-breaking era in Italian cabinetry, and violin makers like Amati, Guarneri, and Stradivari made brilliant use of the new technology. Likewise, 19th century advances in machine tooling made possible the rapid improvement of old-style, valveless brass instruments.”

In Smith’s view, the ideal is not an entirely ghostly orchestra. An accomplished live musician himself, he enjoys live performances—at Avery Fisher and Carnegie Hall—too much to want that. He wants to fill seats that would otherwise be empty. Finally, Smith hopes his perfect instrument of the 21st century will become popular among composers of the avantgarde. “Forget simulation! Use it to expand the musical palette.” Smith is showing the way, with more than 50 compositions for the theater to his credit. Just possibly because his wife is an operatic lyric soprano, he has also ventured into opera. One current project is a chamber opera titled Brecht at HUAC, about the German playwright’s virtuoso testimony before Congress amid the communist witch- hunting after WWII.

After mentioning that he has also been at work on a sequel to The Magic Flute, Smith laughs and ventures a very plausible hypothesis: “If Mozart were around today and heard about this new technology, I’d like to think he’d say, ‘Sounds great—let’s use it!’”