When generating musical sound on a digital computer, it is important to have a good model whose parameters provide a rich source of meaningf ul sound transformations. Three basic model types are used widely today far musical sound generation: instrument models, spectrum models, and abstract models. Instrument models attempt to parameterize a sound at its source, such as a violin, clarinet, or vocal tract. Spectrum models attempt to parameterize a sound at the basilar membrane of the ear, discarding whatever information the ear seems to discard in the spectrum. Abstract models, such as FM, attempt to provide musically useful parameters in an abstract formula. This paper addresses the second category of synthesis technique: spectrum modeling. It describes a technique called spectral modeling synthesis (SMS), that models time-varying spectra as ( 1) a collection of sinusoids controlled through time by piecewise linear amplitude and frequency envelopes ( the de- terministic part), and (2) a time-varying filtered noise component ( the stochastic part). The analysis procedure first extracts the sinusoidal trajectories by tracking peaks in a sequence of short-time Fourier transforms. These peaks are then removed by spectral subtraction. The remaining "noise floor" is then modeled as white noise through a time- varying filter. A piecewise linear approximation to the upper spectral envelope of the noise is computed far each successive spectrum, and the stochastic part is synthesized by means of the overlap- add technique. The SMStechnique has proved to give general, high quality transformations far a wide variety ofmusical signals.