US 5928330 System, device, and method for streaming a multimedia file
ABSTRACT – A system and device for, and method of, presenting multimedia information. In a client-server context, the invention includes a client that receives units of the multimedia information and presents the information on a presentation device. Each unit of information has an importance value assigned to it, which in an exemplary embodiment is indicative of the unit’s importance in relation to the quality of the presentation. The invention includes a mechanism for characterizing the performance capabilities of the system. For example, several conventional statistics may be gathered and analyzed concurrently with the streaming operation and before it begins. The invention includes a mechanism for inferring network conditions from the characterized performance. The server may then stream the units of multimedia information to the client at a streaming rate and adapt the streaming rate of the streaming in response to the importance information and in response to the inferred network conditions.
FIELD OF THE INVENTION
The invention generally relates to real-time multimedia applications and, more particularly, to the streaming of real-time multimedia information over a communication network.
BACKGROUND OF THE INVENTION
Generally speaking, multimedia applications present related media information, such as video, audio, music, etc., on a presentation device, such as a computer having a display and sound system. Some multimedia applications are highly interactive, whereas other applications are far less interactive. For example, a game is a highly interactive application in which the application must respond to many user inputs such as keyboard commands and joystick movements, whereas viewing a video clip is less interactive and may only involve start and stop commands. Moreover, multimedia applications may be directed to standalone single computer contexts, or they may be directed to distributed, network-based contexts.
At a very high level of abstraction, following the producer-consumer paradigm, any multimedia application involves producing and consuming the related multimedia information. The above examples of highly interactive and less interactive and standalone and network-based applications differ in the manner in which the information is produced and consumed and the complexity in controlling the production and consumption .
For example, in PCs and other standalone contexts, the information need only be read off a local CD Rom or the like, and thus, producing the information to the consumer involves relatively predictable characteristics and requires relatively simple control logic. In network-based contexts, on the other hand, the information must be produced over the network and is thus subject to the unpredictable characteristics and intricacies of the network: data may be lost, performance may vary over time, and so on. Consequently, the control logic may need to be relatively complicated.
In either the standalone or network-based context, consuming the information involves presenting the related information to the corresponding presentation components in a controlled manner and in real time. For example, to provide intelligible audio-video clips, the video data must be provided to a video driver and audio data must be provided to a sound card driver within specified timing tolerances to maintain intra- and inter-stream synchronism. Intra-stream synchronism means that a given stream, such as audio, is presented in synchronism within specified time relationships, in short, that the stream itself is coherent. Inter-stream synchronism means that multiple related streams are presented in synchronism with respect to each other. Concerning intra-stream synchronism, all streams, for the most part, should present data in order. However, users are more forgiving if some streams, such as video, leave out certain portions of the data than they are of other streams, such as audio, doing the same. A video stream with missing data may appear a little choppy, but an audio stream with missing data may be completely unintelligible. Concerning inter-stream synchronism, poor control will likely result in poor “lip synch,” making the presentation appear and sound like a poorly dubbed movie.
In the network-based context, one simple model of producing the information involves the consuming entity to request the downloading of the multimedia information for an entire presentation from a server, storing the multimedia information. Once downloaded, the client may then consume, or present, the information. Although relatively simple to implement, this model has the disadvantage of requiring the user to wait for the downloading to complete before the presentation can begin. This delay can be considerable and is especially annoying when a user finds that he or she is only interested in a small portion of the overall presentation.
A more sophisticated model of producing information involves a server at one network site “streaming” the multimedia information over the network to a client at another site. The client begins to present the information as it arrives, rather than waiting for the entire data set to arrive before beginning presentation. This benefit of reduced delay is at the expense of increased complexity. Without the proper control, data overflow and underflow may occur, seriously degrading the quality of the presentation.
Many modern multimedia applications involve the transfer of a large amount of information, placing a considerable load on the resources of the network, server, and client. The use of network-based multimedia applications appears to be growing. As computers become more powerful and more people access network-based multimedia applications, there will be an increased demand for longer, more complicated, more flexible multimedia applications, thereby placing even larger loads and demands on the network, server, and client. The demand placed on servers by these ever-growing multimedia applications is particularly high, as individual servers are called upon to support larger numbers of simultaneous uses: it is not uncommon even today for an Internet server to handle thousands of simultaneous channels. Consequently, there is a need in the art for a device, system, and method that, among other things,
can handle longer, more complicated presentations;
utilize a network’s resources more efficiently; and
utilize a server’s and client’s resources more efficiently.
In short, the invention involves a new file format for organizing related multimedia information and a system and device for, and method of, using the new file format. The invention eases the management and control of multimedia presentations, having various media streams, each of a specific type, each specific type further classified by encoding type, subtype, and encoding rate. Thus, with the invention, an application may support several instances of a particular media type, with each instance having different characteristics. For example, the application may support multiple audio streams, with each stream in a different language, from which a user may select. Analogously, with the invention, the application may choose a given instance of a media stream based on the network’s characteristics; for example, the application may choose an audio subtype that is encoded for a transmission rate that matches the network’s characteristics. The invention reduces a server’s memory and processing requirements, thus allowing a server to simultaneously service more requests and support more channels. And, the invention dynamically adapts the media’s streaming rate to use the network’s resources more efficiently while minimizing the effects of the adaptation on the quality of the presentation.
The invention includes a system and device for, and method of, presenting multimedia information. In a client-server context, the invention includes a client that receives units of the multimedia information and presents the information on a presentation device. Each unit of information has an importance value assigned to it, which in an exemplary embodiment is indicative of the unit’s importance in relation to the quality of the presentation. The invention includes a mechanism for characterizing the performance capabilities of the system. For example, several conventional statistics may be gathered and analyzed concurrently with the streaming operation and before it begins.| The invention includes a mechanism for inferring network conditions from the characterized performance. The server may then stream the units of multimedia information to the client at a streaming rate and adapt the streaming rate of the streaming in response to the importance information and in response to the inferred network conditions.
According to one aspect of the invention, the importance information that is pre-assigned is stored with each unit of information in the file. This importance information is indicative of the unit’s importance to the perceived quality of the presentation
According to another aspect of the invention, the adaptation of the streaming rate includes throttling up the streaming rate if the inferred network condition indicates that there is sufficient bandwidth to receive multiple copies of a given unit of information.
According to yet another aspect of the invention, the adaptation of the streaming rate includes throttling down the streaming rate if the inferred network condition indicates that there is insufficient bandwidth to receive a given unit of information. Similarly, throttling down may occur if the inferred network condition indicates that there is network congestion.
One of the performance characteristics gathered, believed to be novel, is the characterizing of a bit rate throughput value. This may be done, for example, by determining the data rate of a communications device, such as a modem, utilized by the client computer to communicate with the server computer.