Dynamic streaming in Flash Media Server 3.5 – Part 1: Overview of the new capabilities

Dynamic streaming is the process of efficiently delivering streaming video to users by dynamically switching among different streams of varying quality and size during playback. This provides users with the best possible viewing experience their bandwidth and local computer hardware (CPU) can support. Another major goal of dynamic streaming is to make this process smooth and seamless to users, so that if up-scaling or down-scaling the quality of the stream is necessary, it is a smooth and nearly unnoticeable switch without disrupting the continuous playback.

This article provides an overview of the enhanced capabilities and concepts of dynamic streaming with Adobe Flash Player 10 and Adobe AIR 1.5 with the new Adobe Flash Media Server 3.5 (using either Flash Media Streaming Server 3.5 or Flash Media Interactive Server 3.5). Although dynamic streaming was somewhat possible in previous versions of Flash Player and Flash Media Server, the process was very complicated to implement from a developer's perspective and the end result was not a smooth user experience.

After reading this article you should have a good understanding of dynamic streaming concepts. You will know why this feature is important and why dynamic streaming in Flash Player 10, AIR 1.5, and Flash Media Server 3.5 is an exceptional solution to deliver media. You will also learn how to implement dynamic streaming in your Flex or Flash applications. (This is Part 1 of a three-part series. Part 2 covers the ActionScript 3.0 dynamic stream APIs and Part 3 covers integrating dynamic streaming with existing video players.)

With faster-performing client hardware and users with higher bandwidth becoming the norm, the promise of high-definition (HD) video on the web is a reality. HD web video is generally considered larger video starting at 640 × 480 pixel dimensions and increasing up through 720p towards 1080p. The issues facing this trend have been around since the beginning of streaming video. Now Flash Media Server and Flash Player can handle streaming HD video in ways that greatly improve the user's experience without the need for them to do anything besides sit back and enjoy high-quality material.

One of the biggest issues facing publishers trying to stream longer duration (longer than five minutes) and higher quality video—especially HD video—is the standard fluctuations of users' Internet connections. This is a standard issue on most networks and can be exacerbated when multitaskers, wireless network fluctuations, or multiple, simultaneous users sharing a connection are involved.

The end result is a moving target for actual available bandwidth, and this can leave users continually having to rebuffer and wait for their video if the selected stream bandwidth is unsustainable on their network. Dynamic streaming detects fluctuating bandwidth and switches among streams of different bit rates in order to match the content stream to the user's bandwidth (see Figure 1).

On the other hand, some users may start the stream with low available bandwidth, and then free up more bandwidth after the start of the video. In this scenario, dynamic streaming can offer the ability to up-scale the video quality to a higher level, once again improving the user's experience.

In the past, the alternative was to perform initial or frequent bandwidth detection routines. Although better than nothing, these tests were costly in time and often didn't provide the accuracy needed due to the normal fluctuations and changes in bandwidth. Now, with the dynamic streaming capabilities and Quality of Service (QoS) information available, bandwidth detection tests have lost much of their value.

Another issue that can hinder playback, especially with large-dimension HD video and full-screen playback, can be the user's hardware performance limitations. If the CPU cannot decode the video stream fast enough, it will result in dropped frames, which can adversely affect the smoothness of the user's video display. In this case, using a lower-quality video file would enable less strain on the CPU to decode in synch and maintain performance.

Dynamic streaming can provide an optimal solution to network fluctuations and CPU overloading. By continually monitoring key QoS metrics on the client, dynamic streaming can effectively identify when to switch up or down to different-quality streams. The Adobe solution does not require specially encoded files for dynamic streaming, which means existing files already encoded in multiple bit rates can be used. With the dynamic streaming support for H.264 video with AAC/AAC+ as well as FLV files, Adobe supports the standard streaming video formats preferred by nearly all major broadcasters.

In addition, Adobe offers excellent content protection with dynamic streaming with the RTMPE encrypted real-time protocol, as well as new support for streaming rights-managed encrypted content with Flash Media Rights Management Server (FMRMS). Another very powerful feature offered by Adobe with FMS 3.5 and Flash Player 10 is the ability to do dynamic streaming with broadcast (live) video. Paired with the updated free Flash Media Live Encoder (FMLE), which now supports multi-bitrate live publishing, Adobe really has a great suite of products to deliver the ultimate user experience.

Table 1 compares the features and functionality of dynamic streaming offered by Adobe in Flash Media Server 3.5 and Flash Player 10 against Microsoft Silverlight 2.0.

Table 1. Dynamic streaming solutions comparison

Dynamic streaming is made possible by leveraging the new Flash Media Server 3.5 with Flash Player 10 and AIR 1.5. It is available on all server versions, including Flash Media Interactive Server, Flash Media Streaming Server, and Flash Media Development Server, and does not require any special server setup.

Flash Media Server handles the actual switching of the streams for the user based on the client-originated request to do so. Once the server receives the request to switch the user's stream to a different stream (bandwidth), it will wait a short period for the optimal point to switch with minimal playback impact to the user. This occurs on the nearest keyframe for video-only content, the nearest audio sample for audio-only streams, and the combination of the two for video content with audio. This means that video encoded with a keyframe every six seconds could take up to six seconds, once the command to switch streams has been received by the server, to actually switch the content and send the transition-complete notification.

This is also important in considering the available buffer amount. You need to make sure there is enough buffer to cover the transition command sequence, since the buffer will not continue to fill for the old stream once the switch transition request has been received by the server. The client-side application is notified at the key point of the dynamic streaming transition sequence. After the call to the server is received and is beginning to be processed, a NetStatusEvent will be fired with an info Object and a code property (event.info.code) with the value NetStream.Play.Transition. This indicates the server has received the transition command and is awaiting the next keyframe match to switch streams.

An undocumented valuable property is the reason property, also on the info Object of the NetStatusEvent instance (event.info.reason). The reason property provides additional information about the state of the transition request. Normally, this property will contain the value NetStream.Transition.Success, indicating that the transition request is processing normally. If streams that are being switched do not have aligned content/timelines, or if the keyframes are not aligned between the two streams, it is possible that the server will have to force a hard transition. This can also happen with broadcast (live) dynamic streaming if the server cannot find a synchronization point within the two streams. If this happens, the reason property will have the value NetStream.Transition.Forced. This situation can occur under the following circumstances:

• The two streams being switched don't have the same timeline, and therefore the server cannot select a time to perform the switch.
• The keyframe interval of the new stream is longer than the client's playback buffer, which is the maximum amount of time the server will wait for a transition. Since the server did not see a keyframe, it cannot select a frame for the switch.
• The live queue delay for the live streams is longer than the client's playback buffer, which creates a delay similar to a long keyframe interval.

Once the streams have successfully been switched and the new stream is being played by the client application, a play status event is fired and the onPlayStatus() handler of the NetStream client Object is invoked. Its single parameter is a generic Object and will have a code property with a value of NetStream.Play.TransitionComplete, indicating the transition is complete.

Client applications can be run from Flash Player 10 (and later) or AIR 1.5 (and later). Flash Player 9,0,115 achieved over 85% adoption in under six months. With this amazing trend to adoption, Flash Player 10 should be available for more than 85% of the entire worldwide Internet user base within six months. The industry-standard applications that create these rich experiences are Adobe Flash Professional CS4 and Adobe Flex Builder 3, or you can use the Adobe Flex 3 SDK. Within the new development tools, the new ActionScript-based API will provide you with the power both to monitor the performance of streams to determine the need and direction for stream switching, and to invoke the stream switch.

The new client-side ActionScript APIs includes a new info property on the existing NetStream class. When the NetStream.info property is accessed within a NetStream instance, it returns a snapshot of all the available QoS data points and returns them in a NetStreamInfo object (see NetStreamInfo in the ActionScript 3.0 Language and Components Reference). This class contains 19 properties, each providing valuable key points of information regarding the stream connection and playback performance. The wide spectrum of data the NetStreamInfo class provides gives granular real-time metrics about the stream performance, including audio, video, and data streams, both independently and as a whole.

Two of the most important properties it provides are NetStreamInfo.maxBytesPerSecond and NetStreamInfo.droppedFrames. The NetStreamInfo.maxBytesPerSecond property is one of the simplest and most direct ways to see the maximum capacity of the client's network for the stream. NetStreamInfo.droppedFrames indicates the total number of dropped frames to the point of the stream when NetStream.info was accessed. To monitor this property properly, you should likely be polled at a set interval and store the previous count of dropped frames and the target frame rate. If the percent of dropped frames is greater than 20%, you should consider swapping streams if managing manually.

NetStreamInfo public properties

Here are the 19 public properties exposed in the NetStreamInfo object:

• audioBufferByteLength: Number (read-only): Provides the NetStream audio buffer size in bytes.
• audioBufferLength: Number (read-only): Provides NetStream audio buffer size in seconds.
• audioByteCount: Number (read-only): Specifies the total number of audio bytes that have arrived in the queue, regardless of how many have been played or flushed.
• audioBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream audio buffer is filled in bytes per second.
• audioLossRate: Number (read-only): Specifies the audio loss for the NetStream session.
• byteCount: Number (read-only): Specifies the total number of bytes that have arrived into the queue, regardless of how many have been played or flushed.
• currentBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream buffer is filled in bytes per second.
• dataBufferByteLength: Number (read-only): Provides the NetStream data buffer size in bytes.
• dataBufferLength: Number (read-only): Provides the NetStream data buffer size in seconds.
• dataByteCount: Number (read-only): Specifies the total number of bytes of data messages that have arrived in the queue, regardless of how many have been played or flushed.
• dataBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream data buffer is filled in bytes per second.
• droppedFrames: Number (read-only): Returns the number of video frames dropped in the current NetStream playback session.
• maxBytesPerSecond: Number (read-only): Specifies the maximum rate at which the NetStream buffer is filled in bytes per second.
• playbackBytesPerSecond: Number (read-only): Returns the stream playback rate in bytes per second.
• SRTT: Number (read-only): Specifies the Smooth Round Trip Time for the NetStream session.
• videoBufferByteLength: Number (read-only): Provides the NetStream video buffer size in bytes.
• videoBufferLength: Number (read-only): Provides the NetStream video buffer size in seconds.
• videoByteCount: Number (read-only): Specifies the total number of video bytes that have arrived in the queue, regardless of how many have been played or flushed.
• videoBytesPerSecond: Number (read-only): Specifies the rate at which the NetStream video buffer is filled in bytes per second.

Adobe is providing some powerful resources to implement dynamic streaming into your own applications. Most notable are the new DynamicStream and DynamicStreamItem classes. You can download these from the new Flash Media Server Tools page. The classes provide a powerful, clean, and quick way to implement the dynamic streaming functionality without all the heavy lifting of building a custom solution from the ground up. The DynamicStream and DynamicStreamItem classes leverage the key QoS data available from the NetStream.info property and provide a direct and easy-to-use API to stream with multiple bit rates to a video player with all the control a developer woulddesire.

The DynamicStream class extends the NetStream class and is used as an alternative to the NetStream in custom video applications. It also offers powerful benefits, such as integrated quick-start playback, easy implementation, and enhanced control methods. To use the DynamicStream class, you generate an instance of it, passing in a reference to the NetConnection instance you want to communicate through, just as you would with a normal NetStream. The DynamicStream instance has a method called startPlay() to which you can pass an Array of streams encoded at different bit rates in the form of a DynamicStreamItem instance.

Walkthrough 1. Overview of capabilities

In this walkthrough, you will create DynamicStream and DynamicStreamItem objects, and then use those objects to initialize dynamic streaming and see it in action:

1. Open DynamicStreamSample-Simple.fla in the {start files}/DynamicStreamSample - Simple folder
2. Open DynamicStreamSample.as in the {start files}/DynamicStreamSample - Simple/com/realeyes/media/ folder.
3. Review the class file, making sure to note the following objects:
   1. The _nc NetConnection object
   2. The _connClient ConnectionClient object
   3. The _vidDisp Video object
   4. The _uri String property
4. Locate the comment //Declare dynamic stream object.
5. Declare a new private variable named _ds with a type of DynamicStream. Your code should look similar to the following:

6. Locate the comment //Call the method to create the DynamicStream object. Call the _initDynamicStream() method:

7. Locate the comment //Create the DynamicStream ojbect and attach to the video display inside the _initDynamicStream() method.
8. Set the _ds property equal to a new DynamicStream object, passing _nc as a constructor argument:

9. Set the client property of the _nc NetConnection object equal to the _connClient property:

10. Add an event listener for the NetStatusEvent.NET_STATUS event and set the _onNetStatus method as the event handler:

11. Call the attachNetStream() method on the _vidDisp Video object, making sure to pass it the _ds DynamicStream object as a parameter:

12. Under the comment // Add video files, create a new DynamicStreamItem object named dsi:

13. For each of the five following stream names, call the addStream() method on the dsi object and pass it the stream name and bit rate:
    1. name: mp4:sample1_150kbps.f4v
       bit rate: 150
    2. name: mp4:sample1_500kbps.f4v
       bit rate: 500
    3. name: mp4:sample1_700kbps.f4v
       bit rate: 700
    4. name: mp4:sample1_1000kbps.f4v
       bit rate: 1000
    5. name: mp4:sample1_1500kbps.f4v
       bit rate: 1500

14. Optionally, you can set the startRate property on the dsi object to the _startingBitrate property:

15. Call the startPlay() method on the _ds object, passing the dsi variable as the only parameter:

The completed method should look like the following:

16. Compile DynamicStreamSample-Simple.fla to run the sample.

The versatile FLVPlayback component has been updated in Flash CS4 Professional to FLVPlayback 2.5, which encapsulates the DynamicStream classes and offers ease of use and additional API enhancements for dynamic streaming. You can download this now from the Flash Media Server Tools page. Full backwards compatibility has been maintained, but now there is a play2() method available for passing a DynamicStreamItem containing the list of streams with different bit rates as well as a uri property to indicate the connection point.

Walkthrough 2. Simple FLVPlayback 2.5 component

In this walkthrough, you will use an FLVPlayback 2.5 component to play multiple streams with dynamic streaming:

1. Open flvpb_simple_sample.fla in the {start files}/FLVPB - Simple folder.
2. Open the Library Panel (Main Menu > Window > Library).
3. Making sure the flvpb layer is selected, drag the FLVPlayback 2.5 component onto the Stage and give it an instance name of flvpb.
4. Size the component to 640 × 480 pixels and position it in the middle of the Stage (see Figure 2).

5. Select the first frame of the ~AS layer in the timeline and open the Actions Panel (Main Menu > Window > Actions).
6. Import the entire fl.video package:

7. Set the static iNCManagerClass property of the VideoPlayer class equal to NCManagerDynamicStream:

8. Create a new DynamicStreamItem object named dsi:

9. Set the uri property of the dsi obect equal to rtmp://localhost/vod/:

10. For each of the five following stream names, call the addStream() method on the dsi object and pass it the stream name and bit rate:
    1. name: mp4:sample1_150kbps.f4v"
       bit rate: 150
    2. name: mp4:sample1_500kbps.f4v"
       bit rate: 500
    3. name: mp4:sample1_700kbps.f4v"
       bit rate: 700
    4. name: mp4:sample1_1000kbps.f4v"
       bit rate: 1000
    5. name: mp4:sample1_1500kbps.f4v"
       bit rate: 1500

11. Call the play2() method on the flvpb FLVPlayback component making sure to pass the dsi object as the only parameter:

The complete code should look like the following:

12. Save and compile to run the sample.

In addition to the enhanced ActionScript API available in the FLVPlayback 2.5 component, there is now also added support for multi-bitrate dynamic streaming from an external SMIL file. SMIL is an industry-standard XML-based file describing how and what video content should be displayed. This enables easy externalized and configurable control for connecting and playing Flash video supported content with the FLVPlayback component.

Walkthrough 3. FLVPlayback with an external SMIL file

In this walkthrough, you will use an FLVPlayback component to play multiple streams using a SMIL file:

1. Open dyamicStream.smil in the {start files}/FLVPB - SMIL folder.
2. Review the contents of the file for its structure and data.
3. Open flvpb_simple_sample.fla in the {start files}/FLVPB - SMIL folder.
4. Open the Library Panel (Main Menu > Window > Library).
5. Making sure the flvpb layer is selected, drag the FLVPlayback 2.5 component onto the Stage and give it an instance name of flvpb.
6. Size the component to 640 × 480 pixels and position it in the middle of the Stage (see Figure 3).

7. Select the first frame of the ~AS layer in the timeline and open the Actions Panel (Main Menu > Window > Actions).
8. Import the entire fl.video package:

9. Set the static iNCManagerClass property of the VideoPlayer class equal to NCManagerDynamicStream:

10. Set the source property of the flvpb FLVPlayback component equal to dynamicStream.smil:

The completed code should look like the following:

11. Save and compile to run the sample.

Adobe has also supplied a Flex based, embeddable video player with the Flex source code available with installation of any of the Flash Media Server editions. To view the video player, simply launch the Flash Media Server start page at http://[your server name]/, or http://localhost/ if the server is started and you have the integrated Apache server installed. Otherwise, the page can be found at {installation directory}\Flash Media Server 3.5\webroot\index.html.

The HTML code snippet for embedding this video player can also be found on this page. With the code snippet, you can implement this robust video player easily in any HTML page and integrate dynamic streaming content via an external SMIL file simply by updating the HTML parameters. Keep in mind the SWF file referenced by the embed code must be accessible to the server on which you are implementing this embeddable HTML snippet. The Flex source code can be found here: {installation directory}\Flash Media Server 3.5\samples\videoPlayer.

Where to go from here

Dynamic streaming is a powerful new functionality powered by Flash Media Server 3.5 and Flash Player 10. With this new and enhanced capability, there is now an effective, simple way to improve the user experience with video streaming. The problems of continual rebuffering and interrupted viewing experience now have a solid solution that can handle adverse situations and improve the overall standards for the user experience.

Dynamic streaming also opens the door for the continual progression towards HD video on the web and maintains solid position of Flash as the leader of video on the web. The ease of implementation and deep level of control that the new API and dynamic stream classes that Adobe offers have made this a quick and cost-effective tool set to implement.

This is Part 1 of a three-part series on the new dynamic stream classes. You can proceed now to Part 2: ActionScript 3.0 dynamic stream APIs.