In part one of the Flash Player Internals recap we covered how the player is built, in part two we talked about how the player infrastructure works, in part three we examined how ActionScript and the Virtual Machine (VM) are improving in Flash Player 10.1. Now, in the final segment (looking at my notes this will be the longest segment) we will look at how the Flash Player rendering system is being improved and how you can leverage it for multiple devices.
Shake Your Movie-maker
One of the biggest changes in player 10.1 is that all video will now be GPU decoded. This is huge for you video buffs out there because GPU decoding is sick fast and allows lower power devices, like netbooks, to play 1080p video smoothly and efficiently. In the current player, video is decoded using a software based decoder (except in fullscreen, when GPU may be used, but not guaranteed) which is fine when you are on a big 8 core beast, but little machines just can't handle the data.
Now, with a change this big there has to be a few caveats. Well, there is. First, your GPU has to support H.264 decoding in the chipset. If your GPU (ex: video card) doesn't support that, then you are out of luck and the Player will fall back to software decoding. Next, your GPU and GPU drivers have to be certified for the Player to use it. If you have an older driver that is not certified it won't matter if your chipset supports H.264 or not, the Player won't leverage it.
Having the chipset/driver pair white-listed for the Player means that Adobe has to work closely with the manufactures to verify the certification process. Adobe has partnered with all the major manufactures for this process, but this means that we still have a few years before everyone can leverage GPU acceleration. The good news is that mobile devices turn over faster, and pretty much all of the newest devices meet the Player's requirements today.
Another issue is that existing video may need to be re-encoded to allow for GPU acceleration. This issue is created because the H.264 codec version that is used to encode the video must be supported by the GPU/driver combination. Lee recommends that you use H.264 main or baseline codecs because this will guarantee that all GPUs that meet the certification process will be able to render your video. If your videos were not encoded with main or baseline, then you may have to re-encode the video for GPU support.
So, how can you tell if the user can support GPU rendering or not? At the moment, you can't. The Player team is looking at this and they want it for 10.1 but there is a chance that this will not be exposed for the 10.1 release. GPU detection is coming, they will support it, but its up in the air for when the Player will enable this.
Finally, GPU support will be enabled for all wmodes. For those of you that where in attendance for the 2008 Flash Player Internals session, Jim and Lee discussed that only one wmode may be supported in the future. Well, Lee lost the argument and all browser wmodes will support GPU rendering (as long as you meet the previously discussed requirements).
There has been a lot of work done in 10.1 for how the player handles images. Images take up a LOT of memory, and updating how images are managed was one of the keys to getting the memory footprint of applications down, without having the developer change code. For example, the player now handles image instances much better, and allows image reuse without having to copy them. This is an issue, because if you are trying to show the same image multiple times (skins, etc.) the current player would copy the instance for each one on screen. Now, Player 10.1 can leverage the same instance in memory for rendering. Hot!
The Player now allows you, the developer, to determine when an image is decompressed. Currently, when the Player loads a JPEG, it is decompressed right there and then. This becomes an issue because the decompressed image takes up significantly more memory, even if you are not using it yet. This becomes a serious challenge when you are trying to pre-load/cache images but aren't ready to show them yet. In Player 10.1 you will have the ability to tell the Player to load the JPEG, but not to decompress it. This allows for a much better memory management experience within your applications. Lee also mentioned that images that are not being used will now be thrown away. It sounds like the Player would hold on to unused images longer then needed and this is probably one of the many GC tuning issues that we talked about in part three.
For you 3D junkies out there, Player 10.1 supports direct mip-mapping. I am not a 3D guru by any stretch of the imagination, most of my 3D experience was back with Director 8 and 3D Studio Max. So, here are my notes from what Lee said, and if you have a better understanding of mip-mapping feel free to chime in. Lee stated that unused memory is now managed when using mip-mapping, it is extremely efficient with 16-bit images and now it only decodes/renders what is seen, which makes a lot of sense when your read the Wikipedia article on mip-mapping.
Player 10.1 will support GPU vector rendering on mobile platforms. This new engine fully replaces the software rendering of vector graphics and moves the process over the GPU. The issue is that this only works well when the GPU is faster then the CPU. In most desktop environments, this is never the case so this feature is really a mobile only solution. The reason this is great for mobile is that in today's devices the GPU is significantly faster then the CPU for rendering out Vector graphics.
Why So Slow?
Lee had a few recommendations about improving general performance in Flash Player based applications. First, use ActionScript 3 and strong type everything. Using AS2, Generic objects, marking everything * and/or Dynamic based Classes do not perform nearly as well as strong typing and sealed Classes. Use the AVM2 to your advantage and take the time to type. Another tip is to avoid using intervals (timers) when ever possible and don't rush to setting high frame rates. One of the oldest tricks in the Flash book is to crank up your frame rate to "11" (i.e. 100+ frames per second). This strong arming technique used to work but now the AVM and the way it processes frame intervals won't gain you the same benefits as before, in fact in may actually hurt performance. Really take the time to understand the elastic racetrack concept and build your application with this in mind.
You Want Me To Display What?!
When building mobile/handheld based applications, understanding the display list is critical. As developers, especially Flex developers, we get a little nest happy. HBoxes, within VBoxes, within Canvases, etc. This becomes a HUGE problem with mobile devices. Everytime the user interacts with the app the display list has to be traversed and this just kills the CPU. To prevent this, try to keep the display list as flat and as small as possible. If you don't have to nest it, don't, if you can render it as one Sprite, do! Keep your numbers down and this will make your app perform much better.
What this all boils down to is that the utopia of one application on multiple devices is really not tangible. When building an app, you have to understand the target platform/device you are building to. What is the target devices resolution, pixel depth, GPU speed, CPU speed, memory limitations and capabilities. When building mobile applications, what is the cause of any performance issues? Is your display list too deep, are you rendering video correctly or vectors? Is your content truly optimized for your device?
Flash Player 10.1 is giving us a lot of great tools and features to allow us to apply our skills to new and uncharted realms, but with any tool we have to understand what it is good for and how we should use it. I am really excited to see what doors 10.1 opens for developers but I can already foresee some of the challenges we will face. I think that is why I really enjoy Lee's sessions because he helps layout a lot of the new features of the player but he also points out trouble spots that these new abilities can create, before you may stumble upon them yourself.