conferences

With a little less than a month until 360|Flex I thought I would sit down and write a long over due post about the conference.  360|Flex is known in the community as one of those “hard core” conferences where the expectation is to leave sessions with your Flex and Flash IQ increased by a couple digits.

Unfortunately until now, James and I have had to experience the conference through other people’s stories.  For some reason or another the last few years have been pretty hectic, but we made it a mission in 2010 to attend and speak at more events, with 360|Flex being the first on the list.

Session
In our session “Flex 4 Component LifeCycle Best Practices” our goal is to walk folks through the three stages of the Flex lifecycle (birth, mature, death) and discuss what methods can be used to help build efficient and high performance Flex applications following that Flex component design methodology. With the reengineering of the skinning architecture in Flex 4, a few methods have been introduced and others have been more or less deprecated in importance. We will discuss these changes in detail and how you will want to change your development practices. We are extremely excited to share our year long research that began with our widely popular white paper on the Flex 3 lifecycle. Soon after the conference and session, we plan on updating the paper to Flex 4 and will share it with the community.

Attending
As much of an honor it is to speak at 360|Flex what is really exciting us is attending some of the other session and diving deeper into topics we have not had the time to investigate. James and I have been debating our schedule for a few weeks now so far we have compiled a “must-see” list that we have listed below to share with everyone.

Aaron & James’ Must See!

Going deep w/OSMF - OMG!
Level 300
By David Hassoun

RobotLegs on Top of Gaia Flash Framework
Level 100
By Jesse Warden

Building Applications using Test Driven Development (TDD)
By Elad Elrom

Dramatic Effect of Flex Library Linkage
Level 300
By Yakov Fai

Reflex: Rethinking Component Design
Level 300
By Jacob and Tyler Wright

Social Activity
We will be both tweeting as much as we can from the conference and sessions. Hit us up if you would like to discuss our session or any other session in more details. We love meeting new techies!

Aaron Pedersen - @aaronpedersen
James Polanco - @jamespolanco

Also don’t forget to follow 360|Flex for news and updates as we approach the and throughout the conference.

360|Flex - @360flex

Wrap Up
If you are thinking about attending 360|Flex you should book your tickets soon. Last time the conference was held in San Jose it was sold out. This year the conference has parties every night giving you a chance to mingle with the community at large. There are over 40 sessions, 2 panels, and your ticket includes hands on training sessions for all levels on the Sunday before the conference begins. Also, if you are new to Flex, you can take an all day Flex 101 course (one of the hands on training sessions) and be ready for regular sessions on Monday.

Speakers include a wide assortment of Adobe engineers and communities leaders. For a full list check out the following site http://www.360flex.com/blog/category/the-speakers.

Register here. We hope to see you there!

In part one of the Flash Player Internals recap we covered how the player is built, in part two we talked about how the player infrastructure works, in part three we examined how ActionScript and the Virtual Machine (VM) are improving in Flash Player 10.1.  Now, in the final segment (looking at my notes this will be the longest segment) we will look at how the Flash Player rendering system is being improved and how you can leverage it for multiple devices.

Shake Your Movie-maker

One of the biggest changes in player 10.1 is that all video will now be GPU decoded.  This is huge for you video buffs out there because GPU decoding is sick fast and allows lower power devices, like netbooks, to play 1080p video smoothly and efficiently.  In the current player, video is decoded using a software based decoder (except in fullscreen, when GPU may be used, but not guaranteed) which is fine when you are on a big 8 core beast, but little machines just can’t handle the data.

Now, with a change this big there has to be a few caveats.  Well, there is.  First, your GPU has to support H.264 decoding in the chipset.  If your GPU (ex: video card) doesn’t support that, then you are out of luck and the Player will fall back to software decoding.  Next, your GPU and GPU drivers have to be certified for the Player to use it.  If you have an older driver that is not certified it won’t matter if your chipset supports H.264 or not, the Player won’t leverage it.

Having the chipset/driver pair white-listed for the Player means that Adobe has to work closely with the manufactures to verify the certification process.  Adobe has partnered with all the major manufactures for this process, but this means that we still have a few years before everyone can leverage GPU acceleration.  The good news is that mobile devices turn over faster, and pretty much all of the newest devices meet the Player’s requirements today.

Another issue is that existing video may need to be re-encoded to allow for GPU acceleration.  This issue is created because the H.264 codec version that is used to encode the video must be supported by the GPU/driver combination.  Lee recommends that you use H.264 main or baseline codecs because this will guarantee that all GPUs that meet the certification process will be able to render your video.  If your videos were not encoded with main or baseline, then you may have to re-encode the video for GPU support.

So, how can you tell if the user can support GPU rendering or not?  At the moment, you can’t.  The Player team is looking at this and they want it for 10.1 but there is a chance that this will not be exposed for the 10.1 release.  GPU detection is coming, they will support it, but its up in the air for when the Player will enable this.

Finally, GPU support will be enabled for all wmodes.  For those of you that where in attendance for the 2008 Flash Player Internals session, Jim and Lee discussed that only one wmode may be supported in the future.  Well, Lee lost the argument and all browser wmodes will support GPU rendering (as long as you meet the previously discussed requirements).

Picture This

There has been a lot of work done in 10.1 for how the player handles images.  Images take up a LOT of memory, and updating how images are managed was one of the keys to getting the memory footprint of applications down, without having the developer change code.  For example, the player now handles image instances much better, and allows image reuse without having to copy them. This is an issue, because if you are trying to show the same image multiple times (skins, etc.) the current player would copy the instance for each one on screen.  Now, Player 10.1 can leverage the same instance in memory for rendering.  Hot!

The Player now allows you, the developer, to determine when an image is decompressed.  Currently, when the Player loads a JPEG, it is decompressed right there and then.  This becomes an issue because the decompressed image takes up significantly more memory, even if you are not using it yet.  This becomes a serious challenge when you are trying to pre-load/cache images but aren’t ready to show them yet.  In Player 10.1 you will have the ability to tell the Player to load the JPEG, but not to decompress it.  This allows for a much better memory management experience within your applications. Lee also mentioned that images that are not being used will now be thrown away.  It sounds like the Player would hold on to unused images longer then needed and this is probably one of the many GC tuning issues that we talked about in part three.

For you 3D junkies out there, Player 10.1 supports direct mip-mapping.  I am not a 3D guru by any stretch of the imagination, most of my 3D experience was back with Director 8 and 3D Studio Max.  So, here are my notes from what Lee said, and if you have a better understanding of mip-mapping feel free to chime in.  Lee stated that unused memory is now managed when using mip-mapping, it is extremely efficient with 16-bit images and now it only decodes/renders what is seen, which makes a lot of sense when your read the Wikipedia article on mip-mapping.

Vectorize Me!

Player 10.1 will support GPU vector rendering on mobile platforms.  This new engine fully replaces the software rendering of vector graphics and moves the process over the GPU.  The issue is that this only works well when the GPU is faster then the CPU.  In most desktop environments, this is never the case so this feature is really a mobile only solution.  The reason this is great for mobile is that in today’s devices the GPU is significantly faster then the CPU for rendering out Vector graphics.

Why So Slow?

Lee had a few recommendations about improving general performance in Flash Player based applications.  First, use ActionScript 3 and strong type everything.  Using AS2, Generic objects, marking everything * and/or Dynamic based Classes do not perform nearly as well as strong typing and sealed Classes.  Use the AVM2 to your advantage and take the time to type.

Another tip is to avoid using intervals (timers) when ever possible and don’t rush to setting high frame rates.  One of the oldest tricks in the Flash book is to crank up your frame rate to “11″ (i.e. 100+ frames per second).  This strong arming technique used to work but now the AVM and the way it processes frame intervals won’t gain you the same benefits as before, in fact in may actually hurt performance.  Really take the time to understand the elastic racetrack concept and build your application with this in mind.

You Want Me To Display What?!

When building mobile/handheld based applications, understanding the display list is critical.  As developers, especially Flex developers, we get a little nest happy.  HBoxes, within VBoxes, within Canvases, etc.  This becomes a HUGE problem with mobile devices.  Everytime the user interacts with the app the display list has to be traversed and this just kills the CPU.  To prevent this, try to keep the display list as flat and as small as possible.  If you don’t have to nest it, don’t, if you can render it as one Sprite, do!  Keep your numbers down and this will make your app perform much better.

What this all boils down to is that the utopia of one application on multiple devices is really not tangible.  When building an app, you have to understand the target platform/device you are building to.  What is the target devices resolution, pixel depth, GPU speed, CPU speed, memory limitations and capabilities.  When building mobile applications, what is the cause of any performance issues? Is your display list too deep, are you rendering video correctly or vectors?  Is your content truly optimized for your device?

Flash Player 10.1 is giving us a lot of great tools and features to allow us to apply our skills to new and uncharted realms, but with any tool we have to understand what it is good for and how we should use it.  I am really excited to see what doors 10.1 opens for developers but I can already foresee some of the challenges we will face.  I think that is why I really enjoy Lee’s sessions because he helps layout a lot of the new features of the player but he also points out trouble spots that these new abilities can create, before you may stumble upon them yourself.

In part one of the Flash Player Internals recap we covered how the player is built, for part two we talked about how the player infrastructure works.  In part three, we will examine how ActionScript and the Virtual Machine (VM) are improving in Flash Player 10.1.

What’s New, Player-cat?

First off, Lee announced that the ActionScript VM has been updated to use the newest version of nanojit, which is the same VM backend that is used by Firefox 3.5.  This is an important update not only for performance, but because it supports many different processors.  Processor support is critical when you are looking to run the VM on as many devices and platforms as possible and nanojit helps the Player team do this.

The Player now has a much improved internal String class.  Strings have always been a bit of a bottle neck in Flash and the player team has updated the new class to allow better caching, faster string comparison, and most importantly lower memory usage.  This kind of low level update is great because it improves how our applications run without having to change a line of code.  Of course, you need to republish existing applications for 10.1 when it comes out (recall how the player versions SWFs), but the gains should be worth it.

On the memory front, the Player team has spent a lot of time focusing on how to use less of it more efficiently.  The first step they took is to have the VM take up less of a memory footprint.  Lee didn’t get into any details but I would assume nanojit helped with this.  The other area they looked at was Garbage Collection (GC) and how it could be tuned for mobile devices.  What they have found is that memory usage has dropped anywhere from 10-50% for existing applications.  The GC tuning wasn’t the sole update for this huge improvement, we will look at some other updates in part four, but this was a big one.

Garbage in Garbage Out

Anyone that has worked extensively with the Flash Player, starting with Player 9, has eventually run up against the Player’s Garbage Collection (GC) process.  GC is another one of those community hot topics, one that I don’t feel like exploring at the moment.  If you haven’t explored GC in the Flash Player yet, you really, really should. Grant Skinner has an excellent article about how the GC works over on Adobe’s ADC site, it is very much worth your time to read.

The big thing about GC that Lee hit home was that, GC is NOT memory management.  This seems to be a confusion point for a lot of developers and when their app is all bloated with excess memory usage, they point their finger at the GC saying it is the cause.  The reality is, that most memory issues are developer created.  If you don’t take the time to clean up after yourself, i.e. remove event-listeners, strong references, etc. then there is nothing the GC can do to help you.

The key thing about memory management in Flash is to think about it from the very beginning.  The development team that built the New York Times AIR-based reader implemented an IDisposable interface that allowed for self-cleaning of objects before throwing them away.  Aaron and I used a similar approach called IRecyclable for the Scion 2007 redesign.  The idea is that you create a method on your object called dispose() that removes/nulls out all internal references within the class.  If Class being disposed haves children objects that implement the interface, the class’ dispose method call dispose() on the children.  By having this kind of system, you can make sure that your code is self-cleaning at runtime.

Another tip that Lee threw out was using prime numbers when leveraging the Flex Builder profiler.  Create 7 instances of something and then look for the number 21 in your instance counts after the app has been running for a bit.  The theory is that by using prime numbers as an instantiation number, you can find hanging instances faster by looking for multiples of that number.  In the 7/21 case this is caused because you have three instance of the first 7 you generated, which may be a bad thing, that is up to your code and how many instances you expect.

Tada! It’s Alchemy

If performance matters, and we mean really matters, then Lee recommends investing time in learning both Alchemy and C/C++.  For those of you not familiar with Alchemy, it is an Adobe Labs tool that converts C/C++ based libraries to AVM2 byte-code.  This means that you can execute your C/C++ code from within in your Flash application and it is incredibly fast.  Lee talked briefly about how it uses a special memory system based on byte arrays that make the code execute quickly and lightly within the Player. At DevelopmentArc, we haven’t had the chance to really dig into Alchemy yet, but the examples are pretty stunning.

The other technology to leverage for pure performance is PixelBender and the Hydra language.  PixelBender is a new engine within the player that is designed for fast execution of image/math based data.  Its important to note that PixelBender is not supported on a lot of mobile devices (at least not yet) so this is really more of a desktop/browser feature, but if speed is your thing, then you should look into it.

That’s all for part 3, in the final section we look how the Flash video and rendering system is being updated.

[ Read Part 4: Flash Player Internals 10.1 ]

In the first part of our series we discussed how the player code base is being unified and how that will impact future development on the Flash platform.  In part two, we will dive into the infrastructure of the Flash Player and how this affects us as developers.

Understanding The Execution Model

The execution model is all about how the Flash player performs its actions every cycle (i.e. frame).  The first thing is that the Flash Player is actually extremely multi-threaded under the hood, its just not exposed to developers.  This means that conceptually we have to look at the player as single-threaded.  There is a huge debate about the benefits/limitations of this concept, one that I am not going to even try to comment on in this post.  For now, this is the way the player works… ’nuff said.

Lee’s explanation of the execution model is pretty much a recap of what has been said in the past, so read up on the 2007 presentation here and also check out how the elastic racetrack model works.

Timing Priorities

Now that you understand frames, render rates, and the execution model (you did read all those links above right?) there is priority in the Flash Player on what gets executed first.  The king of execution is sound, if you are playing sound the Flash Player will spend as much energy (i.e. CPU cycles) to meet the audio rate demands.  If the player can execute the audio rate properly, then the focus moves on to video.  The player will attempt to get as many frames per second as the video requires.  Once the video is achieved, then the remaining cycles get fed to animation and logic.

There is actually some interesting reasons on why this priority order is defined.  Once again, Chris Giffith provided some great insight into this at SDFUG.  Psychologically, audio is more important to us then video.  Back in the day of over-the-air Television, when the reception was bad as long as the audio was clear we could deal with mostly crummy images. But, once the audio went out people would change the channel.  Well, this applies today with the Flash Player.  We can handle a few video frames dropping but if the audio cuts out the experience is pretty much over.  To prevent this from occurring the player has defined this timing order.

Understanding this timing order is important because it can effect how an application is implemented.  If you want to run high quality audio/video while attempting to perform serious time-sensitive logic, this is more then likely going to be a challenge.  Moving the video quality down, or adjusting the logic system may be required to achieve the best experience for your user.  Understanding what gets priority helps you make a more informed decision when trying to implement or troubleshoot these kinds of applications.

Versioning SWFs (not the Player)

You can read up all about versioning in the first post from the 2007 series, but here is the short and sweet of it:  versioning of a SWF is virtual and can affect how it is executed in the player.  Every version of the Player, has an implementation model of all the previous versions of the Player.  So, if you made a SWF back when Player 6 was out, and try to run it in Player 10.1, the SWF will behave as it did back with Player 6, bugs and all (critical bugs in the player are fixed). That means, if Player 9 fixes a bug that you want in your old SWF, you must recompile the SWF to the new version to take advantage of that, this is all done so not to “break the web”.

What’s New in Player 10.1’s Infrastructure?

First off, Out of Memory is now handled gracefully.  In past versions, if you ran out of memory the Player just crashed.  Not good, especially when you consider out of memory is potentially going to happen a lot more on mobile devices

The other new infrastructure feature that Lee talked about was managing multiple instances of the Player.  One of the challenges of limited CPU and memory availability on devices is having multiple instances of the Player running.  Even if the apps running are the most well thought out, efficient programs; running multiple instances, such as how the Palm Pre allows multiple apps windows at once, will quickly kill the device.  To prevent this the player team is looking at having non-browser players pause other instances of the player when not in focus, or have the other players “downclock” to lower framerates automatically.

What this would do is, as you spawn new instances of the player and they gain focus, the other instances would be aware and then either stop execution or lower their framerate to minimize CPU usage.  The issue the Player team is facing is that this kind of Player “magic” can have all kinds of unwanted repercussions.  For example, Local Connection between the two instances more or less breaks.  If you build an app that communicates with another app on a moblie device and the Players pauses or downclocks this can ruin your experience, which is not a desired effect. Because of these potential issues, this feature is still in development and may not be complete or even part of 10.1 at launch.

Bonus round: global error handling.  Lee didn’t mention this, but it was brought up in one of the keynotes about player 10.1.  You can now place a try/catch at the top of your application to catch all those tricky errors.  This is a huge new feature for developers.  The rouge root error has always been the bane of the Flash Developer’s world since the new AVM2 added them.  I have yet to see an example of how this works, but as soon as we do we will post examples on DevelopmentArc.

That’s all for part 2,  in part 3 we examine how the ActionScript Virtual Machine is being updated and in the final section we look how the Flash video and rendering system is being updated.

[ Read Part 3: Flash Player Internals 10.1 ]
[ Read Part 4: Flash Player Internals 10.1 ]

For the last three years I have attended the Adobe MAX session put on by Lee Thomason and Jim Corbett called “Flash Player Internals”.  This is by far my favorite MAX session, because its extremely advanced, fast paced and they cover as much of the player’s architecture as possible.  This year at the Adobe MAX 2009 conference, Jim was not able to join Lee for the session.  Its unfortunate because Jim and Lee’s interplay is half the fun, but Lee is a great speaker and he handled covering the entire session like pro.

For the 2007 session, I wrote a huge recap article because they covered the first version of the player to support ActionScript 3.  This player changed how the virtual machine (VM) worked and they showed all kinds of tips and tricks based on the VM’s architecture. You can read my previous posts over at my personal blog, VivisectingMedia. The write-up is a multi-part series that starts off covering the player teams motto about not breaking the web.  I skipped writing about 2008, mainly because we where just too busy starting DevelopmentArc, prepping for MAX Europe and the impeding holidays.

Now that MAX 2009 is all wrapped up its time to write up a multi-part series that recaps all the new topics that Lee delved into.  This round, Lee looked at all the upcoming changes in Flash Player 10.1 and I feel taking the time to research and write the posts are worth the effort.  You can watch Lee’s session over at the MAX video site and if you have any questions or comments please send them our way.

Building Flash Player

In the past, the Flash Player was forked for different platforms.  Actually, forked isn’t really correct, it was two very different players built by different teams, that both shared a common root back in the AS2 days.  The FlashLite team (flash for mobile) was a different department within the company and had unique goals, targets, features and deadlines.  I am sure this caused a lot of tension and challenges for both player teams and due to this (and other goals) Adobe has merged the teams into one entity.  This integration means that with the upcoming release of version 10.1 the code base will now be unified.  As Lee said, “One Build to Rule Them All!”.

Lee did mention that around 20% of the code base is still platform specific, i.e. targeting Palm, ActiveX, OS X, Netscape, etc.  There are too many differences on each platform for them to have a 100% unified code base, but this also means that 80% of the code is shared across all platforms.

One But Many

Having a single code base for all target platforms will help Adobe unify their player strategy for mobile, browser, device and desktop development.  Yet, this introduces some new challenges for both the player team and the development community that deploys applications using the Flash Player.

The first big challenge is that we are seeing a huge (and rapidly growing) gap between performance (CPU) and memory availability per platform.  As Lee said, the processing power of an 8 core system vs. the Palm Pre processor is pretty huge. Not only that, but browser/desktop developers have been pretty spoiled when it comes to available memory.  Having multiple available gigabytes of RAM on one platform (i.e. browser) vs. 20 megabytes (i.e. hand-held device) can radically affect how you build an application.  Note:  we are not saying that memory usage isn’t an issue on the browser, its just less of an issue when you compare devices.

The next issue that player unification faces is screen size and pixel density.  As applications start to cross the multi-device bridge, understanding how the screen lays out and is used becomes more important.  Lee didn’t talk about pixel density directly, but Chris Giffith chimed in at the last SDFUG meeting with some interesting side notes.

An LCD monitor pixel density (i.e. pixels per sq. inch) is significantly less then hand-held devices.  This means that a 40 pixel button on your LCD looks great, but becomes too small to use on a Pre.  Pixel density also differs across devices with the same resolution. The iPhone 3Gs has a 3.5 inch, 320×480 resolution, and a 163 pixels per square inch (PPI) screen. The Palm Pre has a 3.1 inch, 320×480 resolution with a 186 PPI screen. If we went solely on resolution, they look the same from a Player perspective so we may design a UI that works well on the iPhone but fails on the Pre due to the density difference.

Another challenge is user interactions.  Does the platform support a mouse and a keyboard?  Does it support multi-touch?  The player has to understand these different kinds of user inputs and we as developers have to understand how it impacts our applications.  A finger is not a single pixel accurate device, we have to understand that when doing development.  The player has to expose the right information to us, from the lowest level granular data to more high-level user actions such as swipe or pinch.

The final major challenge the unification brings to the table is that user’s expectations change base on the device being used.  Performance and usability change based upon whether your user is using a mobile phone vs. a browser.  On a phone, your app could be closed in 10ms because a call is coming in.  If the player doesn’t support this then its impacting the user’s expectations and needs for the device.

The Player is Not the Silver Bullet

What this all leads to is, the Player cannot solve all of these problems.  The team is trying to provide as many of the solutions as possible, but the reality is, as developers we have to really understand our delivery platforms and understand that what works for one user/device combination will not work for another. As we start moving over to more and more radically different platforms, we must develop and design applications with all these consideration in mind.

That’s all for part 1, in part 2 of the series we look at the execution model & player infrastructure, in part 3 we examine how the ActionScript Virtual Machine is being updated and in the final section we look how the Flash video and rendering system is being updated..

[ Read Part 2: Flash Player Internals 10.1 ]
[ Read Part 3: Flash Player Internals 10.1 ]
[ Read Part 4: Flash Player Internals 10.1 ]

This year James and I will be involved in three separate sessions at the Adobe Max conference in LA.   Adobe Max provides a collaborative arena for the communities of designers, developers, project managers, executives and any other project participators to gather and  share ideas, real-life experiences, theories, secrets, and best practices.  Over the past year DevelopmentArc has been involved in a wide variety of projects and we are excited to share our  experience with the community the best we can.   Below is breakdown of the three sessions we are involved in. Two involve a secret (shhhh) project that is to be released next Monday.  Follow James (@jamespolanco) or I (@aaronpedersen) on twitter and we will share as soon as we can.

Delivering Successful Flash Platform Projects (Monday, Oct 5th 11:30am - #adobemax181)

In this session we will be joined by co-author Doug Winnie of Adobe to discuss how during any project the planning phase is often times the most over looked and in most cases the most valuable.  We take you through a fictitious story of two companies as they collaborate on a project.  We plan to highlight exactly why planning when not overlooked, can start a project off on the right foot and actually cost the development team less time, money and resources to complete.

Workflows for the Flash Platform (Monday, Oct 5th 5pm - #adobemax286)

We are joining Doug Winnie again to discuss Flash Platform Workflows. Doug is an expert on the subject and spends most of his days working with various product teams within Adobe trying to enhance the workflow process between applications.  Doug will breakdown the workflow process and demonstrate some very cool and new technologies.

Bird of a Feather (Wednesday, Oct 7th 12:oopm - 1:30pm)

Again joining Mr. Winnie, James and I will be sharing our experiences with whom ever will listen.  We will explain, answer questions about the workflow we used in developing the secret application using Adobe’s soon-to-be released Flex 4 SDK, Flash Builder 4, and Flash Catalyst technologies.  The project had a quick turn around and a small team.  Because of these constraints we employed a modified workflow that leverages Flex 4’s Spark Skinning Architecture along with an unconventional use of Flash Catalyst to enhance a developer-drive workflow. Come by and say hi.