AC for plugins

I have been away from WebKit development for a while. There were some pressing matters in Maemo6 to attend to.

One of the things I am working on is the Flash support for Maemo6. And I am finding that the abstractions of QGV, meegotouch, Qt/WebKit are "adding up" resulting in much slower fps than using just plain Qt.

Some time back, Simon explained to me the concept of Accelerate Compositing. WebKit has no AC support for NPAPI plugins but by doing so we can remove the Qt/WebKit overhead when updating frames. Also, it allows us to implement Flash transparency correctly in QGV.

So, yesterday evening, I started hacking on Accelerated Composition (AC) for NPAPI plugins. With help from Noam, I am happy to report that Flash is rendering with AC :-) Expect it to land sometime next week. Here's a screenie with the fps.



(And oh, that's the first screenshot in this blog :)

UPDATE: You can find the initial code at http://gitorious.org/~girish/webkit/girishs-webkit/commits/plugins_ac_35524. https://bugs.webkit.org/show_bug.cgi?id=35524 is the bugzilla entry.

Youtube on the N900

After my implementation of local rendering, Flash plays quite well on the N900. In fact, with a plain Qt flash loader (npploader - git://git.forwardbias.in/npploader.git), I get ~70fps. With a QWebView, the fps goes down to around ~50fps. With a QGraphicsWebView, the fps goes to around 55fps. Tested with fps tester.

This is not so bad but youtube was still rendering very badly compared to Fennec. With help from Nokia folks, I found out that the codec being served for Fennec is H.263 Sorenson and Qt/WebKit was being served H.264. The difference in quality would have been blindingly obvious had I paid more attention :-) On the n900, Sorenson Spark is the fast codec that is hardware accelerated (Documented here). If you use N900 to watch Youtube, append &fmt=5 to the Youtube url for better viewing.

Youtube serves sorenson based on the UA and I did some elaborate tests to figure what exactly it was looking for in the UA so we can add the same to Qt/WebKit for Maemo5. The long result is below. The short result is that it expects "Firefox/3.5" and "Maemo Browser" in the UA. Unfortunately, we cannot add "Firefox/3.5" as that would mean Qt/WebKit might get served Fennec optimized sites :(

One options, of course, is to contact Youtube and ask them to serve a different codec for Qt/WebKit's UA but we are not pursuing this because this is really not important :) I have a patch to Qt/WebKit that adds "Firefox/3.5 Maemo Browser" to our UA for *.youtube.com by extending the existing QWebPage::userAgentForUrl(). Simon and Harald are brooding over whether we should bother adding this change. Time will tell whether we will apply it.

Results of the test:

Current Qt/WebKit UA - Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3
Current Fennec UA - Mozilla/5.0 (X11; U; Linux armv7l; en-US; rv:1.9.2.1) Gecko/20100126 Firefox/3.5 Maemo Browser 1.7.4.8 RX=51 N900

H.264 desktop UI
-----------------
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5 N900
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 N900
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox Maemo

Mobile UI (same ui as m.youtube.com - 3gp/rtsp)
-----------------------------------------------
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Maemo Browser
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Maemo Browser 1.7.4.8
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Maemo Browser 1.7.4.8 RX=51
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Maemo Browser 1.7.4.8 RX=51 N900
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 N900/Maemo Browser

Sorenson + desktop UI
---------------------------
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5 Maemo Browser
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5 Maemo Browser 1.7.4.8
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5 Maemo Browser 1.7.4.8 RX=51
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5 Maemo Browser 1.7.4.8 RX=51 N900
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox/3.5 N900/Maemo Browser
Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Qt/4.7.0 Mobile Safari/533.3 Firefox Maemo Browser

NPAPI/X11 : Plugin Visibility

For the past week or two, I have been working on getting Flash to work on Maemo5/N900. Read more about it here.

Today, I fixed a part of 44594. For mobile, it would be very efficient if Flash gets notified when it goes out of view or when the browser is minimized. There has been some discussion about this but no no consensus. We have to move on and so we have decided to set an empty clip rect in the NPWindow struct to notify the plugin about it's invisibility (I don't like this, I prefer explicit notifications on what is happenning but oh well).

ATM, clip rect is always set to 0 in windowless mode which breaks the above mechanism. So, I set out to find out it's meaning. I found that Flash doesn't care about the value of clipRect at all which made is quite hard to figure what exactly it's meaning is. A note from an Adobe person in the above thread seems to indicate clipRect is actually respected only on the Mac for Flash!

I now set the clipRect based on my understanding of NPWindow. If reading specs is your thing, please read the spec and see if your understanding matches with mine.

Read the spec? Ok, Quiz time.
1. In windowless mode, if NPWindow.x, y = (10, 10) and if the clipRect is (20, 20, 50x50) in _plugin_ coordinates, what should we set NPWindow.clipRect to?
2. In windowed mode, if a plugin of size 100x100 is at position 50, 50 wrt to the page, and we scroll 60 pixels in either direction, what is NPWindow.x,y and clipRect.

Answers
1. clipRect = (30, 30, 50x50). Atleast as per the doc, the clip rect is wrt to drawable and not the plugin's origin.
2. NPWindow.x,y = -10, -10. clipRect = (0, 0, 90x90). The value of x,y is not debatable :) It has to be set to -10, -10 based on empirical data and it's actually wrt the viewport and not the page as the doc suggestes. So, I made clipRect also wrt the viewport.

Good news is even if your answers to the above are different, it doesn't matter because Flash ignores clipRect :)

66095 is the commit in webkit.