QtWebKit: I'm back!

Hello world!

 

Five years have passed since the last entry in this blog, and almost 3 years since infamous "Changes in QtWebKit development" thread at webkit.org. Fortunately, we've made quite different kind of change in QtWebKit development lately, and it is much more exciting.

QtWebKit is back again!

If you were following QtWebKit development after 2013, you know that actually development have never stopped: each release was getting a bunch of bugfixes and even brand new features. However, WebKit engine itself has not been updated since Qt 5.2 release. That's why it didn't support recent changes in Web standards that happened after 2013, including: new JavaScript language standard ES2015 (also known as ES6), as well as improvements in DOM API and CSS.

However, things have changed in 2016, and now we have revived QtWebKit! Core engine code was updated to it's actual state, and as a result we (and you!) can use all improvements made by WebKit community during these 3 years without any changes in code of existing Qt applications!

WebKit repositories

Here's a cheat sheet for all the WebKit repositories.

SVN: http://svn.webkit.org/repository/webkit. This is always the latest and greatest WebKit code since SVN is the version control used for WebKit code. To checkout trunk, you would do 'svn co http://svn.webkit.org/repository/webkit/trunk'.

Git mirror at webkit.org: git://git.webkit.org/WebKit.git. This git repository is a mirror of the svn repository. If you work with the Qt/WebKit port, you should _not_ choose this (for reasons explained below).

Git mirror at Gitorious.org: git@gitorious.org:webkit/webkit.git. This git repository is a mirror of the svn repository. The difference is that the author/committer names in the commits are "cleaned" up. For example, in the webkit.org's repo it might say 'girish@forwardbias.in@268f45cc-cd09-0410-ab3c-d52691b4dbfc'. In gitorious' repo, this is cleaned up as 'girish@forwardbias.in. As a result of this cleanup, the repository sha1's of gitorious and webkit.org _don't_ match. This mirror is run by Tor Arne. If it is not up to date for some reason, you now know whom to poke.

qtwebkit at gitorious.org: git://gitorious.org/+qtwebkit-developers/webkit/qtwebkit.git. This is the QtWebKit repository. The releases are essentially branches created from the gitorious.org mirror with patches/hot fixes applied. These patches may not have yet landed upstream. And the commits might be in a different order. Since, qtwebkit is branch off gitorious.org, QtWebKit developers would find working with the gitorious.org easier. If you fetch gitorious.org mirror often, you will have lesser objects to fetch to keep qtwebkit uptodate (as opposed to webkit.org git mirror which has nothing in common with qtwebkit).

My usual git-svn workflow:
git fetch webkit-gitorious (get git-svn commits)
git update-ref refs/remotes/trunk webkit-gitorious/master (change the trunk)
git svn rebase (makes git-svn update the svn<->sha1 mapping)

Texture Mapper: Accelerated compositing evolved

In the last year I've been developing and maintaining the Qt implementation of Accelerated Compositing. The accelerated compositing code path in WebKit was originally written by Apple, to allow WebKit ports to accelerate animations, transitions and 3D transforms using the GPU. It was originally using CoreAnimations on Mac and on the iPhone, and something similar on Windows. In January 2010 we introduced the Qt implementation, that uses QGraphicsView and the Qt animation framework, which was then followed by the Chromium implementation, for, as you can guess, Chromium.

The Accelerated Compositing concept comes to optimize for cases where an element would be painted to the screen multiple times without its content changing. For example, a menu sliding into the screen, or a static toolbar on top of a video. It does so by creating a scene graph, a tree of objects (graphics layers), which have properties attached to them - transformation matrix, opacity, position, effects etc., and also a notification when the layer's content needs to be re-rendered.

How Qt Accelerated Compositing Works Today

If you want to know the basics about WebKit's graphics rendering, I recommend reading an entry from the Chromium blog entry from a few months ago: (https://sites.google.com/a/chromium.org/dev/developers/design-documents/gpu-accelerated-compositing-in-chrome).

Our architecture right now is slightly simpler than Chromium, since we don't use their cross process architecture. The idea right now is that each graphics layer is backed by a QGraphicsItem and a QPixmap, and the graphics scene is responsible to know which items to draw, who clips who, apply effects etc. The hardware acceleration is achieved by QtOpenGL, which knows how to accelerate repeated painting of QPixmaps, by caching them into OpenGL textures.

Why this was a good idea, and why it needs to change

Since the Qt APIs are so easy to use, getting Accelerated Compositing running on top of

QGraphicsView

was quite simple. We were up and running with this approach within days, and the feature became stable rather quickly. We learned a lot about how to / how not to do this right during this time, and this wouldn't be possible without the

QGraphicsView

API.

But now, after having more and more people use this feature and push its limits, the

QGraphicsView

approach is showing its limitations. For example, getting 3D transformations to work correctly on top of the 2D graphics view was a pain, and it still doesn't look exactly right because

QGraphicsView

support depth buffering, and isn't as fast as it can be, due to the software 3D->2D projection. Also mask and reflection effects are difficult to optimize, and texture uploads are a bottleneck. But the main problem is that there are too many abstractions - CSS goes to webkit, which goes to

QGraphicsView

, then QPixmap, QPainter, the GL paint engine, and then OpenGL and the driver. Too many assumptions are being made along the way, that are not always correct in the context of the application.

Introducing Texture Mapper

The idea was to remove most of the abstractions, and go from GraphicsLayer directly to OpenGL, making the code that traverses the scene-graph and draws the texture-backed layers a cross-platform WebKit implementation that can be also used by other ports. The original code did exactly that, walked through the tree and issued GL calls, saving a texture per graphics layer. But the need for some abstraction layer came about - some GPUs require different optimizations then others, some have DirectFB and not OpenGL, some can benefit from EGL texture sharing and other proprietary extensions, and we still want this to work on pure software if hardware acceleration is not available.

The current implementation uses an abstraction called TextureMapper: it's a couple of thin abstract classes, one is a BitmapTexture and one is the TextureMapper. A BitmapTexture is equivalent to a GL texture or a QPixmap. Like QPixmap is gives you API to paint into it with software via a GraphicsContext, but it also allows you to use it as the render target for hardware accelerated drawing, like an FBO. This is important when using effects or opacity, for example. To render masks or opacity correctly on an element that has several descendant elements, the sub-tree has to be rendered into an intermediate buffer before the effect is applied. With QGraphicsEffect this is only possible using software, because of the limitations imposed by the many abstraction layers.

The TextureMapper class is equivalent to a QPainter or a GraphicsContext, but it's very limited in scope - you can bind a BitmapTexture to be the current rendering target, set the clip, and draw BitmapTextures.

Abstraction layers are usually a mistake

Abstraction layers are something I'm usually very careful about - people implement things from both sides of the layer, and you become tied to concepts you invented before the scope of the problem was completely clear. TextureMapper is an abstraction layer - which holds the same perils - but I chose to do it here anyway because of the other constraints I had.

To reduce the risks of having an abstraction layer, I limited its scope. The main limitation is that this abstraction layer is not public API, and is not pluggable. That means that, at least for now, the header files for that abstraction might change, as long as all the implementations inside webkit trunk can live with that change. So the abstraction layer just helps us separate parts of the code, it's not an "API".

Acknowledgements

This endeavor became real thanks to Sam Magnuson from Netflix, who did a lot of the work with me, and helped test and optimize it with a real-world application on TV hardware. also it was his idea and I stole it :)

Working across companies in the context of an open source project made a huge difference - they got better user experience for their application, and we got a QtWebkit feature that's functioning much better in a productized context.

Status

This code is now in webkit trunk (as of changeset 70487). To enable it, build webkit trunk with Qt (see http://trac.webkit.org/wiki/QtWebKit), adding the texmap CONFIG option. For example:

WebKit/WebKitTools/Scripts/build-webkit --qt --qmakearg="CONFIG+=texmap"

Most of the code resides under WebCore/platform/graphics/texmap, and the OpenGL implementation is under WebCore/platform/graphics/opengl.