Looking forward to Drupal 8 and using ESI

A couple of days ago I started my first Drupal 7 site for a client, and while I was doing that I was listening to the latest Lullabot podcast and Angie was talking a little about the Drupal 8 (very briefly) and about what Larry Garfield was planning for blocks, in that their will be a completely new model which will be more of a pull model instead of the current push model which is currently in Drupal <= 7. This took me back to Drupalcon SF at the Core Developers Summit and other breakout sessions where we started talking about this.

Basically ATM all the blocks are rendered (depending on restricts) in all pages, even on pages which do not have a region to display. So basically if you have a region 'X' which is only on the front page, and you put blocks into it they will be rendered on all pages even though they will not be displayed. The work around to fix this is to make sure that any block that is within region "X" are restricted to only display on <front>. In the new pull method that is being discussed for Drupal 8, the theme will have more control and regions will only get rendered if the region exists on the page. This will mean that when you doing configure a block correctly and have it being rendered for all pages, it will only be rendered on pages in which the region exists, and leaves your server to do other things instead of wasting time building up blocks they are not going to be using.

This will be a great thing for most sites. Badly configured sites will get the biggest boast where well turned sites will not get so much.

Part of this discussion was the ability to render blocks independently of the rest of the page which would give improvements in being able to running testing which will not be so much of a hack, or even being able to export a Drupal block as a widget which can be embedded into another site. Another thing that this will give is the ability to do Server Side Includes (ESI).

With my work I have a number of large customers which are doing between in excess of 1 million page impressions per month (with my biggest doing close to 20 million), so being able to use ESI really gets me excited.

Most people will be asking what ESI is and why it is so exciting. One of the major limitations of PHP is that it is single threaded. That is it can only do one thing at a time where you are building up a page. Unlike technologies like Java which can be multi-threaded. Being multi-threaded means that you can do multiple things at the same time. So when building up a page in Drupal each part of the page is built one after another, so first the content is rendered, then each of the blocks (your recent posts block, the login block, the who's online block), one after another. With a multi-threaded method before we would have the system render some of the blocks in there own thread. So that resent posts block would be built in the back ground where the other blocks will be created in the foreground which means the overall time to build the page will be less (in theory).

Since we can't use multi-threaded methods in PHP, what it means is that with using something like Varnish or in the higher end Akamai we can use ESI to build the blocks into the page. So when we are building the main page instead of rendering the resent posts block you would add something like this.

&lt;esi:include src="<a href="http://example.com/block/1"">http://example.com/block/1"</a> /&gt;

No real work to add this to the page. Then what happens in Varnish is that it will see this and do a second request to the web server to get this block. And if there are a few of these in the page it will do these all at the same time, giving us parallel building of the page and hopefully make rendering time of the page much smaller.

But this also means that we can do more fun things. So the most recent posts block doesn't need to be built every time the page is rendered, and it doesn't matter which user is viewing the block as it will be the same for everyone (we are ignoring node access) so what we can do is have Drupal tell Varnish via the cache headers to cache this block for 1 hour, so for the next hour Varnish will not go back to the server to get the content of the recent post block but instead use what it had before which means there it will take even less time to build the page.

If the new block/page rendering system work as expected it will mean implementing ESI will be quite easy, and even the administration where I see only a single checkbox which will turn on ESI for a single block, which will mean building larger sites will become easier and even complex smaller sites will be able to cached more logically and speed up the rending of the site.

Drupal 7 is going to be amazing (I say from my Drupal 5.x website) but given the possibilities of what can be done in 8 to help make things faster still and even more flexible is just going to be amazing.

I am starting now to deploy Drupal 7 website so getting out a final release of Drupal 7 is now becoming more important and I will be doing what I can to help get it finished, and I know everyone else will as well.



why wait for D8?

This would be trivial even in drupal 6, you could simply write a callback like module_invoke($module, 'block', 'view', $delta), with appropriate access checks of course.

You do still have a serial step, since the esi:include tag itself won't be visible to the proxy until after the page is rendered.


Allowing Drupal to be more parallel-ized is a great idea in this age of multi-cpu computers. But if Drupal still has to be fully loaded on each page request, I fail to see how another call (or 5-10) to load a block is going to help. I definitely like the idea of "smart" caching the blocks, as that would help a lot.


Blocks is going to suffer dramatic changes for Drupal8, for sure.

In Drupalcon Cph, I went to Crell's session, and the Butler project is really putting a lot of power of building sites with "super"-blocks:

About what you can do already in Drupal6 to improve performance, I would like to add this link of Angie where is better explained:

I agree that is easy to forget about getting out D7 when is so cool stuff being planned for D8. :)

Drupal 7 contrib approach

Drupal 7 also has new block hooks, which allow you to alter the list of blocks. See http://api.drupal.org/api/function/hook_block_list_alter/7 which can be used by a contrib module to implement more of a pull model, albeit in a hackish way.

You can also use http://api.drupal.org/api/function/hook_system_info_alter/7 to define custom regions on top of theme features much like the dashboard module does without those being in the info files.

Regions and blocks got much more flexible in D7, even though they are not yet close to the Butler vision.