Varnish supports ESI. What you've described is ESI.
Just put Varnish in front of your servers and used the Expires headers of the responses to cache stuff accordingly. If you include user information in the cache hash of the personalised chunks of the page (i.e. in the URL or cookie) then you can cache that for a short time too.
I do apply short term caching to a lot of things, mostly to protect against double requests for the same thing when there might be an expensive query behind it.
Everything has had SSI. In fact the way that the BBC used to make their web site in PERL on Apache used SSI for their "Most Read" section.
ESI is different... Edge Side rather than Server Side.
The benefit is that you don't need one server capable of doing everything. You can have a separate web service capable of returning the "Most Read" section on different technology, different machines, elsewhere on the network... and ESI will interleave the output of these web services at the edge of the network.
Varnish supports ESI, and as one of the main reasons to use Varnish is caching you can reasonably expect (and be correct) that Varnish will cache ESI sections.
So Varnish will receive a document, cache the whole of it, and use an entirely different cache policy on a section of it.
It's all swings and roundabouts. I just have a strong preference towards not making any part of the solution complex by trying to make it do anything more than it need do, especially when other layers can be added that do that specific thing (caching in this instance) better.
Thanks for the pointer. There's two advantages to what I'm doing over using Varnish in this way:
1. The component writer knows the needs of the component and can set the caching policy for it there. Other developers can then just include the component without knowing what its needs are. In a way, this means they don't have to design the page for caching.. they kinda get it for free.
2. There are several optimizations I can make using this method, and since it is intrinsic to the need to be able to run things in parallel (components have a clearly defined line between them another components) the caching ability is just a few extra lines of code and so relatively low cost for me to add.
Just put Varnish in front of your servers and used the Expires headers of the responses to cache stuff accordingly. If you include user information in the cache hash of the personalised chunks of the page (i.e. in the URL or cookie) then you can cache that for a short time too.
I do apply short term caching to a lot of things, mostly to protect against double requests for the same thing when there might be an expensive query behind it.