Now what may be the solution that can serve search engines the post-processed HTML and not the templates from server. There are numerous techniques or tricks for it but mainly three popular, feasible and affordable solutions are briefly explained below.
- Bot request some webpage say website.com/some-content.html
- The webserver finds out that the request comes from a bot and send special request (seo-app.website.com/?page=/some-content.html) to SEO-specific app.
- The app (i.e. server side browserless PhantomJS app) can handle the request and render actual content from url “website.com/some-content.html” and pass it to google bot.
The picture below illustrates it clearly.
There are some online services that can handle this type of jobs for you without deploying in your own server.
Another technique is online service called BromBone that crawls your sitemap, generates all the pages statically to stores it on Amazon (may be S3). Using this makes you lot of easier and remove hassle of setting up your own middleware SEO app on your server.
This is similar to the above app except one difference is that BromBone renders pages from sitemap beforehand instead of rendering them on the fly. When google robot crawls the website, you have to proxy it to BromBone page from where it got exact copy of the pages which user can see on their browser. This is instant and bot don’t have to wait for the pages which is also extra benefits SEO-wise.
It may be problematic because of it’s relying on the sitemap. What if the sitemap is not updated frequently ? It can’t give google bots new contents. For solving this problem, there is another service called Prerender which use some tricks.
Prerender is open source library and can be deployed on your own server freely. It also uses PhantomJS to prerenders pages on the fly with some additional tricks to set the current status codes and HTTP headers. Because it also renders content dynamically, there is one disadvantage of performance penalty because google bot has to wait for some time when prerender is hard working for generating the required page on the fly. There is simple solution for it. You can post request manually to prerender to warm up prerender’s cache i.e. it can cache all the pages of the website that google bot may hit on cache beforehand so that there is no extra waiting for bot as pages are directly served from webserver’s cache.
For example a GET request to http://prerender.website.com/http://website.com/some-content.html will prerender the page on the fly and store it on webserver’s cache so that real bots can get the page instantly from it. Similarly if content is updated dynamically, a POST request to http://prerender.website.com/http://website.com/some-content.html will refresh the prerendered content. So this is extra manual work which is necessary if you host this SEO-app on your own server.
You can use online service which has nice solution called SaaS that avoids you doing anything manually from your part and it’s pricing is also very affordable.