Responsive and dynamic websites are all the rage and rightly so. But dynamic content has revealed a major flaw in how Google crawls and caches the internet. One that leaves webmasters creating content which isn’t indexed and scammers with an open door.
Before I explain the flaw in Google Iﾒll briefly cover what dynamic website design is as this will explain how search engines are missing content while some webmasters could use techniques to fool search engine crawlers.
Responsive and dynamic websites for visitors
Responsive and dynamic website designs are often confused so let’s clarify the difference:
- Responsive design is about how content will be displayed.
- Dynamic design is about if content will be displayed and/or which content will be displayed.
These types of design are needed because people now access the internet on a wide range of devices with varying screen sizes but they expect to see well presented sites no matter how they log on to the web.
A website that adjusts how it shows content (responsive) and what content it loads (dynamic) based on the users screen size is one way of meeting these expctations. I say one way because there are many other aspects to responsive and dynamic websites, like showing the appropriate language version based on the users computer settings or geographical location. But in this post Iﾒm going to be talking purely graphical display.
Hereﾒs a responsive example: If a user has a screen with a width greater than 980px wide then a box with ‘share’ icons scrolls with the screen, otherwise the box goes somewhere static so as not to get in the way of the main content on smaller screens.
Hereﾒs a dynamic example: If a user has a laptop or tablet show a right hand bar of content with advertising and links to other parts of the site. But donﾒt do this for smartphone users as it would lead to two very thin columns or a right bar that isnﾒt even visible anyway.
Note we donﾒt just want to hide our right hand bar in the example above, say by using the css code display: none; – this would be a responsive design solution. We want to avoid loading it altogether if the users screen isnﾒt wide enough ﾖ why waste their time waiting for content to load that is then hidden? Hence the need for a dynamic design solution.
Why website designers promote responsive and dynamic solutions
The main draw of responsive and dynamic website design is to avoid the need, and cost, of creating a site for laptops, another for tablets and yet another for smart phones.
It’s always the same website that opens and then the code just figures out what to load and how to display it.
Responsive and dynamic website logic in action
When creating this website, doodleddoes.com, as a responsive and dynamic website we considered what logic should be followed when certain pages (like this one) are opened. Three examples were:
- If the screen is wider than 980 pixels load the right hand bar with recent blog posts, links to my profiles on social networks, etc. If it isnﾒt, donﾒt load it as this could cause poor loading times and wouldnﾒt fit on the screen anyway. This is a dynamic website design characteristic as we are asking if the content should load or not.
- If the screen was wide enough show the main menu options as a horizontal list. If not show one phrase ﾑNavigation menuﾒ which is large enough to be clear on a small screen and easier to navigate on a touch screen. On the surface you might think this is responsive but it is actually dynamic. No menu is shown until the screen size is known so this is an ‘if‘ decision, not a ‘how’. If it was just about changing the styling of the menu that would be responsive but the question of which menu to load makes it dynamic.
- If the screen is deep enough include the share bar as an area which scrolls with the page. But if the screen doesnﾒt have depth just tuck it away at the bottom because it would get in the way of the content otherwise. This is responsive. Itﾒs not about if the content should load, just about how to display it.
The result was that a visitor using a smart phone saw this:
A visitor using a laptop saw this:
What visitors see versus what Google sees
But what Google cached was this:
What visitors see versus what Google sees
Oh dear! Note the blank menu bar and the missing right hand bar content highlighted by the red arrows. In fact when tested Google had absolutely no knowledge of the right bar content even though it had happily indexed the page.
Why canﾒt Google crawl dynamically loaded content?
Dynamically loaded content is called by the browser that opens the page. In a nut shell Googleﾒs crawler, and those of other search engines, donﾒt spend their days indexing the web using, say, Firefox and a 1,000 pixel wide screen!
But because they donﾒt (and they don’t try to emulate a browser) they donﾒt see what a browser really opens and herein lies the problem.
The security issue of not crawling dynamic content
This situation surprised me because there is an obvious security issue here. Googleﾒs crawler may well conclude, based on the content it found, that the page above was about ﾑOnline Marketing Tutorialsﾒ. In this case it would be right!
But what was there to stop me filling the dynamically loaded right bar up with material about gambling or pornography or who knows what. Google just simply wouldnﾒt know so any ﾑfiltersﾒ that a user may have set up to avoid said sites with said content would be bypassed.
Moreover I could put some innocent copy in the right bar and then use something like JQuery, once the page has been opened, to dynamically replace it with evil copy. Or I could even go one step further: Have innocent copy showing as default text for the whole page but after a few seconds replace it with evil content …
Google would only index the innocent copy and its crawler would never fire the code which would load the evil content.
There are probably people doing this already and it wouldnﾒt surprise me if it becomes a large scale form of spamming soon.
Incidently if you want to see this in action I’ve made a demo page which shows the issue.
The SEO/Webmaster issue of not crawling dynamic content
If the ﾓjust create great contentﾔ brigade havenﾒt made you want to scream yet, imagine if you are making great content but Google has absolutely no idea that you are because it is all in dynamic loading areas. That really will make you want to scream.
Responsive websites using dynamic loading can do just that if youﾒre not careful.
How do you know what Google does and doesnﾒt see?
Luckily thatﾒs straightforward. Use fetch as Googlebot in Google Webmaster Tools. This will instantly show you what the Google robot picks up and what it misses or canﾒt read.
If your pages are already in the search results just do a search for the URL and then click the small downward pointing triangle to the right of the green text. Click the ﾑCachedﾒ option to see what Google is currently storing.
One design for mobile, tablet and laptop?
My advice at the moment would be that if you are hoping to rank based on your content then either avoid dynamic loading or create separate websites.
Anything that relies on content being dynamically loaded based on any metric of a users device or browser is content that search engine crawlers are not going to see.
Because this is such an open door to abuse I can’t see this issue continuing in the long term but if you need to rank today, you will need to consider getting round the inability of search engines to crawl dynamically loaded content.