hitGate - Analytics for Business - helps you turn your site into a profitable one
<meta name="description" content="..."> <meta name="keywords" content="keyword, keyword2, ..."> .. ..
Every piece of content you need to display on the page is on specific local server address as http://www.seowisepro.com/seo-features/ or in case of online stores can be dynamically generated as http://www.medik8.ro/catalog/product/view/id/18
This last address does not provide much informations about the content and it is not conclusive for search engines, when they scan your web site.
It is much better if the url of the products (in this case) is http://www.medik8.ro/medik8-retinol-3tr.html, which gives bots (search engine robots) the clue about the product or services
<a href="http://www.medik8.ro/catalog/product/view/id/18" ... > <a href="http://www.medik8.ro/medik8-retinol-3tr.html" ... >
For your server or customer it not matter if your products are listed like this: http://www.seowisepro.com/seo-features/ or in case of online stores can be dynamically generated as http://www.medik8.ro/medik8_retinol_3tr.html instead of http://www.medik8.ro/medik8-retinol-3tr.html
For serch engines it does matter how it the urls, so it advice to avoid underscore in your urls
BAD <a href="http://www.medik8.ro/medik8_retinol_3tr.html" ... > OK <a href="http://www.medik8.ro/medik8-retinol-3tr.html" ... >
We can set here what we want to be indexed and what we need to keep just for us.
Suppose a bot that will check the http://www.medik8.ro/medik8-retinol-3tr.html page.First, it will be looking in http://www.medik8.ro/robots.txt file.
If he found something like:
User-agent: * Disallow: /
the bot will cancel all the pages, because "Disallow: /" tell the bot that there is no available page to search.
On the other hand, if the bot found something like:
User-agent: * Disallow: /my-file/
the bot will exclude the folder "/my-file/" from indexing but will include the rest.