Hi,
any ideas why a homepage make a lot of traffic? But there are not a lot of people on the site? 24 hours and I have 7 Gig traffic ? Some other pages I have made have 4 Gig in the month ?!
Is mit possible that google robots make traffic ? is there a way I can say how often goggle robots search a site ?
thanks
tom
It's probably not the "googlebot" that is crawling your website this much. You have to look at your analytics tracking and see where this traffic is coming from.
If you do want to ask a "bot" to stop crawling your website, you need to put this into your robots.txt file (I'm using the "Ahrefsbot" as an example")
User-agent: AhrefsBot
Disallow: /
However they can choose to ignore this directive...
You can also use the .htaccess file to stop bots
A good article and how to format the .htaccess here
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.