website ranking, search engine included should always concerned, we often talk to the spiders crawl, then how to determine the spiders have not been to your website do?
usually the site log will tell us The information here to Wang Dajun network marketing blog, for example. I use a virtual space, in my website root directory of the log folder there are a number of ex date. Log text file. Wang Dajun here to remind you, if you log in there is no file folder, you may need to download the background to the virtual space the log files to the FTP space, that is downloaded to the root directory of your site folder where the log file. We chose ex101116.log download to a local, this file is Wang Dajun network marketing blog, November 16, 2010 in the log, use Notepad to open the log file, search for May find the following code:
google spiders: googlebot
Baidu Spider: baiduspider
yahoo spider: slurp
soso spider : sosospider
msn spider: msnbot
proper way spiders: YodaoBot and OutfoxBot
Sogou Spider: sougouspider
of course, that these Spider came to your site, otherwise there is no such code.
Baidu spider we choose a code 5.html - 80 - 220.181.108.166 Baiduspider + (+ - 200 ....
I explain below, under the code:
1,2010-11-15 18: 18:10 is the Baidu spider to the date and time.
2, GET / post/5.html of Baidu, said the spider to get access to the page get the meaning.
3,80 is the port
4,220.181.108.166 of Baidu spider ip address
5, Baiduspider +(+
6,200 that capture success, which is Baidu spiders crawling return code.
There are also some other code:
2xx Success
200 normal; request has been completed.
< p> 201 normal; immediately after the POST command.202 normal; has been accepted for processing Cheap MLB Jerseys, but processing is not completed.
203 normal; part of the information - the information is only part of the return .
204 normal; no response - request received, but there is no information to be sent back.
3xx Redirection
301 has been moved - Request data with a new location, and the change is permanent.
302 have been found - the requested data is temporary with different URI.
303 See Other - can be another URI to find the response to the request, and should use the GET method to retrieve the response.
304 Not Modified - failure is expected to modify the document.
305 use a proxy - must place the word section provides a proxy to access the requested resource.
306 not used - no longer used; keep this number for future use.
4xx client errors
400 Bad request - syntax problem with the request Asian Escort agency, or can not meet the request.
401 unauthorized - unauthorized client access to data.
402 to pay models - that billing system has been effective.
403 prohibited - even if authorized does not need to access.
404 not found - server can not find the given resource ; document does not exist.
407 proxy authentication requests - the client must first authenticate itself using a proxy.
410 requested page does not exist (permanent);
415 media type is not supported - server denial of service request because the entity does not support the requested format.
5xx server error
500 Internal error - because unforeseen circumstances , the server can not complete the request.
501 is not running - server does not support the requested tool.
502 Bad Gateway - The server received an invalid response from an upstream server.
503 can not get service - due to a temporary overloading or maintenance, the server can not process the request.
on log analysis today we stop here, if you have other ideas, please discuss the exchange .
author Wang Dajun, pushing a core team members, members of the blue flames. want to see more articles please visit the author Wang Dajun network marketing blog (www.wangdajun.net). Wang Dajun network marketing group: 120 973 316 , Wang Dajun Tencent microblogging exchange group: 112,984,178 welcome free of charge reproduced, reprinted, the original author Beijing travel agency, please keep the information, thank you!
Related Articles: