TIME NOW
World current time now,
CALENDAR
Calendar monthly, yearly
login CONVERT LENGTH
login CONVERT TEMPERATURE
login DICTIONARIES, LISTS
login SCIENCE EDUCATION RELIGION
login WORK CALCULATOR
login CALCULATE LIFE

Internet. programming, web - news, blog, articles

Previous articlePage bottomNext article  ALL TOPICS

Web sites. robots.txt example

 robots.txt example with various criterias. It contains rules for bingbot and others:
User-agent: bingbot
Disallow: /_common_/
Disallow: /html-lessons?*
Disallow: /some-url
Disallow: /?limit=*
Disallow: /?oE=*
Disallow: /?u0=*


User-agent: *
Disallow: /_common_/
Disallow: /html-lessons?*
Disallow: /some-url
Disallow: /?limit=*
Disallow: /?oE=*
Disallow: /?u0=*

Disallow: /_common_/        means to restrict everything in folder /_common_/
Disallow: /some-url             to restrict URL with variables and without them.
Disallow: /html-lessons?*  to restrict URL with variables.  Clean url like http://mainfacts.com/html-lessons will not be restricted


Disallow: /?limit=*
Disallow: /?oE=*
Disallow: /?u0=*                  
restriction of "trash", dublicated content of main page (if it happens of course). For example
                                             http://mainfacts.com/?limit=1&oE=1
                                             http://mainfacts.com/?limit=120000
                                             http://mainfacts.com/?oE=13413&id=1

  
Previous articlePage topNext article  ALL TOPICS



 Use username: Guest, Anonymous, Programmer






QUOTES:
In the confrontation between the stream and the rock, the stream always wins--not through strength but by perseverance.
H. Jackson Browne
People want riches; they need fulfillment.
Robert Conklin
Science is not only compatible with spirituality; it is a profound source of spirituality.
Carl Sagan