Aviran's Place

flowrencemary · Level 1 Joined: 29 Apr 2009 Posts: 1

A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.

sherinmary4 · Level 1 Joined: 16 Apr 2009 Posts: 7

thanks for postings Smile

ryananderson08 · Level 1 Joined: 07 May 2010 Posts: 1

I just want to ask something regarding web crawler. Suppose we are adding some meta tags on a webpage now. My question here is -

<meta "name"="robots" "content"="index, follow"> (for all search engines)

and

<meta "name"="googlebot" "content"="index, follow"> (specially for google)

I understand that both of these meta tags are simply used to allow search engines to index your webpage or they act as a welcome not. Also both of these meta tags are used for web crawlers to index your webpage.

can we use both of them..or only using "robots" meta tag is more than enough ???

seo4meall · Level 1 Joined: 06 Mar 2010 Posts: 3 Location: New Delhi

Web crawlers are search engine bots which traverse the web and gather info of web pages

[url=http://www.seopackages.in]SEO Packages[/url]

alexrock23 · Level 1 Joined: 28 May 2010 Posts: 4

Mostly the search engine crawls on the sites every day to update them in cas of any new content added to it. The search engines saves a copy of a page which are visited mostly so that it becomes easy for them to index later. It is the most basic technique used on the Internet and thus widely successful also.

Lance7_q · Level 1 Joined: 19 Aug 2010 Posts: 1

Hey,
Additional information

Web crawling is an important method for collecting data on, and keeping up with, the rapidly expanding Internet. A vast number of web pages are continually being added every day, and information is constantly changing. A web crawler is a way for the search engines and other users to regularly ensure that their databases are up to date.

On the other hand,When a search engine's web crawler visits a web page, it "reads" the visible text, the hyperlinks, and the content of the various tags used in the site, such as keyword rich meta tags. Using the information gathered from the crawler, a search engine will then determine what the site is about and index the information. The website is then included in the search engine's database and its page ranking process.

Search engines, however, are not the only users of web crawlers. Linguists may use a web crawler to perform a textual analysis; that is, they may comb the Internet to determine what words are commonly used today. Market researchers may use a web crawler to determine and assess trends in a given market. There are numerous nefarious uses of web crawlers as well. In the end, a web crawler may be used by anyone seeking to collect information out on the Internet.

Web crawlers may operate one time only, say for a particular one-time project. If its purpose is for something long term, as is the case with search engines, they may be programed to comb through the Internet periodically to determine whether there has been any significant changes. If a site is experiencing heavy traffic or technical difficulties, the spider may be programmed to note that and revisit the site again, hopefully after the technical issues have subsided.

____________
[url=http://gamegoldreview.com/guild-wars-2-gold]buy gw2 gold[/url]

zwendy5755 · Level 2 Joined: 06 Dec 2010 Posts: 37

where is the forum rules?Anybody can tell me?

===================================================
Author is a dedicated fashion reviewer working for a [url=http://www.fashionlande.com/]wholesale wedding dresses[/url] online shop called FashionLande which is professinal in providing [url=http://www.fashionlande.com/]discount wedding dress[/url] and [url=http://www.fashionlande.com/black-evening-dresses-c-10036.html]black evening dresses[/url].

kethal · Level 1 Joined: 18 Dec 2010 Posts: 10

I want to do thanks to all of you as this one is being very good to crawl all the links you have made above,,, These all are really terrific.. I like all of them

Bretta · Level 1 Joined: 28 Mar 2011 Posts: 1 Location: los angeles

It is right a web crawler is the type of a bot. But they don’t use in spamming a site web crawler is using to check all the back links of the web site you have. I am agreeing with you for the info about the web crawler or the software agent.

mycheapgolf · Level 1 Joined: 22 Feb 2011 Posts: 8

[url=http://www.mycheapgolf.com]Cheap golf clubs[/url]
[url=http://www.mycheapgolf.com]Cheap golf irons[/url]
[url=http://www.mycheapgolf.com]Cheap Golf Drivers[/url]
[url=http://www.mycheapgolf.com/goods-10882-wholesale-cheap-Ping-K15-Driver-on-sale.html]Ping K15 Driver[/url]
[url=http://www.mycheapgolf.com/goods-10973-wholesale-cheap-TaylorMade-R11-Driver-on-sale.html]TaylorMade R11 Driver[/url]
[url=http://www.wowoshoes.com]wholesale jordan shoes[/url]
[url=http://www.wowoshoes.com]wholesale nike shoes[/url]
[url=http://www.wowoshoes.com]wholesale nike air max shoes[/url]
[url=http://www.wowoshoes.com]wholesale gucci[/url]
[url=http://www.airmaxclubs.com]cheap nike air max shoes[/url]
[url=http://www.boots4world.com]Timberland men's shoes[/url]
[url=http://www.cheapguccishoe.com]cheap gucci[/url]
[url=http://www.mycheapfashion.com]cheap MBT shoes[/url]
[url=http://www.mycheapsupra.com]Supra UK[/url]