一个简单的初级爬虫程序通用网络爬虫又称全网爬虫（Scalable Web Crawler），爬行对象从一些种子 URL 扩充到整个 Web，主要为门户站点搜索引擎和大型 Web 服务提供商采集数据。 由于商业原因，它们的技术细节很少公布出来。 这类网络爬虫的爬行范围和数量巨大，对于爬行速度和存储空间要求较高，对于爬行页面的顺序要求相对较低，同时由于待刷新的页面太多，通常采用并行工作方式，但需要较长时间才能刷新一次页面。 虽然存在一定缺陷，通用网络爬虫适用于为搜索引擎搜索广泛的主题，有较强的应用价值.(General web crawler, also known as Crawler Web (Scalable), crawling objects from some seed URL expansion to the entire Web, mainly for portal sites, search engines and large Web service providers to collect data. Their technical details are rarely published for commercial reasons. The range and quantity of this kind of crawling web crawler to crawl speed and huge storage space for higher requirements, requirements of the order page crawling is relatively low, at the same time due to refresh the page too much, usually with parallel, but take a long time to refresh a page. Although there are some defects, the general web crawler is suitable for searching for a wide range of topics for search engines, and has a strong application value.)