CDN加速镜像 | 设为首页 | 加入收藏夹
登陆 | 会员注册
当前位置： 首页 资源下载 WEB源码 JSP源码/Java
收藏

文件名称:WebCollector-master

  • 所属分类：
    Java Develop
  • 标签属性：
    [Java] [源码]
  • 上传时间：
    2014-10-16
  • 文件大小：
    3.71mb
  • 已下载：
    0次
  • 提 供 者：
    吴***
  • 相关连接：
    无
  • 下载说明：
    别用迅雷下载，失败请重下，重下不扣分！
修正介绍说明

介绍说明－－下载内容来自于网络，使用问题请自行百度

基于WebCollector内核，可以自己编写爬虫的http请求、链接解析器、爬取信息更新器、抓取器等模块，WebCollector把这些基于内核编写的模块称作 插件 ，通过不同的插件组合，可以在1分钟内，把WebCollector组装成一个全新的爬虫。



WebCollector内置了一套插件(cn.edu.hfut.dmic.webcollector.plugin.redis)。基于这套插件，可以把WebCollector的任务管理放到redis数据库上，这使得WebCollector可以爬取海量的数据（上亿级别）。



用户可以根据自己的需求，去开发适合自己的插件-Based WebCollector kernel, you can write your own http request reptiles, link resolver, crawling updates, spider and other modules, WebCollector write these kernel-based modules called　plug-in　through a combination of different plug-ins, you can within one minute, the WebCollector assembled into a new reptile.



WebCollector built a set of plug-ins (cn.edu.hfut.dmic.webcollector.plugin.redis). Based on this set of plug-ins, you can put WebCollector task management onto the redis , which makes WebCollector be crawling mass of data (on a million levels).



Users can according to their own needs, to develop their own plug-ins. .
(系统自动生成,下载前可以参看下载内容)

下载文件列表

WebCollector-master/
WebCollector-master/.gitignore
WebCollector-master/LICENSE
WebCollector-master/README.md
WebCollector-master/README.zh-cn.md
WebCollector-master/WebCollector/
WebCollector-master/WebCollector/pom.xml
WebCollector-master/WebCollector/src/
WebCollector-master/WebCollector/src/main/
WebCollector-master/WebCollector/src/main/java/
WebCollector-master/WebCollector/src/main/java/cn/
WebCollector-master/WebCollector/src/main/java/cn/edu/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/crawler/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/crawler/BreadthCrawler.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/DbUpdater.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/Generator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/Injector.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/ParamURLGenerator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/StandardGenerator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/Filter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/IntervalFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/URLRegexFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/UniqueFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/Handler.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/Message.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/AvroModel.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/CommonURL.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/Link.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/output/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/output/FileSystemOutput.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/HtmlParser.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/LinkUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/ParseResult.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/Parser.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/BreadthCrawlerUI.form
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/BreadthCrawlerUI.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Config.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/ConnectionConfig.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/HttpRetry.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/HttpUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Log.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/RandomUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Retry.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Task.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/u

相关说明

  • 本站资源为会员上传分享交流与学习,如有侵犯您的权益,请联系我们删除.
  • 搜珍网是交换下载平台，只提供交流渠道，下载内容来自于网络，除下载问题外，其它问题请自行百度。更多...
  • 本站已设置防盗链，请勿用迅雷、QQ旋风等下载软件下载资源，下载后用WinRAR最新版进行解压.
  • 如果您发现内容无法下载，请稍后再次尝试；或换浏览器；或者到消费记录里找到下载记录反馈给我们.
  • 下载后发现下载的内容跟说明不相乎，请到消费记录里找到下载记录反馈给我们，经确认后退回积分.
  • 如下载前有疑问，可以通过点击"提供者"的名字，查看对方的联系方式，联系对方咨询.
查看所有评论

相关评论

暂无评论内容．

发表评论

*快速评论： 推荐 一般 有密码 和说明不符 不是源码或资料 文件不全 不能解压 纯粹是垃圾
*内　　容：
*验 证 码：
搜珍网 www.dssz.com