基于度排序的P2P IPTV分布式爬虫系统设计与实现
Design and Implementation of a Distributed Crawler System Based on Degree-rank for P2P IPTV
作者：王海舟(四川大学 计算机学院 网络与可信计算研究所)；陈兴蜀(四川大学 计算机学院 网络与可信计算研究所)；王文贤(四川大学 计算机学院 网络与可信计算研究所)；吴小松(四川大学 计算机学院 网络与可信计算研究所)
Author：Wang Haizhou(Network and Trusted Computing Inst.,College of Computer Sci.,Sichuan Univ.)；Chen Xingshu(Network and Trusted Computing Inst.,College of Computer Sci.,Sichuan Univ.)；Wang Wenxian(Network and Trusted Computing Inst.,College of Computer Sci.,Sichuan Univ.)；Wu Xiaosong(Network and Trusted Computing Inst.,College of Computer Sci.,Sichuan Univ.)
收稿日期：2013-12-05 年卷（期）页码：2014,46(3):109-115
期刊名称：工程科学与技术
Journal Name：Advanced Engineering Sciences
关键字：P2P技术；度排序；分布式架构；爬虫系统；PPTV网络电视
Key words：P2P technology;degree-rank;distributed architecture;crawler system;PPTV system
基金项目：国家科技支撑计划资助项目（2012BAH18B05）；国家自然科学基金项目（61272447）
中文摘要
为了开发一款面向P2P网络电视主动测量研究的高性能爬虫系统，提出了一种基于节点度排序的节点信息爬行算法。该算法首先使用类似广度优先遍历的策略对网络电视覆盖网进行节点信息的初始爬行，然后对爬行结果获得的节点列表进行基于节点度大小的排序，最后将排序后的结果作为输入提供给分布式爬虫以实施节点信息的实时采集。实验结果表明，基于度排序的分布式爬虫系统相比现有的其他爬虫具有更快的爬行速度，该爬虫可以在30 s左右完成对一个拥有7 200左右用户规模的流行节目的爬行,为采集准确的PPTV节点信息快照并实施主动测量研究提供了解决方案。
英文摘要
In order to develop a high-performance crawler application for the active measurement of P2P IPTV,based on degree-rank to capture peers’ information，a new algorithm was proposed.Firstly,a similar strategy of breadth-first search (BFS) was adopted to undertake the initial crawling of peers’ information in the overlay.Subsequently,the captured peer list was sorted according to the size of peer’s degree.Finally, the sorted peer list was provided to distributed crawler system as the input so as to start the real-time collection of peers’ information.The experiment results showed that the performance of the proposed distributed crawler system is better than other existing crawler systems.Specifically, the distributed crawler can track a very popular channel with about 7 200 online peers in about 30 seconds. The results can provide a good solution for capturing more accurate snapshots of peers’ information and conducting active measurement study.
【关闭】
