無論是通用搜索還是垂直搜索，其關(guān)鍵的核心技術(shù)之一就是網(wǎng)絡(luò)爬蟲的設(shè)計。本文結(jié)合
HTMLParser 信息提取方法，對生活類垂直搜索引擎中網(wǎng)絡(luò)爬蟲進(jìn)行了詳細(xì)研究。通過深入分析生活類網(wǎng)站網(wǎng)址的樹形結(jié)構(gòu)的構(gòu)架，開發(fā)了收集種子頁面URL 的模擬搜索器，并基于HTMLParser 的信息提取方法，從種子頁面中提取出與生活類主題相關(guān)的目標(biāo)URL。經(jīng)實驗測試證明該爬蟲的爬準(zhǔn)率達(dá)93.552% ，爬全率達(dá)96.720% ，表明該網(wǎng)絡(luò)爬蟲是有效的，達(dá)到中等規(guī)模的垂直搜索企業(yè)級應(yīng)用的要求。
關(guān)鍵詞：網(wǎng)絡(luò)爬蟲；垂直搜索； HTMLParser

Abstract：Whether general search engine or vertical search engine, the design of web crawler is the core technology. In this article, a novel system of life-theme web crawler based on HTMLParser information extraction is thoroughly studied. In this system, a simulation searcher is designed for collecting the seed URL by analyzing tree structure of life-theme website, then, based on the discussion of HTMLParser information extraction, the target URL that relate to life-theme is extracted from the seed pages. Empirical studies show that the Pr ecision = 93.552% and the Re call = 96.720%, proving its effectiveness and achieving requirements for general enterprise-level application of vertical search engine.
Key words：web crawler; vertical search engine;HTMLParser