Some Hints for Nutch

好久没关注Nutch了,看邮件列表,学到了几招关于 Nutch 的小技巧.

  • 如何索引动态 URL 站点?

    调整 regex-urlfilter.txt 或是 crawl-urlfilter.txt 文件. 参见行”# skip URLs containing certain characters as probable queries,后面的内容.

  • 编译 Nutch 需要用到的 Ant 版本至少要 1.6 以上.

  • 验证regex-urlfilter是否正常(by Michael Nebel):

    If you want to know, if your regex-urlfilter works as expectet, you can
    check it with the command:
    cat FILE-WITH-URLS | nutch net/nutch/net/RegexURLFilter
    or by calling "nutch net/nutch/net/RegexURLFilter" and entering the URL
    by hand.
    Everyline line beginning with a "+" ist accepted - a line with a "-" is
    accepted. For example:
    $ echo "http://www.nutch.org" | nutch net/nutch/net/RegexURLFilter
    run with heapsize 256
    -Xmx256m
    050202 173520 loadingfile:/home/nutch/nutch-0.7/conf/nutch-default.xml
    050202 173520 loading file:/home/nutch/nutch-0.7/conf/nutch-site.xml
    050202 173520 found resource regex-urlfilter.txt at
    file:/home/nutch/nutch-0.7/conf/regex-urlfilter.txt