CHAPTER. 6. Crawling the Web with Java. By James Holmes. 167 The Web crawler developed in this chapter is called Search Crawler. Disallow: /login..