The Tiny Search Engine (TSE) is a project that will crawl websites, index their content, and allow users to query for content. It consists of 3 components:
-
The Crawler, which crawls a website and retrieves webpages starting with a specified URL. It parses the initial webpage, extracts any embedded URLs and retrieves those pages, and crawls the pages found at those URLs, but limiting itself to some threshold number of hops from the seed URL (the ‘depth’), and avoiding visiting any given URL more than once. It saves the pages, and the URL and depth for each, in files. When the crawler process is complete, the indexing of the collected documents can begin.
-
The Indexer, which extracts all the words for each stored webpage and creates a lookup table (index) that maps each word found to all the documents (webpages) where the word was found, along with the counts for each word on each document. It saves this index in a file.
-
The Querier, which reads, parses, and responds to user-input queries. It uses the index produced by the indexer to score each crawled webpage based on its relevance to the query, and prints out a list of webpages ranked by their score.
To build everything, simply enter make into the terminal (from the top-level directory)
To clean up all directories, simply enter make clean into the terminal (from the top-level directory)
For testing each component (crawler, indexer, and querier) please see the README.md in those sub-directories.
Notes:
-
We test the overall TSE program by simply testing the querier since the querier relies on outputs from the crawler and indexer. If you would like to test the querier with user-input queries, please see the
README.mdin thequeriersubdirectory for more information. -
Many of the test scripts require the
valgrindcommand-line tool, which is used to check for memory leaks -
Many of the tests use a small set of webpages hosted on the server http://cs50tse.cs.dartmouth.edu. However, this server is not always accessible, which may cause many tests to fail.