Editing Final Report 2012 (section)

===Web Crawler GUI and Pre-indexed Web Data===
[[Image:OldGUI.png|thumb|300px|center|Original Web Crawler GUI Interface.]]

The WebCrawler developed by last year's group working on this project is a highly competent, fully-functional program that does what it has been designed to do very well. However, upon first appearances it is intimidating, with a range of options required to be specified before a search can even be undertaken. There is a detailed and helpful README file provided with the WebCrawler, but it is well-known that most people don't resort to manuals until absolutely desperate. Therefore we sought to redesign only the front end of the WebCrawler in order to encourage more experimentation and greater usage by people interested in  getting their search results as quickly and easily as possible.

Our brief was to design something more like Google, which is immediately accessible and could not be more intuitive. To achieve this without a loss of functionality, we decided to abstract away the more complex options and provide default values for key parameters, so that the user is only required to enter their search string to perform a basic web crawl. However, if they wanted to fine-tune the experience - change the web address the search starts from, save the results to a different location, or define a proxy server connection - they had access to this through an "Advanced" menu. Once either search has commenced, the user is shown a results page which is able to display a greater number of matches at a time - meaning less scrolling and making the results more immediately accessible to the user.

[[Image:NewGUISearchPanel.png|thumb|250px|center|New Web Crawler GUI Interface.]][[Image:NewGUIOptionsPanel.png|thumb|250px|left|Advanced Options.]][[Image:NewGUIResultsPanel.png|thumb|250px|right|Displaying Results.]]


















====Pre-Indexed Web Data====
The internet is a massive place, and the time it would take to crawl every single web page - even by the WebCrawler developed by last year's group - would be prohibitively long. Therefore it was proposed that we search for a means to traverse the web faster. With search engines such as Google regularly trawling the internet and caching the data, it was suggested we find a means to access this pre-indexed data so that search times could be significantly reduced (think how quickly Google returns its search results!). However, the data collected by search engines is not publicly available, so an alternative approach was required. We looked into the possibility of using a service such as Amazon's Elastic Cloud Compute (EC2) to speed the process up. Further investigation indicated that this would require a substantial re-writing of the WebCrawler - which is designed to use a graphical user interface for its input - and only side-stepped the issue. Amazon EC2, and other similar cloud computing solutions, would only provide a means to run large numbers of instances of the WebCrawler simultaneously and, given enough instances, this would be harsh on popular websites with large numbers of links to them, which would be receiving requests from many instances in a short space of time, potentially (accidentally!) causing a Distributed Denial of Service attack as the crawlers use up all the servers' bandwidth.

Given the amount of time and effort required to fully evaluate the possibilities for speeding up trawling the internet, and the reworking of the WebCrawler code, this would make an interesting and useful project in itself for a future group to undertake.