The first search engine created was Archie, created in 1990 by Alan Emtage, a student at McGill University in Montreal. The original intent of the name was "archives," but Unix standards required a shorter filename.
The main way people shared data back then was via File Transfer Protocol (FTP),i.e,to share a file you would need to set up a FTP server and if someone was interested in retrieving the data they could use an FTP client.This process worked effectively in small groups, but the data became as much fragmented as it was collected.
Archie helped solve this data fragmentation problem by combining a script-based data gatherer with a regular expression matcher for retrieving file names matching a user query.
Archie had such popularity that in 1993 the University of Nevada System Computing Services group developed Veronica. Veronica was mirror image to Archie, but it worked on plain text files. Soon another user interface name Jughead appeared with the same purpose as Veronica, both of these were used for files sent via Gopher, which was created as an Archie alternative by Mark McCahill at the University of Minnesota in 1991.
Matthew Gray introduced the World Wide Web Wanderer.It was developed to measure the growth of the web and created this bot to count active web servers. He soon upgraded the bot to capture actual URL's. His database became known as the Wandex.Computer robots are simply programs that automate repetitive tasks at speeds impossible for humans to reproduce.
Bot is defined as anything that interfaces with the user or collects data."Spiders" are used by search engines to search the web for information.