As said above, a spider is a search engine examining a site to see what it contains and what search terms would be most appropriate. The term came about as a “spider” “crawls” the “web” (World Wide Web).
_________________________
“Hiking is just walking where it’s okay to pee. Sometimes old people hike by mistake.” — Demitri Martin