To find new, fresh and relevant content, such as web pages, videos, pictures, and PDF files, search engines utilise web crawlers and spiders to crawl billions of websites on the internet. These spiders visit websites and follow the links to discover new URLs.
The keywords, metadata, user interaction and freshness of all crawled web pages are stored in a giant database called Caffeine. When online users search for something, search engines check the indexed web pages to match the search query with the most relevant content.
Web pages are organised and ranked by search engines based on their relevance to serve the best possible search results to internet users. Therefore, the higher your website appears in SERPs, the more visible and relevant it is to the search query.