Website crawling, indexing and positioning: the guide

The complete guide to understand the differences between crawling, indexing and positioning of a website and to better manage SEO  strategies to make it perform on search engines

Anyone who creates a website must organize content in an SEO NYC perspective if they want it to be found easily by and on Google (and other search engines). However, positioning your site always in the first search results is not easy if you don't have the right skills. Very often, on the Net, there is confusion between crawling , indexing and positioning of a website. Let's find out the meaning of each term together.

What does it mean to crawl a website

When a user does a search on Google, he does it within the index of the web of Google itself, that is in that part of the internet known by the giant of Mountain View.

The phase where Google detects new pages on the internet is called crawling a website . This process starts from the sitemaps provided by the owners of the websites (these are files containing information on the various elements of the site and the related correlations) and from a list of web addresses created thanks to previous scans. The software used by Google, called the crawler , visits these websites and uses the links inside them to detect new pages, then reporting the data relating to the latter to the "Big G" servers.

Google never accepts payments to perform a more frequent scan of a particular site and provides the same tools to everyone, offering owners various choices regarding the crawling procedure: they can provide detailed instructions on how to process their pages, request a new crawl or completely disable the possibility of receiving one using the file called robots.txt .

What is the indexing of a website

The indexing phase of a website is that in which a new site detected by Google's software is inserted into the search engine database. All the information collected is thus organized within the index, keeping track of keywords, site update date and other key information.

When a web page is indexed, it is added to the entries for all the words it contains. The Google Search index is over 100 million gigabytes and includes hundreds of billions of pages.

Knowing how to index a website is important, but it is also knowing how to prevent this from happening: to deny indexing a page or make sure to remove it from the Google index, you need to use the noindex tag , which, put in the page header, signals to the software not to insert it inside the index.

More complicated, for a site owner, is instead managing to make his page appear in the top positions of the Google search results page. This is where the concept of positioning comes into play.

What does it mean to position a website

Positioning a site means making it appear for specific queries in specific positions in the SERP, that is, among the first search results.

In order to show users the best answers, the Mountain View giant uses an algorithm that depends on various factors, ranging from the freshness and originality of the contents to the usability of the site. The fact that other sites considered authoritative by the search engine contain a link to another page, then, is considered an additional indicator of reliability of the linked page.

Appearing in the top positions on the Google search results page is fundamental for a site, because this guarantees superior visibility compared to competitors. In this regard, a distinction is made between organic positioning (ie that obtained through free SEO optimization strategies and techniques, which affect the "Big G" algorithm) and positioning through paid advertisements sold by the search engine.