Indexing Yandex pages. A blog about WordPress plugin settings and search engine optimization for beginner webmasters. How to speed up the indexing process

Until the search engine indexes the site page, it will not participate in its search results. A search engine can find out about a site in two ways:

from the webmaster. Why do you need to add the URL of a web document to the Yandex add-on or Google(from English add url).
by navigating to site pages via links from other indexed web documents.

On this blog, I encountered a situation where, without any problems, Yandex indexed only the Home page, but the internal ones did not participate in the search. Google did not experience similar inattention. There, you could almost instantly notice your article in the search results.

Why is the site not indexed in Yandex?

Google enters all pages of a site into its database: high-quality and low-quality, indiscriminately. But only useful web documents [not to be confused] are included in the ranking. Yandex does not immediately include web junk. You can force it to index any page, but over time the search engine will remove the garbage. Both systems have an additional index. For both systems, low-quality pages affect the ranking of the site as a whole. On the last point there is an official statement [see. Google blog] and simple logic. A particular person’s favorite sites will be in higher positions in his search results. But this same person will have difficulty finding a site that failed to interest him last time. Therefore, first you need to block duplicate web documents from indexing, check whether there are pages with missing content and not allow worthless content into the search results.
"Check URL" in Webmaster.Yandex will remind you what or what the server is delivering.
If you purchased a supported domain that was subject to sanctions, you need to write to the support service something like Hello. On January 1, 2000, the site.ru domain was acquired. On January 20, 2000, the site was added to Webmaster and Addurilka. After three weeks it was not indexed. Please tell me, can a domain cause poor indexing?

How to speed up indexing in Yandex

Confirm rights to manage the site in Yandex.Webmaster.
Publish a link to the article in . Since 2012, Yandex has signed an agreement with him.
Install Yandex.Browser on your computer and “walk” through the pages of the site with its help.
Add . There, in the “Indexing” column, you can enter your URLs manually, just like in Adduril. [not relevant]
Install the Yandex.Metrica code without checking the "Prohibition of sending pages for indexing" checkbox.
Create a Sitemap file. Then, upon arrival to the site, the robot will first check it. This file exists only for him and is not visible to the audience. It consists of a list of page URLs. Newly created or with updated content are at the top. The Sitemap address is written in robots.txt or in the appropriate form in Webmaster - "Indexing settings" - "Sitemap files".

Additional actions when Yandex has indexed only 1 page

The more often the page is updated, the more often the search robot will re-index it. Basically, the content changes periodically on the "site map" and Home pages. The more often new articles are published, the more often the above-mentioned pages will be updated and new material will be indexed through the links inside them.
If you make a cross-cutting block on all pages with the latest written publications, then the search robot can go to a new post from any page of the site it has visited. And it will be faster. Links in an end-to-end block should not be implemented by a script; there is no need for them.
Sites that have , can register in the Yandex.blogs directory, and from there the data will be transferred to the main robot.
Leave links to your articles on frequently updated resources: “LiveJournal”, “, “”, etc.

Database update - issue update(s)- occurs on average once a week. If all the above actions did not help, which I have never had, then you should write a letter to the support service, tell them what has been done, a little about the site, that it is regularly updated, that people are interested in it, give a couple of links to the best articles as examples. If a resource has low traffic due to its narrow subject matter, you need to talk about it. For example, say that the project does not plan for large attendance, since it was created for the society of lepidopterologists who study butterflies of the Lepidoptera family. If the site is commercial, then indicate that there is a real organization behind it.

If there are problems with indexing, first of all you need to check robots.txt and sitemap.xml.

Any search engine has a large database where it lists all sites and new pages. This base is called an "index". Until the robot crawls the HTML document, analyzes it and adds it to the index, it will not appear in the search results. It will be possible to access it only through a link.

What does "indexing" mean?

No one can tell you about this better than Yandex’s indexing specialist:

Indexing is a process during which a search robot crawls a site’s pages and includes (or does not include) these pages in the search engine index. The search bot scans all content, conducts semantic analysis of text content, the quality of links, audio and video files. Based on all this, the search engine draws conclusions and puts the site in the ranking.

While the site is out of the index, no one will know about it, except those to whom you can distribute direct links. That is, the resource is available for viewing, but it is not in the search engine.

Why do you need an index?

The site must be visible in order to promote, grow and develop. A web resource that does not appear in any PS is useless and does not benefit either users or its owner.

In general, here is the full video from the Yandex webmaster school; if you watch it in full, you will become practically an expert in the issue of indexing:

What does indexing speed depend on?

The main points that determine how quickly your site can come to the attention of search robots:

Domain age (the older the domain name, the more favorable bots are to it).
Hosting (PS do not like free hosting at all and often ignore it).
CMS, code cleanliness and validity.
Page refresh speed.

What is a crawl budget?

Each site has a crawling budget - that is, the number of pages beyond which it cannot be included in the index. If the site’s KB is 1000 pages, then even if you have ten thousand of them, there will only be a thousand in the index. The size of this budget depends on how authoritative and useful your site is. And if you have a problem of such a nature that pages do not fall into the index, then as an option, you need, no matter how trivial it may sound, to improve the site!

Site indexing

When creating a new website, you need to correctly fill out the robots.txt file, which tells search engines whether the resource can be indexed, which pages to crawl and which ones not to touch.

The file is created in txt format and placed in the root folder of the site. Proper robots is a separate topic. This file primarily determines what and how bots will analyze on your site.

Typically, it takes search engines from a couple of weeks to a couple of months to evaluate a new site and enter it into the database.

Spiders carefully scan every allowed HTML document, determining the appropriate topic for a new young resource. This action is not carried out in one day. With each new crawl, the PS will add more and more HTML documents to its database. Moreover, from time to time the content will be re-evaluated, as a result of which the positions of pages in search results may change.

The robots meta tag and partly canonical also help manage indexing. When checking the structure and solving problems with indexing, you should always look for their presence.

Google indexes top-level pages first. When a new site with a specific structure needs to be indexed, the home page is the first page to be indexed. After this, without knowing the structure of the site, the search engine will index what is closest to the slash. Later, directories with two slashes are indexed. This means that even if the links in the content are high, they will not necessarily be indexed first. It is important to structure it optimally so that important sections are not behind too many slashes, otherwise Google will decide that this is a low-level page.

Page indexing

When Yandex and Google have already become familiar with the site and “accepted” it into their search database, bots will return to the resource to scan new, added materials. The more frequently and regularly the content is updated, the more closely the spiders will monitor it.

They say that the PDS pinger plugin for Yandex search helps for indexing - https://site.yandex.ru/cms-plugins/. To do this, you first need to install Yandex search on your website. But I didn’t feel much benefit from it.

When a resource is well indexed, it is much easier to display individual, new pages in the search. But nevertheless, the analysis does not always occur uniformly and at the same speed for all simultaneously updated html documents. The most visited and promoted categories of the resource always win.

What sources of information do search engines have about URLs?

Once upon a time, I hired a quick robot to work on a competitor who had not renewed his domain, so that he would be lowered in the search results - this did not give any result.

How to check indexing

Checking the visibility of html documents is carried out differently for Google and Yandex. But in general there is nothing complicated. Even a beginner can do this.

Verification in Yandex

The system offers three main operators that allow you to check how many HTML documents are in the index.

The “site:” operator shows absolutely all resource pages that are already in the database.

Entered into the search bar as follows: site:site

The “host:” operator allows you to see indexed pages from domains and subdomains within the hosting.

Entered into the search bar as follows: host:site

The “url:” operator – shows the specific page requested.

Entered into the search bar as follows: url:site/obo-mne

Checking indexing with these commands always gives accurate results and is the easiest way to analyze the visibility of a resource.

Google check

PS Google allows you to check the visibility of a site using only one command like site:site.

But Google has one peculiarity: it processes commands differently with and without www entered. Yandex does not make such a distinction and gives absolutely the same results, both with and without registered www.

Checking by operators is the most “old-fashioned” method, but for these purposes I use the RDS Bar browser plugin.

Verification with Webmaster

In Google Webmaster and Yandex Webmaster services you can also see how many pages are in the PS database. To do this, you need to be registered in these systems and add your website to them. You can access them using the following links:

The gist of it is this: just enter the page addresses, and the service gives you the results:

The check is not very fast - you will have to wait about 3 minutes, but there are few complaints about the free tool. Just install it in the background window and go about your business, in a few minutes the results will be ready.

Is it possible to speed up indexing?

You can influence the speed of loading HTML documents by search robots. To do this, you should adhere to the following recommendations:

Increase the number of social signals by encouraging users to share links in their profiles. Or you can take tweets from live accounts in Prospero (klout 50+). If you create your own Twitter whitelist, consider that you have received a powerful weapon to speed up indexing;
Add new materials more often;
You can start spinning Direct for the cheapest queries in your topic;
Enter the address of the new page into addurilki immediately after its publication.

High behavioral factors on the site also have a positive effect on the speed of page updating in search. Therefore, do not forget about the quality and usefulness of content for people. A site that users really like will definitely like search robots.

In general, everything is very easy in Google - you can add a page to the index within a few minutes by scanning it in the webmaster panel (item crawl/view as Googlebot/add to index). In the same way, you can quickly reindex the necessary pages.

I also heard stories about guys who sent URLs via Yandex mail so that they would get into the index faster. In my opinion, this is nonsense.

If there is a real problem, and all the previous tips did not help, all that remains is to move on to heavy artillery.

We configure the Last-modified headers (so that the robot checks for updates only documents that have actually changed since its last call);
We remove garbage from the search engine index (this garbage can be found using Comparser);
We hide all unnecessary/junk documents from the robot;
We create additional Sitemap.xml files. Usually robots read up to 50,000 pages from this file, if you have more pages, you need to make more sitemaps;
Setting up the server.

For a young site, fast indexing in search engines is especially important. Because it does not yet have a certain weight (or “trust”). It is especially important to update the site regularly in the first months of its life. The content must also be of high quality.

Fast indexing of a new site in Yandex

In order for your new site to be quickly indexed in Yandex, you need to add it to Yandex Webmaster. Next, select the “Indexation” -> “Page Retraversal” block (see figure).

Re-crawling pages in Yandex Webmaster

For a young site, it is imperative to include the main page in this list. There are frequent cases when a robot goes to the main page and indexes all internal links from it. In this way, you can index much more than 20 pages.

Fast site indexing in Google

Similarly with Yandex, to speed up the indexing of a new site in Google, it must be added to Google Webmaster (Google Search Console). You need to select the “Skinning” tab -> “View as GoogleBot”.

Function View as GoogleBot

And in the form that opens, insert the address of the desired page, click “Scan”. After this, you will see the result of scanning the page and the magic “Request indexing” button will appear.

Functionality Request indexing

Click on the button and you will see a window like this:

How to index a website in Google

Here you must select “crawl this URL and direct links”. In this case, the robot will try to follow all the internal links on the page you specified. With a high degree of probability, they will all be included in the Google index as quickly as possible!

Indexing a new site on an old domain

In this case, the task is not as trivial as it seems. On domains with history, it is often difficult to index a new site. This process may take weeks or months. It depends on the history of the domain: whether sanctions were previously imposed on it and what kind.

The action plan in this case is simple:

Add a site to Yandex and Google webmaster
Request reindexing through the appropriate functionality
Wait 2-3 updates.
If nothing has changed, write to support and resolve this issue individually.

Methods for speeding up indexing of a young site

In addition to the methods I indicated above, there are several more that work:

Sharing material on social networks. I recommend using the following: VKontakte, Facebook, Twitter, Google+ (despite the fact that Google’s social network is virtually dead, it helps speed up the indexing of new pages)
Regular website updates. Over time, the site accumulates statistics on the publication of new materials, which helps index new pages. Update regularly and maybe you will be able to “feed the fastbot” (in this case, indexing new pages will take 1-10 minutes).
For news sites: log in to Yandex News. It's not as difficult as it might seem, but the effect will be amazing. All Yandex News sites have a fast bot.
Competent internal structure of the site. Be sure to use linking, TOP materials, etc. Increasing the number of internal links on the page (within reasonable limits) will also speed up indexing

Fast indexing of new pages of an old site

Questions and answers on indexing young sites

Do you have questions about indexing young sites? Ask them in the comments!

Q: Should a new site be blocked from indexing?
A: I recommend not opening your site to search robots until it is filled with starter content. In my experience, re-indexing existing pages takes much more time than indexing new ones.

Q: How long does it take Yandex to index a new site?
A: On average, this is 1-2 updates (from 1 to 3 weeks). But situations may be different.

Q: What problems might there be with indexing a young site?
A: Probably the main problem is bad content. It is for this reason that the site may not be indexed. There have also been cases when a young but large site with thousands of pages was rolled out. Search still has memories of doorways, so young sites with thousands of pages have a “special attitude”.

Q: How many pages should a new site be indexed from and how often should it be updated?
A: You can open a site for indexing from 1 page. It is important to adhere to a simple rule: do not post thousands of pages on the site at once, because... This may be considered search engine spam, and new materials are regularly added to the site. Let it be 1 material every 3 days, but regularly! This is very important!

Q: How often does Yandex index sites?
A: According to official information from Yandex, the indexing frequency can be from 2 days to several weeks. And about acceleration methods it is written above.

While filling my website with useful information, I constantly thought about how I could make it fast site indexing, especially in Yandex. And I solved this problem. Now my new website pages are indexed by Yandex in 10 minutes. And this is not a joke, but the pure truth. And so in this article I decided to give you information to take action that helped me speed up the indexing of the site.

In fact, I won’t tell you anything complicated or beyond natural, since I don’t really like it myself. I strive for simplicity and ease, but at the same time I strive to bring my actions to a logical conclusion and result.

Yes, perhaps these results were not always positive, but they were, which means that in the future I will become smarter, and my work will become better.

You ask, what am I talking about?

I’m telling you this for a reason, because through such experiments I achieved fast site indexing in Yandex in 10 minutes. It may not seem real, but it is a fact. I don’t really have time to click on my social network buttons before my new website page is already indexed by Yandex.

Unfortunately, there is no such result in Google, no matter how hard I try. And before 6 hours, my new site pages are not indexed in Google. But I will continue to study and work further on the Google search engine.

But in this article I want to talk specifically about Yandex, namely how it happens on my website in 10 minutes.

Perhaps many of the methods that I will now give for quickly indexing a site are very well known (this applies more to more experienced bloggers and webmasters), but for beginners this information will come in handy.

I will now describe to you point by point all the actions that I did to speed up the indexing of my site and which in my case work perfectly. And so let's begin.

How to quickly index a website in Yandex.

1. First of all, after I publish my article on the Internet, I add it to the Yandex webmaster in the “Report a new site” section. This is the first thing everyone should do to speed up the indexing of their website page. Be sure to register there and put a link there to each new article. Of course, this does not give a 100% guarantee of fast indexing of the site in Yandex, but it would not be superfluous to remind Yandex about a new article on your site.

2. On my website at the end of each article there are buttons for social networks that I use. But I don’t use all the social networks that are out there, but only a few of them. I am adding my new page to such social networks as: Twitter, Facebook, Ya.ru and Google Plus. Here are all the main social networks that I use. But before that, you must, accordingly, register on these social networks. I don’t use any more services because I think that’s enough. If you have your own accounts somewhere else, then this will not be superfluous.

4. I add my new site page to the Pingxpertfree program, which distributes my link across its database of sites on the Internet. They are also called rss feeds.

5. I constantly write new and unique articles for my website. Moreover, I write them with a constant frequency of 1 article every 3-4 days, but no less than 2 articles per week. Accordingly, the Yandex robot comes to my site at a given period of time and waits for new posts, and as soon as they appear, Yandex immediately enters them into its database and, as a result, the site is quickly indexed in Yandex.

Try to stick to a schedule for when a new article appears on your site. Train the search robot to constantly visit your site on the day a new article appears on the site. This point is one of the main ones. The most important thing is to constantly fill your site with fresh information, and the robot will constantly visit your site.

6. Make competent and correct internal linking of the site. Its essence is that all your articles are interconnected by links. This is necessary so that the search robot does not leave the page it came to, but moves on to internal links. This is especially good for a young site that has just appeared on the Internet.

7. Try to increase the link mass of your site. Moreover, it is necessary that the links go not only to the main page, but to all pages of the site. This will ensure faster and better indexing of your site in Yandex. But at the very beginning of creating a website or blog, there is no need to use services for pumping up link mass, especially those measured in large quantities in thousandth equivalent. Instead of "run" it is better to use a service like SeoPult, in which links are purchased naturally, gradually, and only high-quality donors who meet strict requirements are selected.

8. You need to constantly monitor the indexing of your pages on the Internet, in particular the search engines Yandex and Google. Control over indexing is important because the faster your article gets into the search results, the faster you will receive the benefit that you were pursuing in your article. Fast indexing of the article is also necessary so that scammers cannot steal your content and index it faster on their website or blog. Sometimes, your pages, which have already been indexed in search engines, after some time may fall out of the search - this may be a glitch of the search engines and soon your articles will return back to the search results, or some problems have been discovered with your website or blog pages and then you you will need to look for the reason(s) why this happened (there can be many reasons).

To speed up the process of monitoring the indexing of your pages, there is a free service https://serphunt.ru/indexing, which will allow you to quickly and free of charge check the indexing of your new or old page in the Yandex and Google search engines. This service allows you to simultaneously check up to 50 pages of a website or blog. This is what the site itself looks like:

These are all the methods that I use for my new written articles so that there is fast site indexing in Yandex in 10 minutes.

But I don’t give you a 100% guarantee that there will be exactly the result that I have, since it is unknown how it will work on your site. Indexing is a delicate thing and it depends on many factors that you need to look for and make work for yourself.

P.S. Don’t be upset if Yandex doesn’t want to index your site for a long time. Perhaps he does not yet have confidence in your site. The main thing is not to stop and constantly write new articles, and perhaps in the future you will teach everyone how to get a site quickly indexed in Yandex. Yandex is a search engine with character and it needs to be guessed.