How to Optimize WordPress Robots.txt File for Search Engine Crawler (SEO)

-
Whenever we talk about WordPress SEO, then the WordPress Robots.txt File plays an important role in Search Engine Ranking. With its help, You can prevent index and crawl the private portion of your site by crawler. However, If you misconfigure the WordPress robots txt file, it may harm your website ranking.

So in this article, I will tell you in details about Robots.txt file. So that you can properly configure robots.txt file for SEO.

By the way, search engine optimization strategies have many ways, but the Robots.txt file is an important part of SEO. The robots.txt file is a small text file found in the root folder of your website that helps to website optimization. But there are many Webmasters or Site owners who are afraid to edit Robots.txt file, but this is not that difficult. Anyone can create and edit their robots.txt file with basic SEO knowledge, and if you are a new blogger, then you are in the right place. In this article, you will know what is the WordPress robots.txt file and How to create robots.txt.

What is WordPress Robots.txt and why should it be used?


What is WordPress Robots.txt and why should it be used?

The robots.txt file helps to understand to search engine bots that which part of the site you want to index and which part Block When search engine spider or web robots come to your site, they follow the robots configuration file on your site and index the part you want to be indexed in search engines.

In WordPress site, the WordPress robots txt file found in the WordPress root folder of WordPress installation. You can save by editing or create new WordPress Robots.txt file via FTP.

Check your WordPress Robots.txt file


As I mentioned above, the Robots.txt file exists in the root folder of your website. You can also view Robots.txt file by using www.domain.com/robots.txt in the search engines or you can also check the robots txt file by going to Google webmaster tools.

wordpress robots txt

How to create your Robot.txt File for WordPress blog or site?


As I mentioned earlier, this robots.txt file is a general text file. If your website has not a Robots.txt file, you can make a robot.txt file by opening Notepad in your pc or laptop.

Add your XML Sitemap to Robots.txt File


If you are using Yoast WordPress SEO plugin or any other plugin to create an XML sitemap, it will try to automatically add a sitemap to the robots.txt file.


However, if it does not add automatically, then you need to manually add the plugin's XML Sitemaps link in your robots.txt file.




Sitemap: http://www.example.com/post-sitemap.xml
Sitemap: http://www.example.com/page-sitemap.xml



Here is a WordPress Robots.txt file sample for any domain. In the Sitemap, replace the Sitemap URL with your blog URL.

sitemap: https://www.example.com/sitemap.xml


User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /archives/
disallow: /*?*
Disallow: *?replytocom
Disallow: /comments/feed/
User-agent: Mediapartners-Google*
Allow: /
User-agent: Googlebot-Image
Allow: /wp-content/uploads/


User-agent: Adsbot-Google
Allow: /


User-agent: Googlebot-Mobile
Allow: /



That is all……

You can comment on any questions or suggestions related to this WordPress Robots.txt article and If you like this article, please do not forget to share it on Twitter, Google+ and Facebook.....

Comments

Post a Comment

Popular posts from this blog

Fix Submitted URL marked ‘noindex’ in Google Search Console Tool 2018

-
Image
Google has recently introduced a Beta version of Google Search Console tool in which webmaster can easily find out their site insight and content indexing. Today in this article, we will discuss New Index coverage issue detected . After launching Beta version, Google is informed every webmaster about Index coverage issue detected via email. If Google has detected Submitted URL marked ‘noindex’ on your site and emailed you then do not ignore it. It may badly hurt your website SEO and Search Engine ranking. Here we are going to discuss with you how to fix Index Coverage Issue in webmaster tool. So let's start article....... How to fix New Index Coverage Issue Detected in Google search console tool First, log in to your Webmaster tool . Then click on the status >> Index coverage >> Submitted URL marked 'noindex '. Here Google will provide 4 ways ( Test Robots.txt Blocking, Fetch as Google, View as Search Result, Submit to Index ) to fix your affected URLs as you ca
Read more

Keep these 12 things in mind before applying to Google AdSense

-
Image
Do you want to earn a lot of money from Google AdSense but it is not easy to get AdSense approval . Here we will tell you some tips that you should use before applying to Google AdSense so that your Google AdSense application can be approved. Some common questions about Google AdSense, which often do bloggers search. How To Apply For AdSense ? How to get Google Adsense approval? I am eligible to apply for AdSense. What is Requirement for Google AdSense? Why AdSense account disapproved? In this article, we will answer you all these questions. This post is a bit longer but it is helpful for you. What is Google AdSense and why should you use it Google AdSense is a great advertising publishing network  owned by Google. It has millions of Satisfied Customers and Publisher. The biggest reason for this, AdSense always pays on time . Another reason to choose Google AdSense, if your blog has high-quality content then you can earn a lot of money through it. Today there are many bloggers who ar
Read more

Best Autoptimize Settings 2019 (Updated)

-
Image
Do you want to improve loading time of your WordPress site ? Well, there are many different ways to improve website loading speed, but today in this tutorial, I'll show you the best Autoptimize settings that optimize CSS and Javascript on your WordPress site and improve site performance  and page load speed. The main goal of the Autoptimize plugin is to minify your website's CSS and Javascript files and improve website performance. In addition, it also optimizes HTML files. So let's start the best Autoptimize settings... Best Autoptimize Settings 2018 Step by Step First, install and activate Autoptimize plugin in your site. After activating the plugin, click on Settings >> Autoptimize. This will take you to the settings page. Here you will see two settings - Basic and Advanced setting. Apart from this, you can configure CDN. Autoptimize Basic setting Here select all checkboxes and click the Save changes button. When you click on Save button after completing Autoptimi
Read more