The Most Popular Extension Builder for Magento 2

With a big catalog of 224+ extensions for your online store

How to Configure Robots.txt in Magento 2

As you know, configuring robot.txt is important to any website that is working on a site’s SEO. Particularly, when you configure the sitemap to allow search engines to index your store, it is necessary to give web crawlers the instructions in the robot.txt file to avoid indexing the disallowed sites. The robot.txt file, that resides in the root of your Magento installation, is directive that search engines such as Google, Yahoo, Bing can recognize and track easily. In this post, I will introduce the guides to configure the robot.txt file so that it works well with your site.

Table of contents:

What is Robots.txt in Magento 2?

The robots.txt file instructs web crawlers to know where to index your website and where to skip. Defining this website robots - website crawlers relationship will help you optimize your website’s ranking. Sometimes you need it to identify and avoid indexing particular parts, which can be done by configuration. It is your decision to use the default settings or set custom instructions for each search engine.

Steps to Configure Magento 2 robots.txt file

Please follow this step-by-step guide to configure your robots.txt file in Magento 2:

  • On the Admin panel, click Stores. In the Settings section, select Configuration.
  • Select Design under General in the panel on the left
  • Open the Search Engine Robots section, and continue with following:
    • In Default Robots, select one of the following:
      • INDEX, FOLLOW
      • NOINDEX, FOLLOW
      • INDEX, NOFOLLOW
      • NOINDEX, NOFOLLOW
    • In the Edit Custom instruction of robots.txt File field, enter custom instructions if needed.
    • In the Reset to Defaults field, click on Reset to Default button if you need to restore the default instructions.
  • When complete, click Save Config.

How to Configure Robots.txt

There are various options for you to choose from. Let’s find out what all default robot options stand for in Magento 2 rbot.txt:

  • In case you want the web crawlers to index and follow the links on a page: INDEX, FOLLOW
  • When you don’t want the web crawlers to index but still follow the links on a page: NOINDEX, FOLLOW
  • If you want the web crawlers to index but don’t follow the links on a page: INDEX, NOFOLLOW
  • If you don’t want the web crawlers to index or follow the links on a page: NOINDEX, NOFOLLOW

Magento 2 Robots.txt Examples

You are also able to hide your pages from the website crawlers by setting custom instructions as follows:

  • Allows Full Access
User-agent:*
Disallow:
  • Disallows Access to All Folders
User-agent:*
Disallow: /

Magento 2 Default Robots.txt

Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Disallow: /*?

# Disable checkout & customer account
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

# Disable Search pages
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/

# Disable common folders
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/

# Disable Tag & Review (Avoid duplicate content)

Disallow: /tag/
Disallow: /review/

# Common files
Disallow: /composer.json
Disallow: /composer.lock
Disallow: /CONTRIBUTING.md
Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow: /COPYING.txt
Disallow: /Gruntfile.js
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /nginx.conf.sample
Disallow: /package.json
Disallow: /php.ini.sample
Disallow: /RELEASE_NOTES.txt

# Disable sorting (Avoid duplicate content)
Disallow: /*?*product_list_mode=
Disallow: /*?*product_list_order=
Disallow: /*?*product_list_limit=
Disallow: /*?*product_list_dir=

# Disable version control folders and others
Disallow: /*.git
Disallow: /*.CVS
Disallow: /*.Zip$
Disallow: /*.Svn$
Disallow: /*.Idea$
Disallow: /*.Sql$
Disallow: /*.Tgz$

More Robots.txt examples

Block Google bot from a folder

User-agent: Googlebot 
Disallow: /subfolder/

Block Google bot from a page

User-agent: Googlebot 
Disallow: /subfolder/page-url.html

Common Web crawlers (Bots) {#common-web-crawlers-(bots)}

Here are some common bots in the internet.

User-agent: Googlebot
User-agent: Googlebot-Image/1.0
User-agent: Googlebot-Video/1.0
User-agent: Bingbot
User-agent: Slurp		# Yahoo
User-agent: DuckDuckBot
User-agent: Baiduspider
User-agent: YandexBot
User-agent: facebot		# Facebook
User-agent: ia_archiver		# Alexa

How to add a sitemap to the robots.txt file in Magento 2?

Much like the robots.txt file, the Magento sitemap plays a crucial role in optimizing your website for search engines. It facilitates a more thorough analysis of your website links by search engines. As robots.txt provides instructions on what to analyze, it is advisable to include information about the sitemap in this file.

To integrate a sitemap into Magento’s robots.txt file, follow these steps:

Go to Store > Configuration > Catalog > XML Sitemap and locate the Search Engine Submission Settings section.

Activate the Submission to Robots.txt option.

Configuration

If you wish to incorporate a custom XML sitemap into robots.txt, proceed to Content >Design > Configuration > select a website > Search Engine Robots. Then, append a custom sitemap to the "Edit custom instruction of the robot.txt File" field.

The bottom line

Configuring Robots.txt is the first step to optimize your search engine rankings, as it enables the search engines to identify which pages to index or not. After that, you can take a look at this guide on how to configure Magento 2 sitemap. If you want a hassle-free solution that works right out of the box for your store with easy installation, check our SEO extension out. In case you need more help with this, contact us and we will handle the rest.


SEO

SEO for M2

Drive more traffic to your online store and boost your rank on search engines

Check it out!


Image Description
Sam is the CEO & co-founder of Mageplaza, a company established to support Magento merchants with different powerful tools and resources. Sam Nguyen is also the CEO & founder of Avada Commerce, an e-commerce solution provider headquartered in Singapore – aiming to support more than a million online businesses to grow and develop.

Looking for
Customization & Development Services?

8+ years of experiences in e-commerce & Magento has prepared us for any challenges, so that we can lead you to your success.

Get free consultant
development service
x

    Explore Our Products:

    People also searched for

    Subscribe

    Stay in the know

    Get special offers on the latest news from Mageplaza.

    Earn $10 in reward now!

    Earn $10 in reward now!

    comment
    iphone
    go up