Boost your customer loyalty effortlessly Learn more

Shopping cart

How to Configure Robots.txt in Magento 2

magento-2-tutorial
how-to
configure

As you know, configuring robot.txt is important to any website that is working on a site’s SEO. Particularly, when you configure the sitemap to allow search engines to index your store, it is necessary to give web crawlers the instructions in the robot.txt file to avoid indexing the disallowed sites. The robot.txt file, that resides in the root of your Magento installation, is directive that search engines such as Google, Yahoo, Bing can recognize and track easily. In this post, I will introduce the guides to configure the robot.txt file so that it works well with your site.

Follow the simple guide here to configure robots.txt in Magento 2

  • On the Admin panel, click Stores. In the Settings section, select Configuration.
  • Select Design under General in the panel on the left
  • Open the Search Engine Robots section, and continue with following:

How to Configure Robots.txt

  • In Default Robots, select one of the following:
    • INDEX, FOLLOW
    • NOINDEX, FOLLOW
    • INDEX, NOFOLLOW
    • NOINDEX, NOFOLLOW
  • In the Edit Custom instruction of robots.txt File field, enter custom instructions if needed.
  • In the Reset to Defaults field, click on Reset to Default button if you need to restore the default instructions.

  • When complete, click Save Config.

Examples of Custom Instructions

  • Allows Full Access
User-agent:*
Disallow:
  • Disallows Access to All Folders
User-agent:*
Disallow: 
  • Default Instructions
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=

Ref: Magento 2 User Guide

It comes to the end of tutorial: How to Configure Robots.txt in Magento 2.

PDF Invoice samples for Magento 2

mageplaza pdf invoice

Want to apply these above Invoice templates for your store?

See how

Please leave comments if you have any questions, feedbacks.

Comments for How to Configure Robots.txt in Magento 2

You also may like these Magento 2 Extensions

One Step Checkout

$199
77 reviews

Layered Navigation

$99
44 reviews

SEO

FREE
29 reviews

Better Blog

FREE
58 reviews

Gift Card

$199
5 reviews

Reward Points

$149
4 reviews

Affiliate

$149
12 reviews