How to Configure Robots.txt in Magento 2

Table of contents:
- Following steps to Configure robots.txt in Magento 2
- Examples of Custom Robots.txt file
- Default Robots.txt for Magento 2
- More Robots.txt examples
- Common Web crawlers (Bots)
As you know, configuring robot.txt is important to any website that is working on a site’s SEO. Particularly, when you configure the sitemap to allow search engines to index your store, it is necessary to give web crawlers the instructions in the robot.txt file to avoid indexing the disallowed sites. The robot.txt file, that resides in the root of your Magento installation, is directive that search engines such as Google, Yahoo, Bing can recognize and track easily. In this post, I will introduce the guides to configure the robot.txt file so that it works well with your site.
Following steps to Configure robots.txt in Magento 2
- On the Admin panel, click
Stores
. In theSettings
section, selectConfiguration
. - Select
Design
underGeneral
in the panel on the left - Open the
Search Engine Robots
section, and continue with following:- In
Default Robots
, select one of the following:- INDEX, FOLLOW
- NOINDEX, FOLLOW
- INDEX, NOFOLLOW
- NOINDEX, NOFOLLOW
- In the
Edit Custom instruction of robots.txt File
field, enter custom instructions if needed. - In the
Reset to Defaults
field, click onReset to Default
button if you need to restore the default instructions.
- In
- When complete, click
Save Config
.
Examples of Custom Robots.txt file
- Allows Full Access
User-agent:*
Disallow:
- Disallows Access to All Folders
User-agent:*
Disallow: /
Default Robots.txt for Magento 2
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
Disallow: /*?
# Disable checkout & customer account
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
# Disable Search pages
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
# Disable common folders
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/
# Disable Tag & Review (Avoid duplicate content)
Disallow: /tag/
Disallow: /review/
# Common files
Disallow: /composer.json
Disallow: /composer.lock
Disallow: /CONTRIBUTING.md
Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow: /COPYING.txt
Disallow: /Gruntfile.js
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /nginx.conf.sample
Disallow: /package.json
Disallow: /php.ini.sample
Disallow: /RELEASE_NOTES.txt
# Disable sorting (Avoid duplicate content)
Disallow: /*?*product_list_mode=
Disallow: /*?*product_list_order=
Disallow: /*?*product_list_limit=
Disallow: /*?*product_list_dir=
# Disable version control folders and others
Disallow: /*.git
Disallow: /*.CVS
Disallow: /*.Zip$
Disallow: /*.Svn$
Disallow: /*.Idea$
Disallow: /*.Sql$
Disallow: /*.Tgz$
More Robots.txt examples
Block Google bot from a folder
User-agent: Googlebot
Disallow: /subfolder/
Block Google bot from a page
User-agent: Googlebot
Disallow: /subfolder/page-url.html
Common Web crawlers (Bots)
Here are some common bots in the internet.
User-agent: Googlebot
User-agent: Googlebot-Image/1.0
User-agent: Googlebot-Video/1.0
User-agent: Bingbot
User-agent: Slurp # Yahoo
User-agent: DuckDuckBot
User-agent: Baiduspider
User-agent: YandexBot
User-agent: facebot # Facebook
User-agent: ia_archiver # Alexa
Abandoned cart email.

Recent Tutorials

Magento 2 Admin Panel
6 Steps to Creating an Extension Package File in Magento
How to build landing pages in Magento store?
Comments for Configure Robots.txt
Explore Our Products:
People also searched for
- magento 2 robots.txt
- magento 2 robots.txt example
- robots.txt magento 2
- magento2 robots.txt
- robot.txt magento 2
- robots.txt for magento 2
- magento 2 robots txt
- magento 2 robots.txt location
- robots txt magento 2
- magento 2.3 robots.txt
- magento 2 robots.txt path
- default robots txt file
- magento robots.txt
- sample robots txt
- robots disallow
- magento robots txt
- magento 2 robots
- magento 2 generate robots.txt
- disable robots txt
- robots.txt disallow all
- how to check robots.txt
- how to configure robots.txt in magento 2
- robots.txt magento
- 2.2.x, 2.3.x, 2.4.x
Stay in the know
Get special offers on the latest news from Mageplaza.
Earn $10 in reward now!