Customizing robots.txt​

The robots.txt file tells search engines which pages on your site can or cannot be crawled. It includes sets of rules, each with three main parts:

• User-agent, specifying which crawler the rule applies to, e.g., adsbot-google.
• The rule itself, indicating which URLs the crawler can or cannot access.
• An optional sitemap URL.

SHOPLINE generates a default robots.txt file suitable for most stores. However, you can customize the robots.txt in the admin panel.

In this tutorial, you will learn how to customize robots.txt.

Requirements​

Edit robots.txt using the following steps:

1. Go to Admin -> Preferences -> Robots.txt Management -> Go to Settings.
2. Open the Robots.txt editor.

Resources​

The robots.txt supports the following Handlebars objects:

• robots
• group
• rule
• user_agent
• sitemap

Customizing robots.txt​

You can make the following changes:

• Add a new rule to an existing group
• Remove a default rule from an existing group
• Add custom rules

While you can replace all template content with plain text rules, it is strongly recommended to use the provided Handlebars objects where possible. Default rules are regularly updated to ensure SEO best practices are always applied.

Add a New Rule to an Existing Group​

To add a new rule to an existing group, adjust the Handlebars to display the default rules, identify the associated group, and insert your rule.

For example, to prevent all crawlers from accessing pages with the URL parameter ?q=:

{{#each robots.default_groups as |group|}}
  {{~ group.user_agent.directive }}: {{ group.user_agent.value }}
  {{#each group.rules as |rule|}}
    {{~ rule.directive }}: {{ rule.value }}
  {{/each}}
  {{#if group.user_agent.value '==' '*'}}
  Disallow: /*?q=*
  {{/if}}
  {{#if group.sitemap}}
    {{~ group.sitemap.directive }}: {{ group.sitemap.value }}
  {{/if}}
{{/each}}

Remove a Default Rule from an Existing Group​

To remove a default rule from an existing group, update the Handlebars to detect the rule and skip it.

For example, to remove the rule blocking crawlers from accessing the /policies/ page:

{{#each robots.default_groups as |group|}}
  {{~ group.user_agent.directive }}: {{ group.user_agent.value }}
  {{#each group.rules as |rule| ~}}
    {{~ rule.directive }}: {{ rule.value }}
  {{/each}}
  {{#if group.user_agent.value '==' '*'~}}
  Disallow: /*?q=*
  {{/if}}
  {{#unless (and (if rule.directive '==' 'Disallow') (if rule.value '==' '/policies/'))~}}
    {{~ rule.directive }}: {{ rule.value }}
  {{/unless}}
  {{#if group.sitemap}}
    {{~ group.sitemap.directive }}: {{ group.sitemap.value }}
  {{/if}}
{{/each}}

Add Custom Rules​

To add new rules not part of the default groups, manually input rules outside the Handlebars to output default rules.

Common examples of custom rules include:

• Block certain crawlers
• Allow certain crawlers
• Add extra sitemap URLs

Block Certain Crawlers​

To block a crawler not in the default ruleset, manually add a rule.

For example, to block the discobot crawler:

User-agent: discobot
Disallow: /

Allow Certain Crawlers​

Similarly, to allow a crawler, manually add a rule.

For example, to allow the discobot crawler:

User-agent: discobot
Allow: /

Add Extra Sitemap URLs​

To include an additional sitemap URL:

Sitemap: [sitemap-url]