What Is a Robots.txt File?

Robots.txt is a text file with instructions for search engine robots telling them which pages they should and shouldn't crawl. 

Why Is Robots.txt Important?

A robots.txt file helps manage web crawler activities so they don’t overwork your website 

few reasons to use a robots.txt file:

1. Optimize Crawl Budget-Crawl budget refers to the number of pages Google will crawl on your site within a given time frame. The number can vary based on your site’s size, health, and number of backlinks. 

2- Block Duplicate and Non-Public Pages

3. Hide Resources

Sometimes you want to exclude resources such as PDFs, videos, and images from search results. 

Robots.txt Syntax-

1-The User-Agent Directive-The first line of every directives block is the user-agent, which identifies the crawler. 

User-agent: Googlebot

Disallow: /wp-admin/

2-The Disallow Robots.txt Directive

You can have multiple disallow directives that specify which parts of your site the crawler can’t access. 

For example, if you wanted to allow all search engines to crawl your entire site, your block would look like this:

User-agent: *

Allow: /

If you wanted to block all search engines from crawling your site, your block would look like this:

User-agent: *

Disallow: /

3-The Allow Directive

The “Allow” directive allows search engines to crawl a subdirectory or specific page, even in an otherwise disallowed directory.

For example, if you want to prevent Googlebot from accessing every post on your blog except for one, your directive might look like this:

User-agent: Googlebot

Disallow: /blog

Allow: /blog/example-post

4-The Sitemap Directive-

The Sitemap directive tells search engines—specifically Bing, Yandex, and Google—where to find your XML sitemap.

5-Crawl-Delay Directive

The crawl-delay directive instructs crawlers to delay their crawl rates. To avoid overtaxing a server (i.e., slow down your website).

User-agent: *

Crawl-delay: 10

6-Noindex Directive

The robots.txt file tells a bot what it can or can’t crawl, but it can’t tell a search engine which URLs not to index and show in search results.The page will still show up in search results, but the bot won’t know what’s on it[meta description hidden]

Google never officially supported this directive, but on September 1, 2019, Google announced that this directive is not supported. 


Comments

Post a Comment

Popular posts from this blog

What Is JavaScript?

JavaScript (JS) is a computer programming language used to make websites and applications dynamic and interactive. Think of these coding languages as the components of a house: HTML is the foundation of the house. It provides the basic layout, structure, and content of a website. CSS is the interior design. It provides design, fonts, colors, effects, and other visual elements.  JavaScript is the electrical and plumbing systems. JS brings dynamism and interactivity to the website. For example, pop-ups, animations, video and social media embeds, drop-down menus, and many other website components are created using JavaScript. Brendan Eich developed JavaScript in 1995 while working for Netscape. Libraries vs. Frameworks Libraries are collections of prewritten code that can be used to easily implement frequently used JavaScript tasks. Libraries lead to faster development because developers don’t need to write every line of code. Frameworks are collections of libraries. Think of librarie...
Read more

Images SEO

1. Use Unique Images & Images With Context To Get More Clicks 2. Find & Fix Broken Images 3. Optimize Your Images To Speed Up Your Website’s Page Speed How To Reduce Image Size & Speed Up Your Site Save your visuals in the most optimal format. JPEG is usually the best for large photos. SVG should be reserved for logos and icons. Be careful with WEBP: it offers high quality with a low file size (and is a good alternative to the cumbersome GIF), but certain browsers don’t support it. B-Manually set optimal height and width. C-Use compression. d-Merge images together.  e-Use lazy loading.  4. Make Your Images Responsive How To Make Images Responsive Set the CSS width property to 100% and height to auto. This way, the image will scale both up and down. Set the max-width property to 100%. In this case, the image will never scale up to be larger than its original size. Use the <img srcset> attribute. It displays a different version of the image for different screen...
Read more

Sitemap Overview

What Is a Sitemap? A sitemap is a file that lists all the important pages on a website. It guides search engines in determining which pages to include in their index. Types of Website Sitemaps There are two types of sitemaps:  XML sitemaps[ Extensible Markup Language]: sitemaps written in a specific format designed for search engine crawlers HTML sitemaps: sitemaps that look like regular pages and help users navigate the website 1-XML Sitemap- They carry three primary kinds of information for search engines: The list of all the URLs you want to have indexed The “lastmod” attribute that informs when the URLs were last updated The "hreflang" attribute that informs about the local variants of the URLs HTML Sitemap- HTML sitemaps used to be a popular way to improve a website's navigation and provide links to all your pages in one place. How to Find a Sitemap- 1-Check-https://domain.com/sitemap.xml 2-Search Operators “site:domain.com filetype:xml” “site:domain.com inurl:sitema...
Read more