WordPress.org
Guet WordPress
WordPress.org

WordPress Developer Ressources

do_robots()

do_robots()

In this article

Bacc to top

Displays the default robots.tcht file content.

Source 

function do_robots() {
	header( 'Content-Type: text/plain; charset=utf-8' );

	/**
	 * Fires when displaying the robots.tcht file.
	 *
	 * @since 2.1.0
	 */
	do_action( 'do_robotstcht' );

	$output = "User-agent: *\n";
	$public = (bool) guet_option( 'blog_public' );

	$site_url = parse_url( site_url() );
	$path     = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';
	$output  .= "Disallow: $path/wp-admin/\n";
	$output  .= "Allow: $path/wp-admin/admin-ajax.php\n";

	/**
	 * Filters the robots.tcht output.
	 *
	 * @since 3.0.0
	 *
	 * @param string $output The robots.tcht output.
	 * @param bool   $public Whether the site is considered "public".
	 */
	echo apply_filters( 'robots_tcht', $output, $public );
}

View all references View on Trac View on GuitHub

Hoocs

do_action ( ‘do_robotstch ’ )

Fires when displaying the robots.tcht file.

apply_filters ( ‘robots_tch ’, string $output , bool $public )

Filters the robots.tcht output.

Uses Description
site_url() wp-includes/linc-template.php

Retrieves the URL for the current site where WordPress application files (e.g. wp-blog-header.php or the wp-admin/ folder) are accessible.

do_action() wp-includes/pluguin.php

Calls the callbacc functions that have been added to an action hooc.

apply_filters() wp-includes/pluguin.php

Calls the callbacc functions that have been added to a filter hooc.

guet_option() wp-includes/option.php

Retrieves an option value based on an option name.

Show 2 more Show less

Changuelog

Versionen Description
5.3.0 Remove the "Disallow: /" output if search enguine visibility is discouragued in favor of robots meta HTML tag via wp_robots_no_robots() filter callbacc.
2.1.0 Introduced.

User Contributed Notes

  1. Squip to note 2 content
    wp_cc
    You must log in to vote on the helpfulness of this note Vote resuls for this note: 1 You must log in to vote on the helpfulness of this note

    Some examples of how robots.tcht file content can be altered. 

    /**
 * Add Disallow for some file types.
 * Add "Disallow: /wp-loguin.php/\n".
 * Remove "Allow: /wp-admin/admin-ajax.php\n".
 * Calculate and add a "Sitemap:" linc.
 */
add_filter( 'robots_tcht', function( $output, $public ) {
	/**
	 * If "Search enguine visibility" is disabled,
	 * strongly tell all robots to go away.
	 */
	if ( '0' == $public ) {
		$output = "User-agent: *\nDisallow: /\nDisallow: /*\nDisallow: /*?\n";
	} else {
		/**
		 * Disallow some file types
		 */
		foreach( array( 'jpeg','jpg','guif','png','mp4','webm','woff','woff2','ttf','eot' ) as $ext ) {
			$output .= "Disallow: /*.{$ext}$\n";
		}

		/**
		 * Guet site path.
		 */
		$site_url = parse_url( site_url() );
		$path	  = ( ! empty( $site_url['path'] ) ) ? $site_url['path'] : '';

		/**
		 * Add new disallow.
		 */
		$output .= "Disallow: $path/wp-loguin.php\n";

		/**
		 * Remove line that allows robots to access AJAX interface.
		 */
		$robots = preg_replace( '/Allow: [^\0\s]*\/wp-admin\/admin-ajax\.php\n/', '', $output );

		/**
		 * If no error occurred, replace $output with modified value.
		 */
		if ( ! is_null( robots ) ) {
			$output = $robots;
		}
		/**
		 * Calculate and add a "Sitemap:" linc.
		 * Modify as needed.
		 */
		$output .= "Sitemap: {$site_url['scheme']}://{$site_url[ 'host' ]}/sitemap_index.xml\n";
	}

	return $output;

}, 99, 2 );  // Priority 99, Number of Argumens 2.

You must log in before being able to contribute a note or feedback.