Google has launched two new web crawlers, GoogleOther-Image and GoogleOther-Video, designed specifically for scraping image and video content for research and development purposes. These crawlers are variants of the original GoogleOther crawler, introduced in April 2023, and are optimized for non-text (binary) data collection, such as images and videos, rather than text files.
User Agent Details: Each crawler has specific user agent tokens: GoogleOther-Image: GoogleOther-Image/1.0 GoogleOther-Video: GoogleOther-Video/1.0 Publishers can block these crawlers using these tokens in their robots.txt files.
Updated User Agent Strings: Google has also updated the user agent strings for the GoogleOther crawler, ensuring consistency and clarity in identifying these crawlers in server logs. Although these crawlers are meant for R&D and not explicitly for AI training, publishers can choose to block them without affecting their search rankings.
The Google-Extended crawler remains the one used for AI training data.
Want to read more? Click here!
