Foren

robots.txt has HTML content

Gaurav Sharma, geändert vor 7 Jahren.

robots.txt has HTML content

New Member Beiträge: 13 Beitrittsdatum: 23.02.15 Neueste Beiträge
I have a virtual site set up. Using the Google Search Console robots.txt Tester, I'm viewing the robots.txt for my site. The issue is that Google alternately sees a good version of my robots.txt and a bad version of my robots.txt.

Good Version
User-agent: *
Allow: /


Bad Version
<pre style="word-wrap: break-word; white-space: pre-wrap;">
User-agent: *
Allow: /</pre>


I've tested the content type using

curl -D - 'https://www.example.com/robots.txt' | grep -i content-type


and the content type is set correctly as


Content-Type: text/plain;charset=UTF-8
X-Content-Type-Options: nosniff