Foren
robots.txt has HTML content
Gaurav Sharma, geändert vor 7 Jahren.
robots.txt has HTML content
New Member Beiträge: 13 Beitrittsdatum: 23.02.15 Neueste Beiträge
I have a virtual site set up. Using the Google Search Console robots.txt Tester, I'm viewing the robots.txt for my site. The issue is that Google alternately sees a good version of my robots.txt and a bad version of my robots.txt.
Good Version
Bad Version
I've tested the content type using
and the content type is set correctly as
Good Version
User-agent: *
Allow: /
Bad Version
<pre style="word-wrap: break-word; white-space: pre-wrap;">
User-agent: *
Allow: /</pre>
I've tested the content type using
curl -D - 'https://www.example.com/robots.txt' | grep -i content-type
and the content type is set correctly as
Content-Type: text/plain;charset=UTF-8
X-Content-Type-Options: nosniff