-
Notifications
You must be signed in to change notification settings - Fork 27.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Next 13 - Sitemap can't fetch on Google Search Console #51649
Comments
same issue.
Maybe some more time needs to pass, so I'll give it a little more time. |
Have you tried putting it in the /public folder instead? |
I've tried, but no..., https://github.com/vercel/commerce/blob/70dcfa9736bb2067713a425e17ee6e59fb3fca2b/app/sitemap.ts#L8 |
Has someone solved this issue ? I'm still stuck on it without any pieces of possible solution... |
Also experiencing this. Putting a |
I tried but even if my new sitemap is valid, nothing changed... |
Has someone solved this issue ? I'm still stuck on it without any pieces of possible solution... i did as same document of next.js has wroten for robots.ts and sitemap.xml and has same problem |
Has someone managed this error in any ways ? Still encounter the problem on my side |
Next 13.4.7 |
After saving the dynamically generated sitemap.xml from the browser and storing it in the public directory, Google Search Console was able to load it. |
Thanks for your feedback, so at the end of the day this is not dynamic anymore... |
That's right... |
So maybe we could create a cron api route that will write this sitemap.xml file every day or week using fs |
Thanks for the very good ideas! |
This is a bit off topic, but it seems that sitemap.ts is built static. If so, it does not have to be cron. |
I'm not sure that sitemap.xml has to be statically generated. The most important thing is to have an up to date version of your sitemap if you have dynamic pages being created. |
I agree. Sorry if I didn't communicate it well. |
Hi guys! I think it is not an error. Neither on Google nor Vercel. Better saying, I'm not sure it is not kinda an error on Google, because I really think it should have a better message to this situation. You can read further info about this in the link below: I spent 30 minutes searching on the web thinking it was a problem. |
If it is not a bug and just due to time needed for google to process the sitemap, all our sitemaps would have been handled by google after a while. The fact is that even after 1 month i still see "can't fetch". So there might be a bigger problem than just a messy error message + time needed for google to handle it. |
It's been quite a while since I posted the first article, but it still hasn't been registered in the sitemap 😓. The following issue states that after changing from I've also noticed someone on X experiencing the same error as me. However, it seems to be happening with Remix as well, so it might be a problem on Google's side. |
I end up create sitemap.xml file in public and copy pasted created sitemap.xml file through nextjs Now google is able to find my sitemap.xml... but i still want valid information too because i don't want copy paste all the time whenever my sitemap changes |
Anyone using the app directory managed to make it work ? |
After encountering the same issues as you had, in my case with I found this article and applied the What happened? ...so I tried Now, sitemap2.xml is working properly in Google Search Console. My |
I tried that export async function GET(req: NextRequest) {
const sitemap: any = await getSitemap();
const toXml = (urls: any) => `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">
${urls
.map((item: any) => {
return `
<url>
<loc>${item.url}</loc>
<lastmod>${item.lastModified}</lastmod>
<changefreq>${item.changeFrequency}</changefreq>
<priority>${item.priority}</priority>
</url>
`;
})
.join('')}
</urlset>`;
return new Response(toXml(sitemap), {
status: 200,
headers: {
'Cache-control': 'public, s-maxage=86400, stale-while-revalidate',
'content-type': 'application/xml'
}
});
} But google can't find it, I also tried the sitemap2.xml inside the public folder and I got the same error. |
Have the same issue with pages router. My sitemap google can't fetch at least 6 months. I tried with sitemap2.xml inside the public folder and it doesn't work too. Does someone have successfully experience with adding sitemap and pages router? |
I used the |
A week after submitting my Next.js 14/app dir sitemap to google search console and it still showed a "Sitemap could not be read" error. I think, though I'm not sure, that I submitted the sitemap before adding the robots.txt file to the app directory. Even though Googlebot was allowed to view the sitemap, perhaps it had an outdated cache of its permissions. With this hunch, I use the "URL inspection" bar in Google Search Console and inspected my sitemap.xml. It gave me an error. I then inspected my robots.txt url without error. I then reinspected my sitemap and this time there was no error. I went back to the sitemap page, removed the existing reference, and resubmitted my sitemap. This time it instantly fetched my sitemap with a success status. Perhaps this will help others with this issue. |
This did not work for me unfortunately. It still "cannot be read" by Google. |
same error here |
I use React with Vite with the sitemap.xml in the public folder and have the same error: |
Interesting. Although, this is issue is not related to Vercel, but interesting to see that React with Vite is not working as well. |
Ok, after making all those changes to my sitemap, I renamed the file to sitemap.xml and today when I entered the Search Console, I realized that the first submission of my sitemap was correct, and waiting to index This confirms that it is something related to Google or something else, the truth is that I am confused. A friend who uses Flutter also had a problem with the sitemap, but after having fetch problems, he went to check the file in the search console and when it came back, the sitemap had been read correctly and everything was in order. I think Google has to fix that, it's really crazy |
It seems to be an issue of Google Search Console. My solution was, on Google Search Console, submitting the sitemap url with a suffix: FYI, I used |
anyone able to solve it ? |
No. Problem might lie with Google Search Console instead. |
I had the problem for a long time, but finally resolved it. I made the files static and put them inside the public folder. Created a sitemap.xml and a robots.txt I also had a middleware that handles languages on route and had to add
Be sure that robots.txt can be crawled going into settings inside Google search console and then Crawling robots and then open report, click on the three dots and request a recrawl, after that I could add the sitemap.xml and it succeeded |
@Tyerlo where did you put this code and what is this file name? |
I have a middleware.ts file that handle languages on routes. Here's the entire file
|
Adding a trailing slash worked for me as well, thank you! |
This comment has been minimized.
This comment has been minimized.
Hey everyone, seems like the latest canary works, please try and lmk! @anthonyjacquelin Please reopen if you face the same issue, but with a valid repro link (current is just codesandbox.com). Thank you! |
@devjiwonchoi can you let me know exactly which version you tested and can we find out, what has been changed to fully understand this issue? Bing seems to read and process the sitemap, so I would really like to understand, what is different, as we also can't just roll-out canary versions in production. |
@JannikZed You are absolutely right, will investigate the root cause of this issue! |
Hey everyone, I've investigated the issue and want to share the result. TL;DRMany issues mentioned in the thread are mostly caused by middleware matcher, sitemap, or robots misconfiguration. Middleware Matcher ConfigurationIf you have middleware in your project and have yet to exclude the sitemap on the matcher config, it may cause an issue when accessed. You can try excluding the sitemap path from your middleware. For clearer guidance we will update the docs for it. export const config = {
matcher: [
/*
* Match all request paths except for the ones starting with:
* - _next/static (static files)
* - _next/image (image optimization files)
* - favicon.ico, sitemap.xml, robots.txt (metadata files)
*/
'/((?!_next/static|_next/image|favicon.ico|sitemap.xml|robots.txt).*)',
],
}
General Sitemap MisconfigurationThis includes:
Based on the tip from John Mueller, editing the URL may do the trick, which is why the tricks in the comments: adding trailing slash, or query param did work.
Also, if you click the "Couldn't fetch" element, you can view why it wasn't fetched.
https://jiwonchoi.dev/sitemap.xml (checkout the lastMod date) If you still have issues, please open a new issue with a reproduction. Thank you! |
@devjiwonchoi thanks for your investigation. But there is another problem for sure. I curl-ed your website and compared to mine and the only difference is you using HTTP/2 and 1.1 on my side. But pretty sure it does not come from here. Plus he Google console does not give any details on why it failed. Unlike on your screenshots, there is no caret with details on my side. Since you mentioned the middleware, could having the next.js app served by an Express server via the For those still having the issue, can you 👍🏽 if you have an express server and 👎🏽 if you haven't ? |
@c100k so I have to say, that it got solved for me. In my case, Google was also not saying a reason for the could not fetch. But it was obviously, that my domain was too fresh. After some time and adding one high-value backlink it just went green. |
This closed issue has been automatically locked because it had no new activity for 2 weeks. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you. |
Verify canary release
Provide environment information
Operating System: Platform: darwin Arch: x64 Version: Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 Binaries: Node: 18.16.0 npm: 9.5.1 Yarn: 1.22.19 pnpm: 7.29.3 Relevant packages: next: 13.4.6 eslint-config-next: 13.2.4 react: 18.2.0 react-dom: 18.2.0 typescript: 4.9.5
Which area(s) of Next.js are affected? (leave empty if unsure)
App directory (appDir: true)
Link to the code that reproduces this issue or a replay of the bug
https://codesandbox.com
To Reproduce
Describe the Bug
Hello,
I'm using Next 13 with the /app directory and trying to configure the sitemap of my project on Google search console.
I have used the documentation as described there: Documentation
I have a sitemap.ts in the root of my /app directory, but it seems not recognized by GSC, and i know the sitemap is valid: URL and i've checked also using this tool
Expected Behavior
I want the /sitemap.xml to be recognized by Google search console.
Which browser are you using? (if relevant)
No response
How are you deploying your application? (if relevant)
No response
The text was updated successfully, but these errors were encountered: