DOMException [OperationError] in crypto module occurs only during GoogleBot requests

[suleymangezsat]

Link to the code that reproduces this issue

https://github.com/suleymangezsat/nextjs-googlebot-repro

To Reproduce

1. Deploy the application to a production environment
2. Verify GoogleBot can access your site by:

• Using Google Search Console to check crawling status
• Or using the "Fetch as GoogleBot" feature

3. Monitor server-side error logs for at least 24 hours
4. To verify the error:

• Check error logs for the specific DOMException [OperationError]: The operation failed for an operation-specific reason
• Confirm the error only occurs with GoogleBot user agent
• Verify the error appears randomly across different pages

Current vs. Expected behavior

Expected Results:

• GoogleBot should successfully crawl all pages without any crypto-related errors

Actual Results:

• Random DOMException [OperationError] errors appear in logs
• Server returns 500 status code when the error occurs
• Error only occurs during GoogleBot requests
• Other crawlers and regular users do not trigger this error

Provide environment information

Operating System:
  Platform: linux
  Arch: x64
  Version: #9-Ubuntu SMP PREEMPT_DYNAMIC Sat Oct  7 01:35:40 UTC 2023
  Available memory (MB): 3907
  Available CPU cores: 2
Binaries:
  Node: 22.13.1
  npm: 10.9.2
  Yarn: 1.22.21
  pnpm: N/A
Relevant Packages:
  next: 14.2.23 // An outdated version detected (latest is 15.1.6), upgrade is highly recommended!
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.7.3

Which area(s) are affected? (Select all that apply)

Not sure

Which stage(s) are affected? (Select all that apply)

Other (Deployed)

Additional context

We're encountering a specific issue that ONLY occurs with GoogleBot requests: random DOMException [OperationError] errors from Node's crypto module. This error does not occur with any other crawlers (e.g., BingBot, YandexBot), user agents, or ASNs. We have confirmed this by analyzing our server logs across all traffic sources. The issue appears despite not using any explicit encryption/decryption operations in our codebase.

Error stack

2025-01-29T08:30:27: DOMException [OperationError]: The operation failed for an operation-specific reason
    at AESCipherJob.onDone (node:internal/crypto/util:462:19)
    at AESCipherJob.callbackTrampoline (node:internal/async_hooks:130:17) {
  digest: '1379131148',
  [cause]: [Error: Cipher job failed]
}

Characteristics of the Issue

• Only occurs with GoogleBot requests
• Happens randomly across all pages
• No explicit crypto operations in the codebase
• Error appears to be related to Next.js's internal AES cipher operations

Additional Context

• No custom middleware is being used
• Using Cloudflare, but issue appears unrelated as it only affects GoogleBot requests
• Error occurs across all pages randomly, not specific to any particular route

Question
Is this a known issue with Next.js's internal crypto operations when handling GoogleBot requests? What could be causing this GoogleBot-specific crypto error in Next.js's internal operations?