[BUG] Submit-PnPSearchQuery Timeout and Errors

[whoisdob]

Notice

Many bugs reported are actually related to the PnP Framework which is used behind the scenes. Consider carefully where to report an issue:

1. Are you using Invoke-PnPSiteTemplate or Get-PnPSiteTemplate? The issue is most likely related to the Provisioning Engine. The Provisioning engine is not located in the PowerShell repo. Please report the issue here: https://github.com/pnp/pnpframework/issues.
2. Is the issue related to the cmdlet itself, its parameters, the syntax, or do you suspect it is the code of the cmdlet that is causing the issue? Then please continue reporting the issue in this repo.
3. If you think that the functionality might be related to the underlying libraries that the cmdlet is calling (We realize that might be difficult to determine), please first double check the code of the cmdlet, which can be found here: https://github.com/pnp/powershell/tree/master/src/Commands. If related to the cmdlet, continue reporting the issue here, otherwise report the issue at https://github.com/pnp/pnpframework/issues

Reporting an Issue or Missing Feature

Please confirm what it is that your reporting

We are trying to use the Submit-PnPSearchQuery command to retrieve files modified in a certain timeframe. This is generally a day, but could be a bigger scope of time as well. When using the Submit-PnPSearchQuery command we find that we often receive one of the following errors:

• Timeout.
• Search has encountered a problem that prevents results from being returned. If the issue persists, please contact your administrator

Both errors are transient and if we wrap the call in a retry loop, it generally will work at some point. To add to to the troubleshooting we've done, we've tried to use Modern Auth (AAD App with Sites Full Control All) and Basic Auth (Admin account with similar level of permissions to AAD App). We find the errors happen more when authenticating via Modern Auth. Basic Auth seems to be more reliable but not infallible. In addition, the errors get more prevalent if we try to use the -All flag to get all results at once. Paginating seems to be better but again the errors appear in almost every call. I did look at the command source code here: https://github.com/pnp/powershell/blob/master/src/Commands/Search/SubmitSearchQuery.cs
I did not see if this is a result of the underlying SP API or not.

Expected behavior

Please describe what output you expect to see from the PnP PowerShell Cmdlets

The expectation is that the Submit-PnPSearchQuery command would gracefully handle issues but also work as I described, the calls do eventually work but after many failures.

Actual behavior

Please describe what you see instead. Please provide samples of output or screenshots.

We use Submit-PnPSearchQuery and receive errors multiple times over before it actually returns results.

Steps to reproduce behavior

Please include complete script or code samples in-line or linked from gists

I should note we have a different script that uses MSAL to generate an access token on behalf of the AAD App. However that works and is not the issue.

##############################################################

Clear-Host
Connect-PnPOnline -Url "https://TENANT.sharepoint.com" -AccessToken "long-access-token-generated-from-auth-service"
#Connect-PnPOnline -Url "https://TENANT.sharepoint.com" -UseWebLogin


# Define the number of results to fetch per page and the maximum number of retries in case of errors
$ResultsPerPage = 500
$MaxRetryCount = 5
$StartRow = 0
$totalResultsFetched = 0 # Counter to keep track of the total number of results fetched
$seenDocIds = @() # List to keep track of DocIDs we've seen

# Indicate the start of the job
Write-Host "Starting job..." -ForegroundColor Green

# Continue fetching results until no more results are returned
while ($true) {
    $retryCount = 0
    $success = $false
    $iterationNumber = ($StartRow / $ResultsPerPage) + 1

    # Indicate the start of a new iteration
    Write-Host "Fetching results for iteration: $iterationNumber..." -ForegroundColor Green

    # Retry fetching results if an error occurs, up to the maximum retry count
    while (-not $success -and $retryCount -lt $MaxRetryCount) {
        try {
            # Fetch search results
            $results = Submit-PnPSearchQuery -Query 'LastModifiedTimeForRetention=09/11/2018..09/11/2018 AND isdocument:true -All -SortList @{DocId="Descending"} -ErrorAction Stop
            #$results = Submit-PnPSearchQuery -Query 'LastModifiedTimeForRetention=09/11/2018..09/11/2018 AND isdocument:true -SortList @{DocId="Descending"} -StartRow $StartRow -MaxResults $ResultsPerPage -ErrorAction Stop
            $success = $true
            
            $results = $results.ResultRows
            # Display each result's title and path
            foreach ($result in $results) {
                # Check if the DocID has been seen before
                if ($seenDocIds -contains $result.DocId) {
                    Write-Host ("Iteration: $iterationNumber" + " | DocID: " + $result.DocId + " | Title: " + $result.Title + " | Path: " + $result.Path) -ForegroundColor Magenta
                } else {
                    Write-Host ("Iteration: $iterationNumber" + " | DocID: " + $result.DocId + " | Title: " + $result.Title + " | Path: " + $result.Path)
                    $seenDocIds += $result.DocId # Add the DocID to the seen list
                }
                $totalResultsFetched++ # Increment the total results counter
            }
        } catch {
            # Handle specific errors and retry fetching results
            if ($_ -like "*Timeout*" -or $_ -like "*Search has encountered a problem*") {
                $retryCount++
                Write-Host "Error occurred: $($_.Exception.Message). Retrying... ($retryCount/$MaxRetryCount)" -ForegroundColor Yellow
                Start-Sleep -Seconds 5 # Optional: Wait for 5 seconds before retrying
            } else {
                throw $_ # If it's another exception, rethrow it 
            }
        }
    }

    # If fetching results failed after all retries, indicate the failure and exit the loop
    if (-not $success) {
        Write-Host "Failed to fetch results after $MaxRetryCount retries." -ForegroundColor Red
        break
    }

    # If the number of results returned is less than the specified per page, indicate the end of the job and exit the loop
    if ($results.Count -lt $ResultsPerPage -or $results.Count -gt $ResultsPerPage) {
        Write-Host "No more results to fetch. Finishing job..." -ForegroundColor Green
        break
    }

    # Move to the next set of results
    $StartRow += $ResultsPerPage
}

# Disconnect from the PnPOnline session
Disconnect-PnPOnline
Write-Host "Disconnected PnPOnline..." -ForegroundColor Green

# Display the total number of results fetched
Write-Host "Total results fetched: $totalResultsFetched" -ForegroundColor Green

##############################################################

What is the version of the Cmdlet module you are running?

(you can retrieve this by executing Get-Module -Name "PnP.PowerShell" -ListAvailable)

Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version PreRelease Name PSEdition ExportedCommands

---

Manifest 2.2.0 PnP.PowerShell Desk {Add-PnPAdaptiveScopeProperty, Add-PnPPropertyBagValue, Add-PnPSiteClassification, Copy-PnPFolder…}

Which operating system/environment are you running PnP PowerShell on?

• [ X ] Windows
• Linux
• MacOS
• Azure Cloud Shell
• Azure Functions
• Other : please specify

[KoenZomers]

I've seen this before at tenants that are heavily hit by all kinds of customizations and/or have large amounts of data. Could you try the SharePoint Search Query tool to see if you get similar behavior?