Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripts using PNP.PowerShell do not work in VSCode #3510

Closed
4 tasks done
Tom-CK opened this issue Aug 17, 2021 · 65 comments
Closed
4 tasks done

Scripts using PNP.PowerShell do not work in VSCode #3510

Tom-CK opened this issue Aug 17, 2021 · 65 comments
Assignees
Labels
Area-Extension Terminal Bug: PowerShell 5.1 Bugs when using PowerShell 5.1. Issue-Bug A bug to squash. Resolution-External Will close automatically.

Comments

@Tom-CK
Copy link

Tom-CK commented Aug 17, 2021

Prerequisites

  • I have written a descriptive issue title.
  • I have searched all issues to ensure it has not already been reported.
  • I have read the troubleshooting guide.
  • I have verified that I am using the latest version of Visual Studio Code and the PowerShell extension.

Summary

I've installed PNP.PowerShell-Module Version 1.7.0.
When working within a normal PowerShell, all CMDLets work fine - for example running "Get-PBPListItem" returns the expected items.

When I work in VSCode with installed PowerShell Extension (v 2021.8.0), ALL PNP-PowerShell CMDLets throw an exception (TargetInvocationException).

This used to work perfect earlier ....

PowerShell Version

Name                           Value
----                           -----
PSVersion                      5.1.19041.1023
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.1023
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Visual Studio Code Version

1.59.0
379476f0e13988d90fab105c5c19e7abc8b1dea8
x64

Extension Version

Steps to Reproduce

  • Install PNP.PowerShell module
  • run a CMDLet like Get-PNListItem in a regular powershell window -> working !
  • run a CMDLet like Get-PNListItem in a vscode powershell window -> TargetInvocationException

Visuals

image

Logs

No response

@Tom-CK Tom-CK added the Issue-Bug A bug to squash. label Aug 17, 2021
@ghost ghost added the Needs: Triage Maintainer attention needed! label Aug 17, 2021
@rjmholt
Copy link
Contributor

rjmholt commented Aug 17, 2021

Can you provide the output of $error[0] | fl * -force after the error occurs?

@Tom-CK
Copy link
Author

Tom-CK commented Aug 17, 2021

 $error[0] | fl * -force



writeErrorStream      : True
PSMessageDetails      : 
Exception             : System.Reflection.TargetInvocationException: Ein Aufrufziel hat einen Ausnahmefehler verursacht. ---> System.MissingMethodException:  
                        Methode nicht gefunden: "Int32 System.Text.Encodings.Web.TextEncoder.FindFirstCharacterToEncodeUtf8(System.ReadOnlySpan`1<Byte>)".    
                           bei System.Text.Json.JsonWriterHelper.NeedsEscaping(ReadOnlySpan`1 value, JavaScriptEncoder encoder)
                           bei System.Text.Json.JsonEncodedText.EncodeHelper(ReadOnlySpan`1 utf8Value, JavaScriptEncoder encoder)
                           bei System.Text.Json.JsonPropertyInfo.DeterminePropertyName()
                           bei System.Text.Json.JsonPropertyInfo.GetPolicies()
                           bei System.Text.Json.JsonClassInfo.CreateProperty(Type declaredPropertyType, Type runtimePropertyType, Type 
                        implementedPropertyType, PropertyInfo propertyInfo, Type parentClassType, JsonConverter converter, JsonSerializerOptions options)     
                           bei System.Text.Json.JsonClassInfo.AddProperty(Type propertyType, PropertyInfo propertyInfo, Type classType, JsonSerializerOptions 
                        options)
                           bei System.Text.Json.JsonClassInfo..ctor(Type type, JsonSerializerOptions options)
                           bei System.Text.Json.JsonSerializerOptions.GetOrAddClass(Type classType)
                           bei System.Text.Json.JsonSerializer.ReadCore(Type returnType, JsonSerializerOptions options, Utf8JsonReader& reader)
                           bei System.Text.Json.JsonSerializer.Deserialize(String json, Type returnType, JsonSerializerOptions options)
                           bei System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
                           bei PnP.Framework.Utilities.TokenHelper.GetMetadataDocument(String realm)
                           bei PnP.Framework.Utilities.TokenHelper.GetStsUrl(String realm)
                           bei PnP.Framework.Utilities.TokenHelper.GetAppOnlyAccessToken(String targetPrincipalName, String targetHost, String targetRealm)    
                           bei PnP.Framework.Utilities.ACSTokenGenerator.GetToken(Uri siteUrl)
                           bei PnP.Framework.AuthenticationManager.<>c__DisplayClass75_0.<GetAccessTokenContext>b__0(Object sender, WebRequestEventArgs args)  
                           bei Microsoft.SharePoint.Client.ClientRuntimeContext.OnExecutingWebRequest(WebRequestEventArgs args)
                           --- Ende der internen Ausnahmestapelüberwachung ---
                           bei System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
                           bei System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
                           bei System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, 
                        CultureInfo culture)
                           bei Microsoft.SharePoint.Client.ClientContextExtensions.<>c__DisplayClass10_0.<Clone>b__0(Object oSender, WebRequestEventArgs       
                        webRequestEventArgs)
                           bei System.EventHandler`1.Invoke(Object sender, TEventArgs e)
                           bei Microsoft.SharePoint.Client.ClientRuntimeContext.OnExecutingWebRequest(WebRequestEventArgs args)
                           bei Microsoft.SharePoint.Client.ClientContext.GetWebRequestExecutor()
                           bei Microsoft.SharePoint.Client.ClientContext.<GetFormDigestInfoPrivateAsync>d__37.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                           bei Microsoft.SharePoint.Client.ClientContext.<EnsureFormDigestAsync>d__36.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                           bei Microsoft.SharePoint.Client.ClientContext.<ExecuteQueryAsync>d__28.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei Microsoft.SharePoint.Client.ClientContextExtensions.<ExecuteQueryImplementation>d__6.MoveNext()
                        --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
                           bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                           bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                           bei Microsoft.SharePoint.Client.ClientContextExtensions.ExecuteQueryRetry(ClientRuntimeContext clientContext, Int32 retryCount,     
                        String userAgent)
                           bei PnP.PowerShell.Commands.PnPWebCmdlet.GetWeb()
                           bei PnP.PowerShell.Commands.Lists.GetListItem.ExecuteCmdlet()
                           bei PnP.PowerShell.Commands.PnPSharePointCmdlet.ProcessRecord()
TargetObject          :
CategoryInfo          : WriteError: (:) [Get-PnPListItem], TargetInvocationException
FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Lists.GetListItem
ErrorDetails          : Ein Aufrufziel hat einen Ausnahmefehler verursacht.
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : bei <ScriptBlock>, <Keine Datei>: Zeile 1
PipelineIterationInfo : {0, 1}

@Tom-CK
Copy link
Author

Tom-CK commented Aug 17, 2021

Maybe a typo in a configuration ?
I did not change anything in config.

@andyleejordan
Copy link
Member

Can you confirm that $PSVersion from the integrated console session itself? I'm wondering if maybe you're in PowerShell 7 there...(I mean, it should still work, but it would help narrow things down).

@rjmholt Look at the accepted answer in https://docs.microsoft.com/en-us/answers/questions/302548/jsonserializer-throws-exception-in-xamarinios.html, very curious.

@Tom-CK
Copy link
Author

Tom-CK commented Aug 18, 2021

This is the result of $PSVersionTable in PowerShell Integrated Console of vscode:

Name Value


PSVersion 5.1.19041.1023
PSEdition Desktop
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 10.0.19041.1023
CLRVersion 4.0.30319.42000
WSManStackVersion 3.0
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1

@Tom-CK
Copy link
Author

Tom-CK commented Aug 18, 2021

Maybe same problem here:
pnp/powershell#727

@Tom-CK
Copy link
Author

Tom-CK commented Aug 18, 2021

I have removed module "PNP.PowerShell" in Version 1.7.0 and installed this module in version 1.5.0.
The error went away.
Then I updated module "PNP.PowerShell" back to Version 1.7.0 - the error was there again.
This can be reproduced.

Conclusion:
The problem must have something to do with module PNP.PowerShell Version 1.7.0.

But strange anyway: The exception is only thrown when running CMDLets in VSCode-PowerShell.

I will post an issue in PNP.PowerShell.

@Tom-CK Tom-CK closed this as completed Aug 18, 2021
@rjmholt
Copy link
Contributor

rjmholt commented Aug 18, 2021

Going to reopen the issue here since it's quite possible this is at least partially an issue in PSES

@rjmholt rjmholt reopened this Aug 18, 2021
@andyleejordan
Copy link
Member

Just for reference, the module sources: https://github.com/pnp/powershell

@andyleejordan
Copy link
Member

@Tom-CK Can you check if this works with PowerShell 7? We have a theory right now...

@andyleejordan
Copy link
Member

Hey @erwinvanhunen, this bug appears between 1.5.0 and 1.7.0, and we think it was the introduction of your own ALC that may have caused this unfortunate interaction. PSES's assembly load logic when running with Windows PowerShell (so full .NET) does not check for the assembly version, it only resolves by name, and we think that's the issue. Since PSES has its own resolve assembly event, the one in pnp/powershell@fd65170#diff-05a628297ec69a1e93f2c84d148c6fac638d6499eda9b04d32a914b628fbe660 isn't being fired. We fear that a lot might break if we change our assembly loader to start checking for name + version. 🧐

@rjmholt
Copy link
Contributor

rjmholt commented Aug 24, 2021

It looks like PnP also does some dependency wrangling and registers their own assembly resolve event, but neither of us checks the required assembly version.

I've had a go implementing that. Give the extension here a try and let me know what happens.

pses.zip

@TobyLeduc
Copy link

TobyLeduc commented Aug 24, 2021

I loaded the zip, disabled the original powershell extension and it did not fix the issue for me. My issue might be different because so far I am only having issues using the "Teams" commands. The SharePoint commands I have used so far have not caused me any issue.

PS> code --list-extensions --show-versions | Select-String powershell
[email protected]
[email protected]

PS> connect-pnponline -Url "https://m365x167780.sharepoint.com/sites/<site>" 
-interactive
PS> Get-PnPTeamsTeam
PS>

The return should have been a list of 20-30 sites, I get nothing in the integrated console.

@ghost ghost added the Needs: Maintainer Attention Maintainer attention needed! label Aug 24, 2021
@andyleejordan
Copy link
Member

My issue might be different because so far I am only having issues using the "Teams" commands. The SharePoint commands I have used so far have not caused me any issue.

Well, that's good to know. 🤔

@rjmholt
Copy link
Contributor

rjmholt commented Aug 24, 2021

The return should have been a list of 20-30 sites, I get nothing in the integrated console.

Ok that's much stranger than getting an exception, and harder to debug unfortunately

@TobyLeduc
Copy link

The return should have been a list of 20-30 sites, I get nothing in the integrated console.

Ok that's much stranger than getting an exception, and harder to debug unfortunately

Here is the error when I try to target a specific site as this actually gives an error.

PS> Connect-PnPOnline -url "https://<tenant>.sharepoint.com/teams/TLTestTeam" -Interactive
PS> Get-PnPTeamsTeam -Identity "TLTestTeam"

Get-PnPTeamsTeam : Team not found
At C:\Users\tleduc\OneDrive - City of Kelowna\SharePoint Online\scripts\test2.ps1:2 char:1
+ Get-PnPTeamsTeam -Identity "TLTestTeam"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (:) [Get-PnPTeamsTeam], PSArgumentException
    + FullyQualifiedErrorId : PnP.PowerShell.Commands.Graph.GetTeamsTeam

PS> $error[0] | fl * -force
writeErrorStream      : True
PSMessageDetails      :
Exception             : System.Management.Automation.PSArgumentException: Team not found
TargetObject          :
CategoryInfo          : ObjectNotFound: (:) [Get-PnPTeamsTeam], PSArgumentException
FullyQualifiedErrorId : PnP.PowerShell.Commands.Graph.GetTeamsTeam
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, C:\Users\tleduc\OneDrive - City of Kelowna\SharePoint Online\scripts\test2.ps1: line 2
PipelineIterationInfo : {0, 1}

@Tom-CK
Copy link
Author

Tom-CK commented Aug 25, 2021

Sorry for my late reply, I was on holidays for some days:

Please remind:

  • the error is thrown only when working VSCode-PowerShell. It it not thrown when using regular PowerShell.
  • after installing package PNP.PowerShell in Version 1.5.0 everything worked fine. Went back to 1.7.0 the error was thrown
    in VSCode PowerShell-Extension only.

@TobyLeduc
Copy link

Sorry for my late reply, I was on holidays for some days:

Please remind:

  • the error is thrown only when working VSCode-PowerShell. It it not thrown when using regular PowerShell.
  • after installing package PNP.PowerShell in Version 1.5.0 everything worked fine. Went back to 1.7.0 the error was thrown
    in VSCode PowerShell-Extension only.

For me the error only exists in the "Powershell Integrated Console" terminal, switching terminals to "Powershell" I can run the commands and they work.
image

Yes 1.5.0 is the last working version for this issue.

@dietrichmeyer
Copy link

Using Windows, PowerShell 7.3.2 and Visual Studio Code 1.74.3 and PowerShell for Visual Studio Code v2022.11.0 and also tried v2023.1.0. PnP.PowerShell 1.12.0.
Running code in Visual Studio Code, sometimes my script works (posted below), but most often I get this error:
Connect-PnPOnline: Could not load file or assembly 'Microsoft.Identity.Client, Version=4.36.1.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae'. Could not
find or load a specific file. (0x80131621).
In above posts it sounds like this cannot be fixed. I'm new to PowerShell 7 and Visual Code. Is there a workaround? I'm starting to get used to Visual Studio Code, but should I go back to the Windows PowerShell ISE?
##code is below
Install-Module -Name Microsoft.PowerShell.SecretManagement, Microsoft.PowerShell.SecretStore
Get-Secret -Name 'PS PnP'
Get-SecretInfo -Name 'PS PnP' | Select-Object -ExpandProperty Metadata
$certificatePassword = Get-Secret -Name 'PS PnP'
$siteurl = "https://XXXX.sharepoint.com/sites/yyyy"
$listname= "User Setup"
Connect-PnPOnline -Url $siteurl
-Tenant XXXX.onmicrosoft.com -ClientId "xxxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb"
-CertificatePath "C:\Data\PS\creds\BBBB PnP.pfx" `
-CertificatePassword $certificatePassword

@andyleejordan
Copy link
Member

I will try to spend the next community day (Monday) looking at this again, though I'm not hopeful. I think it's on PnP.PowerShell to either implement or fix their own Assembly Load Context.

@gautamdsheth
Copy link

@andschwa - thank you taking the time to investigate it. We have launched a new 2.x nightly which has dropped support PS5 and is based completely on .NET 6 , maybe it can help ?

The nightly builds are available via PSGallery and using -Allowprerelease parameter to install the module

@andyleejordan
Copy link
Member

@gautamdsheth that sounds super promising!

@andyleejordan
Copy link
Member

@gautamdsheth have you seen pnp/powershell#1814 (reply in thread)? I'm going to see if we load Microsoft.Identity.Client.dll but that comment sure seems to be making a lot of sense.

@andyleejordan
Copy link
Member

andyleejordan commented Feb 15, 2023

Just dumping some info here, I can confirm that PnP.PowerShell (both 2.x and 1.x) depend on older versions of Microsoft.Identity.Client than requested by ExchangeOnlineManagement and MicrosoftTeams modules. And so far as I've been able to dig, none of PowerShell Editor Services, Secret Management, nor Secret Store depend on this module.

@gautamdsheth is there a way to get PnP.PowerShell to accept 4.0 and up? I think you might have some "must be this version" logic that's causing the 4.29 from Teams and the 4.44 from Exchange to be rejected, but I'm willing to bet that if it did allow it to be used, it might just work fine. I think that's something to do with bindingRedirects right?

Screenshot 2023-02-14 at 3 00 50 PM

@andyleejordan
Copy link
Member

Considering people are having issues with PowerShell 7 and only in the Extension Terminal, where we have an ALC, I'm thinking Rob may have been onto something when he surmised that it's because we don't check assembly versions in PSES. But this is far outside my expertise, I'll chat with @SeeminglyScience.

@gautamdsheth
Copy link

@andschwa - Yes, we are going to bump the MSAL.NET to the latest version , mostly 4.49 as part of the update to v2. Hopefully, will fix things.

@PowerBugi
Copy link

@gautamdsheth have you seen pnp/powershell#1814 (reply in thread)? I'm going to see if we load Microsoft.Identity.Client.dll but that comment sure seems to be making a lot of sense.

Hello @andschwa & @gautamdsheth
I posted a similar message 4 days ago here, I think it went unnoticed because it is not an issue :

pnp/powershell#1814 (comment)

I hope version 2 will be able solve this problem... It's really annoying to have module cohabitation problems because of this DLL.

@JustinGrote
Copy link
Collaborator

JustinGrote commented Feb 16, 2023

Since you now have a .net 6 only version, you really should look into using assemblyLoadContext to abstract your dependencies from the PowerShell shared assembly space to avoid this problem.
https://github.com/rjmholt/ModuleDependencyIsolationExample

EDIT: Sorry this was directed at PnP development, I forgot what repo I was in, oops.

@andyleejordan
Copy link
Member

@JustinGrote thing is, they are using an ALC. Somehow the combination of being loaded in the extension terminal where PSES is using an ALC breaks their ALC. Yes, it should "just be working" because of module dependency isolation...but for reasons I don't know where to begin to find out, it's not.

@andyleejordan
Copy link
Member

Oh, ohhh, oh! @gautamdsheth I was finding PnP.PowerShell's ALC to link here again, which is nearly an exact copy of the example from Rob that @JustinGrote just shared...and with a fresh set of eyes and a cup of coffee in me, I think I spied the issue:

        protected override Assembly Load(AssemblyName assemblyName)
        {
            string assemblyFileName = $"{assemblyName.Name}.dll";

            // Make sure we allow other common PowerShell dependencies to be loaded by PowerShell
            // But specifically exclude Microsoft.ApplicationInsightssince we want to use a different version here
            if (!assemblyName.Name.Equals("Microsoft.ApplicationInsights", StringComparison.OrdinalIgnoreCase))
            {
                string psHomeAsmPath = Path.Join(s_psHome, assemblyFileName);
                if (File.Exists(psHomeAsmPath))
                {
                    // With this API, returning null means nothing is loaded
                    return null;
                }
            }

            // Now try to load the assembly from the dependency directory
            string dependencyAsmPath = Path.Join(_dependencyDirPath, assemblyFileName);
            if (File.Exists(dependencyAsmPath))
            {
                return LoadFromAssemblyPath(dependencyAsmPath);
            }

            return null;
        }

My understanding of this is that Microsoft.ApplicationInsights is the only dependency being loaded from the module's dependency directory path, everything else (such as Microsoft.Identity.Client, the problem) is being loaded from the shared path of PowerShell (or from a previous resolution).

@lucahost
Copy link

I think you are quite close to the issue @andschwa, the problem occurs, if shared dependencies exist in s_psHome which are requested. Then they are returned / loaded from PnP's ALC even if the requested Version does not match.
I'm not so sure, but a Version-Check as in PsesLoadContext could do the trick.
For dependencies not existing in s_psHome the code is partially working, they are loaded from the modules-directory but could have the wrong version as well.

@SydneyhSmith SydneyhSmith added the Resolution-External Will close automatically. label Mar 7, 2023
@ghost ghost closed this as completed Mar 7, 2023
@ghost
Copy link

ghost commented Mar 7, 2023

This issue has been marked as external. It has been automatically closed for housekeeping purposes.

@andyleejordan
Copy link
Member

andyleejordan commented Jun 8, 2023

@lucahost I think you are right and that it's the version check that's missing, presumably causing an the older version of Microsoft.Identity.Client shipped with PnP.PowerShell to be returned as resolved even when the requestor requires a newer version.

It's actually worse than this. The AssemblyLoadContext as implemented in PnP.PowerShell is only isolating Microsoft.ApplicationInsights, and PnP.PowerShell has a lot more dependencies (namely Microsoft.Identity.Client) that conflict and need to be isolated in the same manner.

pnp/powershell#488 (comment)

@all2001
Copy link

all2001 commented Oct 19, 2023

any suggestions or workarounds for this?
This issue seems to be going for a years and I stucked with my development with this
I can do nothing in VS Code which is the one IDE I can use for Powershell 7.x

@andyleejordan
Copy link
Member

You can use a regular pwsh terminal in VS Code, just not the "Extension Terminal." Click the + button to spawn a new terminal in the pane.

@all2001
Copy link

all2001 commented Oct 20, 2023

You can use a regular pwsh terminal in VS Code, just not the "Extension Terminal." Click the + button to spawn a new terminal in the pane.

Thank you, this workaround works for me

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Extension Terminal Bug: PowerShell 5.1 Bugs when using PowerShell 5.1. Issue-Bug A bug to squash. Resolution-External Will close automatically.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

15 participants