Native invocation using `ArgumentList` by JamesWTruher · Pull Request #14692 · PowerShell/PowerShell

James Truher (JamesWTruher) · 2021-02-02T01:23:03Z

This adds a new behavior for native executables which uses ArgumentList as the command arguments for the native executable.

PR Summary

The experimental feature PSNativeCommandArgumentPassing when enabled will use the ArgumentList property of the StartProcessInfo object rather than our current mechanism of reconstructing a string when invoking a native executable.
$PSNativeCommandArgumentPassing is a new automatic variable which allows for runtime selection of behavior. $PSNativeCommandArgumentPassing may be set to either Legacy or Standard, Legacy is the historic behavior. The default when the experimental feature is enabled is the new Standard behavior.

This new behavior is a breaking change from current behavior. This may break scripts and automation which work-around the various issues when invoking native applications. Historically, quotes must be escaped and it is not possible to provide empty arguments to a native application. This PR attempts to address these issues.

new behaviors made available by this change

literal or expandable strings with embedded quotes the quotes are now preserved

PS > $a = 'a" "b'
PS > $PSNativeCommandArgumentPassing = "Legacy"
PS > testexe -echoargs $a 'a" "b' a" "b    
Arg 0 is <a b>
Arg 1 is <a b>
Arg 2 is <a b>
PS > $PSNativeCommandArgumentPassing = "Standard"
PS > testexe -echoargs $a 'a" "b' a" "b    
Arg 0 is <a" "b>
Arg 1 is <a" "b>
Arg 2 is <a b>

empty strings as arguments are now preserved:

PS>  $PSNativeCommandArgumentPassing = "Legacy"
PS> testexe -echoargs '' a b ''           
Arg 0 is <a>
Arg 1 is <b>
PS> $PSNativeCommandArgumentPassing = "Standard"
PS> testexe -echoargs '' a b ''           
Arg 0 is <>
Arg 1 is <a>
Arg 2 is <b>
Arg 3 is <>

the new behavior does not change invocations which look like this:

PS> $PSNativeCommandArgumentPassing = "Legacy"                                            
PS> testexe -echoargs -k com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect          
Arg 0 is <-k>
Arg 1 is <com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect>
PS> $PSNativeCommandArgumentPassing = "Standard"                                  
PS> testexe -echoargs -k com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect
Arg 0 is <-k>
Arg 1 is <com:port=\\devbox\pipe\debug,pipe,resets=0,reconnect>

Additionally, parameter tracing is now provided so Trace-Command will provide useful information for debugging

PS> $PSNativeCommandArgumentPassing = "Legacy"                                           
PS> trace-command -PSHOST -Name ParameterBinding { testexe -echoargs $a 'a" "b' a" "b }
DEBUG: 2021-02-01 17:19:53.6438 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/src/github/forks/jameswtruher/PowerShell-1/test/tools/TestExe/bin/testexe]
DEBUG: 2021-02-01 17:19:53.6440 ParameterBinding Information: 0 :     BIND argument [-echoargs a" "b a" "b "a b"]
DEBUG: 2021-02-01 17:19:53.6522 ParameterBinding Information: 0 : CALLING BeginProcessing
Arg 0 is <a b>
Arg 1 is <a b>
Arg 2 is <a b>
PS> $PSNativeCommandArgumentPassing = "Standard"                                            
PS> trace-command -PSHOST -Name ParameterBinding { testexe -echoargs $a 'a" "b' a" "b }
DEBUG: 2021-02-01 17:20:01.9829 ParameterBinding Information: 0 : BIND NAMED native application line args [/Users/james/src/github/forks/jameswtruher/PowerShell-1/test/tools/TestExe/bin/testexe]
DEBUG: 2021-02-01 17:20:01.9829 ParameterBinding Information: 0 :     BIND cmd line arg [-echoargs] to position [0]
DEBUG: 2021-02-01 17:20:01.9830 ParameterBinding Information: 0 :     BIND cmd line arg [a" "b] to position [1]
DEBUG: 2021-02-01 17:20:01.9830 ParameterBinding Information: 0 :     BIND cmd line arg [a" "b] to position [2]
DEBUG: 2021-02-01 17:20:01.9831 ParameterBinding Information: 0 :     BIND cmd line arg [a b] to position [3]
DEBUG: 2021-02-01 17:20:01.9908 ParameterBinding Information: 0 : CALLING BeginProcessing
Arg 0 is <a" "b>
Arg 1 is <a" "b>
Arg 2 is <a b>

PR Context

Invoking native applications is sometimes very problematic because of how the application is invoked. A new property is available on the ProcessStartInfo object which hopes to improve the experience. This PR takes advantage of this new property.

PR Checklist

The impact on completions may be affected if those completers are calling native applications.

Ilya (iSazonov) · 2021-02-02T18:34:49Z

/cc Michael Klement (@mklement0) Please look the PR.

Michael Klement (mklement0) · 2021-02-03T20:55:38Z

While switching to .ArgumentList is definitely called for and fully solves the problem on Unix-like platforms, on Windows there are also vital accommodations that we need to make for common use cases, notably calling batch files and misexec-style CLIs, as detailed in ~~this comment~~ [update: see Implement accommodations for Windows CLIs, notably batch files and misexec-style programs as part of experimental feature PSNativeCommandArgumentPassing #15143].
- If there's disagreement about these accommodations, let's have a discussion, but none has happened yet (except with other community members), so, yes, perhaps an RFC is necessary.
Both improvements should be part of the same PR, to once and for all solve 100% of all argument-passing problems on Unix, and 99% of all problems on Windows (the remaining problems cannot be fixed generically, and must be addressed by users via --%).
That is, only one (opt-in) breaking change should be made, and it should be controlled by a single preference variable (which is the next best solution if the breaking change is deemed unacceptable).
As for the preference variable:
- First, the name $PSNativeApplicationUsesArgumentList is too technical to begin with (end users who expect argument-passing in their shell to "just work" cannot be expected not know about plumbing details such as .Arguments vs. .ArgumentList, which would make discovery difficult); also, the variable should be [bool]-typed (while assigning 1 works through implicit conversion, we should show $true / $false in the docs).
- Second, with the PR encompassing the Windows-only accommodations too, a broader name is called for. I'm struggling to come up with a good suggestion. $PSFixNativeArgumentPassing?

Steve Lee (SteveL-MSFT) · 2021-02-03T23:38:46Z

For the variable, I prefer $PSNativeCommandArgumentParsing as an enum with values ArgumentList, and String (can't think of something better...).

Michael Klement (mklement0) · 2021-02-04T15:25:23Z

Steve Lee (@SteveL-MSFT):

I prefer $PSNativeCommandArgumentParsing

Not sure if it's just a typo, but to be clear: this isn't about argument parsing, it's about argument passing (which, on Windows only, also involves encoding, as a single string - which .ArgumentList will do for us, but the suggested accommodations will have to be coded manually).

an enum with values ArgumentList, and String (can't think of something better...).

I can see why you would want to avoid the word "fix" in the variable name - even though that's what it comes down to.

Again, what matters to the user is only the following - not technical characterizations based on plumbing users may not even know of:

Is the old behavior in effect, where I need to apply workarounds?
Or is the new behavior in effect, where things work as expected?

Thus, if you do want an enum, I suggest a $PSNativeCommandArgumentPassing variable with the following enumeration values, corresponding to the above:

Legacy
Standard (or similar)

Steve Lee (SteveL-MSFT) · 2021-03-03T23:41:17Z

@PowerShell/powershell-committee reviewed this, we agreed on automatic variable $PSNativeCommandArgumentPassing as an enum with values Legacy and Standard (default). This feature needs to be an ExperimentalFeature.

Ilya (iSazonov) · 2021-03-31T18:36:48Z

Michael Klement (@mklement0) Do you want to review the PR?

…Binder.cs Co-authored-by: Ilya <darpa@yandex.ru>

Travis Plunk (TravisEz13) · 2021-03-31T19:10:48Z

PoshChan-Bot (@PoshChan) Please remind me in 1 hour

PoshChan-Bot (PoshChan) · 2021-03-31T20:11:06Z

Travis Plunk (@TravisEz13), this is the reminder you requested 1 hour ago

Ilya (iSazonov) · 2021-04-01T13:33:31Z

Since this PR introduces a breaking change anyway and is being implemented as an experimental feature, I would like to understand why don't we implement mklement0's proposal in full as an experimental feature and close the long history?
See:

Also I asked previously and still wonder why Michael Klement (@mklement0) is not in a PowerShell Committee. I believe MSFT team would have saved a lot of time and effort in this case.

Travis Plunk (TravisEz13) · 2021-04-01T20:30:35Z

Ilya (@iSazonov) can you open a new issue w.r.t. #14692 (comment) and mark it for committee review?

Travis Plunk (TravisEz13) · 2021-04-01T20:31:20Z

PoshChan-Bot (@PoshChan) Please remind me in 1 Hour

Michael Klement (mklement0) · 2021-04-01T20:36:13Z

The (hopefully) final proposal is now in #14747 (comment)

PoshChan-Bot (PoshChan) · 2021-04-01T21:31:40Z

Travis Plunk (@TravisEz13), this is the reminder you requested 1 Hour ago

Deleted user (ghost) · 2021-04-14T23:59:47Z

🎉v7.2.0-preview.5 has been released which incorporates this pull request.:tada:

Handy links:

Release Notes

@davidBar-On

[7.2.0-preview.5] - 2021-04-14 * Breaking Changes - Make PowerShell Linux deb and RPM packages universal (#15109) - Enforce AppLocker Deny configuration before Execution Policy Bypass configuration (#15035) - Disallow mixed dash and slash in command line parameter prefix (#15142) (Thanks @davidBar-On!) * Experimental Features - `PSNativeCommandArgumentPassing`: Use `ArgumentList` for native executable invocation (breaking change) (#14692) * Engine Updates and Fixes - Add `IArgumentCompleterFactory` for parameterized `ArgumentCompleters` (#12605) (Thanks @powercode!) * General Cmdlet Updates and Fixes - Fix SSH remoting connection never finishing with misconfigured endpoint (#15175) - Respect `TERM` and `NO_COLOR` environment variables for `$PSStyle` rendering (#14969) - Use `ProgressView.Classic` when Virtual Terminal is not supported (#15048) - Fix `Get-Counter` issue with `-Computer` parameter (#15166) (Thanks @krishnayalavarthi!) - Fix redundant iteration while splitting lines (#14851) (Thanks @hez2010!) - Enhance `Remove-Item -Recurse` to work with OneDrive (#14902) (Thanks @iSazonov!) - Change minimum depth to 0 for `ConvertTo-Json` (#14830) (Thanks @kvprasoon!) - Allow `Set-Clipboard` to accept empty string (#14579) - Turn on and off `DECCKM` to modify keyboard mode for Unix native commands to work correctly (#14943) - Fall back to `CopyAndDelete()` when `MoveTo()` fails due to an `IOException` (#15077) * Code Cleanup <details> <summary> We thank the following contributors! @xtqqczze, @iSazonov, @ZhiZe-ZG </summary> <ul> <li>Update .NET to <code>6.0.0-preview.3</code> (#15221)</li> <li>Add space before comma to hosting test to fix error reported by <code>SA1001</code> (#15224)</li> <li>Add <code>SecureStringHelper.FromPlainTextString</code> helper method for efficient secure string creation (#14124) (Thanks @xtqqczze!)</li> <li>Use static lambda keyword (#15154) (Thanks @iSazonov!)</li> <li>Remove unnecessary <code>Array</code> -> <code>List</code> -> <code>Array</code> conversion in <code>ProcessBaseCommand.AllProcesses</code> (#15052) (Thanks @xtqqczze!)</li> <li>Standardize grammar comments in Parser.cs (#15114) (Thanks @ZhiZe-ZG!)</li> <li>Enable <code>SA1001</code>: Commas should be spaced correctly (#14171) (Thanks @xtqqczze!)</li> <li>Refactor <code>MultipleServiceCommandBase.AllServices</code> (#15053) (Thanks @xtqqczze!)</li> </ul> </details> * Tools - Use Unix line endings for shell scripts (#15180) (Thanks @xtqqczze!) * Tests - Add the missing tag in Host Utilities tests (#14983) - Update `copy-props` version in `package.json` (#15124) * Build and Packaging Improvements <details> <summary> We thank the following contributors! @JustinGrote </summary> <ul> <li>Fix <code>yarn-lock</code> for <code>copy-props</code> (#15225)</li> <li>Make package validation regex accept universal Linux packages (#15226)</li> <li>Bump NJsonSchema from 10.4.0 to 10.4.1 (#15190)</li> <li>Make MSI and EXE signing always copy to fix daily build (#15191)</li> <li>Sign internals of EXE package so that it works correctly when signed (#15132)</li> <li>Bump Microsoft.NET.Test.Sdk from 16.9.1 to 16.9.4 (#15141)</li> <li>Update daily release tag format to work with new Microsoft Update work (#15164)</li> <li>Feature: Add Ubuntu 20.04 Support to install-powershell.sh (#15095) (Thanks @JustinGrote!)</li> <li>Treat rebuild branches like release branches (#15099)</li> <li>Update WiX to 3.11.2 (#15097)</li> <li>Bump NJsonSchema from 10.3.11 to 10.4.0 (#15092)</li> <li>Allow patching of preview releases (#15074)</li> <li>Bump Newtonsoft.Json from 12.0.3 to 13.0.1 (#15084, #15085)</li> <li>Update the <code>minSize</code> build package filter to be explicit (#15055)</li> <li>Bump NJsonSchema from 10.3.10 to 10.3.11 (#14965)</li> </ul> </details> * Documentation and Help Content - Merge `7.2.0-preview.4` changes to master (#15056) - Update `README` and `metadata.json` (#15046) - Fix broken links for `dotnet` CLI (#14937) [7.2.0-preview.5]: v7.2.0-preview.4...v7.2.0-preview.5

Co-authored-by: Ilya <darpa@yandex.ru>

@davidBar-On

[7.2.0-preview.5] - 2021-04-14 * Breaking Changes - Make PowerShell Linux deb and RPM packages universal (PowerShell#15109) - Enforce AppLocker Deny configuration before Execution Policy Bypass configuration (PowerShell#15035) - Disallow mixed dash and slash in command line parameter prefix (PowerShell#15142) (Thanks @davidBar-On!) * Experimental Features - `PSNativeCommandArgumentPassing`: Use `ArgumentList` for native executable invocation (breaking change) (PowerShell#14692) * Engine Updates and Fixes - Add `IArgumentCompleterFactory` for parameterized `ArgumentCompleters` (PowerShell#12605) (Thanks @powercode!) * General Cmdlet Updates and Fixes - Fix SSH remoting connection never finishing with misconfigured endpoint (PowerShell#15175) - Respect `TERM` and `NO_COLOR` environment variables for `$PSStyle` rendering (PowerShell#14969) - Use `ProgressView.Classic` when Virtual Terminal is not supported (PowerShell#15048) - Fix `Get-Counter` issue with `-Computer` parameter (PowerShell#15166) (Thanks @krishnayalavarthi!) - Fix redundant iteration while splitting lines (PowerShell#14851) (Thanks @hez2010!) - Enhance `Remove-Item -Recurse` to work with OneDrive (PowerShell#14902) (Thanks @iSazonov!) - Change minimum depth to 0 for `ConvertTo-Json` (PowerShell#14830) (Thanks @kvprasoon!) - Allow `Set-Clipboard` to accept empty string (PowerShell#14579) - Turn on and off `DECCKM` to modify keyboard mode for Unix native commands to work correctly (PowerShell#14943) - Fall back to `CopyAndDelete()` when `MoveTo()` fails due to an `IOException` (PowerShell#15077) * Code Cleanup <details> <summary> We thank the following contributors! @xtqqczze, @iSazonov, @ZhiZe-ZG </summary> <ul> <li>Update .NET to <code>6.0.0-preview.3</code> (PowerShell#15221)</li> <li>Add space before comma to hosting test to fix error reported by <code>SA1001</code> (PowerShell#15224)</li> <li>Add <code>SecureStringHelper.FromPlainTextString</code> helper method for efficient secure string creation (PowerShell#14124) (Thanks @xtqqczze!)</li> <li>Use static lambda keyword (PowerShell#15154) (Thanks @iSazonov!)</li> <li>Remove unnecessary <code>Array</code> -> <code>List</code> -> <code>Array</code> conversion in <code>ProcessBaseCommand.AllProcesses</code> (PowerShell#15052) (Thanks @xtqqczze!)</li> <li>Standardize grammar comments in Parser.cs (PowerShell#15114) (Thanks @ZhiZe-ZG!)</li> <li>Enable <code>SA1001</code>: Commas should be spaced correctly (PowerShell#14171) (Thanks @xtqqczze!)</li> <li>Refactor <code>MultipleServiceCommandBase.AllServices</code> (PowerShell#15053) (Thanks @xtqqczze!)</li> </ul> </details> * Tools - Use Unix line endings for shell scripts (PowerShell#15180) (Thanks @xtqqczze!) * Tests - Add the missing tag in Host Utilities tests (PowerShell#14983) - Update `copy-props` version in `package.json` (PowerShell#15124) * Build and Packaging Improvements <details> <summary> We thank the following contributors! @JustinGrote </summary> <ul> <li>Fix <code>yarn-lock</code> for <code>copy-props</code> (PowerShell#15225)</li> <li>Make package validation regex accept universal Linux packages (PowerShell#15226)</li> <li>Bump NJsonSchema from 10.4.0 to 10.4.1 (PowerShell#15190)</li> <li>Make MSI and EXE signing always copy to fix daily build (PowerShell#15191)</li> <li>Sign internals of EXE package so that it works correctly when signed (PowerShell#15132)</li> <li>Bump Microsoft.NET.Test.Sdk from 16.9.1 to 16.9.4 (PowerShell#15141)</li> <li>Update daily release tag format to work with new Microsoft Update work (PowerShell#15164)</li> <li>Feature: Add Ubuntu 20.04 Support to install-powershell.sh (PowerShell#15095) (Thanks @JustinGrote!)</li> <li>Treat rebuild branches like release branches (PowerShell#15099)</li> <li>Update WiX to 3.11.2 (PowerShell#15097)</li> <li>Bump NJsonSchema from 10.3.11 to 10.4.0 (PowerShell#15092)</li> <li>Allow patching of preview releases (PowerShell#15074)</li> <li>Bump Newtonsoft.Json from 12.0.3 to 13.0.1 (PowerShell#15084, PowerShell#15085)</li> <li>Update the <code>minSize</code> build package filter to be explicit (PowerShell#15055)</li> <li>Bump NJsonSchema from 10.3.10 to 10.3.11 (PowerShell#14965)</li> </ul> </details> * Documentation and Help Content - Merge `7.2.0-preview.4` changes to master (PowerShell#15056) - Update `README` and `metadata.json` (PowerShell#15046) - Fix broken links for `dotnet` CLI (PowerShell#14937) [7.2.0-preview.5]: PowerShell/PowerShell@v7.2.0-preview.4...v7.2.0-preview.5

ghost assigned Travis Plunk (TravisEz13) Feb 2, 2021

James Truher (JamesWTruher) added CL-BreakingChange Indicates that a PR should be marked as a breaking change in the Change Log Review - Committee The PR/Issue needs a review from the PowerShell Committee WG-Engine-ParameterBinder labels Feb 2, 2021

Michael Klement (mklement0) mentioned this pull request Feb 3, 2021

Arguments for external executables aren't correctly escaped #1995

Closed

James Truher (JamesWTruher) mentioned this pull request Feb 9, 2021

Use ArgumentList when invoking native executables #14747

Closed

Michael Klement (mklement0) mentioned this pull request Feb 17, 2021

Cmdlet.ThrowTerminatingError() is not affected by -ErrorAction #14819

Closed

James Truher (JamesWTruher) force-pushed the NativeInvokeWithArglist01 branch from 7f6581d to dcb79f5 Compare February 18, 2021 01:50

Steg17 approved these changes Feb 18, 2021

View reviewed changes

Joey Aiello (joeyaiello) mentioned this pull request Mar 2, 2021

Create RFC####-Improve-generation-of-argument-string-for-executables PowerShell/PowerShell-RFC#90

Closed

Steve Lee (SteveL-MSFT) added Committee-Reviewed PS-Committee has reviewed this and made a decision and removed Review - Committee The PR/Issue needs a review from the PowerShell Committee labels Mar 3, 2021

James Truher (JamesWTruher) force-pushed the NativeInvokeWithArglist01 branch from dcb79f5 to 0985d70 Compare March 4, 2021 19:41

Steve Lee (SteveL-MSFT) requested changes Mar 4, 2021

View reviewed changes

ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 4, 2021

James Truher (JamesWTruher) force-pushed the NativeInvokeWithArglist01 branch from eb105bd to 337877e Compare March 9, 2021 17:38

ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Mar 10, 2021

Dongbo Wang (daxian-dbw) reviewed Mar 10, 2021

View reviewed changes

Comment thread src/System.Management.Automation/engine/InitialSessionState.cs Outdated

James Truher (JamesWTruher) force-pushed the NativeInvokeWithArglist01 branch from da24160 to e4f7558 Compare March 11, 2021 19:21

James Truher (JamesWTruher) changed the title ~~WIP: Native invocation using ArgumentList~~ Native invocation using ArgumentList Mar 17, 2021