Replace Pester submodule by download from the gallery - #4618

Travis failures are due to

The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over).
The job has been terminated

Did pester 4 output to become more verbose?

I see dll not found error:

981Microsoft.PowerShell.Core\Import-Module : The specified module '/home/travis/bu

982ild/PowerShell/PowerShell/src/powershell-unix/bin/Linux/netcoreapp2.0/ubuntu.14

983.04-x64/publish/Modules/Pester/4.0.6\lib\Gherkin.dll' was not loaded because 

984no valid module file was found in any module directory.

985At /home/travis/build/PowerShell/PowerShell/src/powershell-unix/bin/Linux/netco

986reapp2.0/ubuntu.14.04-x64/publish/Modules/Pester/4.0.6/Functions/Gherkin.ps1:2 

987char:1

Also

7786No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

I work to achieve Pester compatibility with Linux, macOS and PSCore on Windows. The current status you can follow pester/Pester#639.

The error with not loading is resolved in my development branch now.

Thank you Wojciech Sciesinski (@it-praktyk) !
I just rebased the PR and I believe now it should be good to go with current Pester at 4.1.1

sergei (@vors), Pester 4.x introduced some breaking changes and our tests need to be updated before we can start relying on 4.x

The migration process is described at Pester wiki Migrating from Pester 3 to Pester 4.

Wojciech Sciesinski (@it-praktyk) thanks, but we're aware of the work involved, just a matter of prioritization and doing it :) Currently planned to be addressed after 6.0.0 release.

Steve Lee (@SteveL-MSFT), I just updated some of them - I'll open PR to sergei (@vors)'s branch.

Can we close this because #5623 was merged?

Ilya (@iSazonov) isn't this PR about using the PS gallery Pester vs psl-pester? We removed the sub module, but i believe this project still relies on psl-pester.

#3002 I don't understand who really works on this sergei (@vors) or James Truher (@JamesWTruher) ?

Mark Kraus (@markekraus),

PS gallery Pester vs psl-pester

psl-pester means Pester what is it currently bundled with PowerShell code?

I think, because of Pester since v. 4.1.0 is supported on PSCore 6.0, bundling of other/older version with PowerShell is not necessary.

OK, I answer myself - without editing my previous comment.

I think that using official Pester is the better idea than using a fork of old code. Of course, some work needs to be done to update and verify tests.

I prepared the pull request that resolves braking changes between Pester 3.x and 4.x vors@039194d .

Wojciech Sciesinski (@it-praktyk) awesome, thank you! I cherry-picked your changes to the branch head instead of merging it thru github, because it required one more rebase on master for build.psm1 and I tried to make a clean history.

Can we close this because #5623 was merged?

As Mark Kraus (@markekraus) said: this one is different and should remove dependency on the old and devirgent fork of pester (aka psl-pester).

#3002 I don't understand who really works on this sergei (@vors) or James Truher (@JamesWTruher) ?

Not sure, I would not say I working actively to move this forward. My hope was that it would be a simple change with a drop-in replacement. Hopefully Wojciech Sciesinski (@it-praktyk) just fixed all the migration problems, we will see when CI finishes the build.

Hopefully Wojciech Sciesinski (@it-praktyk) just fixed all the migration problems, we will see when CI finishes the build.
I've fixed tests for some - most obvious - breaking changes.

Now, I'm in progress of local tests and after that, I'll publish some conclusion.

The first "draft" of them for a build of the current state of PowerShell made and tested on Ubuntu 16.04 in headless mode.

psl-pester
Tests completed in 1538.69s
Passed: 6038 Failed: 14 Skipped: 1211 Pending: 341

Pester 4.2.0-alpha3
Tests completed in 2487.44s
Failed: 38, Pending: 341, Tests Passed: 6007, Skipped: 1211, Inconclusive: 0

Failures are https://travis-ci.org/PowerShell/PowerShell/jobs/326257638#L3384

  Describing Import-Module should be case insensitive
    [-] Import-Module can import a module using different casing using 'TESTMODULE/1.1' and manifest:True 112ms
      Expected: {TESTMODULE}
      But was:  {TestModule TESTMODULE}
      195:         $m.Name | Should Be "TESTMODULE"
      at Invoke-LegacyAssertion, /home/travis/build/PowerShell/PowerShell/src/powershell-unix/bin/Linux/netcoreapp2.0/linux-x64/publish/Modules/Pester/4.1.1/Functions/Assertions/Should.ps1: line 190
      at <ScriptBlock>, /home/travis/build/PowerShell/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Core/Import-Module.Tests.ps1: line 195
    [-] Import-Module can import a module using different casing using 'TESTMODULE' and manifest:True 89ms
      Expected: {TESTMODULE}
      But was:  {TestModule TESTMODULE TESTMODULE}
      195:         $m.Name | Should Be "TESTMODULE"
      at Invoke-LegacyAssertion, /home/travis/build/PowerShell/PowerShell/src/powershell-unix/bin/Linux/netcoreapp2.0/linux-x64/publish/Modules/Pester/4.1.1/Functions/Assertions/Should.ps1: line 190
      at <ScriptBlock>, /home/travis/build/PowerShell/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Core/Import-Module.Tests.ps1: line 195
    [-] Import-Module can import a module using different casing using 'TESTMODULE' and manifest:False 43ms
      Expected: {TESTMODULE}
      But was:  {TestModule TESTMODULE TESTMODULE}
      195:         $m.Name | Should Be "TESTMODULE"
      at Invoke-LegacyAssertion, /home/travis/build/PowerShell/PowerShell/src/powershell-unix/bin/Linux/netcoreapp2.0/linux-x64/publish/Modules/Pester/4.1.1/Functions/Assertions/Should.ps1: line 190
      at <ScriptBlock>, /home/travis/build/PowerShell/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Core/Import-Module.Tests.ps1: line 195
Executing script /home/travis/build/PowerShell/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Core/Job.Tests.ps1
  Describing Job Cmdlet Tests
    Context Simple Jobs

Also on AppVeyor the produced log is sparse, looks like we loose information along the way:

Executing script C:\projects\powershell\test\powershell\engine\Api\TypeInference.Tests.ps1
  Describing Type inference Tests
+
274ms
+
11ms
+
8ms
+
8ms
+
37ms
+
68ms
+
43ms
+

I didn't see it in other projects that use Pester 4, so it's probably caused by some wrappers from build.psm1 that run tests. https://ci.appveyor.com/project/PowerShell/powershell/build/v6.1.0-preview.7466

I didn't analyze it yet deeply but changes my for psl-pester can be related to your last comment.

Summary of my tests on Ubuntu Linux 16.04.

OS: Ubuntu 16.04 x64 - Vagrant bento/ubuntu-16.04 - v201801.02.0 - provider: vmware_workstation

bash

sudo su
curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list | sudo tee /etc/apt/sources.list.d/microsoft.list
apt-get update
apt-get install -y powershell
cd /vagrant/
git clone https://github.com/PowerShell/PowerShell.git
cd PowerShell/
git submodule init
git submodule update
pwsh

pwsh

cd /vagrant/PowerShell
ipmo ./build.psm1                                                                                                                                                                                            
Start-PSBootstrap
Start-PSBuild
Restore-PSPester
Start-PSPester -PassThru | Export-Clixml /vagrant/psl-pester-20180117a.xml
cd ..                                                                                                                                                                                                        
git clone https://github.com/Pester/Pester.git
cd /vagrant/PowerShell/src/powershell-unix/bin/Linux/netcoreapp2.0/linux-x64/publish/Modules/
mv Pester psl-pester
ln -s /vagrant/Pester ./Pester
cd /vagrant/PowerShell/                                                                                                                                                                                 
git remote add vors https://github.com/vors/PowerShell
git fetch vors                                                                                                                                                                                               
git log vors/replace-pester
git cherry-pick c75e2f3ac14e25fc27efa0f9f37ed285e16850b2
Start-PSPester -PassThru | Export-Clixml /vagrant/pester-4.0.2-alpha3-20180117a.xml
(Import-Clixml /vagrant/psl-pester-20180117a.xml).TestResult | Where { $_.Result -eq 'Failed'} | Out-File psl-pester-failed-20180117a.xml                                                                    
(Import-Clixml /vagrant/pester-4.0.2-alpha3-20180117a.xml).TestResult | Where { $_.Result -eq 'Failed'} | Out-File pester-4.0.2-alpha3-failed-20180117a.xml   

Files with results of failed tests were published as a gist.

Under execution of tests using Pester 4.2.0-alpha3 I observed that execution of tests stopped and wait for an interactive input.

Executing script /home/vagrant/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1

<OUTPUT_PARTIALY_OMITTED>

Describing Handling of globbing patterns

Context Handling of Unix [ab] globbing patterns in literal paths

cmdlet AfterEach at command pipeline position 1

Supply values for the following parameters:

Scriptblock:

<ENTER_FROM_A_KEYBOARD>

            [-] Error occurred in Context block 179.58s
        PSInvalidCastException: Cannot convert the "" value of type "System.String" to type "System.Management.Automation.ScriptBlock".
        ArgumentTransformationMetadataException: Cannot convert the "" value of type "System.String" to type "System.Management.Automation.ScriptBlock".
        ParameterBindingArgumentTransformationException: Cannot process argument transformation on parameter 'Scriptblock'. Cannot convert the "" value of type "System.String" to type "System.Management.Automation.ScriptBlock".
        at <ScriptBlock>, /home/vagrant/PowerShell/test/powershell/Modules/Microsoft.PowerShell.Management/FileSystem.Tests.ps1: line 381
        at DescribeImpl, /home/vagrant/PowerShell/src/powershell-unix/bin/Linux/netcoreapp2.0/linux-x64/publish/Modules/Pester/Functions/Describe.ps1: line 161

cmdlet AfterEach at command pipeline position 1

I think the reason is that the curly brace is moved to the next line

Ilya (@iSazonov), the very good suggestion.

Should I prepare a pull request for this small update for the PowerShell/master branch or do you prefer to update this via the vors/replace-pester branch?

Wojciech Sciesinski (@it-praktyk) feel free to take over this work and open a new PR. You can cherry-pick the build.psm1 changes from this one and continue working on the tests changes. Does that sound like a plan?

Wojciech Sciesinski (@it-praktyk) It would be great if you took this job. (It seems sergei (@vors) is very busy in other projects)

There was a big change to build.psm1 which was just merged which will have to be resolved.

Travis Plunk (@TravisEz13), thank you for heads-up. I just analyze what was changed.

sergei (@vors), Ilya (@iSazonov) I'll try to continue work on this subject. I hope to share something valuable soon.

This is fixed by #6064

Awesome! I think Restore-PSPester and Restore-GitModule should be removed from build.psm1.

Restore-PSPester is useful to get pester for testing. We are still not shipping pester.

Restore-GitModule was removed.