[BUG] Import numpy as np failure

[gmenziesint]

Describe the bug
Import numpy as np failure during backup start, I haven't changed my pipeline for a few months now and it was working fine until this morning.

To Reproduce

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
  none
schedules:
  - cron: '0 */6 * * *'
    displayName: "Every 6 hours"
    branches:
      include:
      - main
    always: true

pool:
  vmImage: ubuntu-latest

jobs:
  - job: backup_document
    displayName: Backup Intune configuration
    pool:
      vmImage: ubuntu-latest
    continueOnError: false
    steps:
    - checkout: self
      persistCredentials: true
    
#Remove existing prod-backup directory
    - task: Bash@3
      displayName: Remove existing prod-backup directory
      inputs:
        targetType: 'inline'
        script: |
          rm -rfv "$(Build.SourcesDirectory)/prod-backup"
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false
        
# Install IntuneCD
# https://github.com/almenscorner/IntuneCD
    - task: Bash@3
      displayName: Install IntuneCD Package V2.3.4
      inputs:
        targetType: 'inline'
        script: |
          pip3 install IntuneCD==2.3.4
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true

# Backup the latest configuration, using the current directory
    - task: Bash@3
      displayName: IntuneCD backup Start
      inputs:
        targetType: 'inline'
        script: |
          mkdir -p "$(Build.SourcesDirectory)/prod-backup"
          IntuneCD-startbackup \
          --mode=1 \
          --path "$(Build.SourcesDirectory)/prod-backup" \
          --exclude ConditionalAccess \
          --ignore-omasettings \
          --append-id \
          --autopilot \
          --audit
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true
      env:
        TENANT_NAME: $(TENANT_NAME)
        CLIENT_ID: $(CLIENT_ID)
        CLIENT_SECRET: $(CLIENT_SECRET)
        
# Set git settings
    - task: Bash@3
      displayName: Configure Git Global Settings
      inputs:
        targetType: 'inline'
        script: |
          git config user.name $(USER_NAME)
          git config user.email $(USER_EMAIL)
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true
# Commit changes and push to Azure DevOps repo
    - task: Bash@3
      displayName: Commit changes to Azure DevOps repo
      name: commitAndsetVariable
      inputs:
        targetType: 'inline'
        script: |
          DATEF=`date +%Y.%m.%d`
          git add --all
          # modified files in folder prod-backup
          var=$(git diff --name-only --staged -- prod-backup)
          echo "##vso[task.setVariable variable=CHANGE_DETECTED;isOutput=true;]$var"
          git commit -m "Intune config backup $DATEF"
          git push origin HEAD:main
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false
        
# Create markdown documentation
    - task: Bash@3
      displayName: Generate markdown document
      inputs:
        targetType: 'inline'
        script: |
          if [ ! -z "$(commitAndsetVariable.CHANGE_DETECTED)" ]
          then
            INTRO="Intune backup and documentation generated at $(Build.Repository.Uri) <img align=\"right\" width=\"96\" height=\"96\" src=\"./logo.png\">"
            IntuneCD-startdocumentation \
                --path="$(Build.SourcesDirectory)/prod-backup" \
                --outpath="$(Build.SourcesDirectory)/prod-as-built.md" \
                --tenantname=$TENANT_NAME \
                --intro="$INTRO" \
                #--split=Y
          else
            echo "no configuration backup change detected in the last commit, documentation will not be created"
          fi
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true
      env:
        TENANT_NAME: $(TENANT_NAME)

# Commit changes and push to repo
    - task: Bash@3
      displayName: Commit changes
      inputs:
        targetType: 'inline'
        script: |
          DATEF=`date +%Y.%m.%d`
          git add --all
          git commit -m "Intune config as-built $DATEF"
          git push origin HEAD:main
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false

  - job: tag
    displayName: Tag repo
    dependsOn: backup_document
    condition: and(succeeded(), ne(dependencies.backup_document.outputs['commitAndsetVariable.CHANGE_DETECTED'], ''))
    pool:
      vmImage: ubuntu-latest
    continueOnError: false
    steps:
    - checkout: self
      persistCredentials: true

# Set git global settings
    - task: Bash@3
      displayName: Configure Git
      inputs:
        targetType: 'inline'
        script: |
          git config user.name $(USER_NAME)
          git config user.email $(USER_EMAIL)
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true

    - task: Bash@3
      displayName: Pull origin
      inputs:
        targetType: 'inline'
        script: |
          git pull origin main
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: false

    - task: PowerShell@2
      displayName: Git tag
      inputs:
        targetType: 'inline'
        script: |
          # change in configuration backup folder detected, create TAG
          $DATEF= Get-Date -format "yyyy-MM-dd_THH-mm"
          git tag -a "v$DATEF" -m "Microsoft Intune configuration snapshot $DATEF"
          git push origin "v$DATEF" *> $null # even status information goes to stderr :(
        failOnStderr: true
        pwsh: false
        workingDirectory: '$(Build.SourcesDirectory)'

  - job: publish
    displayName: Publish as-built artifacts
    dependsOn: tag
    condition: and(succeeded(), ne(dependencies.backup_document.outputs['commitAndsetVariable.CHANGE_DETECTED'], ''))
    pool:
      vmImage: ubuntu-latest
    continueOnError: false
    steps:
    - checkout: self
      persistCredentials: true

Expected behavior
Expected to run correctly.

Screenshots
If applicable, add screenshots to help explain your problem.

  File "/home/vsts/.local/bin/IntuneCD-startbackup", line 8, in <module>
    sys.exit(start())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 319, in start
    run_backup(
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 272, in run_backup
    backup_intune(results, path, output, exclude, token, prefix, append_id, args)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup_intune.py", line 9, in backup_intune
    from .intunecdlib.process_scope_tags import ProcessScopeTags
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/process_scope_tags.py", line 2, in <module>
    from .BaseGraphModule import BaseGraphModule
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/BaseGraphModule.py", line 10, in <module>
    from deepdiff import DeepDiff
  File "/home/vsts/.local/lib/python3.10/site-packages/deepdiff/__init__.py", line 10, in <module>
    from .diff import DeepDiff
  File "/home/vsts/.local/lib/python3.10/site-packages/deepdiff/diff.py", line 30, in <module>
    from deepdiff.distance import DistanceMixin, logarithmic_similarity
  File "/home/vsts/.local/lib/python3.10/site-packages/deepdiff/distance.py", line 1, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

##[error]Bash exited with code '1'.
##[error]Bash wrote one or more lines to the standard error stream.
##[error]Traceback (most recent call last):
  File "/home/vsts/.local/bin/IntuneCD-startbackup", line 8, in <module>
    sys.exit(start())
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 319, in start

##[error]    run_backup(
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/run_backup.py", line 272, in run_backup
    backup_intune(results, path, output, exclude, token, prefix, append_id, args)
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/backup_intune.py", line 9, in backup_intune
    from .intunecdlib.process_scope_tags import ProcessScopeTags
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/process_scope_tags.py", line 2, in <module>
    from .BaseGraphModule import BaseGraphModule
  File "/home/vsts/.local/lib/python3.10/site-packages/IntuneCD/intunecdlib/BaseGraphModule.py", line 10, in <module>
    from deepdiff import DeepDiff
  File "/home/vsts/.local/lib/python3.10/site-packages/deepdiff/__init__.py", line 10, in <module>
    from .diff import DeepDiff
  File "/home/vsts/.local/lib/python3.10/site-packages/deepdiff/diff.py", line 30, in <module>
    from deepdiff.distance import DistanceMixin, logarithmic_similarity
  File "/home/vsts/.local/lib/python3.10/site-packages/deepdiff/distance.py", line 1, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

Run type (please complete the following information):

• Mode: [e.g. 0 or 1]
• Client [e.g. Pipeline, local machine]
• Version [e.g. 1.0.2]

Additional context
Add any other context about the problem here.

[gmenziesint]

I added the below to my pipeline before installing IntuneCD and this seems to have resolved the issue.

# Install numPy
    - task: Bash@3
      displayName: Install numPY
      inputs:
        targetType: 'inline'
        script: |
          pip3 install numpy
        workingDirectory: '$(Build.SourcesDirectory)'
        failOnStderr: true

I initially thought it was MS moving ubuntu-latest to 24.04 but manually setting the VM to be 22.04 didn't make a difference.

[almenscorner]

It seems to be an issue with the DeepDiff package as reported here: seperman/deepdiff#478

Installing an older version of DeepDiff prior to IntuneCD might work as well

[gmenziesint]

It seems to be an issue with the DeepDiff package as reported here: seperman/deepdiff#478

Installing an older version of DeepDiff prior to IntuneCD might work as well

Thanks for that, will keep an eye on that issue :)

[almenscorner]

Next update I think I will add version lock on dependencies so this does not happen

[almenscorner]

A bugfix has been pushed to the dependency package and pipeline is now completing successfully without any additional steps