Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(linux): add logic to parse the upstream package name #88

Closed
wants to merge 2 commits into from
+116,506 −38
Closed

fix(linux): add logic to parse the upstream package name #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
917d590
fix(linux): add logic to parse the upstream package name
JamieMagee Apr 4, 2022
77c6d60
Merge branch 'main' into users/jamagee/linux-package-source-name
JamieMagee Apr 5, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 34 additions & 1 deletion src/Microsoft.ComponentDetection.Detectors/linux/LinuxScanner.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,12 @@ public async Task<IEnumerable<LayerMappedLinuxComponents>> ScanLinuxAsync(string
.DistinctBy(artifact => (artifact.Name, artifact.Version))
.Where(artifact => AllowedArtifactTypes.Contains(artifact.Type))
.Select(artifact =>
(Component: new LinuxComponent(syftOutput.Distro.Name, syftOutput.Distro.Version, artifact.Name, artifact.Version), layerIds: artifact.Locations.Select(location => location.LayerId).Distinct()));
(Component: new LinuxComponent(
syftOutput.Distro.Name,
syftOutput.Distro.Version,
ExtractPackageSourceName(artifact),
artifact.Version),
layerIds: artifact.Locations.Select(location => location.LayerId).Distinct()));

foreach (var (component, layers) in linuxComponentsWithLayers)
{
Expand Down Expand Up @@ -135,5 +140,33 @@ public async Task<IEnumerable<LayerMappedLinuxComponents>> ScanLinuxAsync(string
return null;
}
}

/// <summary>
/// Extracts a package's upstream source name.
/// For example some distributions package openssl as openssl-dev, libssl, openssl1.0, etc.
/// </summary>
private string ExtractPackageSourceName(Package package)
{
var name = package.Name;
switch (package.MetadataType)
{
case "ApkMetadata":
name = package.Metadata.OriginPackage ?? package.Metadata.Package;
break;
case "DpkgMetadata":
name = string.IsNullOrWhiteSpace(package.Metadata.Source)
? package.Metadata.Package
: package.Metadata.Source;
break;
case "RpmdbMetadata":
name = package.Metadata.Name;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can consistently choose either the binary package name or the source package name for all RPM based distros. Mariner publishes vulnerabilities under source package names, as does SUSE. But RHEL (also used for centos vulns) publish vulnerabilities under the binary package names.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but we just need a more granular decision tree here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. (Though per my email, I'm not sure if component governance might have a need for both binary and source package name.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that component-detection doesn't do vulnerability scanning itself, and supports usages other than vulnerability scanning (e.g SBOMs, ad-hoc inventory analysis) I don't think component-detection

  • should discard the binary package name
  • have the API for the Name field be dependent on the current CVE security behaviour of various distros

I've proposed an alternative in #126 which is to keep both binary and source package name.

break;
default:
Logger.LogWarning($"Unknown metadata type: {package.MetadataType}");
break;
}

return name;
}
}
}
15 changes: 10 additions & 5 deletions test/Directory.Build.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,14 @@
<PackageReference Include="FluentAssertions"/>
</ItemGroup>

<PropertyGroup>
<CodeAnalysisRuleSet>$(MSBuildThisFileDirectory)../analyzers.ruleset</CodeAnalysisRuleSet>
<TargetLatestRuntimePatch>true</TargetLatestRuntimePatch>
<NoWarn>NU1608,NU5119,NU5100</NoWarn>
</PropertyGroup>
<PropertyGroup>
<CodeAnalysisRuleSet>$(MSBuildThisFileDirectory)../analyzers.ruleset</CodeAnalysisRuleSet>
<TargetLatestRuntimePatch>true</TargetLatestRuntimePatch>
<NoWarn>NU1608,NU5119,NU5100</NoWarn>
</PropertyGroup>

<ItemGroup>
<None Include="Resources\**"
CopyToOutputDirectory="PreserveNewest"/>
</ItemGroup>
</Project>
70 changes: 41 additions & 29 deletions test/Microsoft.ComponentDetection.Detectors.Tests/LinuxScannerTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FluentAssertions;
using Microsoft.ComponentDetection.Contracts;
using Microsoft.ComponentDetection.Contracts.BcdeModels;
using Microsoft.ComponentDetection.Detectors.Linux;
using Microsoft.ComponentDetection.TestsUtilities;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;

Expand All @@ -16,26 +18,6 @@ namespace Microsoft.ComponentDetection.Detectors.Tests
[TestCategory("Governance/ComponentDetection")]
public class LinuxScannerTests
{
private const string SyftOutput = @"{
""distro"": {
""name"":""test-distribution"",
""version"":""1.0.0""
},
""artifacts"": [
{
""name"":""test"",
""version"":""1.0.0"",
""type"":""deb"",
""locations"": [
{
""path"": ""/var/lib/dpkg/status"",
""layerID"": ""sha256:f95fc50d21d981f1efe1f04109c2c3287c271794f5d9e4fdf9888851a174a971""
}
]
}
]
}";

private LinuxScanner linuxScanner;
private Mock<IDockerService> mockDockerService;

Expand All @@ -46,23 +28,53 @@ public void TestInitialize()
mockDockerService.Setup(service => service.CanPingDockerAsync(It.IsAny<CancellationToken>()))
.ReturnsAsync(true);
mockDockerService.Setup(service => service.TryPullImageAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()));
mockDockerService.Setup(service => service.CreateAndRunContainerAsync(It.IsAny<string>(), It.IsAny<List<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync((SyftOutput, string.Empty));

linuxScanner = new LinuxScanner { DockerService = mockDockerService.Object };
}

[TestMethod]
public async Task TestLinuxScanner()
public async Task Should_ParseAlpine()
{
var syftOutput = await ResourceUtilities.LoadTextAsync(Path.Join("linux", "alpine.syft.json"));
mockDockerService.Setup(service => service.CreateAndRunContainerAsync(It.IsAny<string>(), It.IsAny<List<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync((syftOutput, string.Empty));

var result = await linuxScanner.ScanLinuxAsync("fake_hash", new[] { new DockerLayer { LayerIndex = 0, DiffId = "sha256:a1c01e366b99afb656cec4b16561b6ab299fa471011b4414826407af3a5884f8" } }, 0);

result.Should().HaveCount(1);
var linuxComponents = result.First().LinuxComponents;
linuxComponents.Should().HaveCount(14);
linuxComponents.Should().Contain(linuxComponent => linuxComponent.Name == "openssl");
}

[TestMethod]
public async Task Should_ParseMariner()
{
var result = (await linuxScanner.ScanLinuxAsync("fake_hash", new[] { new DockerLayer { LayerIndex = 0, DiffId = "sha256:f95fc50d21d981f1efe1f04109c2c3287c271794f5d9e4fdf9888851a174a971" } }, 0)).First().LinuxComponents;
var syftOutput = await ResourceUtilities.LoadTextAsync(Path.Join("linux", "mariner.syft.json"));
mockDockerService.Setup(service => service.CreateAndRunContainerAsync(It.IsAny<string>(), It.IsAny<List<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync((syftOutput, string.Empty));

var result = await linuxScanner.ScanLinuxAsync("fake_hash", new[] { new DockerLayer { LayerIndex = 0, DiffId = "sha256:09951eca1ed8c40810aa3e6995a344827e21fbd06b74f35325a59b605d672764" } }, 0);

result.Should().HaveCount(1);
var linuxComponents = result.First().LinuxComponents;
linuxComponents.Should().HaveCount(103);
linuxComponents.Should().Contain(linuxComponent => linuxComponent.Name == "openssl");
}

[TestMethod]
public async Task Should_ParseUbuntu()
{
var syftOutput = await ResourceUtilities.LoadTextAsync(Path.Join("linux", "ubuntu.syft.json"));
mockDockerService.Setup(service => service.CreateAndRunContainerAsync(It.IsAny<string>(), It.IsAny<List<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync((syftOutput, string.Empty));

var result = await linuxScanner.ScanLinuxAsync("fake_hash", new[] { new DockerLayer { LayerIndex = 0, DiffId = "sha256:9f54eef412758095c8079ac465d494a2872e02e90bf1fb5f12a1641c0d1bb78b" } }, 0);

result.Should().HaveCount(1);
var package = result.First();
package.Name.Should().Be("test");
package.Version.Should().Be("1.0.0");
package.Release.Should().Be("1.0.0");
package.Distribution.Should().Be("test-distribution");
var linuxComponents = result.First().LinuxComponents;
linuxComponents.Should().HaveCount(92);
linuxComponents.Should().Contain(linuxComponent => linuxComponent.Name == "gnutls28");
}
}
}
6 changes: 3 additions & 3 deletions ...ft.ComponentDetection.Detectors.Tests/Microsoft.ComponentDetection.Detectors.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="NuGet.Versioning"/>
<PackageReference Include="System.Reactive"/>
<PackageReference Include="packageurl-dotnet"/>
<PackageReference Include="NuGet.Versioning" />
<PackageReference Include="System.Reactive" />
<PackageReference Include="packageurl-dotnet" />
</ItemGroup>

<ItemGroup>
Expand Down
Loading