Skip to content
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
483248e
code changes for cargo package package identifier
crvreddy Sep 3, 2025
b1b50f9
Updated for identify right dependency
crvreddy Sep 10, 2025
b5ee6b7
Updated as per metadata.json file
crvreddy Sep 15, 2025
8fca48f
Updated code changes
crvreddy Sep 15, 2025
4b0ed7b
updated changes for cargo property
crvreddy Sep 16, 2025
c39f7af
Add Cargo package type support In Artifactory Uploader
Sep 16, 2025
9f2f052
added unit test
malavikakrishnan123 Sep 19, 2025
a299d4b
updated code for identify direct dependencies
crvreddy Sep 19, 2025
fee76d9
Added unit tests and integration tests for Cargo
Sep 19, 2025
7be7bc0
Updated CA_UsageDocument.md
Nikkireddy1 Sep 19, 2025
abbf173
Update CA_UsageDocument.md
Nikkireddy1 Sep 19, 2025
03a1628
Updated appSettings.json with Cargo
Nikkireddy1 Sep 19, 2025
42d59d4
Updated a few more tests for cargo
Sep 21, 2025
85b8fed
Merge branch 'test/PackageIdentifierCargoTests' of https://github.com…
Sep 21, 2025
b5740bf
sonar issues fix
crvreddy Sep 22, 2025
f658abc
Merge remote-tracking branch 'origin/feature/PackageIdentifierForCarg…
Sep 22, 2025
5264b9e
sonar issues fix
crvreddy Sep 22, 2025
1d92600
Merge remote-tracking branch 'origin/feature/PackageIdentifierForCarg…
Sep 22, 2025
2f3d7f7
Pushed UTs for BomHelper and DisplayInfo
Sep 22, 2025
09d8f31
Improved code coverage
Sep 23, 2025
37a7e15
Fixed the failing tests
Sep 23, 2025
eb6f3ef
updated for identify dev dependency
crvreddy Sep 23, 2025
844cd0f
changed nameing conversion
crvreddy Sep 23, 2025
f8e68a3
Added coverage and updated usage doc
Sep 23, 2025
3f0d9e6
Merge remote-tracking branch 'origin/feature/PackageIdentifierForCarg…
Sep 23, 2025
798076d
Merge branch 'development' into feature/PackageIdentifierForCargo
crvreddy Sep 23, 2025
7fc871d
Merge pull request #345 from siemens/test/PackageIdentifierCargoTests
crvreddy Sep 23, 2025
f915dfd
Added unit test
malavikakrishnan123 Sep 23, 2025
e158ecb
Merge branch 'feature/PackageIdentifierForCargo' into feature/Artifac…
Sep 24, 2025
4c0c9a2
unit test cases updated
crvreddy Sep 24, 2025
52ee303
integration test and unit test
malavikakrishnan123 Sep 26, 2025
611df53
removed the duplication code
malavikakrishnan123 Sep 29, 2025
2cec9f6
fixed unit test
malavikakrishnan123 Sep 30, 2025
338651d
fixed integration test
malavikakrishnan123 Sep 30, 2025
fedda6b
Merge branch 'feature/PackageIdentifierForCargo' into feature/Artifac…
sumanthkb44 Oct 5, 2025
a3d17c1
updated change cli
crvreddy Oct 6, 2025
23265d2
removed empty statement
malavikakrishnan123 Oct 6, 2025
9962c65
Merge branch 'feature/ArtifactoryUploader_ForCargo' of https://github…
malavikakrishnan123 Oct 6, 2025
6d6565a
fixed the maintainability issue
malavikakrishnan123 Oct 6, 2025
d0be70d
Merge branch 'feature/PackageIdentifierForCargo' into feature/Artifac…
crvreddy Oct 7, 2025
4a5d79e
Merge pull request #355 from siemens/feature/ArtifactoryUploader_ForC…
crvreddy Oct 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11,437 changes: 11,437 additions & 0 deletions TestFiles/IntegrationTestFiles/SpdxTestFiles/Cargo/Cargo.spdx.sbom.json

Large diffs are not rendered by default.

Large diffs are not rendered by default.

21 changes: 16 additions & 5 deletions doc/UsageDoc/CA_UsageDocument.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@


# Introduction
Welcome to the Continuous Clearing Tool, your automated solution for streamlining the SW360 clearing process. Designed with Project Managers and Developers in mind, this tool efficiently manages third-party components across various platforms, including NPM, NuGet, Maven, Python, Conan, Alpine, and Debian.
Welcome to the Continuous Clearing Tool, your automated solution for streamlining the SW360 clearing process. Designed with Project Managers and Developers in mind, this tool efficiently manages third-party components across various platforms, including NPM, NuGet, Maven, Python, Conan, Cargo, Alpine, and Debian.

## Key Features
- **Automated Scanning and Identification**: The tool automatically scans and identifies third-party components in your projects.
Expand All @@ -77,17 +77,17 @@ Simply integrate the Continuous Clearing Tool into your project workflow to expe

# Continuous Clearing Tool workflow diagram
* Package Identifier
* [NPM/NUGET/MAVEN/PYTHON/CONAN](../usagedocimg/packageIdentifiernpmnuget.PNG)
* [NPM/NUGET/MAVEN/PYTHON/CONAN/CARGO](../usagedocimg/packageIdentifiernpmnuget.PNG)
* [Debian/Alpine](../usagedocimg/packageIdentifierdebianalpine.PNG)
* [BasicSBOM](../usagedocimg/PackageidentifierBasicSBOMflowdiagram.png)

* SW360 Package Creator
* [NPM/NUGET/MAVEN/PYTHON/CONAN](../usagedocimg/packageCreatirnpmnuget.PNG)
* [NPM/NUGET/MAVEN/PYTHON/CONAN/CARGO](../usagedocimg/packageCreatirnpmnuget.PNG)
* [Debian](../usagedocimg/packagecreatordebian.PNG)
* [Alpine](../usagedocimg/ComponentcreaterforAlpine.PNG)

* Artifactory Uploader
* [NPM/NUGET/MAVEN/PYTHON/CONAN](../usagedocimg/artifactoryuploader.PNG)
* [NPM/NUGET/MAVEN/PYTHON/CONAN?CARGO](../usagedocimg/artifactoryuploader.PNG)

# Prerequisite
To ensure a smooth operation of the Continuous Clearing Tool, please follow these prerequisites:
Expand Down Expand Up @@ -196,6 +196,17 @@ Users have the flexibility to generate a basic SBOM even if connections to SW360
* **Project Type :** **Conan**

* Input file repository should contain **conan.lock** file.

* **Project Type :** **Cargo**

* Run the command given below (i.e., To generate a metadata file for your project, run the following command in your project directory (where your Cargo.toml is located)) .

* For creating metadata.json file you can use the format version 1.

**Example**: cargo metadata --format-version 1 > cargo.metadata.json
After successful execution, *.metadata.json file will be created in specified directory .

Resulted cargo.metadata.json file will be having the list of installed packages and the same file will be used as an input to Continuous clearing tool - Package identifier via the input directory parameter. The remaining process is same as other project types.

* **Project Type :** **Debian & Alpine**

Expand Down Expand Up @@ -254,7 +265,7 @@ Description for the settings in appSettings.json file
| S.No | Argument Name | Description | Mandatory | Example |
| ---- | ----------------------------------------- | ------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------ |
| 1 | TimeOut | Timeout in seconds | No | 400 |
| 2 | ProjectType | Type of the project | Yes | `Nuget`, `NPM`, `Poetry`, `Conan`, `Alpine`, `Debian`, `Maven` |
| 2 | ProjectType | Type of the project | Yes | `Nuget`, `NPM`, `Poetry`, `Conan`, `Alpine`, `Debian`, `Maven`, `Cargo` |
| 3 | MultipleProjectType | Whether multiple project types are supported | No | `False` |
| 4 | Telemetry.Enable | Enable telemetry | No | `False` |
| 5 | Telemetry.ApplicationInsightsConnectionString | Application Insights instrumentation key | No | `123-456-789-123-123` |
Expand Down
12 changes: 12 additions & 0 deletions src/LCT.APICommunications.UTest/JfrogAqlApiCommunicationUTest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,18 @@ public void BuildAqlQuery_EmptyComponentType_ReturnsBasicQuery()
Assert.That(query, Is.EqualTo(expectedQuery));
}

[Test]
public void JfrogAqlApiCommunication_GetCargoComponentDataByRepo_ReturnsInvalidOperationException()
{
// Arrange
ArtifactoryCredentials repoCredentials = new ArtifactoryCredentials();
string invalidDomainName = ""; // Invalid domain name
int timeout = 30; // Timeout in seconds
string invalidRepoName = "invalid-npm-repo"; // Invalid repo name

JfrogAqlApiCommunication jfrogApiCommunication = new JfrogAqlApiCommunication(invalidDomainName, repoCredentials, timeout);


}
}
}
1 change: 1 addition & 0 deletions src/LCT.APICommunications/ApiConstant.cs
Original file line number Diff line number Diff line change
Expand Up @@ -76,5 +76,6 @@ public static class ApiConstant
public const string ArtifactoryRepoName = "ArtifactoryRepoName";
public const string JfrogArtifactoryApiSearchAql = $"/api/search/aql";
public static readonly List<int> APIRetryIntervals = [5, 10, 30]; // in seconds
public const string CargoExtension = ".crate";
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ public interface IJfrogAqlApiCommunication
/// <param name="repoName">repoName</param>
/// <returns>HttpResponseMessage</returns>
Task<HttpResponseMessage> GetPypiComponentDataByRepo(string repoName);
/// <summary>
/// Gets the internal component data based on repo name
/// </summary>
/// <param name="repoName">repoName</param>
/// <returns>HttpResponseMessage</returns>
Task<HttpResponseMessage> GetCargoComponentDataByRepo(string repoName);


/// <summary>
Expand Down
55 changes: 20 additions & 35 deletions src/LCT.APICommunications/JfrogAqlApiCommunication.cs
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,7 @@ public async Task<HttpResponseMessage> CheckConnection()
return await httpClient.GetAsync(url);
}

/// <summary>
/// Gets the Internal Component Data By Repo name
/// </summary>
/// <param name="repoName"></param>
/// <returns></returns>
public async Task<HttpResponseMessage> GetInternalComponentDataByRepo(string repoName)
private async Task<HttpResponseMessage> GetComponentDataByRepo(string repoName, string includeFields)
{
HttpClient httpClient = GetHttpClient(ArtifactoryCredentials);
TimeSpan timeOutInSec = TimeSpan.FromSeconds(TimeoutInSec);
Expand All @@ -58,44 +53,36 @@ public async Task<HttpResponseMessage> GetInternalComponentDataByRepo(string rep
StringBuilder query = new();
query.Append("items.find({\"repo\":\"");
query.Append($"{repoName}");
query.Append("\"}).include(\"repo\", \"path\", \"name\", \"actual_sha1\",\"actual_md5\",\"sha256\")");
query.Append("\"}).include(");
query.Append($"{includeFields}");
query.Append(')');

string aqlQueryToBody = query.ToString();
string uri = $"{DomainName}{ApiConstant.JfrogArtifactoryApiSearchAql}";
HttpContent httpContent = new StringContent(aqlQueryToBody);
return await httpClient.PostAsync(uri, httpContent);
}

/// <summary>
/// Gets the Internal Component Data By Repo name
/// </summary>
/// <param name="repoName"></param>
/// <returns></returns>
public async Task<HttpResponseMessage> GetInternalComponentDataByRepo(string repoName)
{
return await GetComponentDataByRepo(repoName, "\"repo\", \"path\", \"name\", \"actual_sha1\",\"actual_md5\",\"sha256\"");
}
public async Task<HttpResponseMessage> GetNpmComponentDataByRepo(string repoName)
{
HttpClient httpClient = GetHttpClient(ArtifactoryCredentials);
TimeSpan timeOutInSec = TimeSpan.FromSeconds(TimeoutInSec);
httpClient.Timeout = timeOutInSec;

StringBuilder query = new();
query.Append("items.find({\"repo\":\"");
query.Append($"{repoName}");
query.Append("\"}).include(\"repo\", \"path\", \"name\",\"@npm.name\",\"@npm.version\", \"actual_sha1\",\"actual_md5\",\"sha256\")");

string aqlQueryToBody = query.ToString();
string uri = $"{DomainName}{ApiConstant.JfrogArtifactoryApiSearchAql}";
HttpContent httpContent = new StringContent(aqlQueryToBody);
return await httpClient.PostAsync(uri, httpContent);
return await GetComponentDataByRepo(repoName, "\"repo\", \"path\", \"name\",\"@npm.name\",\"@npm.version\", \"actual_sha1\",\"actual_md5\",\"sha256\"");
}
public async Task<HttpResponseMessage> GetPypiComponentDataByRepo(string repoName)
{
HttpClient httpClient = GetHttpClient(ArtifactoryCredentials);
TimeSpan timeOutInSec = TimeSpan.FromSeconds(TimeoutInSec);
httpClient.Timeout = timeOutInSec;

StringBuilder query = new();
query.Append("items.find({\"repo\":\"");
query.Append($"{repoName}");
query.Append("\"}).include(\"repo\", \"path\", \"name\",\"@pypi.normalized.name\",\"@pypi.version\", \"actual_sha1\",\"actual_md5\",\"sha256\")");

string aqlQueryToBody = query.ToString();
string uri = $"{DomainName}{ApiConstant.JfrogArtifactoryApiSearchAql}";
HttpContent httpContent = new StringContent(aqlQueryToBody);
return await httpClient.PostAsync(uri, httpContent);
return await GetComponentDataByRepo(repoName, "\"repo\", \"path\", \"name\",\"@pypi.normalized.name\",\"@pypi.version\", \"actual_sha1\",\"actual_md5\",\"sha256\"");
}
public async Task<HttpResponseMessage> GetCargoComponentDataByRepo(string repoName)
{
return await GetComponentDataByRepo(repoName, "\"repo\", \"path\", \"name\",\"@crate.name\",\"@crate.version\", \"actual_sha1\",\"actual_md5\",\"sha256\"");
}

/// <summary>
Expand Down Expand Up @@ -204,8 +191,6 @@ public static string BuildAqlQuery(ComponentsToArtifactory component)
return query.ToString();
}



}

private async Task<HttpResponseMessage> ExecuteSearchAqlAsync(string uri, HttpContent httpContent)
Expand Down
1 change: 1 addition & 0 deletions src/LCT.Common/CommonAppSettings.cs
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ public string ProjectType
public Config Alpine { get; set; }
public Config Poetry { get; set; }
public Config Conan { get; set; }
public Config Cargo { get; set; }
public string Mode { get; set; } = string.Empty;
public bool IsTestMode
{
Expand Down
3 changes: 2 additions & 1 deletion src/LCT.Common/CommonHelper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,8 @@ public static string[] GetRepoList(CommonAppSettings appSettings)
{ "NUGET", () => appSettings.Nuget },
{ "POETRY", () => appSettings.Poetry },
{ "DEBIAN", () => appSettings.Debian },
{ "MAVEN", () => appSettings.Maven }
{ "MAVEN", () => appSettings.Maven },
{ "CARGO", () => appSettings.Cargo }
};
if (projectTypeMappings.TryGetValue(appSettings.ProjectType.ToUpperInvariant(), out var getConfig))
{
Expand Down
1 change: 1 addition & 0 deletions src/LCT.Common/Constants/Dataconstant.cs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ public static class Dataconstant
{"POETRY", "pkg:pypi"},
{"CONAN", "pkg:conan"},
{"ALPINE", "pkg:apk/alpine"},
{"CARGO", "pkg:cargo"},
};

//Identified types
Expand Down
2 changes: 2 additions & 0 deletions src/LCT.Common/Constants/FileConstant.cs
Original file line number Diff line number Diff line change
Expand Up @@ -58,5 +58,7 @@ public static class FileConstant
public static readonly string[] Nuget_DeploymentType_DetectionTags = ["SelfContained"];
public const string backUpKey = "Backup";
public const string SPDXFileExtension = ".spdx.sbom.json";
public const string CargoFileExtension = "metadata.json";

}
}
29 changes: 28 additions & 1 deletion src/LCT.Common/appSettings.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
// --------------------------------------------------------------------------------------------------------------------

{
"TimeOut": 400,
"TimeOut": 400,
"ProjectType": "<Insert ProjectType>",
"MultipleProjectType": false,
"Telemetry": {
Expand Down Expand Up @@ -219,5 +219,32 @@
},
"ReleaseRepo": "<Insert Conan Release RepoName>",
"DevDepRepo": "<Insert Conan DevDep RepoName>"
},
"Cargo": {
"Include": [
"*.metadata.json",
"*.cdx.json",
"*.spdx.sbom.json"
],
"Exclude": [],
"Artifactory": {
"ThirdPartyRepos": [
{
"Name": "<Insert Cargo 3rd party Repo Name>",
"Upload": true
}
],
"InternalRepos": [
"<Insert Cargo Internal Repo Names>"
],
"DevRepos": [
"<Insert Cargo Dev Repo Name>"
],
"RemoteRepos": [
"<Insert Cargo Remote Repo Name>"
]
},
"ReleaseRepo": "<Insert Cargo Release RepoName>",
"DevDepRepo": "<Insert Cargo DevDep RepoName>"
}
}
6 changes: 6 additions & 0 deletions src/LCT.Facade/Interfaces/IJfrogAqlApiCommunicationFacade.cs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,12 @@ public interface IJfrogAqlApiCommunicationFacade
/// <param name="repoName">repoName</param>
/// <returns>HttpResponseMessage</returns>
Task<HttpResponseMessage> GetPypiComponentDataByRepo(string repoName);
/// <summary>
/// Gets the Internal Component Data By Repo Name
/// </summary>
/// <param name="repoName">repoName</param>
/// <returns>HttpResponseMessage</returns>
Task<HttpResponseMessage> GetCargoComponentDataByRepo(string repoName);


/// <summary>
Expand Down
9 changes: 9 additions & 0 deletions src/LCT.Facade/JfrogAqlApiCommunicationFacade.cs
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,15 @@ public async Task<HttpResponseMessage> GetPypiComponentDataByRepo(string repoNam
{
return await m_jfrogAqlApiCommunication.GetPypiComponentDataByRepo(repoName);
}
/// <summary>
/// Gets the Internal Component Data By Repo Name
/// </summary>
/// <param name="repoName">repoName</param>
/// <returns>HttpResponseMessage</returns>
public async Task<HttpResponseMessage> GetCargoComponentDataByRepo(string repoName)
{
return await m_jfrogAqlApiCommunication.GetCargoComponentDataByRepo(repoName);
}

/// <summary>
/// Gets the package information in the repo, via the name or path
Expand Down
Loading
Loading