feat: add source-merge subcommand #6
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
|
|
Urist-McGit
reviewed
Sep 8, 2025
Member
Author
Which name do you propose instead for the command? |
fmoessbauer
force-pushed
the
fm/merge-tars
branch
from
96356e5 to
6921220
Compare
Collaborator
|
fmoessbauer
force-pushed
the
fm/merge-tars
branch
3 times, most recently
from
31b1f44 to
88886bf
Compare
Urist-McGit
changed the title
fmoessbauer
force-pushed
the
fm/merge-tars
branch
from
88886bf to
58d1c4d
Compare
Urist-McGit
reviewed
Sep 10, 2025
Urist-McGit
reviewed
Sep 10, 2025
Urist-McGit
reviewed
Sep 10, 2025
On repeated executions some files already have been downloaded. Currently these are already skipped when downloading again, but they show up in the statistics. We now split the statistics into the total data and the data we already have. This gives a better overview what still is missing and needs to be downloaded. We further change the return type to a named tuple to make it easier for downstream users to access the various values. Signed-off-by: Felix Moessbauer <[email protected]>
All debian source packages come with a .dsc file that provides the links to the other files (e.g. .orig.tar or .debian.tar), along with some other information. As a safety measure, we check on downloading if every source package has this file. Signed-off-by: Felix Moessbauer <[email protected]>
Signed-off-by: Felix Moessbauer <[email protected]>
Signed-off-by: Felix Moessbauer <[email protected]>
A source package consist of multiple individual parts (e.g. policy 4.x diff files, .orig and .debian tarballs). To create a single artifact which can be used for license clearing, we need to merge these into a single archive. For that, we introduce the SourceArchiveMerger class which performs the merge based on the .dsc data. Signed-off-by: Felix Moessbauer <[email protected]>
We add the merge subcommand which runs against a download directory and creates combined archive files, which can be used as input to license clearing tools that only support a single archive per component. Signed-off-by: Felix Moessbauer <[email protected]>
We add a unit test that checks the various debian formats, as well as compressing the output tar with all supported compressors. Signed-off-by: Felix Moessbauer <[email protected]>
The mirror might have the same file under various names and paths. Currently, we only return the first instance, but this is not sufficient if other files link to a filename that is not a first instance. We now expand this list and return all instances. Signed-off-by: Felix Moessbauer <[email protected]>
As the snapshot client now returns all file instances, we also download them multiple times. To optimize this, we check if we already have a file with that hash and just link it. Signed-off-by: Felix Moessbauer <[email protected]>
fmoessbauer
force-pushed
the
fm/merge-tars
branch
from
58d1c4d to
3ff44a8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We add the merge subcommand which runs against a download directory and creates combined archive files, which can be used as input to license clearing tools that only support a single archive per component.