Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move back to nx archivemanager #659

Merged
merged 6 commits into from
Sep 28, 2023
Merged

Conversation

halgari
Copy link
Collaborator

@halgari halgari commented Sep 21, 2023

Currenty this still fails due to a ZStd decompression error in the SDV tests. The unit tests for the archive manager work fine. But something with how SDV is doing things is causing this to fail.

@halgari halgari linked an issue Sep 21, 2023 that may be closed by this pull request
@halgari
Copy link
Collaborator Author

halgari commented Sep 21, 2023

@Sewer56 mind taking a look at NxArchiveManager.ChunkedArchiveStream.*? I'm sure I'm missing something. If you run the SDV tests you'll see "no installer found for X". But if you then run with debugging enabled you'll see a ZStd error being thrown by the NX Compression.Decompress method. I'm sure I'm doing something wrong here, but I can't figure out what it is.

@codecov
Copy link

codecov bot commented Sep 21, 2023

Codecov Report

Merging #659 (a2a1b2f) into main (a74a37b) will increase coverage by 0.26%.
The diff coverage is 98.14%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #659      +/-   ##
==========================================
+ Coverage   64.29%   64.55%   +0.26%     
==========================================
  Files         633      633              
  Lines       18054    18145      +91     
  Branches     1172     1176       +4     
==========================================
+ Hits        11607    11714     +107     
+ Misses       6150     6131      -19     
- Partials      297      300       +3     
Flag Coverage Δ
Linux 63.89% <98.14%> (+0.32%) ⬆️
Windows 63.74% <98.14%> (+0.22%) ⬆️
clean_environment_tests 64.54% <98.14%> (+0.26%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/NexusMods.DataModel/Services.cs 96.72% <100.00%> (ø)
src/NexusMods.DataModel/NxArchiveManager.cs 94.19% <98.13%> (+94.19%) ⬆️

... and 3 files with indirect coverage changes

@Sewer56
Copy link
Member

Sewer56 commented Sep 21, 2023

Sure, I got some time to spare. I'll have a look very soon.

The advanced installer suggestion stuff I'm finishing up is dependent on AL's current work (their work changes API I have to interface with); so it'd be more efficient for me to fully wait until that's finalised anyway, as I've been working off of an incomplete PR.

@Al12rs
Copy link
Contributor

Al12rs commented Sep 25, 2023

Is this ready for review?

@Sewer56
Copy link
Member

Sewer56 commented Sep 25, 2023

No, there's something weird going on with this and I need to investigate.

@github-actions
Copy link
Contributor

This PR conflicts with main. You need to rebase the PR before it can be merged.

Copy link
Member

@Sewer56 Sewer56 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address the changes above then it's good for merging.
I'll detail the performance gimmick I mentioned earlier in a new issue. We can adjust the text of that issue as needed.


private void MakeBlocks(int chunkSize)
{
// Slow due to copy to stack, but not that big a deal here.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated code for this shouldn't incur a stack copy; it would use the heap allocated value stored inside of ChunkedArchiveStream directly (basically by pointer).

If FileEntry was a readonly field and JIT couldn't determine if called method is pure, a stack copy could occur, but that is not the case.

settings.MaxNumThreads = 1;
var settings = new UnpackerSettings
{
MaxNumThreads = 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a leftover from developing/testing?
The other overload that's actually used by the LayoutSynchronizer runs on all threads.


unpacker.ExtractFiles(infos.Select(o => (IOutputDataProvider)o.Item2).ToArray(), settings);
foreach (var info in infos)
foreach (var (hash, output, size) in infos)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that IOutputDataProvider should be disposed by the user (via IDisposable).

In this case, it is not necessary because the output type is OutputArrayProvider, and there's nothing to dispose for that type, but this should probably be noted as a comment just in case.

src/NexusMods.DataModel/NxArchiveManager.cs Show resolved Hide resolved
DecompressSize = _entry.DecompressedBlockOffset + (int)Math.Min(remainingDecompSize, (ulong)chunkSize)
};

_blocks.Add(block);
Copy link
Member

@Sewer56 Sewer56 Sep 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be preallocated as array rather than using list; since the number of blocks (chunkCount) is known ahead of time. Could make this method return array that directly assigns to _blocks field.

# Conflicts:
#	Directory.Packages.props
@github-actions
Copy link
Contributor

This PR doesn't conflict with main anymore. It can be merged after all status checks have passed and it has been reviewed.

@halgari halgari merged commit 8d696b8 into main Sep 28, 2023
6 checks passed
@halgari halgari deleted the 640-move-back-to-nx-archivemanager branch September 28, 2023 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
3 participants