There were some pretty big changes added with v1.0.3 of blockframe.
Removed mmap conditions and thresholds to enable default mmap for file read in tier 2 commits. Previously mmap would be selected if a file was greater than 10mb, use_mmap would be checked within the segment loop constantly, even though use_mmap was a constant. Presence of conditional mmap meant doing variable sized buffer fills which would eventually lead to short reads and corruption of segments due to filling EOF segment with dummy data as the buffer would be statically sized for reusability.
Added padding to generate_parity_segmented. The Reed-Solomon erasure coding is ran on SIMD instructions, meaning the data which is being processed on the specific lanes must be 64 bytes. Prior if 65 bytes were passed, the 1 left over bit wouldn't be filled into the SIMD vector which created undefined behaviour. Added next multiple of 64 padding for exception chunks.
Changed commit boundaries for tier 1 from 10mb to 25mb.
Changed sha256 function name to blake3_hash_bytes.
DeusCodex•3w ago
Removed mmap conditions and thresholds to enable default mmap for file read in tier 2 commits. Previously mmap would be selected if a file was greater than 10mb, use_mmap would be checked within the segment loop constantly, even though use_mmap was a constant. Presence of conditional mmap meant doing variable sized buffer fills which would eventually lead to short reads and corruption of segments due to filling EOF segment with dummy data as the buffer would be statically sized for reusability.
Added padding to generate_parity_segmented. The Reed-Solomon erasure coding is ran on SIMD instructions, meaning the data which is being processed on the specific lanes must be 64 bytes. Prior if 65 bytes were passed, the 1 left over bit wouldn't be filled into the SIMD vector which created undefined behaviour. Added next multiple of 64 padding for exception chunks.
Changed commit boundaries for tier 1 from 10mb to 25mb. Changed sha256 function name to blake3_hash_bytes.