jakeISAC/bifrost
jakeISAC/bifrost
1
0
Fork 0
Code Issues Pull requests Projects Releases 11 Packages Wiki Activity Actions

testing-features #72

Merged
jakeISAC merged 94 commits from testing-features into dev 2026-05-31 19:55:43 +02:00
Conversation 0 Commits 94 Files changed 31 +1188 -448
jakeISAC commented 2026-05-31 19:55:30 +02:00
Owner
Copy link
No description provided. 
Reviewed-on: jakeISAC/heimdal#27
Reviewed-on: jakeISAC/heimdal#28
Reviewed-on: jakeISAC/heimdal#29
Reviewed-on: jakeISAC/heimdal#30
Reviewed-on: jakeISAC/heimdal#31
Reviewed-on: jakeISAC/heimdal#32
Reviewed-on: jakeISAC/heimdal#33
Reviewed-on: jakeISAC/heimdal#34
Reviewed-on: jakeISAC/heimdal#35
Reviewed-on: jakeISAC/heimdal#36
Reviewed-on: jakeISAC/heimdal#37
Reviewed-on: jakeISAC/heimdal#38
Reviewed-on: jakeISAC/heimdal#39
Reviewed-on: jakeISAC/heimdal#40
Reviewed-on: jakeISAC/heimdal#41
Reviewed-on: jakeISAC/heimdal#42
Reviewed-on: jakeISAC/heimdal#43
Reviewed-on: jakeISAC/heimdal#44
Reviewed-on: jakeISAC/heimdal#45
Reviewed-on: jakeISAC/heimdal#46
Reviewed-on: #47
Reviewed-on: #48
Reviewed-on: #49
Reviewed-on: #50
Reviewed-on: #51
Reviewed-on: #52
Reviewed-on: #53
Reviewed-on: #54
Reviewed-on: #55
Reviewed-on: #57
Reviewed-on: #59
Reviewed-on: #61
Reviewed-on: #62
Reviewed-on: #64
Reviewed-on: #65
Reviewed-on: #67
Reviewed-on: #68
Reviewed-on: #69
- Replace asynchronous k-means and scheduling functions with synchronous implementations using Rayon.
- Optimize data processing using `par_iter` and `par_bridge` for parallel execution.
- Remove Tokio dependency from k-means and scheduling logic.
- Update tests to reflect synchronous function changes.
- Bump version to 1.2.10 in Cargo.toml.
Reviewed-on: #70
- Implement k-means++ initialization for improved centroid selection.
- Replace natural logarithm with base-2 logarithm in `create_log_space`.
- Refine distance calculations for better precision using `OrderedFloat`.
- Remove `NOISE_RANGE` constant and redundant noise injection mechanism.
- Fix minor sorting logic in `dir_preprocessor.rs`.
- Adjust default schedule parameter in test to reflect updated logic.
- Bump version to 1.2.11 in Cargo.toml.
- Introduce `--threads` argument to control maximum concurrent tasks.
- Limit task spawning in `copy_multithreaded` to reduce memory usage.
- Add semaphore for concurrent group control to enhance stability.
- Updated argument names to better reflect their functionality.
- Adjusted related scheduler and copying logic accordingly.
- Improved comments and logs for enhanced code readability and maintainability.
- Implement `process_file_split` with support for `Smart` and `Basic` split strategies.
- Introduce `FileReconstructionData` struct for handling file chunk metadata.
- Add initial stub for `basic_split` function.
- Update `mod.rs` to include the new `file_chunking` module.
- Remove obsolete TODO comment from `path_preprocessor.rs`.
- Extend CLI arguments to include `--preserve-mode`.
- Update `copy_multithreaded` to handle permissions and ownership.
- Implement basic permission and ownership handling logic in `dir_preprocessor.rs`.
- Add MacOS build target and corresponding linker configuration in Makefile and `.cargo/config.toml`.
- Change FileData getters to return references to avoid cloning
- Remove the file_chunking module and related code
- Add a permissions module with skeleton set_dir_permissions and
  set_dir_ownership
- Update dir_preprocessor to use borrowed PathBuf keys and depth-based
  path graph
- Improve archive_mode logging to display path strings when available
- Implement `set_dir_permissions` and `set_dir_ownership` with depth-based ordering.
- Extend `--preserve-mode` support for sequential permission/ownership updates.
- Add `BTreeMap` for optimized metadata storage.
- Update tests and refactor related modules for improved consistency.
- Introduced `--preserve-mode` support for both sequential and multithreaded file copying.
- Added Unix-specific `chown` implementation in `copy_sequential` and `copy_multithreaded`.
- Integrated `jemallocator` for Linux builds and added debug allocator logging.
- Updated Makefile to include test-specific build targets.
- Enhanced platform-specific configurations in `.cargo/config.toml` for Linux and macOS.
- Refactored module imports with conditional compilation for Unix targets.
- Fixed typos and improved comments for better clarity and maintainability.
- Replace `HashMap` with `BTreeMap` for ordered path graph storage.
- Add early exit for empty relative paths in directory analysis.
- Fix edge case for empty file paths to prevent incorrect handling.
- Improve comments for better readability and debugging insights.
- Enhance comment clarity for K-Means++ centroid initialization algorithm.
- Add reference to external implementation for better context.
- Adjust comments in archive_mode to clarify lazy evaluations and new file insertion logic.
- Introduce `multithreaded_dir_analysis` with Rayon thread pool for concurrent processing.
- Add `generate_file_data` macro to simplify and standardize file metadata generation.
- Optimize single-file and directory handling logic in `analysis`.
- Update test to reflect new directory analysis behavior.
- Move debug timing logic closer to `multithreaded_dir_analysis`.
- Remove redundant timing logic in `root_contents` processing.
- Extend `analysis` to support mode selection and provide analysis duration.
- Add `analysis_mode` and `multi_threaded_analysis` options to CLI for flexible analysis operations.
- Update tests and logging to accommodate new analysis behavior.
- Add elapsed time display for analysis in standard output.
- Add `num_walkers` parameter to `analysis` for improved flexibility in multithreaded directory analysis.
- Update CLI to include `number_of_walkers` option.
- Modify tests and associated logic to support the new parameter.
- Replace sequential directory processing with parallelized chunk processing using Rayon.
- Add `process` helper function for better modularity and reuse.
- Improve edge case handling for empty directories with detailed logging.
- Minor formatting improvements in `analysis` function signature and related logic.
- Update `destination` argument logic to depend on either `completions` or `analysis_mode`.
- Minor whitespace cleanup.
- Reorganize CLI arguments for improved clarity and functionality.
- Refactor source and destination validation to accommodate `analysis_mode`.
- Enhance analysis output with walker count and formatted duration display.
- Reintegrate scheduler validation and destination preprocessing after analysis logic.
- Replace static walker count with `LazyCell` for dynamic CPU-based initialization.
- Add ARM64 Windows release build target.
- Update `build-all` target to include ARM64 Windows.
- Extend hash calculation in `Makefile` for ARM64 Linux builds.
- Replace `cargo build` with `cargo xwin build` for `build-release-windows-arm64`.
- Include hash calculation for ARM64 Windows release binaries.
- Add `human_readable_number` macro for formatted output.
- Refactor source and destination validation logic.
- Improve analysis output with resolved paths, human-readable counts, and walker info.
- Simplify scheduler and destination preprocessing.
- Remove temporary fixes and clean up unused code.
- Trim and standardize scheduler value parsing.
- Update dependencies and version to `1.2.17`.
Simplified permission setting function by removing an unecessarry
creation of BTreeMap. Instead we simply reverse an iterator on an
already sorted extracted_paths --> shalow to deep --> deep to shalow.

Added some extract comments and docs.
Schedulers were redesigned with Rayon in mind a while a ago and keeping
the function singature async did not make much sense.

Made some additonal optimazations: removed the human_readbale_macro,
since I have to use indicatif anyway.
- Deleted `disk_speed_probe.rs` and its reference in `mod.rs`.
- Refined k-means and CFS scheduler docs for clarity, added detailed error and result descriptions.
- Minor adjustments to file operation and directory analysis comments for consistency.
- added some hints in form of `[cold]` and `[inline]` to help compiler
  optimze the code
- formatted some code and docs to make it more alligned
- added some new comments and docks
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jakeISAC/bifrost!72
No description provided.