Jiahao XU
c916814e7e
Refactor: Extract find_version into mod
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-21 13:40:36 +10:00
Jiahao XU
5d79af545b
Add doc for Vfs
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-21 13:38:24 +10:00
Jiahao XU
784a24577b
Refactor: Rm extract_impl
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:37:50 +10:00
Jiahao XU
db22d7d041
Fix codestyle in async_extracter
...
Use consistent codestyle for specifing trait bounds.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:35:01 +10:00
Jiahao XU
53bf76104b
Revert refactor: Put fetch_crate_cratesio back to drivers
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:27:08 +10:00
Jiahao XU
b14b71135e
Revert refactor: Put find_version back to drivers
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:25:50 +10:00
Jiahao XU
a5879e3d65
Rm unnecessary to_owned call in extract_*
...
It was called before because `spawn_blocking` requires that, but we now
switches to `block_in_place` which no longer needs this.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:24:01 +10:00
Jiahao XU
9de8a4841f
Update doc of mod async_extracter
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:12:48 +10:00
Jiahao XU
b152358175
Rm unused trait bound in extract_impl
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:12:13 +10:00
Jiahao XU
621a641529
Rm unused items in mod helpers
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:10:22 +10:00
Jiahao XU
c15d99c6f0
Run downloader & extracter on the same thread
...
This have the following advantage:
- Remove the mpsc channel, which:
- Remove synchronization required for mpsc.
- Remove the internal buffering of the mpsc channel, which avoid potentially OOM situation.
- Improve data locality since it no longer needs to be sent over thread.
- It uses `block_in_place` to avoid creating an additional blocking
thread.
The disadvantages would be that the downloader can no longer be run in parallel to the extracter.
If the bottleneck is the decompressor, then the downloader should also pause and wait
for the decompressor to consume the data.
But if the bottleneck is the network, then that might be an issue.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:07:46 +10:00
Jiahao XU
aba1ba7b6d
Manually impl From<io::Error> for BinstallError
...
so that if the `io::Error` wraps a `BinstallError`, we would just unwrap
it and return the inner `BinstallError`.
Otherwise, just wrap the `io::Error` in a `BinstallError`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 18:05:15 +10:00
Jiahao XU
1161a60968
Simplify create_tar_decoder: Ret io::Result
...
instead of `Result<Archive, BinstallError>`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 17:56:17 +10:00
Jiahao XU
0480e99460
Impl newtype StreamReadable
...
It wraps a `Stream<Item = Result<Bytes, E>>` and implements `Read` and
`BufRead` on it so that it can be used on sync context.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-18 17:29:50 +10:00
Jiahao XU
39ab334da5
Add a simple optimization to normalize_path
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-15 17:52:49 +10:00
Jiahao XU
30b9a78520
Optimize normalize_path: Avoid copy if possible
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-15 17:45:50 +10:00
Jiahao XU
282805c3ac
Add reference to the src of path_ext
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-15 17:23:00 +10:00
Jiahao XU
9eb1128f9f
Rm unused DummyVisitor in extract_tar_based_stream
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-13 19:32:55 +10:00
Jiahao XU
2091345ce0
Refactor: Mv TarEntriesVisitor to mod async_extracter
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-13 01:14:22 +10:00
Jiahao XU
467f7f6834
Refactor: Call create_tar_decoder directly
...
in `extract_tar_based_stream*`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-13 01:12:21 +10:00
Jiahao XU
9584c8d35e
Refactor: Extract create_tar_decoder
...
from `extract_compressed_from_readable`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-13 01:06:12 +10:00
Jiahao XU
8ef1e56fcc
Take Receiver by value in ReadableRx::new
...
It would remove the lifetime and make reasoning the code much easier.
It would also unblock the next commit I am going to make.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-13 01:05:07 +10:00
Jiahao XU
b4e61161f2
Derive strum_macros::Display on TarBasedFmt
...
sp that it can be printed.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-13 00:53:08 +10:00
Jiahao XU
c9b0c0c59c
Add .DS_Store to .gitignore
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 22:38:34 +10:00
Jiahao XU
94c77c32b4
Make debug! message more consistent
...
in `extract_compressed_from_readable`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 22:38:00 +10:00
Jiahao XU
225cf74cd9
Refactor: Ret impl Stream in create_request
...
Since both `download*` function takes a `impl Stream` and the
`Response::bytes_stream` takes `Response` by value, thus there is no
lifetime issue and we can return `impl Stream` instead of `Response`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 20:01:43 +10:00
Jiahao XU
baf9784b82
Update doc of mod async_extracter
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 19:52:03 +10:00
Jiahao XU
3a1038c80b
Optimize binary size/compilation time
...
by reducing generics monomorphization using `Box<dyn Trait>`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 19:44:59 +10:00
Jiahao XU
bd39ce754f
Fix Vfs::add_path: Use to_string_lossy
...
instead of `to_str` to be compatible with the implementation in
`cargo_toml`:
https://docs.rs/cargo_toml/0.11.5/src/cargo_toml/afs.rs.html#24
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 19:23:03 +10:00
Jiahao XU
b88e384f95
Fix Vfs::add_path: Add insert filename instead of path
...
into the `HashSet<Box<Path>>`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 19:17:06 +10:00
Jiahao XU
b879c15c70
Update doc of PathExt::normalize_path
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:22:20 +10:00
Jiahao XU
989be49cb0
Fix confusing doc of Vfs::add_path
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:13:48 +10:00
Jiahao XU
24b1941c1a
Simplify fetch_crate_cratesio: Rm unused param temp_dir
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:09:17 +10:00
Jiahao XU
e39549f470
Improve debug! logging in download_and_extract
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:08:22 +10:00
Jiahao XU
f25306ff97
Simplify download_tar_based_and_visit: Rm unused param
...
`path`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:07:29 +10:00
Jiahao XU
e376b71cf4
Simplify extract_tar_based_stream_and_visit
...
Rm unused param `path` and the unnecessary
`fs::create_dir_all` since the tar will not be extracted to disk.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:05:35 +10:00
Jiahao XU
17fcac7e63
Refactor: Simplify untar with new enum Op
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 17:03:39 +10:00
Jiahao XU
b2c34137cc
Mark extract_compressed_from_readable & unzip to be pub(super)
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 16:56:41 +10:00
Jiahao XU
f82890cba3
Rm download_and_extract_with_filter
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 16:42:32 +10:00
Jiahao XU
e68eea35fe
Mark type ManifestVisitor as pub(super)
...
Since mod `cretesio` is the only one need to have access to it.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 16:40:51 +10:00
Jiahao XU
5bb5d12949
Optimize fetch_crate_cratesio using ManifestVisitor
...
and `download_tar_based_and_visit`.
By using these two items, we avoid any I/O altogether.
Everything happens in memory, thus there will be no i/o related errors
or cost.
This commit does not regress anything because
`helpers::load_manifest_path` calls `Manifest::from_path_with_metadata`,
which read in the whole `Cargo.toml` file at once.
Thus this commit would not cause any OOM when the the original code
would not.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 16:37:53 +10:00
Jiahao XU
f3d3c488e3
Impl new type `ManifestVisitor
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 16:32:36 +10:00
Jiahao XU
44d43113f4
Forward impl AbstractFilesystem for &Vfs
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 16:08:41 +10:00
Jiahao XU
3c30722a06
Impl new type Vfs which impl AbstractFilesystem
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 01:57:34 +10:00
Jiahao XU
c6687edf48
Fix visbility of PathExt: Mark it as pub
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 01:54:15 +10:00
Jiahao XU
cb2be5a882
Add new trait PathExt & impl for Path
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-12 01:46:10 +10:00
Jiahao XU
0162f5f462
Add doc for TarEntriesVisitor
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-11 22:53:34 +10:00
Jiahao XU
0eb9424f17
Set vis of find_version to pub(super)
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-11 22:47:47 +10:00
Jiahao XU
6c6055da69
Refactor mod drivers: Extract out sub mods
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-11 22:43:45 +10:00
Jiahao XU
f8c8c66f57
Impl new fn helpers::download_tar_based_and_visit
...
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-06-11 20:38:11 +10:00
