Fixed#806
- Add new feature flag `cross-lang-fat-lto` and enable it on release for linux
- Enable `-C linker-plugin-lto` for linux
- Only use `-Z gcc-ld=lld` on non-windows targets when `cargo-zigbuild` is not enabled
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
which would cause the `StreamReadable` to return eof even if the
underlying stream is still open and has not sent EOF yet.
Fixed#777
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Support for custom root cert in `binstalk_downloader::remote::Client`
* Support adding root cert via env `CARGO_HTTP_CAINFO`, `SSL_CERT_{FILE, PATH}`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#779#791
- Retry request on timeout
- Retry for `StatusCode::{REQUEST_TIMEOUT, GATEWAY_TIMEOUT}`
- Add `DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT` for 503/429
if 503/429 does not give us a header or give us an invalid header on
when to retry, we would default to
`DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT`.
- Fix `Client::get_redirected_final_url`: Retry using `GET` on status code 400..405 + 410
- Rename remote_exists => remote_gettable & support fallback to GET
if HEAD fails due to status code 400..405 + 410.
- Improve `Client::get_stream`: Include url & method in the err of the stream returned
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
`futures-util` has too many dependencies and it contains a lot of code
of which we only use a tiny bit of them.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#747
- Add dep compact_str v0.6.1 to binstalk-downloader
- Impl new type `DelayRequest`
- Handle 503/429 with wait duration > `MAX_RETRY_DURATION` by simply taking the min
- Fix `Client::send_request_inner`: Ensure 503/429 get propagated to other requests
even if the current requests reach its maximum retry and decides to
simply return an error.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
- Refactor: Mv fn `utils::asyncify` into mod `utils`
- Improve err msg for task failure in `utils::asyncify`
- Make sure `asyncify` always returns the same annoymous type
that implements `Future` if the `T` is same.
- Rewrite `extract_bin` to avoid `block_in_place`
support cancellation by dropping
- Rm unused dep scopeguard from binstalk-downloader
- Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
- Unbox `extract_future` in `async_extracter::extract_zip`
- Refactor `Download` API: Remove `CancellationFuture` as param
since all futures returned by `Download::and_*` does not call
`block_in_place`, so they can be cancelled by drop instead of using this
cumbersome hack.
- Fix exports from mod `async_tar_visitor`
- Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
- Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
and rm fn `wait_on_cancellation_signal_inner`
- Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
- Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
- Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
- Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
- Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
- Optimize `extract_zip_entry`: Spawn only one blocking task per fn call
by using a mspc queue for the data to be written to the `outfile`.
This would improve efficiency as using `tokio::fs::File` is expensive:
It spawns a new blocking task, which needs one heap allocation and then
pushed to a mpmc queue, and then wait for it to be done on every loop.
This also fix a race condition where the unix permission is set before
the whole file is written, which might be used by attackers.
- Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
To avoid frequent allocation and deallocation.
- Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`
Performs the reserve before sending the buf over mpsc queue to
increase the possibility of reusing the previous allocation.
NOTE: `BytesMut` only reuses the previous allocation if it is the
only one holds the reference to it, which is either on the first
allocation or all the `Bytes` in the mpsc queue has been consumed,
written to the file and dropped.
Since reading from entry would have to wait for external file I/O,
this would give the blocking thread some time to flush `Bytes`
out.
- Disable unused feature fs of dep tokio
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Add new dep async_zip v0.0.9 to binstalk-downloader
with features "gzip", "zstd", "xz", "bzip2", "tokio".
* Refactor: Simplify `async_extracter::extract_*` API
* Refactor: Create newtype wrapper of `ZipError`
so that the zip can be upgraded without affecting API of this crate.
* Enable feature fs of dep tokio in binstalk-downloader
* Rewrite `extract_zip` to use `async_zip::read::stream::ZipFileReader`
which avoids writing the zip file to a temporary file and then read it
back into memory.
* Refactor: Impl new fn `await_on_option` and use it
* Optimize `tokio::select!`: Make them biased and check for cancellation first
to make cancellation takes effect ASAP.
* Rm unused dep zip from binstalk-downloader
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Add new dep tokio-tar v0.3.0 to binstalk-downloader
* Add new dep tokio-util v0.7.4 with feat io to binstalk-downloader
* Add dep async-trait v0.1.59 to binstalk-downloader
* Add new dep async-compression v0.3.15 to binstalk-downloader
with features "gzip", "zstd", "xz", "bzip2", "tokio".
* Rewrite `Download::and_visit_tar` to use `tokio-tar`
to avoid the cumbersome `block_in_place`.
* Apply temporary workaround: Rm use of let-else in mod visitor
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
as there is no need to randomize the first one to be polled.
For `cancel_on_user_sig_term` and `StreamReadable::fill_buf`, the
cancellation future should always to be polled first so that user would
feel responsive.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
- bin and binstalk uses new let-else lang feature
- dep zstd of binstalk-downloader just bumped its msrv to v1.64.0
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
