It also uses `max_stable_version` in the json downloaded from https://crates.io/api/v1/crates/$name if possible, which is equivalent to the version shown on https://crates.io/crates/$name .
- Add new feat `json` to `binstalk-downloader`
- Impl new async fn `Response::json`
- use `Response::json` in `GhApiClient` impl
- Mark all err types in binstalk-downloader as `non_exhaustive`
- Ret `remote::Error` in `remote::Certificate::{from_pem, from_der}` instead of `ReqwestError`.
- Refactor `BinstallError`: Merge variant `Unzip`, `Reqwest` & `Http`
into one variant `Download`.
- Manually download and parse json from httos://crates.io/api/v1
- Remove unused deps `crates_io_api`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#776
- Add new feature gh-api-client to binstalk-downloader
- Impl new type `binstalk_downloader::remote::{RequestBuilder, Response}`
- Impl `binstalk_downloader::gh_api_client::GhApiClient`, exposed if `cfg(feature = "gh-api-client")` and add e2e and unit tests for it
- Use `binstalk_downloader::gh_api_client::GhApiClient` to speedup `cargo-binstall`
- Add new option `--github-token` to supply the token for GitHub restful API, or read from env variable `GITHUB_TOKEN` if not present.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Refactor: Extract new mod `ops::resolve::resolution`
* Refactor: Extract new type `ResolutionFetch`
* Refactor: Extract new type `ResolutionSource`
* Improve `Resolution::print`: Provides more details on which crate
is the resolution for.
* Make `Resolution::print` a pub fn
* Refactor: Extract new fn `ResolutionFetch::install`
* Refactor: Extract new async fn `ResolutionSource::install`
* Optimize `ResolutionSource::install` avoiding `OsStr::to_string_lossy`.
* Add new dep command-group v2.0.0 to binstalk with feat with-tokio
* Fix `ResolutionSource::install`: Use `AsyncCommandGroup::group_spawn`
instead of `tokio::process::Command::spawn` to ensure all the processes
spanwed by the `cargo` process can be terminated with one signal sent to
the `cargo` process.
* Fix printing resolution: Make sure they are printed without interleaving
* Refactor `entry::install_crates`
* Improve dry-run output for `ResolutionSource::install`
* Refactor: Extract new fn `ResolutionFetch::print`
* Refactor: Extract new fn `ResolutionSource::print`
* Optimize `Resolution`: Box unit variant `Fetch`
* Improve formatting of `tokio::process::Command`
Prints out sth like `cargo install ...` instead of the `Debug::fmt`.
* Improve dry-run output
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
- Refactor: Mv fn `utils::asyncify` into mod `utils`
- Improve err msg for task failure in `utils::asyncify`
- Make sure `asyncify` always returns the same annoymous type
that implements `Future` if the `T` is same.
- Rewrite `extract_bin` to avoid `block_in_place`
support cancellation by dropping
- Rm unused dep scopeguard from binstalk-downloader
- Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
- Unbox `extract_future` in `async_extracter::extract_zip`
- Refactor `Download` API: Remove `CancellationFuture` as param
since all futures returned by `Download::and_*` does not call
`block_in_place`, so they can be cancelled by drop instead of using this
cumbersome hack.
- Fix exports from mod `async_tar_visitor`
- Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
- Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
and rm fn `wait_on_cancellation_signal_inner`
- Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
- Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
- Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
- Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
- Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
- Optimize `extract_zip_entry`: Spawn only one blocking task per fn call
by using a mspc queue for the data to be written to the `outfile`.
This would improve efficiency as using `tokio::fs::File` is expensive:
It spawns a new blocking task, which needs one heap allocation and then
pushed to a mpmc queue, and then wait for it to be done on every loop.
This also fix a race condition where the unix permission is set before
the whole file is written, which might be used by attackers.
- Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
To avoid frequent allocation and deallocation.
- Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`
Performs the reserve before sending the buf over mpsc queue to
increase the possibility of reusing the previous allocation.
NOTE: `BytesMut` only reuses the previous allocation if it is the
only one holds the reference to it, which is either on the first
allocation or all the `Bytes` in the mpsc queue has been consumed,
written to the file and dropped.
Since reading from entry would have to wait for external file I/O,
this would give the blocking thread some time to flush `Bytes`
out.
- Disable unused feature fs of dep tokio
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Optimize `BinstallError::CratesIoApi`: Extract new type `errors::CratesIoApiError` and box it
Also improve `<CratesIoApiError as Display>::fmt` impl.
* Optimize `BinstallError::SubProcess`: Use `Box<str>` instead of `String`
* Optimize `BinstallError::CargoManifest`: Box `CargoTomlError`
* Optimize `BinstallError::VersionParse`: Extract `VersionParseError` and box it
Also improve `<VersionParseError as Display>::fmt` impl.
* Optimize `BinstallError::CrateContext`: Extract `CrateContextError` and box it in
* Optimize `install_from_source`: Only format `cmd` on err
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Optimize `Fetcher::find`: Make it non-async to avoid boxing
and return `AutoAbortJoinHandle<...>` instead.
Since spawning a new task in tokio box the future anyway, boxing the
returned future again in `Fetcher::find` merely adds unnecessary
overheads.
* Optimize `QuickInstall::report`: Make it async fn instead of ret `JoinHandle`
This provides several benefits:
- On debug build, no task will be spawned
- The calls to `self.stats_url()` can be evaluated in parallel.
- The task now returns `()` so that the task spawned is smaller.
* Fix `QuickInstall::find`: `warn!` if quickinstall report fails
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Optimization: Rm `debug!` in `find_version`
printing all version iterated obviously doesn't help much in debugging
in the problem but rather just confusing.
Also this makes it hard for the compiler to optimize the iterators.
* Use let-else in `ManifestVisitor`
* Optimize `BinFile::preview_{bin, link}` for zero-copy
Return `impl Display` that lazily format instead of allocating a `String`
* Optimize `infer_bin_dir_template`: Generate dir lazily
* Optimize `find_version`: Lazily clone `version_req` only on err
* Refactor `find_version`: Use `bool::then_some`
* Add dep either v1.8.0 to binstalk
* Optimize `GhCrateMeta::find`: Avoid cloning and `Vec` creation
by using `Either`
* Optimize `ops::install::install_from_package`: Make it a regular fn
instead of async fn since it does not `.await` on any async fn.
* Optimize `QuickInstall`: Rm field `target`
since `Arc<Data>` already contains that field.
* Optimize `Fetcher`s: Extract new struct `TargetData`
so that `Data` can be shared by all fetchers, regardless of the target.
* Optimize `QuickInstall`: Rm unused field `data`
* Optimize `Resolution::print`: Replace branching with conditional move
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Avoid potential panicking in `args::parse` by using `Vec::get` instead of indexing
* Refactor: Simplify `opts::{resolve, install}` API
Many parameters can be shared and put into `opts::Options` intead and
that would also avoid a few `Arc<Path>`.
* Optimize `get_install_path`: Avoid cloning `install_path`
* Optimize `LazyJobserverClient`: Un`Arc` & remove `Clone` impl
to avoid additional boxing
* Optimize `find_version`: Avoid cloning `semver::Version`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`
return `impl Iterator<Item = impl Future<Output = ...>>`
instead of `impl Iterator<Item = AutoAbortJoinHandle<...>>`
to avoid unnecessary spawning.
Each task spawned has to be boxed and then polled by tokio runtime.
They might also be moved.
While they increase parallelism, spawning these futures does not justify
the costs because:
- Each `Future` only calls `remote_exists`
- Each `remote_exists` call send requests to the same domain, which is
likely to share the same http2 connection.
Since the conn is shared anyway, spawning does not speedup anything
but merely add communication overhead.
- Plus the tokio runtime spawning cost
* Optimize `install_crates`: Destruct `Args` before any `.await` point
to reduce size of the future
* Refactor `logging`: Replace param `arg` with `log_level` & `json_output`
to avoid dep on `Args`
* Add dep strum & strum_macros to crates/bin
* Derive `strum_macros::EnumCount` for `Strategy`
* Optimize strategies parsing in `install_crates`
* Fix panic in `install_crates` when `Compile` is not the last strategy specified
* Optimize: Take `Vec<Self>` instead of slice in `CrateName::dedup`
* Refactor: Extract new fn `compute_resolvers`
* Refactor: Extract new fn `compute_paths_and_load_manifests`
* Refactor: Extract new fn `filter_out_installed_crates`
* Reorder `install_crates`: Only run target detection if args are valid
and there are some crates to be installed.
* Optimize `filter_out_installed_crates`: Avoid allocation
by returning an `Iterator`
* Fix user_agent of `remote::Client`: Let user specify it
* Refactor: Replace `UIThread` with `ui::confirm`
which is much simpler.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Optimize `Download::and_extract`: Avoid dup monomorphization
* Increase buffer size for binstall_crates_v1 to `4096 * 5`
* Optimize `opts::resolve`: Avoid unnecessary `clone`s
* Fix reserve in `opts::resolve`: Do not over-reserve
* Rename field `opts::Options::resolver` => `resolvers`
* Refactor: Extract new type `resolve::PackageInfo`
- which makes `opts::resolve_inner` easier to understand
- reduce number of parameters required for `download_extract_and_verify` and
`collect_bin_files`
- reducing size of future returned by `opts::resolve_inner` by dropping
`cargo_toml::{Manifest, Package}` as early as possible since
`Manifest` is 3000 Bytes large while `Package` is 600 Bytes large.
* Optimize `fetchers::Data`: Use `CompactString` for field name & version
since they are usually small enough to fit in inlined version of
`CompactString`.
* Optimize `gh_crate_meta`: Avoid unnecessary allocation
in `RepositoryHost::get_default_pkg_url_template`.
* Refacator: Use `Itertools::cartesian_product` in `apply_filenames_to_paths`
* Optimize `ops::resolve`: Avoid unnecessary `clone` & reduce future size
by calling `fetcher.target_meta()` to obtain final metadata after
downloaded and extracted the binaries.
* Optimize `ops::resolve`: Avoid unnecessary allocation
in `download_extract_and_verify`: Replace `Itertools::join` with
`Itertools::format` to avoid allocating the string.
* Fix disabling cargo-install fallback
* Simplify `BinFile::from_product`: Takes `&str` instead of `&Product`
since we only need `product.name`
* Rename `BinFile::from_product` => `BinFile::new`
* Refactor: Create newtype `ops::resolve::Bin`
so that we don't need to `unwrap()` on `Product::name`
and reduce memory usage.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Refactor: Extract new crate binstalk-types
* Optimize: Rm field `CrateInfo::other`
which also removes dep serde-tuple-vec-map and serde-json from
binstalk-types.
This also makes `CrateInfo` easier to use, more generic and can be used
over any `Serializer`, not just `serde_json::Value`.
* Mark all errors in `binstalk-manifests` as non_exhaustive
* Reduce size of `CvsParseError` by using `Box<str>`
instead of `String` for variant `UnknownSourceType`.
* Reduce size of `CratesTomlParseError` to 16 bytes on 64bit platform
by boxing variants `TomlWrite` and `CvsParse` as these two fields are
significantly larger than other variants.
* Unify import style in mod `binstall_crates_v1`
* Replace dep binstalk-manifests with binstalk-types in binstalk-downloader
to reduce its transitive dependencies and enables binstalk-downloader to
be built in parallel to binstak-manifests.
* Replace dep binstalk-manifests with binstalk-types in binstalk
to reduce transitive dependencies and enables binstalk to be built in
parallel to binstalk-manifests.
This is benefitial because binstalk-manifests pulls in toml_edit, which
could takes up to 15s to be built on M1 (7-9s for codegen).
* Add dep binstalk-manifests to crates/bin
* Update dependabot and GHA release-pr
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Resolved#522
* Add dep tracing v0.1.37 to binstalk
* Use `tracing` instead of `log` for logging in binstalk
* Rm dev dep `env_logger` since `log` is no longer used
* Rm unused dep `log` from binstalk
* Replace use of `log` with `tracing` in crates/bin
* Enable feat std of dep log in crates/bin
* Add dep tracing v0.1.37 to binstalk-downloader
* Replace use of `log` with `tracing` in binstalk-downloader
* Rm unused dep `log` from binstalk-downlaoder
* Wrap `ops::{install, resolve}` in `tracing::instrument`
* Wrap `Download::and_{extract, visit_tar}` in `instrument`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#136
* Impl opt-out optioins in binstalk
* Replace field `Options::{gh_crate, quickinstall}_fetcher` with `resolver`
which can determine order of resolver used.
* Add new field `Args::{disable_}strategies`
* Add new variant `BinstallError::InvalidStrategies`
* Add variant `BinstallError::NoFallbackToCargoInstall`
* Add code for supporting strategies in mod entry
* Test `--disable-strategies` in `tests.sh`
* Test for `--strategies` in `tests.sh`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Extraction wasn't cancellable by `cancel_on_user_sig_term` used in `entry` since it calls `block_in_place`.
This PR adds cancellation support to it by adding a `static` variable `OnceCell` to `wait_on_cancellation_signal` so that once it returns `Ok(())`, all other calls to it after that point also returns `Ok(())` immediately.
`StreamReadable`, which is used in cancellation process, then stores a boxed future of `wait_on_cancellation_signal` and polled it in `BufReader::fill_buf`.
Note that for zip, the extraction process takes `File` instead of `StreamReadable` due to `io::Seek` requirement, so it cancelling during extraction for zip is still not possible.
This PR also optimized `extract_bin` and `extract_zip` by using `StreamReadable::copy` introduced to this PR instead of `io::copy`, which allocates an internal buffer on stack, which imposes extra copy.
It also fixed `StreamReadable::fill_buf` by ensuring that empty buffer is only returned on eof.
* Make `wait_on_cancellation_signal` pub
* Enable feature `parking_lot` of dep tokio
* Mod `wait_on_cancellation_signal`: Use `OnceCell` internally
to archive the effect that once call to it return `Ok(())`, all calls to
it after that also returns `Ok(())`.
* Impl `From<BinstallError>` for `io::Error`
* Impl cancellation on user signal in `StreamReadable`
* Fix err msg when cancelling during extraction in `ops::resolve`
* Optimize: Impl & use `StreamReadable::copy`
which is same as `io::copy` but does not allocate any internal buffer
since `StreamReadable` is buffered.
* Fix `next_stream`: Return non-empty bytes on `Ok(Some(bytes))`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
If user interrupts `cargo-binstall` by signal while it is running `cargo-install`, then the `cargo-install` would continue to run at background even after `cargo-binstall` terminates.
Setting `cmd` to be kill_on_drop fixed this.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Refactor: Avoid parsing `package.version()` twice in `ops::resolve` and `ops::install`
* Optimize Resolution: Replace `Package<Meta>` with two `CompactStrings`: `name` and `version`
* Use `CompactString` for `BinstallError::CratesIoApi::crate_name`
* Use `CompactString` for `BinstallError::VersionParse::v`
* Use `CompactString` for `BinstallError::VersionReq::req`
* Use `CompactString` for `BinstallError::VersionUnavailable::crate_name`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Run `$CARGO -vV` in `detect-targets` if env `CARGO` present
* Improve crate doc for `detect-targets`
* Use env var `CARGO` in `install_from_source` if present
* Test use of `CARGO` env in `tests.sh`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Make binaries with `required-features` optional so that crates like `sccache` would work as expected.
* Fix `infer_bin_dir_template`: concat with `data.bin_path` introduced in #417
* Always `chmod +x $bin` on unix in case the archive (e.g. `sccache`) did not set the executable bit of fmt == Bin.
* CI: Install `sccache` but do not run it since it requires cargo env to be ready
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Extract `infer_bin_dir_template` and call it once in `collect_bin_files`
* Ensure `product.name.is_some()` is true
* Avoid cloning `str` in `BinFile::from_product`
* Optimize: Take reference in `bins::Data`
* Optimize: Construct `CompactString` only when needed
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
From @passcod :
* Make bin-dir an Option
* Use cargo-release as a test case
* WIP: Try multiple default bin dirs
* Update bins.rs
* Update cargo_toml_binstall.rs
From @NobodyXu :
* Optimize & Prepare for support multi `bin-path`s
* Ensure quickinstall bin_dir work as usual.
* Rm dup entries in `FULL_FILENAMES`
* Add second version of `NOVERSION_FILENAMES`
* Impl multiple default `bin-dir`s
* Improve doc for `BinFile::from_product`
* Fix tests.sh
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
* Refactor: Extract new fn `BinFile::check_source_exists`
* Impl new async fn `AutoAbortJoinHandle::flattened_join`
* Impl new fn `Fetcher::fetcher_name`
* Verify that `bin_files` exist in `resolve` stage
To ensure that the installation stage won't fail because of missing
binaries.
* Rm unused `MultiFecther`
* Simplify `Future` impl for `AutoAbortJoinHandle`
* Add new variant `BinstallError::CargoTomlMissingPackage`
* Replace `unwrap` in `resolve_inner` with proper error handling
* Make `Fetcher::new` as a regular function
instead of an `async` function.
* Ret `Arc<dyn Fetcher>` in trait fn `Fetcher::new`
* Refactor `resolve_inner`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
