Commit graph

54 commits

Author SHA1 Message Date
Jiahao XU
8e08c65946
Fix fallback to cargo-install: Use the default target unless user override (#1430)
* Fix fallback to `cargo-install`: Use the default target

which is guaranteed to work, instead of providing a custom `--target`
from `detect-targets`.

The one from `detect-targets` might report musl even if the `rustc`
reports gnu, since OSes like NixOS could use a custom glibc location
that prevent glibc binaries built for other OSes be run for them.

However, if we are compiling the crate and `rustc` defaults to glibc,
then it should be guaranteed to work.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl fn `DesiredTargets::get_initialized`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix fallback to `cargo-install`: Use user-provided override targets

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-30 00:22:21 +00:00
Félix Saparelli
680accd0d3
Check quickinstall signatures (#1382)
* Check quickinstall signatures

* e2e test

* We can no longer check presence of signing section early

* Improve UI a little
2023-09-23 08:35:49 +00:00
Félix Saparelli
32beba507b
Initial signing support (#1345)
* Add CLI options

* Add manifest types

* Thread signature policy through to fetchers

* Thread signing section through from metadata

* Implement signing validation

* Clippy

* Attempt testing

* Yes and

* Why

* fmt

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/gh_crate_meta.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* fixes

* Finish feature

* Document

* Include all fields in the signing.file template

* Readme document

* Review fixes

* Fail on non-utf8 sig

* Thank goodness for tests

* Run test in ci

* Add rsign2 commands

* Log utf8 error

* Update e2e-tests/signing.sh

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests/signing.sh` MacOS CI failure

Move the tls cert creation into `signing.sh` and sleep for 10s to wait
for https server to start.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor e2e-tests-signing files

 - Use a tempdir generated by `mktemp` for all certificates-related
   files
 - Put other checked-in files into `e2e-tests/signing`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` connection err in MacOS CI

Wait for server to start up by trying to connect to it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` passing `-subj` to `openssl` on Windows

Use single quote instead of double quote to avoid automatic expansion
from bash

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` waiting for server to startup

Remove `timeout` since it is not supported on MacOS.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Try to fix windows CI by setting `MSYS_NO_PATHCONV=1` on `openssl` cmds

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` on windows

By using double `//` for the value passed to option `-subj`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed infinite loop in `signing/wait-for-server` on Windows

Pass `--ssl-revoke-best-effort` to prevent schannel from checking ssl
revocation status.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add cap on retry attempt in `signing/wait-for-server.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Let `singing/server.py` print output to stderr

so that we can see the error message there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `signing/server.py` on MacOS CI

use `python3` since macos-latest still has python2 installed and
`python` is a symlink to `python2` there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-23 04:02:56 +00:00
Jiahao XU
3c5641610a
feat: Improve resolution logging and verbose logging (#1341)
* feat: Improve resolution logging

Fixed #1336

Log target of the pre-built binaries which will be installed.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix args parsing: `quiet` & `verbose` cannot be set at the same time

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* feat: Implies `--log-level debug` if `--verbose` is set

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-02 09:38:24 +00:00
Jiahao XU
e5567ae526
Fix clippy lint and format (#1328)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-25 08:53:50 +00:00
Jiahao XU
0f7828569a
Add error logging if achieve doesn't contain bin required (#1298)
* Add `error` logging if achieve doesn't contain bin required

So that users know that this is an upstream issue and will report to
upstream.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk/src/ops/resolve.rs

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-16 23:13:21 +00:00
Jiahao XU
0c5a65fb35
Refactor: Extract new crate binstalk-bins (#1294)
To reduce `binstalk` codegen and enable reuse of it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-15 21:48:42 +00:00
Jiahao XU
76c72469eb
Refactor: Extract new crate binstalk-fetchers (#1291)
To reduce `binstalk` codegen and enable better reuse.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-14 03:20:34 +00:00
Jiahao XU
fbed317df5
feat: git::Repository cancellation support (#1288)
feat: `git::Repository` support cancellation.

To make sure users can cancel git operation via signal, e.g. when the
git operation fail or users no longer want to install.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-12 14:18:02 +00:00
Jiahao XU
8ff13c1b36
Refactor: Extract cargo_toml_workspace as a new crate (#1287)
To reduce codegen time of `binstalk` and also enable others to reuse
this crate.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-12 12:05:10 +00:00
Jiahao XU
aeacebcf83
feat: Support passing workspace to --manifest-path (#1246)
Previously it will load the root `Cargo.toml` and treat it as the
manifest for the crate, now it will check its `package.name` and would
search for the workspace if the `package.name` does not match the crate
name.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-03 21:12:06 +00:00
Jiahao XU
01a87ac606
feat: Impl support for alternative registries (#1184)
Fixed #1168

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-30 03:52:40 +00:00
Jiahao XU
ca00cbaccc
feat: Support install directly from git repo (#1162)
Fixed #3

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-24 01:01:31 +00:00
Jiahao XU
d8419ea5a2
feat: Add more variables for template (#1160)
Fixed #775

 - Add dep target-lexicon v0.12.7
 - Add `target-{family, arch, libc, vendor}` to
   `package.metadata.binstall`.

For `{universal, universal2}-apple-darwin`, the `target-arch` is set to
`universal`.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-20 22:05:13 +00:00
Jiahao XU
1432093dcc
feat: Add new cmdline option --no-track (#1111)
Same as `cargo-install`'s `--no-track`.
It is also passed to `cargo-install` if it is invoked.

Also fixed `fs::atomic_symlink_file` which on Windows could fallback to
non-atomic install if symlinking failed.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-03 09:15:18 +00:00
Jiahao XU
a27d5aebf6
Use leon for template in binstalk & detect malformed pkg-url/pkg-fmt early (#933)
Fixed #851

* Add new dep leon to crate binstalk
* Add new variant `BinstallError::Template{Parse, Render}Error`
* Use `leon::Template` in mod `bins`
* Use `leon::Template` in mod `fetchers::gh_crate_meta`
* Refactor mod `bins`: Rm unused associated fn & fields from `Context`
* Simplify unit testing in mod `fetchers::gh_crate_meta`
* Rm soft-deprecated field `fetchers::gh_crate_meta::Context::format`
  and change the `match` to resolve `archive` => `self.archive_format`.
* Make macro_rules `leon::template!` far easier to use
* Construct `leon::Template<'_>` as constant in `gh_crate_meta::hosting`
* Simplify `leon::Values` trait
   Change its method `get_value` signature to
   
   ```rust
       fn get_value(&self, key: &str) -> Option<Cow<'_, str>>;
   ```
   
   Now, `ValuesFn` also accepts non-`'static` function, but now
   `leon::Values` is only implemented for `&ValuesFn<F>` now.
   
   This makes it a little bit more cumbersome to use but I think it's a
   reasonable drawback.
* Rm `Send` bound req from `ValuesFn`
* Impl new fn `leon::Template::cast`
   for casting `Template<'s>` to `Template<'t>` where `'s: 't`
* Rename `leon::Template::has_keys` => `has_any_of_keys`
* Make checking whether format related keys are present more robust
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Skip checking all fmt ext
   if none of the format related keys ("format", "archive-format",
   "archive-suffix") are present.
* Only ret `.exe` in `PkgFmt::extensions` on windows
   by adding a new param `is_windows: bool`
* Improve debug msg in `GhCrateMeta::fetch_and_extract`
* Add warnings to `GhCrateMeta::find`
* Rm dep tinytemplate
* `impl<'s, 'rhs: 's> ops::AddAssign<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::AddAssign<Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<&Item<'item>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::Add<Template<'rhs>> for Template<'s>` (improved existing `Add` impl)
* `impl<'s, 'rhs: 's> ops::Add<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<&Item<'item>> for Template<'s>`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-03-26 16:11:10 +11:00
Jiahao XU
cc78ff3b90
Fix fetcher::QuickInstallstats report sending (#918)
Fixed https://github.com/cargo-bins/cargo-quickinstall/issues/195

 - Fix `fetchers::QuickInstall`: Stop sending stats for `universal-apple-darwin`
   since quickinstall only supports targets officially supports by rust.
 - Only send stats report to quickinstall if the `Fetcher::find` is `.await`ed on
   
   This prevents stats report to be sent for cases where the `QuickInstall` fetcher
   is actually unused, e.g. resolved to `GhCrateMeta` or other `QuickInstall` fetcher
   with different target.
   
   This also reduces amount of http requests created in background.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-17 21:05:19 +11:00
Jiahao XU
d118fa8fdb
Obey crates.io crawler policy: Only make one request per sec (#908)
Fixed https://github.com/taiki-e/install-action/issues/89

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-14 17:11:22 +11:00
Jiahao XU
89a47cbec9
Fix extract_tar_based_stream: Normalize path before adding them to ExtractedFiles (#901)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-13 21:27:50 +11:00
Jiahao XU
44e81c7e1a
Fix: Pass value of --roots to cargo-install as --root (#860)
Pass value of `--roots` to `cargo-install` as `--root`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-06 11:52:28 +11:00
Jiahao XU
9c7da6a179
Return a list of files written to disk in binstalk_downloader::download::Download::and_extract (#856)
to avoid collecting extracted files from disk again in resolution stage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-03 23:31:27 +11:00
Jiahao XU
8eee318ccd
Replace dep crates_io_api with in-house solution (#846)
It also uses `max_stable_version` in the json downloaded from https://crates.io/api/v1/crates/$name if possible, which is equivalent to the version shown on https://crates.io/crates/$name .

 - Add new feat `json` to `binstalk-downloader`
 - Impl new async fn `Response::json`
 - use `Response::json` in `GhApiClient` impl
 - Mark all err types in binstalk-downloader as `non_exhaustive`
 - Ret `remote::Error` in `remote::Certificate::{from_pem, from_der}` instead of `ReqwestError`.
 - Refactor `BinstallError`: Merge variant `Unzip`, `Reqwest` & `Http`
    into one variant `Download`.
 - Manually download and parse json from httos://crates.io/api/v1
 - Remove unused deps `crates_io_api`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 18:25:34 +11:00
Jiahao XU
599bcaf333
Impl GhApiClient and use it in cargo-binstall to speedup resolution process (#832)
Fixed #776

 - Add new feature gh-api-client to binstalk-downloader
 - Impl new type `binstalk_downloader::remote::{RequestBuilder, Response}`
 - Impl `binstalk_downloader::gh_api_client::GhApiClient`, exposed if `cfg(feature = "gh-api-client")` and add e2e and unit tests for it
 - Use `binstalk_downloader::gh_api_client::GhApiClient` to speedup `cargo-binstall`
 - Add new option `--github-token` to supply the token for GitHub restful API, or read from env variable `GITHUB_TOKEN` if not present.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 12:04:22 +11:00
Jiahao XU
96ebcfd08f
Add new option --locked that is passed through to cargo-install (#830)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-22 22:27:17 +11:00
Jiahao XU
2e118b3044
Optimize GhCrateMeta::find: Cache url resolution result (#810)
So that we don't have to create multiple HEAD/GET http request to get
the final, redirected url.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-16 14:51:47 +13:00
Jiahao XU
270bf08a24
Optimize use of tinytemplate (#647)
* Optimize `GhCrateMeta::find`: Cache compiled template `pkg_url`
* Optimize `collect_bin_files`: Cache compiled template `bin_dir`
* Refactor: Improve readability of `gh_crate_meta`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Dedup urls before checking
* Optimize `BinFile::new`: Avoid one heap alloc when creating `dest`
* Improve `warn!` msg in `GhCrateMeta::launch_baseline_find_tasks`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-26 17:33:49 +13:00
Jiahao XU
7b34178bcc
Optimize resolve_inner: Avoid VersionReq::clone (#656)
* Add new dep maybe-owned v0.3.4 to binstalk
* Optimize `resolve_inner`: Avoid `VersionReq::clone`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-06 13:47:45 +13:00
Jiahao XU
6bdb26930e
Refactor, improvements and bugfix for resolution and installation process (#619)
* Refactor: Extract new mod `ops::resolve::resolution`
* Refactor: Extract new type `ResolutionFetch`
* Refactor: Extract new type `ResolutionSource`
* Improve `Resolution::print`: Provides more details on which crate
   is the resolution for.
* Make `Resolution::print` a pub fn
* Refactor: Extract new fn `ResolutionFetch::install`
* Refactor: Extract new async fn `ResolutionSource::install`
* Optimize `ResolutionSource::install` avoiding `OsStr::to_string_lossy`.
* Add new dep command-group v2.0.0 to binstalk with feat with-tokio
* Fix `ResolutionSource::install`: Use `AsyncCommandGroup::group_spawn`
   instead of `tokio::process::Command::spawn` to ensure all the processes
   spanwed by the `cargo` process can be terminated with one signal sent to
   the `cargo` process.
* Fix printing resolution: Make sure they are printed without interleaving
* Refactor `entry::install_crates`
* Improve dry-run output for `ResolutionSource::install`
* Refactor: Extract new fn `ResolutionFetch::print`
* Refactor: Extract new fn `ResolutionSource::print`
* Optimize `Resolution`: Box unit variant `Fetch`
* Improve formatting of `tokio::process::Command`
   Prints out sth like `cargo install ...` instead of the `Debug::fmt`.
* Improve dry-run output

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-28 00:18:07 +00:00
Jiahao XU
db45f2fb7f
Refactor binstalk-downloader APIs: Remove cancellation_future plus optimizations (#591)
- Refactor: Mv fn `utils::asyncify` into mod `utils`
 - Improve err msg for task failure in `utils::asyncify`
 - Make sure `asyncify` always returns the same annoymous type
   that implements `Future` if the `T` is same.
 - Rewrite `extract_bin` to avoid `block_in_place`
   support cancellation by dropping
 - Rm unused dep scopeguard from binstalk-downloader
 - Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
 - Unbox `extract_future` in `async_extracter::extract_zip`
 - Refactor `Download` API: Remove `CancellationFuture` as param

   since all futures returned by `Download::and_*` does not call
   `block_in_place`, so they can be cancelled by drop instead of using this
   cumbersome hack.
 - Fix exports from mod `async_tar_visitor`
 - Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
 - Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
   and rm fn `wait_on_cancellation_signal_inner`
 - Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
 - Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
   Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
 - Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
 - Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
 - Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
 - Optimize `extract_zip_entry`: Spawn only one blocking task per fn call

   by using a mspc queue for the data to be written to the `outfile`.

   This would improve efficiency as using `tokio::fs::File` is expensive:
   It spawns a new blocking task, which needs one heap allocation and then
   pushed to a mpmc queue, and then wait for it to be done on every loop.

   This also fix a race condition where the unix permission is set before
   the whole file is written, which might be used by attackers.
 - Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
   To avoid frequent allocation and deallocation.
 - Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`

   Performs the reserve before sending the buf over mpsc queue to
   increase the possibility of reusing the previous allocation.

   NOTE: `BytesMut` only reuses the previous allocation if it is the
   only one holds the reference to it, which is either on the first
   allocation or all the `Bytes` in the mpsc queue has been consumed,
   written to the file and dropped.

   Since reading from entry would have to wait for external file I/O,
   this would give the blocking thread some time to flush `Bytes`
   out.
 - Disable unused feature fs of dep tokio

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-12 03:15:30 +00:00
Jiahao XU
8a5577297e
Optimize BinstallError: Reduce size from 40B to 32B (#581)
* Optimize `BinstallError::CratesIoApi`: Extract new type `errors::CratesIoApiError` and box it
   Also improve `<CratesIoApiError as Display>::fmt` impl.
* Optimize `BinstallError::SubProcess`: Use `Box<str>` instead of `String`
* Optimize `BinstallError::CargoManifest`: Box `CargoTomlError`
* Optimize `BinstallError::VersionParse`: Extract `VersionParseError` and box it
   Also improve `<VersionParseError as Display>::fmt` impl.
* Optimize `BinstallError::CrateContext`: Extract `CrateContextError` and box it in
* Optimize `install_from_source`: Only format `cmd` on err

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-01 20:05:00 +13:00
Jiahao XU
ab3e47c42b
Optimize Fetcher::find and fix quickinstall reporting (#579)
* Optimize `Fetcher::find`: Make it non-async to avoid boxing
   and return `AutoAbortJoinHandle<...>` instead.
   
   Since spawning a new task in tokio box the future anyway, boxing the
   returned future again in `Fetcher::find` merely adds unnecessary
   overheads.
* Optimize `QuickInstall::report`: Make it async fn instead of ret `JoinHandle`
   
   This provides several benefits:
    - On debug build, no task will be spawned
    - The calls to `self.stats_url()` can be evaluated in parallel.
    - The task now returns `()` so that the task spawned is smaller.
* Fix `QuickInstall::find`: `warn!` if quickinstall report fails

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-30 16:15:22 +13:00
Jiahao XU
66abf1b8c1
More mionr optimizations (#553)
* Optimize `BinFile::preview_bin`: Use `Path::display` instead of `to_string_lossy` to avoid allocation.
* Refactor: Rm useless `AsRef::as_ref` call
* Optimize `GhCrateMeta::find`: Reduce fut size by converting `Url` to `String`
   `Url` is much larger than `String`.
* Refactor `PackageInfo::resolve`: Destruct `Meta::binstall`
* Optimize `resolve::load_manifest_path`: Avoid allocation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-22 19:35:16 +13:00
Jiahao XU
bdb4b2070d
Minor optimization (#544)
* Optimization: Rm `debug!` in `find_version`
   printing all version iterated obviously doesn't help much in debugging
   in the problem but rather just confusing.
   
   Also this makes it hard for the compiler to optimize the iterators.
* Use let-else in `ManifestVisitor`
* Optimize `BinFile::preview_{bin, link}` for zero-copy
   Return `impl Display` that lazily format instead of allocating a `String`
* Optimize `infer_bin_dir_template`: Generate dir lazily
* Optimize `find_version`: Lazily clone `version_req` only on err
* Refactor `find_version`: Use `bool::then_some`
* Add dep either v1.8.0 to binstalk
* Optimize `GhCrateMeta::find`: Avoid cloning and `Vec` creation
   by using `Either`
* Optimize `ops::install::install_from_package`: Make it a regular fn
   instead of async fn since it does not `.await` on any async fn.
* Optimize `QuickInstall`: Rm field `target`
   since `Arc<Data>` already contains that field.
* Optimize `Fetcher`s: Extract new struct `TargetData`
   so that `Data` can be shared by all fetchers, regardless of the target.
* Optimize `QuickInstall`: Rm unused field `data`
* Optimize `Resolution::print`: Replace branching with conditional move

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-21 10:32:46 +13:00
Jiahao XU
50b6e62164
Minor refactor and optimization (#543)
* Avoid potential panicking in `args::parse` by using `Vec::get` instead of indexing
* Refactor: Simplify `opts::{resolve, install}` API
   Many parameters can be shared and put into `opts::Options` intead and
   that would also avoid a few `Arc<Path>`.
* Optimize `get_install_path`: Avoid cloning `install_path`
* Optimize `LazyJobserverClient`: Un`Arc` & remove `Clone` impl
   to avoid additional boxing
* Optimize `find_version`: Avoid cloning `semver::Version`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`
   return `impl Iterator<Item = impl Future<Output = ...>>`
   instead of `impl Iterator<Item = AutoAbortJoinHandle<...>>`
   to avoid unnecessary spawning.
   
   Each task spawned has to be boxed and then polled by tokio runtime.
   They might also be moved.
   
   While they increase parallelism, spawning these futures does not justify
   the costs because:
    - Each `Future` only calls `remote_exists`
    - Each `remote_exists` call send requests to the same domain, which is
      likely to share the same http2 connection.
      Since the conn is shared anyway, spawning does not speedup anything
      but merely add communication overhead.
    - Plus the tokio runtime spawning cost
* Optimize `install_crates`: Destruct `Args` before any `.await` point
   to reduce size of the future
* Refactor `logging`: Replace param `arg` with `log_level` & `json_output`
   to avoid dep on `Args`
* Add dep strum & strum_macros to crates/bin
* Derive `strum_macros::EnumCount` for `Strategy`
* Optimize strategies parsing in `install_crates`
* Fix panic in `install_crates` when `Compile` is not the last strategy specified
* Optimize: Take `Vec<Self>` instead of slice in `CrateName::dedup`
* Refactor: Extract new fn `compute_resolvers`
* Refactor: Extract new fn `compute_paths_and_load_manifests`
* Refactor: Extract new fn `filter_out_installed_crates`
* Reorder `install_crates`: Only run target detection if args are valid
   and there are some crates to be installed.
* Optimize `filter_out_installed_crates`: Avoid allocation
   by returning an `Iterator`
* Fix user_agent of `remote::Client`: Let user specify it
* Refactor: Replace `UIThread` with `ui::confirm`
   which is much simpler.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-19 20:00:27 +13:00
Jiahao XU
325cb5cc19
Optimizations plus bug fix (#541)
* Optimize `Download::and_extract`: Avoid dup monomorphization
* Increase buffer size for binstall_crates_v1 to `4096 * 5`
* Optimize `opts::resolve`: Avoid unnecessary `clone`s
* Fix reserve in `opts::resolve`: Do not over-reserve
* Rename field `opts::Options::resolver` => `resolvers`
* Refactor: Extract new type `resolve::PackageInfo`
    - which makes `opts::resolve_inner` easier to understand
    - reduce number of parameters required for `download_extract_and_verify` and
      `collect_bin_files`
    - reducing size of future returned by `opts::resolve_inner` by dropping
      `cargo_toml::{Manifest, Package}` as early as possible since
      `Manifest` is 3000 Bytes large while `Package` is 600 Bytes large.
* Optimize `fetchers::Data`: Use `CompactString` for field name & version
   since they are usually small enough to fit in inlined version of
   `CompactString`.
* Optimize `gh_crate_meta`: Avoid unnecessary allocation
   in `RepositoryHost::get_default_pkg_url_template`.
* Refacator: Use `Itertools::cartesian_product` in `apply_filenames_to_paths`
* Optimize `ops::resolve`: Avoid unnecessary `clone` & reduce future size
   by calling `fetcher.target_meta()` to obtain final metadata after
   downloaded and extracted the binaries.
* Optimize `ops::resolve`: Avoid unnecessary allocation
   in `download_extract_and_verify`: Replace `Itertools::join` with
   `Itertools::format` to avoid allocating the string.
* Fix disabling cargo-install fallback
* Simplify `BinFile::from_product`: Takes `&str` instead of `&Product`
   since we only need `product.name`
* Rename `BinFile::from_product` => `BinFile::new`
* Refactor: Create newtype `ops::resolve::Bin`
   so that we don't need to `unwrap()` on `Product::name`
   and reduce memory usage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 23:59:35 +00:00
Jiahao XU
d9cc3ce219
Refactor: Extract new crate binstalk-types plus other misc refactor and optimization (#535)
* Refactor: Extract new crate binstalk-types
* Optimize: Rm field `CrateInfo::other`
   which also removes dep serde-tuple-vec-map and serde-json from
   binstalk-types.
   
   This also makes `CrateInfo` easier to use, more generic and can be used
   over any `Serializer`, not just `serde_json::Value`.
* Mark all errors in `binstalk-manifests` as non_exhaustive
* Reduce size of `CvsParseError` by using `Box<str>`
   instead of `String` for variant `UnknownSourceType`.
* Reduce size of `CratesTomlParseError` to 16 bytes on 64bit platform
   by boxing variants `TomlWrite` and `CvsParse` as these two fields are
   significantly larger than other variants.
* Unify import style in mod `binstall_crates_v1`
* Replace dep binstalk-manifests with binstalk-types in binstalk-downloader
   to reduce its transitive dependencies and enables binstalk-downloader to
   be built in parallel to binstak-manifests.
* Replace dep binstalk-manifests with binstalk-types in binstalk
   to reduce transitive dependencies and enables binstalk to be built in
   parallel to binstalk-manifests.
   
   This is benefitial because binstalk-manifests pulls in toml_edit, which
   could takes up to 15s to be built on M1 (7-9s for codegen).
* Add dep binstalk-manifests to crates/bin
* Update dependabot and GHA release-pr

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 13:46:27 +13:00
Jiahao XU
4e875874b6
Replace use of log with tracing (#527)
Resolved #522

* Add dep tracing v0.1.37 to binstalk
* Use `tracing` instead of `log` for logging in binstalk
* Rm dev dep `env_logger` since `log` is no longer used
* Rm unused dep `log` from binstalk

* Replace use of `log` with `tracing` in crates/bin
* Enable feat std of dep log in crates/bin

* Add dep tracing v0.1.37 to binstalk-downloader
* Replace use of `log` with `tracing` in binstalk-downloader
* Rm unused dep `log` from binstalk-downlaoder


* Wrap `ops::{install, resolve}` in `tracing::instrument`
* Wrap `Download::and_{extract, visit_tar}` in `instrument`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-13 21:45:37 +13:00
Jiahao XU
9e80cf0700
Feature: Impl opt-out options (#510)
Fixed #136

* Impl opt-out optioins in binstalk
* Replace field `Options::{gh_crate, quickinstall}_fetcher` with `resolver`
  which can determine order of resolver used.
* Add new field `Args::{disable_}strategies`
* Add new variant `BinstallError::InvalidStrategies`
* Add variant `BinstallError::NoFallbackToCargoInstall`
* Add code for supporting strategies in mod entry
* Test `--disable-strategies` in `tests.sh`
* Test for `--strategies` in `tests.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-11 05:24:46 +00:00
Jiahao XU
9ed04cf89b
Fix clippy error on unit test CI (#503)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-28 13:48:16 +13:00
Jiahao XU
aa6012baae
Make extraction cancellable for bin and tar based formats (#481)
Extraction wasn't cancellable by `cancel_on_user_sig_term` used in `entry` since it calls `block_in_place`.

This PR adds cancellation support to it by adding a `static` variable `OnceCell` to `wait_on_cancellation_signal` so that once it returns `Ok(())`, all other calls to it after that point also returns `Ok(())` immediately.

`StreamReadable`, which is used in cancellation process, then stores a boxed future of `wait_on_cancellation_signal` and polled it in `BufReader::fill_buf`.

Note that for zip, the extraction process takes `File` instead of `StreamReadable` due to `io::Seek` requirement, so it cancelling during extraction for zip is still not possible.

This PR also optimized `extract_bin` and `extract_zip` by using `StreamReadable::copy` introduced to this PR instead of `io::copy`, which allocates an internal buffer on stack, which imposes extra copy.

It also fixed `StreamReadable::fill_buf` by ensuring that empty buffer is only returned on eof.

* Make `wait_on_cancellation_signal` pub
* Enable feature `parking_lot` of dep tokio
* Mod `wait_on_cancellation_signal`: Use `OnceCell` internally
   to archive the effect that once call to it return `Ok(())`, all calls to
   it after that also returns `Ok(())`.
* Impl `From<BinstallError>` for `io::Error`
* Impl cancellation on user signal in `StreamReadable`
* Fix err msg when cancelling during extraction in `ops::resolve`
* Optimize: Impl & use `StreamReadable::copy`
   which is same as `io::copy` but does not allocate any internal buffer
   since `StreamReadable` is buffered.
* Fix `next_stream`: Return non-empty bytes on `Ok(Some(bytes))`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-13 13:31:13 +13:00
Jiahao XU
fdc617d870
Fix install_from_source: Set cmd to be kill_on_drop (#479)
If user interrupts `cargo-binstall` by signal while it is running `cargo-install`, then the `cargo-install` would continue to run at background even after `cargo-binstall` terminates.

Setting `cmd` to be kill_on_drop fixed this.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-12 17:56:26 +13:00
Jiahao XU
76bc030f90
Disable tcp_nodelay for reqwest::Client and add rate limiting for https requests (#458) 2022-10-07 15:51:34 +11:00
Jiahao XU
3421403e75
Refactor and Optimizations (#459)
* Refactor: Avoid parsing `package.version()` twice in `ops::resolve` and `ops::install`
* Optimize Resolution: Replace `Package<Meta>` with two `CompactStrings`: `name` and `version`
* Use `CompactString` for `BinstallError::CratesIoApi::crate_name`
* Use `CompactString` for `BinstallError::VersionParse::v`
* Use `CompactString` for `BinstallError::VersionReq::req`
* Use `CompactString` for `BinstallError::VersionUnavailable::crate_name`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-06 04:19:12 +13:00
dependabot[bot]
405e41eda4
Bump cargo_toml from 0.12.2 to 0.12.4 (#455)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-04 02:18:54 +13:00
Jiahao XU
ec2bdb551e
Use CARGO env variable if present (#453)
* Run `$CARGO -vV` in `detect-targets` if env `CARGO` present
* Improve crate doc for `detect-targets`
* Use env var `CARGO` in `install_from_source` if present
* Test use of `CARGO` env in `tests.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-02 04:52:25 +13:00
Jiahao XU
abd50a72ec
Fix skipping optional dep: Skip fetcher if they provide 0 bin (#446)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-29 03:51:11 +13:00
Jiahao XU
f482e362ba
Make binaries with required-features optional (#438)
* Make binaries with `required-features` optional so that crates like `sccache` would work as expected.
* Fix `infer_bin_dir_template`: concat with `data.bin_path` introduced in #417 
* Always `chmod +x $bin` on unix in case the archive (e.g. `sccache`) did not set the executable bit of fmt == Bin.
* CI: Install `sccache` but do not run it since it requires cargo env to be ready

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 13:46:55 +10:00
Jiahao XU
c15903684f
Feature/fix no symlink (#432)
* Refactor `BinFile::from_product`: Simplify logic flow.
* Fix `--no-symlink` behavior

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 02:32:22 +00:00
Jiahao XU
b8d2184ee9
Optimize applying PkgOverrides into PkgMeta (#429)
* Refactor: Rm `PkgMeta::clone_without_overrides`
* Impl new fn `PkgMeta::merge_overrides` to avoid unnecessary clone

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-26 19:05:14 +13:00
Jiahao XU
2cc12f9b69
Optimize bins::BinFile (#422)
* Extract `infer_bin_dir_template` and call it once in `collect_bin_files`
* Ensure `product.name.is_some()` is true
* Avoid cloning `str` in `BinFile::from_product`
* Optimize: Take reference in `bins::Data`
* Optimize: Construct `CompactString` only when needed

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-26 09:39:41 +13:00