Commit graph

55 commits

Author SHA1 Message Date
Jiahao XU
6809601273
Add new option to --disable-telemetry to disable quickinstall statistics collection (#1831)
* Add new option `--disable-quick-install-stats`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Send quickinstall stats in parallel

This commit also make sure'the stats is always sent for
each fetcher, regardless of whether it is picked or not,
to make sure that the quick-install stats collection gets
the full infomration of possible targets.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename option to `--disable-telemetry`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update args.rs

Co-authored-by: Félix Saparelli <felix@passcod.name>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Update args.rs

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2024-07-26 06:03:38 +00:00
Jiahao XU
3f29fbe83a
Feature: SupportdDisable of strategies for crate using Cargo.toml (#1828)
* Refactor: Move `Strategy` to `binstalk-types`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add serialisation test for `Strategy`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add support to disable strategies via crate `Cargo.toml`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add e2e-test

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `Cargo.toml` disabled strategy checking for compile strategy

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Optimize `resolve_inner`: Cache meta override

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add compile-time length checking for `Strategy`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* More optimization

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix order of override: cli options alwayus takes precedence

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add missing manifest for e2e-test

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-07-23 14:05:22 +00:00
Jiahao XU
eba07fb147
Fix v1 manifest format for git and local path (#1821)
* Bump simple-git to v0.2.10

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix v1 manifest format for git and local path

Fixed #1815

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix v1 format for custom registry

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unused functions

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix compilation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update git.sh

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Fixed git.sh for windows

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* fixx git.sh for win

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Update git.sh

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* fix git.sh

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
2024-07-17 05:34:12 +00:00
Jiahao XU
ac7bac651d
Run artifact discover in sequential instead of in parallel (#1796)
* Perform artifact discovery in sequential

Run different `fetcher.find()` in sequential

* FuturesResolver: Fallback to other future if one error

* Fix typo

* Apply cargo fmt

* Parallelise `<QuickInstall as Fetcher>::find`

Check for signature in parallel to the package

* Download signature in `<QuickInstall as Fetcher>::find`

So that the signature download can be done in parallel.

* Bump msrv for binstalk-fetchers to 1.70

* Update crates/binstalk-fetchers/src/futures_resolver.rs

Co-authored-by: Félix Saparelli <felix@passcod.name>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* cargo fmt

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2024-06-23 10:42:03 +00:00
Jiahao XU
fff6aa8122
Improve use of github token (#1769)
* Add new dep zeroize

* Use Zeroizing to avoid leaking the token

* Optimize gh-auth-token

Spawn it as a task, and only await it
when using GhApiClient

* Fix binstalk-git-repo-api unit tests
2024-06-15 05:42:09 +00:00
Tomas Olvecky
dfa230f039
Use binary name when searching for artifacts (#1747)
* Use binary name when searching for artifacts

When there is a single binary declared in the manifest and
it differs from the package name, add it to the list of handles
used for pre-built artifact fetching.

* Simplify `binary_name` assignment with a `match`

* Add e2e test

* Only attempt to use the binary name with `GhCrateMeta` fetcher

* Avoid too much over-allocating.

Technically it should also check if gh-crate-meta resolver is enabled, but it is unlikely for it to be disabled and overallocating for extra n-target should be fine, it is an improvement over doubling the space allocated if the binary_name is Some.

* Fix fmt in crates/binstalk/src/ops/resolve.rs

---------

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-06-14 04:52:37 +00:00
Remo Senekowitsch
250814f530
fix(resolve): prefer official binaries (#1741)
* prefer official binaries with secondary targets over quickinstall binaries

* chore(test): add jj-cli to live e2e test
2024-06-13 18:26:30 +10:00
Jiahao XU
1dbd2460a3
Support private github repository (#1690)
* Refactor: Create new crate binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix CI lint warnings

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `just check`: Rm deleted features from `cargo-hack` check

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract  new mod error

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Ret artifact url in `has_release_artifact`

So that we can use it to download from private repositories.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Move `test_graph_ql_error_type` to mod `error`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `cargo test` in `binstalk-git-repo-api``

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unnecessary import in mod `error::test`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename mod `request`` to `release_artifacts`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl draft version of fetching repo info

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Move `HasReleaseArtifacts` failure variants into `GhApiError`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Use `GhRepo` in `GhRelease`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Return `'static` future

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Make sure `'static` Future is returned

To make it easier to create generic function

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add logging to unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract new fn `GhApiClient::do_fetch`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rm unused `percent_encode_http_url_path`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `cargo test` run on CI

`cargo test` run all tests in one process.

As such, `set_global_default` would fail on the second call.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Optimize `GhApiClient::do_fetch`: Avoid unnecessary restful API call

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Rm param `auth_token` for restful API fn

which is always set to `None`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit test for `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor testing: Parameter-ize testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelise `test_get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Create parameter-ised `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelize `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: `gh_api_client::test::create_client` shall not be `async`

as there is no `.await` in it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Return `Url` in `GhApiClient::has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::download_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unused deps added to binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new API `GhApiClient::remote_client`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhApiClient::has_gh_token`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhRepo::try_extract_from_url`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename `ReleaseArtifactUrl` to `GhReleaseArtifactUrl`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new fn `Download::with_data_verifier`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* feature: Support private repository

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add e2e-test/private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `launch_baseline_find_tasks`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix test failure: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Temporarily enable debug output for e2e-test-private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `get_repo_info`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Improve `debug!` logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debugging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Apply suggestions from code review

* Fix compilation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix cargo fmt

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add crate binstalk-git-repo-api to release-pr.yml

* Update crates/binstalk-git-repo-api/Cargo.toml

* Apply suggestions from code review

* Update crates/binstalk/Cargo.toml

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-06-10 06:02:12 +00:00
Ilkka Poutanen
6dd4a155fb
Typo fix (#1631) 2024-02-23 11:21:05 +00:00
FlareFlo
857a2caf34
Fix missing spaces in error log (#1577) 2024-01-19 02:20:37 +00:00
Félix Saparelli
680accd0d3
Check quickinstall signatures (#1382)
* Check quickinstall signatures

* e2e test

* We can no longer check presence of signing section early

* Improve UI a little
2023-09-23 08:35:49 +00:00
Félix Saparelli
32beba507b
Initial signing support (#1345)
* Add CLI options

* Add manifest types

* Thread signature policy through to fetchers

* Thread signing section through from metadata

* Implement signing validation

* Clippy

* Attempt testing

* Yes and

* Why

* fmt

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/gh_crate_meta.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* fixes

* Finish feature

* Document

* Include all fields in the signing.file template

* Readme document

* Review fixes

* Fail on non-utf8 sig

* Thank goodness for tests

* Run test in ci

* Add rsign2 commands

* Log utf8 error

* Update e2e-tests/signing.sh

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests/signing.sh` MacOS CI failure

Move the tls cert creation into `signing.sh` and sleep for 10s to wait
for https server to start.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor e2e-tests-signing files

 - Use a tempdir generated by `mktemp` for all certificates-related
   files
 - Put other checked-in files into `e2e-tests/signing`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` connection err in MacOS CI

Wait for server to start up by trying to connect to it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` passing `-subj` to `openssl` on Windows

Use single quote instead of double quote to avoid automatic expansion
from bash

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` waiting for server to startup

Remove `timeout` since it is not supported on MacOS.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Try to fix windows CI by setting `MSYS_NO_PATHCONV=1` on `openssl` cmds

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` on windows

By using double `//` for the value passed to option `-subj`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed infinite loop in `signing/wait-for-server` on Windows

Pass `--ssl-revoke-best-effort` to prevent schannel from checking ssl
revocation status.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add cap on retry attempt in `signing/wait-for-server.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Let `singing/server.py` print output to stderr

so that we can see the error message there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `signing/server.py` on MacOS CI

use `python3` since macos-latest still has python2 installed and
`python` is a symlink to `python2` there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-23 04:02:56 +00:00
Jiahao XU
e5567ae526
Fix clippy lint and format (#1328)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-25 08:53:50 +00:00
Jiahao XU
0f7828569a
Add error logging if achieve doesn't contain bin required (#1298)
* Add `error` logging if achieve doesn't contain bin required

So that users know that this is an upstream issue and will report to
upstream.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk/src/ops/resolve.rs

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-16 23:13:21 +00:00
Jiahao XU
0c5a65fb35
Refactor: Extract new crate binstalk-bins (#1294)
To reduce `binstalk` codegen and enable reuse of it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-15 21:48:42 +00:00
Jiahao XU
76c72469eb
Refactor: Extract new crate binstalk-fetchers (#1291)
To reduce `binstalk` codegen and enable better reuse.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-14 03:20:34 +00:00
Jiahao XU
fbed317df5
feat: git::Repository cancellation support (#1288)
feat: `git::Repository` support cancellation.

To make sure users can cancel git operation via signal, e.g. when the
git operation fail or users no longer want to install.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-12 14:18:02 +00:00
Jiahao XU
8ff13c1b36
Refactor: Extract cargo_toml_workspace as a new crate (#1287)
To reduce codegen time of `binstalk` and also enable others to reuse
this crate.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-12 12:05:10 +00:00
Jiahao XU
aeacebcf83
feat: Support passing workspace to --manifest-path (#1246)
Previously it will load the root `Cargo.toml` and treat it as the
manifest for the crate, now it will check its `package.name` and would
search for the workspace if the `package.name` does not match the crate
name.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-03 21:12:06 +00:00
Jiahao XU
01a87ac606
feat: Impl support for alternative registries (#1184)
Fixed #1168

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-30 03:52:40 +00:00
Jiahao XU
ca00cbaccc
feat: Support install directly from git repo (#1162)
Fixed #3

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-24 01:01:31 +00:00
Jiahao XU
d8419ea5a2
feat: Add more variables for template (#1160)
Fixed #775

 - Add dep target-lexicon v0.12.7
 - Add `target-{family, arch, libc, vendor}` to
   `package.metadata.binstall`.

For `{universal, universal2}-apple-darwin`, the `target-arch` is set to
`universal`.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-20 22:05:13 +00:00
Jiahao XU
a27d5aebf6
Use leon for template in binstalk & detect malformed pkg-url/pkg-fmt early (#933)
Fixed #851

* Add new dep leon to crate binstalk
* Add new variant `BinstallError::Template{Parse, Render}Error`
* Use `leon::Template` in mod `bins`
* Use `leon::Template` in mod `fetchers::gh_crate_meta`
* Refactor mod `bins`: Rm unused associated fn & fields from `Context`
* Simplify unit testing in mod `fetchers::gh_crate_meta`
* Rm soft-deprecated field `fetchers::gh_crate_meta::Context::format`
  and change the `match` to resolve `archive` => `self.archive_format`.
* Make macro_rules `leon::template!` far easier to use
* Construct `leon::Template<'_>` as constant in `gh_crate_meta::hosting`
* Simplify `leon::Values` trait
   Change its method `get_value` signature to
   
   ```rust
       fn get_value(&self, key: &str) -> Option<Cow<'_, str>>;
   ```
   
   Now, `ValuesFn` also accepts non-`'static` function, but now
   `leon::Values` is only implemented for `&ValuesFn<F>` now.
   
   This makes it a little bit more cumbersome to use but I think it's a
   reasonable drawback.
* Rm `Send` bound req from `ValuesFn`
* Impl new fn `leon::Template::cast`
   for casting `Template<'s>` to `Template<'t>` where `'s: 't`
* Rename `leon::Template::has_keys` => `has_any_of_keys`
* Make checking whether format related keys are present more robust
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Skip checking all fmt ext
   if none of the format related keys ("format", "archive-format",
   "archive-suffix") are present.
* Only ret `.exe` in `PkgFmt::extensions` on windows
   by adding a new param `is_windows: bool`
* Improve debug msg in `GhCrateMeta::fetch_and_extract`
* Add warnings to `GhCrateMeta::find`
* Rm dep tinytemplate
* `impl<'s, 'rhs: 's> ops::AddAssign<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::AddAssign<Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<&Item<'item>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::Add<Template<'rhs>> for Template<'s>` (improved existing `Add` impl)
* `impl<'s, 'rhs: 's> ops::Add<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<&Item<'item>> for Template<'s>`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-03-26 16:11:10 +11:00
Jiahao XU
cc78ff3b90
Fix fetcher::QuickInstallstats report sending (#918)
Fixed https://github.com/cargo-bins/cargo-quickinstall/issues/195

 - Fix `fetchers::QuickInstall`: Stop sending stats for `universal-apple-darwin`
   since quickinstall only supports targets officially supports by rust.
 - Only send stats report to quickinstall if the `Fetcher::find` is `.await`ed on
   
   This prevents stats report to be sent for cases where the `QuickInstall` fetcher
   is actually unused, e.g. resolved to `GhCrateMeta` or other `QuickInstall` fetcher
   with different target.
   
   This also reduces amount of http requests created in background.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-17 21:05:19 +11:00
Jiahao XU
d118fa8fdb
Obey crates.io crawler policy: Only make one request per sec (#908)
Fixed https://github.com/taiki-e/install-action/issues/89

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-14 17:11:22 +11:00
Jiahao XU
89a47cbec9
Fix extract_tar_based_stream: Normalize path before adding them to ExtractedFiles (#901)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-13 21:27:50 +11:00
Jiahao XU
9c7da6a179
Return a list of files written to disk in binstalk_downloader::download::Download::and_extract (#856)
to avoid collecting extracted files from disk again in resolution stage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-03 23:31:27 +11:00
Jiahao XU
8eee318ccd
Replace dep crates_io_api with in-house solution (#846)
It also uses `max_stable_version` in the json downloaded from https://crates.io/api/v1/crates/$name if possible, which is equivalent to the version shown on https://crates.io/crates/$name .

 - Add new feat `json` to `binstalk-downloader`
 - Impl new async fn `Response::json`
 - use `Response::json` in `GhApiClient` impl
 - Mark all err types in binstalk-downloader as `non_exhaustive`
 - Ret `remote::Error` in `remote::Certificate::{from_pem, from_der}` instead of `ReqwestError`.
 - Refactor `BinstallError`: Merge variant `Unzip`, `Reqwest` & `Http`
    into one variant `Download`.
 - Manually download and parse json from httos://crates.io/api/v1
 - Remove unused deps `crates_io_api`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 18:25:34 +11:00
Jiahao XU
599bcaf333
Impl GhApiClient and use it in cargo-binstall to speedup resolution process (#832)
Fixed #776

 - Add new feature gh-api-client to binstalk-downloader
 - Impl new type `binstalk_downloader::remote::{RequestBuilder, Response}`
 - Impl `binstalk_downloader::gh_api_client::GhApiClient`, exposed if `cfg(feature = "gh-api-client")` and add e2e and unit tests for it
 - Use `binstalk_downloader::gh_api_client::GhApiClient` to speedup `cargo-binstall`
 - Add new option `--github-token` to supply the token for GitHub restful API, or read from env variable `GITHUB_TOKEN` if not present.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 12:04:22 +11:00
Jiahao XU
2e118b3044
Optimize GhCrateMeta::find: Cache url resolution result (#810)
So that we don't have to create multiple HEAD/GET http request to get
the final, redirected url.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-16 14:51:47 +13:00
Jiahao XU
270bf08a24
Optimize use of tinytemplate (#647)
* Optimize `GhCrateMeta::find`: Cache compiled template `pkg_url`
* Optimize `collect_bin_files`: Cache compiled template `bin_dir`
* Refactor: Improve readability of `gh_crate_meta`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Dedup urls before checking
* Optimize `BinFile::new`: Avoid one heap alloc when creating `dest`
* Improve `warn!` msg in `GhCrateMeta::launch_baseline_find_tasks`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-26 17:33:49 +13:00
Jiahao XU
7b34178bcc
Optimize resolve_inner: Avoid VersionReq::clone (#656)
* Add new dep maybe-owned v0.3.4 to binstalk
* Optimize `resolve_inner`: Avoid `VersionReq::clone`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-06 13:47:45 +13:00
Jiahao XU
6bdb26930e
Refactor, improvements and bugfix for resolution and installation process (#619)
* Refactor: Extract new mod `ops::resolve::resolution`
* Refactor: Extract new type `ResolutionFetch`
* Refactor: Extract new type `ResolutionSource`
* Improve `Resolution::print`: Provides more details on which crate
   is the resolution for.
* Make `Resolution::print` a pub fn
* Refactor: Extract new fn `ResolutionFetch::install`
* Refactor: Extract new async fn `ResolutionSource::install`
* Optimize `ResolutionSource::install` avoiding `OsStr::to_string_lossy`.
* Add new dep command-group v2.0.0 to binstalk with feat with-tokio
* Fix `ResolutionSource::install`: Use `AsyncCommandGroup::group_spawn`
   instead of `tokio::process::Command::spawn` to ensure all the processes
   spanwed by the `cargo` process can be terminated with one signal sent to
   the `cargo` process.
* Fix printing resolution: Make sure they are printed without interleaving
* Refactor `entry::install_crates`
* Improve dry-run output for `ResolutionSource::install`
* Refactor: Extract new fn `ResolutionFetch::print`
* Refactor: Extract new fn `ResolutionSource::print`
* Optimize `Resolution`: Box unit variant `Fetch`
* Improve formatting of `tokio::process::Command`
   Prints out sth like `cargo install ...` instead of the `Debug::fmt`.
* Improve dry-run output

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-28 00:18:07 +00:00
Jiahao XU
db45f2fb7f
Refactor binstalk-downloader APIs: Remove cancellation_future plus optimizations (#591)
- Refactor: Mv fn `utils::asyncify` into mod `utils`
 - Improve err msg for task failure in `utils::asyncify`
 - Make sure `asyncify` always returns the same annoymous type
   that implements `Future` if the `T` is same.
 - Rewrite `extract_bin` to avoid `block_in_place`
   support cancellation by dropping
 - Rm unused dep scopeguard from binstalk-downloader
 - Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
 - Unbox `extract_future` in `async_extracter::extract_zip`
 - Refactor `Download` API: Remove `CancellationFuture` as param

   since all futures returned by `Download::and_*` does not call
   `block_in_place`, so they can be cancelled by drop instead of using this
   cumbersome hack.
 - Fix exports from mod `async_tar_visitor`
 - Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
 - Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
   and rm fn `wait_on_cancellation_signal_inner`
 - Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
 - Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
   Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
 - Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
 - Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
 - Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
 - Optimize `extract_zip_entry`: Spawn only one blocking task per fn call

   by using a mspc queue for the data to be written to the `outfile`.

   This would improve efficiency as using `tokio::fs::File` is expensive:
   It spawns a new blocking task, which needs one heap allocation and then
   pushed to a mpmc queue, and then wait for it to be done on every loop.

   This also fix a race condition where the unix permission is set before
   the whole file is written, which might be used by attackers.
 - Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
   To avoid frequent allocation and deallocation.
 - Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`

   Performs the reserve before sending the buf over mpsc queue to
   increase the possibility of reusing the previous allocation.

   NOTE: `BytesMut` only reuses the previous allocation if it is the
   only one holds the reference to it, which is either on the first
   allocation or all the `Bytes` in the mpsc queue has been consumed,
   written to the file and dropped.

   Since reading from entry would have to wait for external file I/O,
   this would give the blocking thread some time to flush `Bytes`
   out.
 - Disable unused feature fs of dep tokio

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-12 03:15:30 +00:00
Jiahao XU
8a5577297e
Optimize BinstallError: Reduce size from 40B to 32B (#581)
* Optimize `BinstallError::CratesIoApi`: Extract new type `errors::CratesIoApiError` and box it
   Also improve `<CratesIoApiError as Display>::fmt` impl.
* Optimize `BinstallError::SubProcess`: Use `Box<str>` instead of `String`
* Optimize `BinstallError::CargoManifest`: Box `CargoTomlError`
* Optimize `BinstallError::VersionParse`: Extract `VersionParseError` and box it
   Also improve `<VersionParseError as Display>::fmt` impl.
* Optimize `BinstallError::CrateContext`: Extract `CrateContextError` and box it in
* Optimize `install_from_source`: Only format `cmd` on err

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-01 20:05:00 +13:00
Jiahao XU
ab3e47c42b
Optimize Fetcher::find and fix quickinstall reporting (#579)
* Optimize `Fetcher::find`: Make it non-async to avoid boxing
   and return `AutoAbortJoinHandle<...>` instead.
   
   Since spawning a new task in tokio box the future anyway, boxing the
   returned future again in `Fetcher::find` merely adds unnecessary
   overheads.
* Optimize `QuickInstall::report`: Make it async fn instead of ret `JoinHandle`
   
   This provides several benefits:
    - On debug build, no task will be spawned
    - The calls to `self.stats_url()` can be evaluated in parallel.
    - The task now returns `()` so that the task spawned is smaller.
* Fix `QuickInstall::find`: `warn!` if quickinstall report fails

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-30 16:15:22 +13:00
Jiahao XU
66abf1b8c1
More mionr optimizations (#553)
* Optimize `BinFile::preview_bin`: Use `Path::display` instead of `to_string_lossy` to avoid allocation.
* Refactor: Rm useless `AsRef::as_ref` call
* Optimize `GhCrateMeta::find`: Reduce fut size by converting `Url` to `String`
   `Url` is much larger than `String`.
* Refactor `PackageInfo::resolve`: Destruct `Meta::binstall`
* Optimize `resolve::load_manifest_path`: Avoid allocation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-22 19:35:16 +13:00
Jiahao XU
bdb4b2070d
Minor optimization (#544)
* Optimization: Rm `debug!` in `find_version`
   printing all version iterated obviously doesn't help much in debugging
   in the problem but rather just confusing.
   
   Also this makes it hard for the compiler to optimize the iterators.
* Use let-else in `ManifestVisitor`
* Optimize `BinFile::preview_{bin, link}` for zero-copy
   Return `impl Display` that lazily format instead of allocating a `String`
* Optimize `infer_bin_dir_template`: Generate dir lazily
* Optimize `find_version`: Lazily clone `version_req` only on err
* Refactor `find_version`: Use `bool::then_some`
* Add dep either v1.8.0 to binstalk
* Optimize `GhCrateMeta::find`: Avoid cloning and `Vec` creation
   by using `Either`
* Optimize `ops::install::install_from_package`: Make it a regular fn
   instead of async fn since it does not `.await` on any async fn.
* Optimize `QuickInstall`: Rm field `target`
   since `Arc<Data>` already contains that field.
* Optimize `Fetcher`s: Extract new struct `TargetData`
   so that `Data` can be shared by all fetchers, regardless of the target.
* Optimize `QuickInstall`: Rm unused field `data`
* Optimize `Resolution::print`: Replace branching with conditional move

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-21 10:32:46 +13:00
Jiahao XU
50b6e62164
Minor refactor and optimization (#543)
* Avoid potential panicking in `args::parse` by using `Vec::get` instead of indexing
* Refactor: Simplify `opts::{resolve, install}` API
   Many parameters can be shared and put into `opts::Options` intead and
   that would also avoid a few `Arc<Path>`.
* Optimize `get_install_path`: Avoid cloning `install_path`
* Optimize `LazyJobserverClient`: Un`Arc` & remove `Clone` impl
   to avoid additional boxing
* Optimize `find_version`: Avoid cloning `semver::Version`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`
   return `impl Iterator<Item = impl Future<Output = ...>>`
   instead of `impl Iterator<Item = AutoAbortJoinHandle<...>>`
   to avoid unnecessary spawning.
   
   Each task spawned has to be boxed and then polled by tokio runtime.
   They might also be moved.
   
   While they increase parallelism, spawning these futures does not justify
   the costs because:
    - Each `Future` only calls `remote_exists`
    - Each `remote_exists` call send requests to the same domain, which is
      likely to share the same http2 connection.
      Since the conn is shared anyway, spawning does not speedup anything
      but merely add communication overhead.
    - Plus the tokio runtime spawning cost
* Optimize `install_crates`: Destruct `Args` before any `.await` point
   to reduce size of the future
* Refactor `logging`: Replace param `arg` with `log_level` & `json_output`
   to avoid dep on `Args`
* Add dep strum & strum_macros to crates/bin
* Derive `strum_macros::EnumCount` for `Strategy`
* Optimize strategies parsing in `install_crates`
* Fix panic in `install_crates` when `Compile` is not the last strategy specified
* Optimize: Take `Vec<Self>` instead of slice in `CrateName::dedup`
* Refactor: Extract new fn `compute_resolvers`
* Refactor: Extract new fn `compute_paths_and_load_manifests`
* Refactor: Extract new fn `filter_out_installed_crates`
* Reorder `install_crates`: Only run target detection if args are valid
   and there are some crates to be installed.
* Optimize `filter_out_installed_crates`: Avoid allocation
   by returning an `Iterator`
* Fix user_agent of `remote::Client`: Let user specify it
* Refactor: Replace `UIThread` with `ui::confirm`
   which is much simpler.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-19 20:00:27 +13:00
Jiahao XU
325cb5cc19
Optimizations plus bug fix (#541)
* Optimize `Download::and_extract`: Avoid dup monomorphization
* Increase buffer size for binstall_crates_v1 to `4096 * 5`
* Optimize `opts::resolve`: Avoid unnecessary `clone`s
* Fix reserve in `opts::resolve`: Do not over-reserve
* Rename field `opts::Options::resolver` => `resolvers`
* Refactor: Extract new type `resolve::PackageInfo`
    - which makes `opts::resolve_inner` easier to understand
    - reduce number of parameters required for `download_extract_and_verify` and
      `collect_bin_files`
    - reducing size of future returned by `opts::resolve_inner` by dropping
      `cargo_toml::{Manifest, Package}` as early as possible since
      `Manifest` is 3000 Bytes large while `Package` is 600 Bytes large.
* Optimize `fetchers::Data`: Use `CompactString` for field name & version
   since they are usually small enough to fit in inlined version of
   `CompactString`.
* Optimize `gh_crate_meta`: Avoid unnecessary allocation
   in `RepositoryHost::get_default_pkg_url_template`.
* Refacator: Use `Itertools::cartesian_product` in `apply_filenames_to_paths`
* Optimize `ops::resolve`: Avoid unnecessary `clone` & reduce future size
   by calling `fetcher.target_meta()` to obtain final metadata after
   downloaded and extracted the binaries.
* Optimize `ops::resolve`: Avoid unnecessary allocation
   in `download_extract_and_verify`: Replace `Itertools::join` with
   `Itertools::format` to avoid allocating the string.
* Fix disabling cargo-install fallback
* Simplify `BinFile::from_product`: Takes `&str` instead of `&Product`
   since we only need `product.name`
* Rename `BinFile::from_product` => `BinFile::new`
* Refactor: Create newtype `ops::resolve::Bin`
   so that we don't need to `unwrap()` on `Product::name`
   and reduce memory usage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 23:59:35 +00:00
Jiahao XU
4e875874b6
Replace use of log with tracing (#527)
Resolved #522

* Add dep tracing v0.1.37 to binstalk
* Use `tracing` instead of `log` for logging in binstalk
* Rm dev dep `env_logger` since `log` is no longer used
* Rm unused dep `log` from binstalk

* Replace use of `log` with `tracing` in crates/bin
* Enable feat std of dep log in crates/bin

* Add dep tracing v0.1.37 to binstalk-downloader
* Replace use of `log` with `tracing` in binstalk-downloader
* Rm unused dep `log` from binstalk-downlaoder


* Wrap `ops::{install, resolve}` in `tracing::instrument`
* Wrap `Download::and_{extract, visit_tar}` in `instrument`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-13 21:45:37 +13:00
Jiahao XU
9e80cf0700
Feature: Impl opt-out options (#510)
Fixed #136

* Impl opt-out optioins in binstalk
* Replace field `Options::{gh_crate, quickinstall}_fetcher` with `resolver`
  which can determine order of resolver used.
* Add new field `Args::{disable_}strategies`
* Add new variant `BinstallError::InvalidStrategies`
* Add variant `BinstallError::NoFallbackToCargoInstall`
* Add code for supporting strategies in mod entry
* Test `--disable-strategies` in `tests.sh`
* Test for `--strategies` in `tests.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-11 05:24:46 +00:00
Jiahao XU
aa6012baae
Make extraction cancellable for bin and tar based formats (#481)
Extraction wasn't cancellable by `cancel_on_user_sig_term` used in `entry` since it calls `block_in_place`.

This PR adds cancellation support to it by adding a `static` variable `OnceCell` to `wait_on_cancellation_signal` so that once it returns `Ok(())`, all other calls to it after that point also returns `Ok(())` immediately.

`StreamReadable`, which is used in cancellation process, then stores a boxed future of `wait_on_cancellation_signal` and polled it in `BufReader::fill_buf`.

Note that for zip, the extraction process takes `File` instead of `StreamReadable` due to `io::Seek` requirement, so it cancelling during extraction for zip is still not possible.

This PR also optimized `extract_bin` and `extract_zip` by using `StreamReadable::copy` introduced to this PR instead of `io::copy`, which allocates an internal buffer on stack, which imposes extra copy.

It also fixed `StreamReadable::fill_buf` by ensuring that empty buffer is only returned on eof.

* Make `wait_on_cancellation_signal` pub
* Enable feature `parking_lot` of dep tokio
* Mod `wait_on_cancellation_signal`: Use `OnceCell` internally
   to archive the effect that once call to it return `Ok(())`, all calls to
   it after that also returns `Ok(())`.
* Impl `From<BinstallError>` for `io::Error`
* Impl cancellation on user signal in `StreamReadable`
* Fix err msg when cancelling during extraction in `ops::resolve`
* Optimize: Impl & use `StreamReadable::copy`
   which is same as `io::copy` but does not allocate any internal buffer
   since `StreamReadable` is buffered.
* Fix `next_stream`: Return non-empty bytes on `Ok(Some(bytes))`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-13 13:31:13 +13:00
Jiahao XU
76bc030f90
Disable tcp_nodelay for reqwest::Client and add rate limiting for https requests (#458) 2022-10-07 15:51:34 +11:00
Jiahao XU
3421403e75
Refactor and Optimizations (#459)
* Refactor: Avoid parsing `package.version()` twice in `ops::resolve` and `ops::install`
* Optimize Resolution: Replace `Package<Meta>` with two `CompactStrings`: `name` and `version`
* Use `CompactString` for `BinstallError::CratesIoApi::crate_name`
* Use `CompactString` for `BinstallError::VersionParse::v`
* Use `CompactString` for `BinstallError::VersionReq::req`
* Use `CompactString` for `BinstallError::VersionUnavailable::crate_name`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-06 04:19:12 +13:00
dependabot[bot]
405e41eda4
Bump cargo_toml from 0.12.2 to 0.12.4 (#455)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-04 02:18:54 +13:00
Jiahao XU
abd50a72ec
Fix skipping optional dep: Skip fetcher if they provide 0 bin (#446)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-29 03:51:11 +13:00
Jiahao XU
f482e362ba
Make binaries with required-features optional (#438)
* Make binaries with `required-features` optional so that crates like `sccache` would work as expected.
* Fix `infer_bin_dir_template`: concat with `data.bin_path` introduced in #417 
* Always `chmod +x $bin` on unix in case the archive (e.g. `sccache`) did not set the executable bit of fmt == Bin.
* CI: Install `sccache` but do not run it since it requires cargo env to be ready

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 13:46:55 +10:00
Jiahao XU
c15903684f
Feature/fix no symlink (#432)
* Refactor `BinFile::from_product`: Simplify logic flow.
* Fix `--no-symlink` behavior

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 02:32:22 +00:00
Jiahao XU
b8d2184ee9
Optimize applying PkgOverrides into PkgMeta (#429)
* Refactor: Rm `PkgMeta::clone_without_overrides`
* Impl new fn `PkgMeta::merge_overrides` to avoid unnecessary clone

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-26 19:05:14 +13:00