Commit graph

68 commits

Author SHA1 Message Date
David Laban
c8dec953cc
report to new stats server (with status) (#1912)
* docstrings for things that I was curious about

* report to new stats server, with status

* Arc<Self> needs a clone

* fix stats url in docs

* fix stats url handling
2024-09-11 08:15:15 +00:00
Jiahao XU
a53d51c083
Send telemetry report to quickinstall if no pre-built is found (#1905)
* Send telemetry report to quickinstall if no pre-built is found

quickinstall telemtry is overloaded cargo-bins/cargo-quickinstall#268, it only needs report when the pre-built is not available in upstream.

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Fix resolve.rs

And optimize sending telemtry: Remove unnecessary `<Arc as Clone>::clone`

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Fix fmt of resolve.rs

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

---------

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
2024-09-05 14:43:38 +00:00
Jiahao XU
52f2db4f57
Add --maximum-resolution-timeout (#1862)
* Add --maximum-resolution-timeout

Fixed #1823

* Display default value for `--maximum-resolution-timeout` in help
2024-08-04 02:13:18 +00:00
Jiahao XU
6809601273
Add new option to --disable-telemetry to disable quickinstall statistics collection (#1831)
* Add new option `--disable-quick-install-stats`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Send quickinstall stats in parallel

This commit also make sure'the stats is always sent for
each fetcher, regardless of whether it is picked or not,
to make sure that the quick-install stats collection gets
the full infomration of possible targets.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename option to `--disable-telemetry`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update args.rs

Co-authored-by: Félix Saparelli <felix@passcod.name>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Update args.rs

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2024-07-26 06:03:38 +00:00
Jiahao XU
3f29fbe83a
Feature: SupportdDisable of strategies for crate using Cargo.toml (#1828)
* Refactor: Move `Strategy` to `binstalk-types`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add serialisation test for `Strategy`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add support to disable strategies via crate `Cargo.toml`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add e2e-test

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `Cargo.toml` disabled strategy checking for compile strategy

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Optimize `resolve_inner`: Cache meta override

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add compile-time length checking for `Strategy`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* More optimization

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix order of override: cli options alwayus takes precedence

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add missing manifest for e2e-test

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-07-23 14:05:22 +00:00
Jiahao XU
eba07fb147
Fix v1 manifest format for git and local path (#1821)
* Bump simple-git to v0.2.10

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix v1 manifest format for git and local path

Fixed #1815

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix v1 format for custom registry

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unused functions

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix compilation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update git.sh

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Fixed git.sh for windows

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* fixx git.sh for win

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* Update git.sh

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* fix git.sh

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
2024-07-17 05:34:12 +00:00
Jiahao XU
ac7bac651d
Run artifact discover in sequential instead of in parallel (#1796)
* Perform artifact discovery in sequential

Run different `fetcher.find()` in sequential

* FuturesResolver: Fallback to other future if one error

* Fix typo

* Apply cargo fmt

* Parallelise `<QuickInstall as Fetcher>::find`

Check for signature in parallel to the package

* Download signature in `<QuickInstall as Fetcher>::find`

So that the signature download can be done in parallel.

* Bump msrv for binstalk-fetchers to 1.70

* Update crates/binstalk-fetchers/src/futures_resolver.rs

Co-authored-by: Félix Saparelli <felix@passcod.name>
Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>

* cargo fmt

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <30436523+NobodyXu@users.noreply.github.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2024-06-23 10:42:03 +00:00
Jiahao XU
fff6aa8122
Improve use of github token (#1769)
* Add new dep zeroize

* Use Zeroizing to avoid leaking the token

* Optimize gh-auth-token

Spawn it as a task, and only await it
when using GhApiClient

* Fix binstalk-git-repo-api unit tests
2024-06-15 05:42:09 +00:00
Tomas Olvecky
dfa230f039
Use binary name when searching for artifacts (#1747)
* Use binary name when searching for artifacts

When there is a single binary declared in the manifest and
it differs from the package name, add it to the list of handles
used for pre-built artifact fetching.

* Simplify `binary_name` assignment with a `match`

* Add e2e test

* Only attempt to use the binary name with `GhCrateMeta` fetcher

* Avoid too much over-allocating.

Technically it should also check if gh-crate-meta resolver is enabled, but it is unlikely for it to be disabled and overallocating for extra n-target should be fine, it is an improvement over doubling the space allocated if the binary_name is Some.

* Fix fmt in crates/binstalk/src/ops/resolve.rs

---------

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-06-14 04:52:37 +00:00
Remo Senekowitsch
250814f530
fix(resolve): prefer official binaries (#1741)
* prefer official binaries with secondary targets over quickinstall binaries

* chore(test): add jj-cli to live e2e test
2024-06-13 18:26:30 +10:00
Jiahao XU
1dbd2460a3
Support private github repository (#1690)
* Refactor: Create new crate binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix CI lint warnings

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `just check`: Rm deleted features from `cargo-hack` check

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract  new mod error

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Ret artifact url in `has_release_artifact`

So that we can use it to download from private repositories.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Move `test_graph_ql_error_type` to mod `error`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `cargo test` in `binstalk-git-repo-api``

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unnecessary import in mod `error::test`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename mod `request`` to `release_artifacts`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl draft version of fetching repo info

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Move `HasReleaseArtifacts` failure variants into `GhApiError`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Use `GhRepo` in `GhRelease`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Return `'static` future

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Make sure `'static` Future is returned

To make it easier to create generic function

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add logging to unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract new fn `GhApiClient::do_fetch`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rm unused `percent_encode_http_url_path`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `cargo test` run on CI

`cargo test` run all tests in one process.

As such, `set_global_default` would fail on the second call.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Optimize `GhApiClient::do_fetch`: Avoid unnecessary restful API call

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Rm param `auth_token` for restful API fn

which is always set to `None`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit test for `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor testing: Parameter-ize testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelise `test_get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Create parameter-ised `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelize `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: `gh_api_client::test::create_client` shall not be `async`

as there is no `.await` in it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Return `Url` in `GhApiClient::has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::download_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unused deps added to binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new API `GhApiClient::remote_client`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhApiClient::has_gh_token`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhRepo::try_extract_from_url`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename `ReleaseArtifactUrl` to `GhReleaseArtifactUrl`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new fn `Download::with_data_verifier`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* feature: Support private repository

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add e2e-test/private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `launch_baseline_find_tasks`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix test failure: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Temporarily enable debug output for e2e-test-private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `get_repo_info`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Improve `debug!` logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debugging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Apply suggestions from code review

* Fix compilation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix cargo fmt

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add crate binstalk-git-repo-api to release-pr.yml

* Update crates/binstalk-git-repo-api/Cargo.toml

* Apply suggestions from code review

* Update crates/binstalk/Cargo.toml

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-06-10 06:02:12 +00:00
Ilkka Poutanen
6dd4a155fb
Typo fix (#1631) 2024-02-23 11:21:05 +00:00
FlareFlo
857a2caf34
Fix missing spaces in error log (#1577) 2024-01-19 02:20:37 +00:00
Jiahao XU
c08b8d232a
feat: Impl new option --continue-on-failure (#1559)
* feat: Impl new option `--continue-on-failure`

Resolve #1548

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new e2e-tests continue-on-failure

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rm dup line ion `e2e-tests/live.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix shellcheck

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `BinstallError::crate_errors` if `errors.len()` is 1

In that case, it should return `Some(Self::CrateContext(_))` instead of
`Some(Self::Errors(_))`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more tests to `e2e-tests/continue-on-failure.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Propagate crate errors on `confirm()` err

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Test having two errors in `e2e-tests/continue-on-failure.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2024-01-08 12:09:45 +00:00
Jiahao XU
8e08c65946
Fix fallback to cargo-install: Use the default target unless user override (#1430)
* Fix fallback to `cargo-install`: Use the default target

which is guaranteed to work, instead of providing a custom `--target`
from `detect-targets`.

The one from `detect-targets` might report musl even if the `rustc`
reports gnu, since OSes like NixOS could use a custom glibc location
that prevent glibc binaries built for other OSes be run for them.

However, if we are compiling the crate and `rustc` defaults to glibc,
then it should be guaranteed to work.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl fn `DesiredTargets::get_initialized`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix fallback to `cargo-install`: Use user-provided override targets

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-30 00:22:21 +00:00
Félix Saparelli
680accd0d3
Check quickinstall signatures (#1382)
* Check quickinstall signatures

* e2e test

* We can no longer check presence of signing section early

* Improve UI a little
2023-09-23 08:35:49 +00:00
Félix Saparelli
32beba507b
Initial signing support (#1345)
* Add CLI options

* Add manifest types

* Thread signature policy through to fetchers

* Thread signing section through from metadata

* Implement signing validation

* Clippy

* Attempt testing

* Yes and

* Why

* fmt

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/gh_crate_meta.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* fixes

* Finish feature

* Document

* Include all fields in the signing.file template

* Readme document

* Review fixes

* Fail on non-utf8 sig

* Thank goodness for tests

* Run test in ci

* Add rsign2 commands

* Log utf8 error

* Update e2e-tests/signing.sh

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests/signing.sh` MacOS CI failure

Move the tls cert creation into `signing.sh` and sleep for 10s to wait
for https server to start.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor e2e-tests-signing files

 - Use a tempdir generated by `mktemp` for all certificates-related
   files
 - Put other checked-in files into `e2e-tests/signing`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` connection err in MacOS CI

Wait for server to start up by trying to connect to it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` passing `-subj` to `openssl` on Windows

Use single quote instead of double quote to avoid automatic expansion
from bash

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` waiting for server to startup

Remove `timeout` since it is not supported on MacOS.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Try to fix windows CI by setting `MSYS_NO_PATHCONV=1` on `openssl` cmds

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` on windows

By using double `//` for the value passed to option `-subj`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed infinite loop in `signing/wait-for-server` on Windows

Pass `--ssl-revoke-best-effort` to prevent schannel from checking ssl
revocation status.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add cap on retry attempt in `signing/wait-for-server.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Let `singing/server.py` print output to stderr

so that we can see the error message there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `signing/server.py` on MacOS CI

use `python3` since macos-latest still has python2 installed and
`python` is a symlink to `python2` there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-23 04:02:56 +00:00
Jiahao XU
3c5641610a
feat: Improve resolution logging and verbose logging (#1341)
* feat: Improve resolution logging

Fixed #1336

Log target of the pre-built binaries which will be installed.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix args parsing: `quiet` & `verbose` cannot be set at the same time

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* feat: Implies `--log-level debug` if `--verbose` is set

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-09-02 09:38:24 +00:00
Jiahao XU
e5567ae526
Fix clippy lint and format (#1328)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-25 08:53:50 +00:00
Jiahao XU
0f7828569a
Add error logging if achieve doesn't contain bin required (#1298)
* Add `error` logging if achieve doesn't contain bin required

So that users know that this is an upstream issue and will report to
upstream.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk/src/ops/resolve.rs

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-16 23:13:21 +00:00
Jiahao XU
0c5a65fb35
Refactor: Extract new crate binstalk-bins (#1294)
To reduce `binstalk` codegen and enable reuse of it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-15 21:48:42 +00:00
Jiahao XU
76c72469eb
Refactor: Extract new crate binstalk-fetchers (#1291)
To reduce `binstalk` codegen and enable better reuse.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-14 03:20:34 +00:00
Jiahao XU
fbed317df5
feat: git::Repository cancellation support (#1288)
feat: `git::Repository` support cancellation.

To make sure users can cancel git operation via signal, e.g. when the
git operation fail or users no longer want to install.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-12 14:18:02 +00:00
Jiahao XU
8ff13c1b36
Refactor: Extract cargo_toml_workspace as a new crate (#1287)
To reduce codegen time of `binstalk` and also enable others to reuse
this crate.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-12 12:05:10 +00:00
Jiahao XU
aeacebcf83
feat: Support passing workspace to --manifest-path (#1246)
Previously it will load the root `Cargo.toml` and treat it as the
manifest for the crate, now it will check its `package.name` and would
search for the workspace if the `package.name` does not match the crate
name.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-03 21:12:06 +00:00
Jiahao XU
01a87ac606
feat: Impl support for alternative registries (#1184)
Fixed #1168

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-30 03:52:40 +00:00
Jiahao XU
ca00cbaccc
feat: Support install directly from git repo (#1162)
Fixed #3

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-24 01:01:31 +00:00
Jiahao XU
d8419ea5a2
feat: Add more variables for template (#1160)
Fixed #775

 - Add dep target-lexicon v0.12.7
 - Add `target-{family, arch, libc, vendor}` to
   `package.metadata.binstall`.

For `{universal, universal2}-apple-darwin`, the `target-arch` is set to
`universal`.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-20 22:05:13 +00:00
Jiahao XU
1432093dcc
feat: Add new cmdline option --no-track (#1111)
Same as `cargo-install`'s `--no-track`.
It is also passed to `cargo-install` if it is invoked.

Also fixed `fs::atomic_symlink_file` which on Windows could fallback to
non-atomic install if symlinking failed.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-03 09:15:18 +00:00
Jiahao XU
a27d5aebf6
Use leon for template in binstalk & detect malformed pkg-url/pkg-fmt early (#933)
Fixed #851

* Add new dep leon to crate binstalk
* Add new variant `BinstallError::Template{Parse, Render}Error`
* Use `leon::Template` in mod `bins`
* Use `leon::Template` in mod `fetchers::gh_crate_meta`
* Refactor mod `bins`: Rm unused associated fn & fields from `Context`
* Simplify unit testing in mod `fetchers::gh_crate_meta`
* Rm soft-deprecated field `fetchers::gh_crate_meta::Context::format`
  and change the `match` to resolve `archive` => `self.archive_format`.
* Make macro_rules `leon::template!` far easier to use
* Construct `leon::Template<'_>` as constant in `gh_crate_meta::hosting`
* Simplify `leon::Values` trait
   Change its method `get_value` signature to
   
   ```rust
       fn get_value(&self, key: &str) -> Option<Cow<'_, str>>;
   ```
   
   Now, `ValuesFn` also accepts non-`'static` function, but now
   `leon::Values` is only implemented for `&ValuesFn<F>` now.
   
   This makes it a little bit more cumbersome to use but I think it's a
   reasonable drawback.
* Rm `Send` bound req from `ValuesFn`
* Impl new fn `leon::Template::cast`
   for casting `Template<'s>` to `Template<'t>` where `'s: 't`
* Rename `leon::Template::has_keys` => `has_any_of_keys`
* Make checking whether format related keys are present more robust
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Skip checking all fmt ext
   if none of the format related keys ("format", "archive-format",
   "archive-suffix") are present.
* Only ret `.exe` in `PkgFmt::extensions` on windows
   by adding a new param `is_windows: bool`
* Improve debug msg in `GhCrateMeta::fetch_and_extract`
* Add warnings to `GhCrateMeta::find`
* Rm dep tinytemplate
* `impl<'s, 'rhs: 's> ops::AddAssign<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::AddAssign<Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<&Item<'item>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::Add<Template<'rhs>> for Template<'s>` (improved existing `Add` impl)
* `impl<'s, 'rhs: 's> ops::Add<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<&Item<'item>> for Template<'s>`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-03-26 16:11:10 +11:00
Jiahao XU
cc78ff3b90
Fix fetcher::QuickInstallstats report sending (#918)
Fixed https://github.com/cargo-bins/cargo-quickinstall/issues/195

 - Fix `fetchers::QuickInstall`: Stop sending stats for `universal-apple-darwin`
   since quickinstall only supports targets officially supports by rust.
 - Only send stats report to quickinstall if the `Fetcher::find` is `.await`ed on
   
   This prevents stats report to be sent for cases where the `QuickInstall` fetcher
   is actually unused, e.g. resolved to `GhCrateMeta` or other `QuickInstall` fetcher
   with different target.
   
   This also reduces amount of http requests created in background.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-17 21:05:19 +11:00
Jiahao XU
d118fa8fdb
Obey crates.io crawler policy: Only make one request per sec (#908)
Fixed https://github.com/taiki-e/install-action/issues/89

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-14 17:11:22 +11:00
Jiahao XU
89a47cbec9
Fix extract_tar_based_stream: Normalize path before adding them to ExtractedFiles (#901)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-13 21:27:50 +11:00
Jiahao XU
44e81c7e1a
Fix: Pass value of --roots to cargo-install as --root (#860)
Pass value of `--roots` to `cargo-install` as `--root`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-06 11:52:28 +11:00
Jiahao XU
9c7da6a179
Return a list of files written to disk in binstalk_downloader::download::Download::and_extract (#856)
to avoid collecting extracted files from disk again in resolution stage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-03 23:31:27 +11:00
Jiahao XU
8eee318ccd
Replace dep crates_io_api with in-house solution (#846)
It also uses `max_stable_version` in the json downloaded from https://crates.io/api/v1/crates/$name if possible, which is equivalent to the version shown on https://crates.io/crates/$name .

 - Add new feat `json` to `binstalk-downloader`
 - Impl new async fn `Response::json`
 - use `Response::json` in `GhApiClient` impl
 - Mark all err types in binstalk-downloader as `non_exhaustive`
 - Ret `remote::Error` in `remote::Certificate::{from_pem, from_der}` instead of `ReqwestError`.
 - Refactor `BinstallError`: Merge variant `Unzip`, `Reqwest` & `Http`
    into one variant `Download`.
 - Manually download and parse json from httos://crates.io/api/v1
 - Remove unused deps `crates_io_api`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 18:25:34 +11:00
Jiahao XU
599bcaf333
Impl GhApiClient and use it in cargo-binstall to speedup resolution process (#832)
Fixed #776

 - Add new feature gh-api-client to binstalk-downloader
 - Impl new type `binstalk_downloader::remote::{RequestBuilder, Response}`
 - Impl `binstalk_downloader::gh_api_client::GhApiClient`, exposed if `cfg(feature = "gh-api-client")` and add e2e and unit tests for it
 - Use `binstalk_downloader::gh_api_client::GhApiClient` to speedup `cargo-binstall`
 - Add new option `--github-token` to supply the token for GitHub restful API, or read from env variable `GITHUB_TOKEN` if not present.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 12:04:22 +11:00
Jiahao XU
96ebcfd08f
Add new option --locked that is passed through to cargo-install (#830)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-22 22:27:17 +11:00
Jiahao XU
2e118b3044
Optimize GhCrateMeta::find: Cache url resolution result (#810)
So that we don't have to create multiple HEAD/GET http request to get
the final, redirected url.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-16 14:51:47 +13:00
Jiahao XU
270bf08a24
Optimize use of tinytemplate (#647)
* Optimize `GhCrateMeta::find`: Cache compiled template `pkg_url`
* Optimize `collect_bin_files`: Cache compiled template `bin_dir`
* Refactor: Improve readability of `gh_crate_meta`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Dedup urls before checking
* Optimize `BinFile::new`: Avoid one heap alloc when creating `dest`
* Improve `warn!` msg in `GhCrateMeta::launch_baseline_find_tasks`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-26 17:33:49 +13:00
Jiahao XU
7b34178bcc
Optimize resolve_inner: Avoid VersionReq::clone (#656)
* Add new dep maybe-owned v0.3.4 to binstalk
* Optimize `resolve_inner`: Avoid `VersionReq::clone`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-06 13:47:45 +13:00
Jiahao XU
6bdb26930e
Refactor, improvements and bugfix for resolution and installation process (#619)
* Refactor: Extract new mod `ops::resolve::resolution`
* Refactor: Extract new type `ResolutionFetch`
* Refactor: Extract new type `ResolutionSource`
* Improve `Resolution::print`: Provides more details on which crate
   is the resolution for.
* Make `Resolution::print` a pub fn
* Refactor: Extract new fn `ResolutionFetch::install`
* Refactor: Extract new async fn `ResolutionSource::install`
* Optimize `ResolutionSource::install` avoiding `OsStr::to_string_lossy`.
* Add new dep command-group v2.0.0 to binstalk with feat with-tokio
* Fix `ResolutionSource::install`: Use `AsyncCommandGroup::group_spawn`
   instead of `tokio::process::Command::spawn` to ensure all the processes
   spanwed by the `cargo` process can be terminated with one signal sent to
   the `cargo` process.
* Fix printing resolution: Make sure they are printed without interleaving
* Refactor `entry::install_crates`
* Improve dry-run output for `ResolutionSource::install`
* Refactor: Extract new fn `ResolutionFetch::print`
* Refactor: Extract new fn `ResolutionSource::print`
* Optimize `Resolution`: Box unit variant `Fetch`
* Improve formatting of `tokio::process::Command`
   Prints out sth like `cargo install ...` instead of the `Debug::fmt`.
* Improve dry-run output

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-28 00:18:07 +00:00
Jiahao XU
db45f2fb7f
Refactor binstalk-downloader APIs: Remove cancellation_future plus optimizations (#591)
- Refactor: Mv fn `utils::asyncify` into mod `utils`
 - Improve err msg for task failure in `utils::asyncify`
 - Make sure `asyncify` always returns the same annoymous type
   that implements `Future` if the `T` is same.
 - Rewrite `extract_bin` to avoid `block_in_place`
   support cancellation by dropping
 - Rm unused dep scopeguard from binstalk-downloader
 - Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
 - Unbox `extract_future` in `async_extracter::extract_zip`
 - Refactor `Download` API: Remove `CancellationFuture` as param

   since all futures returned by `Download::and_*` does not call
   `block_in_place`, so they can be cancelled by drop instead of using this
   cumbersome hack.
 - Fix exports from mod `async_tar_visitor`
 - Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
 - Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
   and rm fn `wait_on_cancellation_signal_inner`
 - Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
 - Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
   Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
 - Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
 - Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
 - Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
 - Optimize `extract_zip_entry`: Spawn only one blocking task per fn call

   by using a mspc queue for the data to be written to the `outfile`.

   This would improve efficiency as using `tokio::fs::File` is expensive:
   It spawns a new blocking task, which needs one heap allocation and then
   pushed to a mpmc queue, and then wait for it to be done on every loop.

   This also fix a race condition where the unix permission is set before
   the whole file is written, which might be used by attackers.
 - Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
   To avoid frequent allocation and deallocation.
 - Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`

   Performs the reserve before sending the buf over mpsc queue to
   increase the possibility of reusing the previous allocation.

   NOTE: `BytesMut` only reuses the previous allocation if it is the
   only one holds the reference to it, which is either on the first
   allocation or all the `Bytes` in the mpsc queue has been consumed,
   written to the file and dropped.

   Since reading from entry would have to wait for external file I/O,
   this would give the blocking thread some time to flush `Bytes`
   out.
 - Disable unused feature fs of dep tokio

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-12 03:15:30 +00:00
Jiahao XU
8a5577297e
Optimize BinstallError: Reduce size from 40B to 32B (#581)
* Optimize `BinstallError::CratesIoApi`: Extract new type `errors::CratesIoApiError` and box it
   Also improve `<CratesIoApiError as Display>::fmt` impl.
* Optimize `BinstallError::SubProcess`: Use `Box<str>` instead of `String`
* Optimize `BinstallError::CargoManifest`: Box `CargoTomlError`
* Optimize `BinstallError::VersionParse`: Extract `VersionParseError` and box it
   Also improve `<VersionParseError as Display>::fmt` impl.
* Optimize `BinstallError::CrateContext`: Extract `CrateContextError` and box it in
* Optimize `install_from_source`: Only format `cmd` on err

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-01 20:05:00 +13:00
Jiahao XU
ab3e47c42b
Optimize Fetcher::find and fix quickinstall reporting (#579)
* Optimize `Fetcher::find`: Make it non-async to avoid boxing
   and return `AutoAbortJoinHandle<...>` instead.
   
   Since spawning a new task in tokio box the future anyway, boxing the
   returned future again in `Fetcher::find` merely adds unnecessary
   overheads.
* Optimize `QuickInstall::report`: Make it async fn instead of ret `JoinHandle`
   
   This provides several benefits:
    - On debug build, no task will be spawned
    - The calls to `self.stats_url()` can be evaluated in parallel.
    - The task now returns `()` so that the task spawned is smaller.
* Fix `QuickInstall::find`: `warn!` if quickinstall report fails

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-30 16:15:22 +13:00
Jiahao XU
66abf1b8c1
More mionr optimizations (#553)
* Optimize `BinFile::preview_bin`: Use `Path::display` instead of `to_string_lossy` to avoid allocation.
* Refactor: Rm useless `AsRef::as_ref` call
* Optimize `GhCrateMeta::find`: Reduce fut size by converting `Url` to `String`
   `Url` is much larger than `String`.
* Refactor `PackageInfo::resolve`: Destruct `Meta::binstall`
* Optimize `resolve::load_manifest_path`: Avoid allocation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-22 19:35:16 +13:00
Jiahao XU
bdb4b2070d
Minor optimization (#544)
* Optimization: Rm `debug!` in `find_version`
   printing all version iterated obviously doesn't help much in debugging
   in the problem but rather just confusing.
   
   Also this makes it hard for the compiler to optimize the iterators.
* Use let-else in `ManifestVisitor`
* Optimize `BinFile::preview_{bin, link}` for zero-copy
   Return `impl Display` that lazily format instead of allocating a `String`
* Optimize `infer_bin_dir_template`: Generate dir lazily
* Optimize `find_version`: Lazily clone `version_req` only on err
* Refactor `find_version`: Use `bool::then_some`
* Add dep either v1.8.0 to binstalk
* Optimize `GhCrateMeta::find`: Avoid cloning and `Vec` creation
   by using `Either`
* Optimize `ops::install::install_from_package`: Make it a regular fn
   instead of async fn since it does not `.await` on any async fn.
* Optimize `QuickInstall`: Rm field `target`
   since `Arc<Data>` already contains that field.
* Optimize `Fetcher`s: Extract new struct `TargetData`
   so that `Data` can be shared by all fetchers, regardless of the target.
* Optimize `QuickInstall`: Rm unused field `data`
* Optimize `Resolution::print`: Replace branching with conditional move

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-21 10:32:46 +13:00
Jiahao XU
50b6e62164
Minor refactor and optimization (#543)
* Avoid potential panicking in `args::parse` by using `Vec::get` instead of indexing
* Refactor: Simplify `opts::{resolve, install}` API
   Many parameters can be shared and put into `opts::Options` intead and
   that would also avoid a few `Arc<Path>`.
* Optimize `get_install_path`: Avoid cloning `install_path`
* Optimize `LazyJobserverClient`: Un`Arc` & remove `Clone` impl
   to avoid additional boxing
* Optimize `find_version`: Avoid cloning `semver::Version`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`
   return `impl Iterator<Item = impl Future<Output = ...>>`
   instead of `impl Iterator<Item = AutoAbortJoinHandle<...>>`
   to avoid unnecessary spawning.
   
   Each task spawned has to be boxed and then polled by tokio runtime.
   They might also be moved.
   
   While they increase parallelism, spawning these futures does not justify
   the costs because:
    - Each `Future` only calls `remote_exists`
    - Each `remote_exists` call send requests to the same domain, which is
      likely to share the same http2 connection.
      Since the conn is shared anyway, spawning does not speedup anything
      but merely add communication overhead.
    - Plus the tokio runtime spawning cost
* Optimize `install_crates`: Destruct `Args` before any `.await` point
   to reduce size of the future
* Refactor `logging`: Replace param `arg` with `log_level` & `json_output`
   to avoid dep on `Args`
* Add dep strum & strum_macros to crates/bin
* Derive `strum_macros::EnumCount` for `Strategy`
* Optimize strategies parsing in `install_crates`
* Fix panic in `install_crates` when `Compile` is not the last strategy specified
* Optimize: Take `Vec<Self>` instead of slice in `CrateName::dedup`
* Refactor: Extract new fn `compute_resolvers`
* Refactor: Extract new fn `compute_paths_and_load_manifests`
* Refactor: Extract new fn `filter_out_installed_crates`
* Reorder `install_crates`: Only run target detection if args are valid
   and there are some crates to be installed.
* Optimize `filter_out_installed_crates`: Avoid allocation
   by returning an `Iterator`
* Fix user_agent of `remote::Client`: Let user specify it
* Refactor: Replace `UIThread` with `ui::confirm`
   which is much simpler.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-19 20:00:27 +13:00
Jiahao XU
325cb5cc19
Optimizations plus bug fix (#541)
* Optimize `Download::and_extract`: Avoid dup monomorphization
* Increase buffer size for binstall_crates_v1 to `4096 * 5`
* Optimize `opts::resolve`: Avoid unnecessary `clone`s
* Fix reserve in `opts::resolve`: Do not over-reserve
* Rename field `opts::Options::resolver` => `resolvers`
* Refactor: Extract new type `resolve::PackageInfo`
    - which makes `opts::resolve_inner` easier to understand
    - reduce number of parameters required for `download_extract_and_verify` and
      `collect_bin_files`
    - reducing size of future returned by `opts::resolve_inner` by dropping
      `cargo_toml::{Manifest, Package}` as early as possible since
      `Manifest` is 3000 Bytes large while `Package` is 600 Bytes large.
* Optimize `fetchers::Data`: Use `CompactString` for field name & version
   since they are usually small enough to fit in inlined version of
   `CompactString`.
* Optimize `gh_crate_meta`: Avoid unnecessary allocation
   in `RepositoryHost::get_default_pkg_url_template`.
* Refacator: Use `Itertools::cartesian_product` in `apply_filenames_to_paths`
* Optimize `ops::resolve`: Avoid unnecessary `clone` & reduce future size
   by calling `fetcher.target_meta()` to obtain final metadata after
   downloaded and extracted the binaries.
* Optimize `ops::resolve`: Avoid unnecessary allocation
   in `download_extract_and_verify`: Replace `Itertools::join` with
   `Itertools::format` to avoid allocating the string.
* Fix disabling cargo-install fallback
* Simplify `BinFile::from_product`: Takes `&str` instead of `&Product`
   since we only need `product.name`
* Rename `BinFile::from_product` => `BinFile::new`
* Refactor: Create newtype `ops::resolve::Bin`
   so that we don't need to `unwrap()` on `Product::name`
   and reduce memory usage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 23:59:35 +00:00
Jiahao XU
d9cc3ce219
Refactor: Extract new crate binstalk-types plus other misc refactor and optimization (#535)
* Refactor: Extract new crate binstalk-types
* Optimize: Rm field `CrateInfo::other`
   which also removes dep serde-tuple-vec-map and serde-json from
   binstalk-types.
   
   This also makes `CrateInfo` easier to use, more generic and can be used
   over any `Serializer`, not just `serde_json::Value`.
* Mark all errors in `binstalk-manifests` as non_exhaustive
* Reduce size of `CvsParseError` by using `Box<str>`
   instead of `String` for variant `UnknownSourceType`.
* Reduce size of `CratesTomlParseError` to 16 bytes on 64bit platform
   by boxing variants `TomlWrite` and `CvsParse` as these two fields are
   significantly larger than other variants.
* Unify import style in mod `binstall_crates_v1`
* Replace dep binstalk-manifests with binstalk-types in binstalk-downloader
   to reduce its transitive dependencies and enables binstalk-downloader to
   be built in parallel to binstak-manifests.
* Replace dep binstalk-manifests with binstalk-types in binstalk
   to reduce transitive dependencies and enables binstalk to be built in
   parallel to binstalk-manifests.
   
   This is benefitial because binstalk-manifests pulls in toml_edit, which
   could takes up to 15s to be built on M1 (7-9s for codegen).
* Add dep binstalk-manifests to crates/bin
* Update dependabot and GHA release-pr

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 13:46:27 +13:00