Commit graph

80 commits

Author SHA1 Message Date
Jiahao XU
22b3419fce
feat: Add GraphQL support to GhApiClient (#1124)
Fixed #868

 - Add new fn `remote::Client::post`
 - Add new fn `remote::RequestBuilder::body`
 - Re-export `reqwest::Body` in `remote`
 - Add dep percent-encoding v2.2.0 to binstalk-downloader
 - Add dep serde-tuple-vec-map v1.0.1 to binstalk-downloader
 - Add GraphQL to `GhApiClient`, fallback to Restful API if token is not
   provided or authorization failed.
 - Fixed `GhReleaseArtifact::try_extract_artifact_from_str`: decode
   percent encoded http url path and add regression tests
 - Added variant `GhApiError::Context` & `GhApiContextError`
 - Added variant `GhApiError::GraphQLErrors` & `GhGraphQLErrors`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-04 14:09:49 +00:00
Jiahao XU
1432093dcc
feat: Add new cmdline option --no-track (#1111)
Same as `cargo-install`'s `--no-track`.
It is also passed to `cargo-install` if it is invoked.

Also fixed `fs::atomic_symlink_file` which on Windows could fallback to
non-atomic install if symlinking failed.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-03 09:15:18 +00:00
Jiahao XU
e96477a116
Optimize fetch_crate_cratesio for exact version (#1089)
If `version_req` requests a specific version instead of a range, then
there is no need to pull all versions available from
https://crates.io/api/v1/crates

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-05-25 12:56:31 +00:00
Jiahao XU
0261d12d9d
Support subcrate in repo (#991)
Fixed #838

 - Add new key `subcrate` for rendering `pkg-url`
 - Add new release paths in GitHub, GitLab & SourceForge using key `subcrate` for auto-detection
 - Add subcrate detection for GitHub and GitLab
 - Add `debug!` when using gh api token in `GhApiClient::new`
 - Add subcrate testing to `e2e-tests/subcrate.sh`
 - Bump cargo-release to 0.24.9 in e2e-tests/live.sh
   to fix test failure on MacOS without libssl installed in `/usr/local/`.
 - Optimize GhCrateMeta: Detect subcrate and repo-host in `Data::get_repo_info`
    to cache the result and avoid duplicate works, this also makes the code
    more ergonomic by removing the need to some `unwrap()` plus making it
    more efficient since we don't need to clone the url just to modify it.
 - Add instrument to `Data::get_repo_info`
 - Fix `shellcheck` err in `e2e-tests/*.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-04-24 19:41:20 +10:00
Jiahao XU
5683ca2476
Add new crate leon-macros that provide template! with identical syntax as runtime parsing (#946)
`leon_macros::template!` can parse template at compile-time.
It accepts a utf-8 string literal and uses `leon` internally to parse it, then generate code that evaluates to `Template<'static>`.

 - Exclude fuzz from crate leon when publishing
 - Impl fn-like proc-macro `leon_macros::template!`
 - Add dep `leon-macros` to binstalk
 - Use `leon_macros::template!` in `binstalk::fetchers::gh_crate_meta::hosting`
 - Add doc for `leon-macros` in `leon`
 - Improve `std::fmt::Display` impl for `leon::ParseError`
 - Fixed broken infra link in leon

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-04-17 14:50:58 +10:00
Jiahao XU
a27d5aebf6
Use leon for template in binstalk & detect malformed pkg-url/pkg-fmt early (#933)
Fixed #851

* Add new dep leon to crate binstalk
* Add new variant `BinstallError::Template{Parse, Render}Error`
* Use `leon::Template` in mod `bins`
* Use `leon::Template` in mod `fetchers::gh_crate_meta`
* Refactor mod `bins`: Rm unused associated fn & fields from `Context`
* Simplify unit testing in mod `fetchers::gh_crate_meta`
* Rm soft-deprecated field `fetchers::gh_crate_meta::Context::format`
  and change the `match` to resolve `archive` => `self.archive_format`.
* Make macro_rules `leon::template!` far easier to use
* Construct `leon::Template<'_>` as constant in `gh_crate_meta::hosting`
* Simplify `leon::Values` trait
   Change its method `get_value` signature to
   
   ```rust
       fn get_value(&self, key: &str) -> Option<Cow<'_, str>>;
   ```
   
   Now, `ValuesFn` also accepts non-`'static` function, but now
   `leon::Values` is only implemented for `&ValuesFn<F>` now.
   
   This makes it a little bit more cumbersome to use but I think it's a
   reasonable drawback.
* Rm `Send` bound req from `ValuesFn`
* Impl new fn `leon::Template::cast`
   for casting `Template<'s>` to `Template<'t>` where `'s: 't`
* Rename `leon::Template::has_keys` => `has_any_of_keys`
* Make checking whether format related keys are present more robust
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Skip checking all fmt ext
   if none of the format related keys ("format", "archive-format",
   "archive-suffix") are present.
* Only ret `.exe` in `PkgFmt::extensions` on windows
   by adding a new param `is_windows: bool`
* Improve debug msg in `GhCrateMeta::fetch_and_extract`
* Add warnings to `GhCrateMeta::find`
* Rm dep tinytemplate
* `impl<'s, 'rhs: 's> ops::AddAssign<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::AddAssign<Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<&Item<'item>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::Add<Template<'rhs>> for Template<'s>` (improved existing `Add` impl)
* `impl<'s, 'rhs: 's> ops::Add<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<&Item<'item>> for Template<'s>`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-03-26 16:11:10 +11:00
Jiahao XU
f28c05b0ee
Fix QuickInstall report bug for target universal2-apple-darwin (#921)
This bug was reintroduced by #919 and it was fixed by #918

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-18 22:35:48 +13:00
Jiahao XU
cc78ff3b90
Fix fetcher::QuickInstallstats report sending (#918)
Fixed https://github.com/cargo-bins/cargo-quickinstall/issues/195

 - Fix `fetchers::QuickInstall`: Stop sending stats for `universal-apple-darwin`
   since quickinstall only supports targets officially supports by rust.
 - Only send stats report to quickinstall if the `Fetcher::find` is `.await`ed on
   
   This prevents stats report to be sent for cases where the `QuickInstall` fetcher
   is actually unused, e.g. resolved to `GhCrateMeta` or other `QuickInstall` fetcher
   with different target.
   
   This also reduces amount of http requests created in background.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-17 21:05:19 +11:00
Jiahao XU
d118fa8fdb
Obey crates.io crawler policy: Only make one request per sec (#908)
Fixed https://github.com/taiki-e/install-action/issues/89

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-14 17:11:22 +11:00
Jiahao XU
89a47cbec9
Fix extract_tar_based_stream: Normalize path before adding them to ExtractedFiles (#901)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-13 21:27:50 +11:00
Jiahao XU
72789874f0
Use Quickinstall new release schema (#867)
- Refactor: Extract new fn `binstalk::helpers::remote::does_url_exist`
 - Use new quickinstall release schema in `binstalk::fetchers::QuickInstall`
 - Optimize `fetchers::QuickInstall`: Generate url once in `Fetcher::new`
    Avoid repeated string allocation plus `Url` parsing.
    This also makes changing package_url and stats_url easier.
 - Optimize `QuickInstall::report`: Use HEAD instead of GET

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-06 08:13:08 +00:00
Jiahao XU
01c8ecb778
Fix zip extraction and bins::infer_bin_dir_template (#869)
- Fix zip extraction code: Ensure dir is rwx and file is readable for curr user
 - Add more integration test for `ExtractedFiles`
 - Fix `bins::infer_bin_dir_template` introduced in #856

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-06 17:54:31 +11:00
Jiahao XU
44e81c7e1a
Fix: Pass value of --roots to cargo-install as --root (#860)
Pass value of `--roots` to `cargo-install` as `--root`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-06 11:52:28 +11:00
Jiahao XU
9c7da6a179
Return a list of files written to disk in binstalk_downloader::download::Download::and_extract (#856)
to avoid collecting extracted files from disk again in resolution stage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-03 23:31:27 +11:00
Jiahao XU
8eee318ccd
Replace dep crates_io_api with in-house solution (#846)
It also uses `max_stable_version` in the json downloaded from https://crates.io/api/v1/crates/$name if possible, which is equivalent to the version shown on https://crates.io/crates/$name .

 - Add new feat `json` to `binstalk-downloader`
 - Impl new async fn `Response::json`
 - use `Response::json` in `GhApiClient` impl
 - Mark all err types in binstalk-downloader as `non_exhaustive`
 - Ret `remote::Error` in `remote::Certificate::{from_pem, from_der}` instead of `ReqwestError`.
 - Refactor `BinstallError`: Merge variant `Unzip`, `Reqwest` & `Http`
    into one variant `Download`.
 - Manually download and parse json from httos://crates.io/api/v1
 - Remove unused deps `crates_io_api`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 18:25:34 +11:00
Jiahao XU
599bcaf333
Impl GhApiClient and use it in cargo-binstall to speedup resolution process (#832)
Fixed #776

 - Add new feature gh-api-client to binstalk-downloader
 - Impl new type `binstalk_downloader::remote::{RequestBuilder, Response}`
 - Impl `binstalk_downloader::gh_api_client::GhApiClient`, exposed if `cfg(feature = "gh-api-client")` and add e2e and unit tests for it
 - Use `binstalk_downloader::gh_api_client::GhApiClient` to speedup `cargo-binstall`
 - Add new option `--github-token` to supply the token for GitHub restful API, or read from env variable `GITHUB_TOKEN` if not present.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-02 12:04:22 +11:00
Jiahao XU
96ebcfd08f
Add new option --locked that is passed through to cargo-install (#830)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-22 22:27:17 +11:00
Jiahao XU
2e118b3044
Optimize GhCrateMeta::find: Cache url resolution result (#810)
So that we don't have to create multiple HEAD/GET http request to get
the final, redirected url.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-16 14:51:47 +13:00
Jiahao XU
9874dca454
Fix bin names in .crates.toml on win (#802)
`bins::BinFile::new` does not include `binary_ext` in the
`BinFile::base_name`.

Also add regression e2e-test.

Fixed #801

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-15 17:24:50 +13:00
Jiahao XU
04113e9dd9
Rm NOVERSION_FILENAMES "{ name }{ archive-suffix }" (#795)
which does not contain `target`, hence it might report the binary being
available for any target we specified.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-13 13:44:24 +11:00
Jiahao XU
87686cb2f7
Feature: Better retry policy in binstalk-downloader (#794)
Fixed #779 #791 

 - Retry request on timeout
 - Retry for `StatusCode::{REQUEST_TIMEOUT, GATEWAY_TIMEOUT}`
 - Add `DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT` for 503/429
   if 503/429 does not give us a header or give us an invalid header on
   when to retry, we would default to
   `DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT`.
 - Fix `Client::get_redirected_final_url`: Retry using `GET` on status code 400..405 + 410
 - Rename remote_exists => remote_gettable & support fallback to GET
   if HEAD fails due to status code 400..405 + 410.
 - Improve `Client::get_stream`: Include url & method in the err of the stream returned

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-13 13:43:48 +11:00
Jiahao XU
d9dd611b9e
Improve fmt::Display impl for all Error types (#781)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-10 14:01:22 +11:00
Jiahao XU
a13c01b769
Replace dep futures-util with helpers::FuturesResolver (#765)
futures-util has too many dependencies and it contains a lot of code of
which we only use `futures_util::stream::{FuturesUnordered, StreamExt}`.

We don't even need most of the functionalities in `FuturesUnordered` as
we just need the output of first future that either returns `Err(_)` or
`Ok(Some(_))`.

So we replace it with ou own homebrew solution (~80 loc) and it's easier
to use.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-05 16:26:25 +11:00
Félix Saparelli
6bc1fb4983
Fix #728: fallback to copy when symlinking on windows (#763)
* Fix #728: fallback to copy when symlinking on windows

* Signature hmm?

* Squash warnings on windows

* Just don’t use generics here
2023-02-04 14:14:08 +13:00
Jiahao XU
e510511487
Improve GhCrateMeta: Detect cases where pkg-fmt is not specified (#757)
* Fix fmt of mod `<GhCrateMeta as Fetcher>::find`
* Add new variant `BinstallError::InvalidPkgFmt`
* Impl new fn `PkgFmt::guess_pkg_format`
* Improve `GhCrateMeta`: Detect cases where `pkg-fmt` is not specified
  
  but `pkg-url` also does not contain format, archive-format or
  archive-suffix which is required for automatically deducing the pkg-fmt.
  
  In these cases, we would call `PkgFmt::guess_pkg_format` to try out best
  to figure out the pkg-fmt, otherwise we just return an error.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-04 10:47:57 +13:00
Jiahao XU
270bf08a24
Optimize use of tinytemplate (#647)
* Optimize `GhCrateMeta::find`: Cache compiled template `pkg_url`
* Optimize `collect_bin_files`: Cache compiled template `bin_dir`
* Refactor: Improve readability of `gh_crate_meta`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Dedup urls before checking
* Optimize `BinFile::new`: Avoid one heap alloc when creating `dest`
* Improve `warn!` msg in `GhCrateMeta::launch_baseline_find_tasks`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-26 17:33:49 +13:00
Jiahao XU
eb95c5b799
Minor optimization for bin and binstalk (#646)
* Refactor `logging`: Extract `report_err`
* Optimize `get_default_pkg_url_template`: Return iter instead of `Vec`
   to avoid heap allocation.
* Refactor `GhCrateMeta::find`: Rm `launch_baseline_find_tasks`
* Optimize `GhCrateMeta::find`: Avoid cloning `Cow<'_, str>`
* Improve `report_err` output: Print newline after msg

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-25 01:58:07 +13:00
Jiahao XU
ab4da7f059
Fix self-update on windows (#712)
Fixed #702


* Add new dep windows v0.44.0 for windows only
* Add regression test for #702
* Impl `win::replace_file`
* Use `ReplaceFileW` on windows if `fs::rename` in `fs::atomic_install` fails
* Improve logging and err messages
* Add more regression tests
* Make `BinFile::install_link` atomic: Do not remove file if dst exists
* Fallback to `ReplaceFileW` in `fs::atomic_symlink_file`
* Refactor: Extract new fn `fs::persist`
* Use `fs::persist` instead of `TempFile::persist` in `fs::atomic_install`, which fallbacks to `ReplaceFileW` on windows

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-01-17 18:05:07 +11:00
Jiahao XU
7b34178bcc
Optimize resolve_inner: Avoid VersionReq::clone (#656)
* Add new dep maybe-owned v0.3.4 to binstalk
* Optimize `resolve_inner`: Avoid `VersionReq::clone`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-06 13:47:45 +13:00
Jiahao XU
daec880008
Rm unused variant binstalk_downloader::download::DownloadError::UserAbort (#653)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-06 02:08:57 +13:00
Jiahao XU
959b465d81
Optimize Download::and_visit_tar: Use trait object to avoid monomorphization (#644)
by removing method `TarEntriesVisitor::finish` and associated type
`TarEntriesVisitor::Target`.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-04 13:11:10 +13:00
Jiahao XU
0ecb30dc9d
Update quickinstall url (#638)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-29 00:59:13 +00:00
dependabot[bot]
287ae03a51
Bump cargo_toml from 0.13.0 to 0.14.0 (#632)
* Bump cargo_toml from 0.13.0 to 0.14.0

Bumps [cargo_toml](https://gitlab.com/crates.rs/cargo_toml) from 0.13.0 to 0.14.0.
- [Release notes](https://gitlab.com/crates.rs/cargo_toml/tags)
- [Commits](https://gitlab.com/crates.rs/cargo_toml/commits/main)

---
updated-dependencies:
- dependency-name: cargo_toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix use of `Manifest::complete_from_abstract_filesystem`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-28 04:16:24 +00:00
Jiahao XU
6bdb26930e
Refactor, improvements and bugfix for resolution and installation process (#619)
* Refactor: Extract new mod `ops::resolve::resolution`
* Refactor: Extract new type `ResolutionFetch`
* Refactor: Extract new type `ResolutionSource`
* Improve `Resolution::print`: Provides more details on which crate
   is the resolution for.
* Make `Resolution::print` a pub fn
* Refactor: Extract new fn `ResolutionFetch::install`
* Refactor: Extract new async fn `ResolutionSource::install`
* Optimize `ResolutionSource::install` avoiding `OsStr::to_string_lossy`.
* Add new dep command-group v2.0.0 to binstalk with feat with-tokio
* Fix `ResolutionSource::install`: Use `AsyncCommandGroup::group_spawn`
   instead of `tokio::process::Command::spawn` to ensure all the processes
   spanwed by the `cargo` process can be terminated with one signal sent to
   the `cargo` process.
* Fix printing resolution: Make sure they are printed without interleaving
* Refactor `entry::install_crates`
* Improve dry-run output for `ResolutionSource::install`
* Refactor: Extract new fn `ResolutionFetch::print`
* Refactor: Extract new fn `ResolutionSource::print`
* Optimize `Resolution`: Box unit variant `Fetch`
* Improve formatting of `tokio::process::Command`
   Prints out sth like `cargo install ...` instead of the `Debug::fmt`.
* Improve dry-run output

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-28 00:18:07 +00:00
Jiahao XU
73a794dc13
Optimization: Box fut Remote::get_redirected_final_url in GhCrateMeta::find (#600)
* Box fut `Remote::get_redirected_final_url` in `GhCrateMeta::find`
   Since the other await point in `GhCrateMeta::find` only needs
   `Arc<Self>` and `handles` to be saved, which is much smaller than the
   future returned by `Remote::get_redirected_final_url`
* Refactor: Simplify `wait_on_cancellation_signal`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-12 08:42:11 +00:00
Jiahao XU
db45f2fb7f
Refactor binstalk-downloader APIs: Remove cancellation_future plus optimizations (#591)
- Refactor: Mv fn `utils::asyncify` into mod `utils`
 - Improve err msg for task failure in `utils::asyncify`
 - Make sure `asyncify` always returns the same annoymous type
   that implements `Future` if the `T` is same.
 - Rewrite `extract_bin` to avoid `block_in_place`
   support cancellation by dropping
 - Rm unused dep scopeguard from binstalk-downloader
 - Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
 - Unbox `extract_future` in `async_extracter::extract_zip`
 - Refactor `Download` API: Remove `CancellationFuture` as param

   since all futures returned by `Download::and_*` does not call
   `block_in_place`, so they can be cancelled by drop instead of using this
   cumbersome hack.
 - Fix exports from mod `async_tar_visitor`
 - Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
 - Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
   and rm fn `wait_on_cancellation_signal_inner`
 - Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
 - Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
   Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
 - Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
 - Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
 - Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
 - Optimize `extract_zip_entry`: Spawn only one blocking task per fn call

   by using a mspc queue for the data to be written to the `outfile`.

   This would improve efficiency as using `tokio::fs::File` is expensive:
   It spawns a new blocking task, which needs one heap allocation and then
   pushed to a mpmc queue, and then wait for it to be done on every loop.

   This also fix a race condition where the unix permission is set before
   the whole file is written, which might be used by attackers.
 - Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
   To avoid frequent allocation and deallocation.
 - Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`

   Performs the reserve before sending the buf over mpsc queue to
   increase the possibility of reusing the previous allocation.

   NOTE: `BytesMut` only reuses the previous allocation if it is the
   only one holds the reference to it, which is either on the first
   allocation or all the `Bytes` in the mpsc queue has been consumed,
   written to the file and dropped.

   Since reading from entry would have to wait for external file I/O,
   this would give the blocking thread some time to flush `Bytes`
   out.
 - Disable unused feature fs of dep tokio

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-12 03:15:30 +00:00
Jiahao XU
23a5937aff
Optimization: Rewrite Download::and_visit_tar to use tokio-tar (#587)
* Add new dep tokio-tar v0.3.0 to binstalk-downloader
* Add new dep tokio-util v0.7.4 with feat io to binstalk-downloader
* Add dep async-trait v0.1.59 to binstalk-downloader
* Add new dep async-compression v0.3.15 to binstalk-downloader
   with features "gzip", "zstd", "xz", "bzip2", "tokio".
* Rewrite `Download::and_visit_tar` to use `tokio-tar`
   to avoid the cumbersome `block_in_place`.
* Apply temporary workaround: Rm use of let-else in mod visitor

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-04 14:31:34 +13:00
Jiahao XU
887ebb6f6f
Fix BinstallError::CrateContext: Forward diagnostics of field CrateContextError::err (#586)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-03 12:12:04 +00:00
Jiahao XU
a69db83aa6
Fix error reporting in main and move all arg validation into fn args::parse (#585)
* Fix reporting parsing error from `args::parse`
   Report it in `args::parse` by using `Args::command().error(...).exit()`
   instead of returning `BinstallError`.
* Rm unused variant `BinstallError::OverrideOptionUsedWithMultiInstall`
* Refactor: Move `strategies` validation into `args::parse`
* Rm unused variant `BinstallError::InvalidStrategies`
* Add new unit test `args::test::verify_cli`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-03 11:57:50 +00:00
Jiahao XU
8a5577297e
Optimize BinstallError: Reduce size from 40B to 32B (#581)
* Optimize `BinstallError::CratesIoApi`: Extract new type `errors::CratesIoApiError` and box it
   Also improve `<CratesIoApiError as Display>::fmt` impl.
* Optimize `BinstallError::SubProcess`: Use `Box<str>` instead of `String`
* Optimize `BinstallError::CargoManifest`: Box `CargoTomlError`
* Optimize `BinstallError::VersionParse`: Extract `VersionParseError` and box it
   Also improve `<VersionParseError as Display>::fmt` impl.
* Optimize `BinstallError::CrateContext`: Extract `CrateContextError` and box it in
* Optimize `install_from_source`: Only format `cmd` on err

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-12-01 20:05:00 +13:00
Jiahao XU
ab3e47c42b
Optimize Fetcher::find and fix quickinstall reporting (#579)
* Optimize `Fetcher::find`: Make it non-async to avoid boxing
   and return `AutoAbortJoinHandle<...>` instead.
   
   Since spawning a new task in tokio box the future anyway, boxing the
   returned future again in `Fetcher::find` merely adds unnecessary
   overheads.
* Optimize `QuickInstall::report`: Make it async fn instead of ret `JoinHandle`
   
   This provides several benefits:
    - On debug build, no task will be spawned
    - The calls to `self.stats_url()` can be evaluated in parallel.
    - The task now returns `()` so that the task spawned is smaller.
* Fix `QuickInstall::find`: `warn!` if quickinstall report fails

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-30 16:15:22 +13:00
Jiahao XU
ff737730f4
Optimize use of tokio::select!: Use biased selection (#580)
as there is no need to randomize the first one to be polled.

For `cancel_on_user_sig_term` and `StreamReadable::fill_buf`, the
cancellation future should always to be polled first so that user would
feel responsive.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-30 16:05:52 +13:00
Jiahao XU
b5002de9c4
Return Future + 'static in GhCrateMeta::launch_baseline_find_tasks (#573)
So that the borrow checker know that `FuturesUnordered<F>` holds no
reference to any data and thus it can further optimize future returned
by `GhCrateMeta::find`.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-29 16:26:54 +13:00
Jiahao XU
66abf1b8c1
More mionr optimizations (#553)
* Optimize `BinFile::preview_bin`: Use `Path::display` instead of `to_string_lossy` to avoid allocation.
* Refactor: Rm useless `AsRef::as_ref` call
* Optimize `GhCrateMeta::find`: Reduce fut size by converting `Url` to `String`
   `Url` is much larger than `String`.
* Refactor `PackageInfo::resolve`: Destruct `Meta::binstall`
* Optimize `resolve::load_manifest_path`: Avoid allocation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-22 19:35:16 +13:00
Jiahao XU
83b2de3ea2
Enforce strict strategies parsing rule: Reject duplicate strategy in --strategies (#545)
`cargo-binstall` should reject duplicate strategy in `--strategies` as it is illegal input and it cannot be interpreted reasonably.

* Reject duplicate strategy in `--strategies`
* Optimize `compute_resolvers`: Use `Vec::retain`
   instead of `Iterator::filter` plus collecting into `Vec`.
* Optimize `compute_resolvers`: Reject `strategies.len() > Strategy::COUNT`
* Improve err msg for cases where user disabled all strategies
* Simplify `BinstallError::InvalidStrategies`: Takes `&'static str`
   instead of `&'static &'static str` since other variants are larger than 8B.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-21 12:02:58 +11:00
Jiahao XU
bdb4b2070d
Minor optimization (#544)
* Optimization: Rm `debug!` in `find_version`
   printing all version iterated obviously doesn't help much in debugging
   in the problem but rather just confusing.
   
   Also this makes it hard for the compiler to optimize the iterators.
* Use let-else in `ManifestVisitor`
* Optimize `BinFile::preview_{bin, link}` for zero-copy
   Return `impl Display` that lazily format instead of allocating a `String`
* Optimize `infer_bin_dir_template`: Generate dir lazily
* Optimize `find_version`: Lazily clone `version_req` only on err
* Refactor `find_version`: Use `bool::then_some`
* Add dep either v1.8.0 to binstalk
* Optimize `GhCrateMeta::find`: Avoid cloning and `Vec` creation
   by using `Either`
* Optimize `ops::install::install_from_package`: Make it a regular fn
   instead of async fn since it does not `.await` on any async fn.
* Optimize `QuickInstall`: Rm field `target`
   since `Arc<Data>` already contains that field.
* Optimize `Fetcher`s: Extract new struct `TargetData`
   so that `Data` can be shared by all fetchers, regardless of the target.
* Optimize `QuickInstall`: Rm unused field `data`
* Optimize `Resolution::print`: Replace branching with conditional move

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-21 10:32:46 +13:00
Jiahao XU
50b6e62164
Minor refactor and optimization (#543)
* Avoid potential panicking in `args::parse` by using `Vec::get` instead of indexing
* Refactor: Simplify `opts::{resolve, install}` API
   Many parameters can be shared and put into `opts::Options` intead and
   that would also avoid a few `Arc<Path>`.
* Optimize `get_install_path`: Avoid cloning `install_path`
* Optimize `LazyJobserverClient`: Un`Arc` & remove `Clone` impl
   to avoid additional boxing
* Optimize `find_version`: Avoid cloning `semver::Version`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`
   return `impl Iterator<Item = impl Future<Output = ...>>`
   instead of `impl Iterator<Item = AutoAbortJoinHandle<...>>`
   to avoid unnecessary spawning.
   
   Each task spawned has to be boxed and then polled by tokio runtime.
   They might also be moved.
   
   While they increase parallelism, spawning these futures does not justify
   the costs because:
    - Each `Future` only calls `remote_exists`
    - Each `remote_exists` call send requests to the same domain, which is
      likely to share the same http2 connection.
      Since the conn is shared anyway, spawning does not speedup anything
      but merely add communication overhead.
    - Plus the tokio runtime spawning cost
* Optimize `install_crates`: Destruct `Args` before any `.await` point
   to reduce size of the future
* Refactor `logging`: Replace param `arg` with `log_level` & `json_output`
   to avoid dep on `Args`
* Add dep strum & strum_macros to crates/bin
* Derive `strum_macros::EnumCount` for `Strategy`
* Optimize strategies parsing in `install_crates`
* Fix panic in `install_crates` when `Compile` is not the last strategy specified
* Optimize: Take `Vec<Self>` instead of slice in `CrateName::dedup`
* Refactor: Extract new fn `compute_resolvers`
* Refactor: Extract new fn `compute_paths_and_load_manifests`
* Refactor: Extract new fn `filter_out_installed_crates`
* Reorder `install_crates`: Only run target detection if args are valid
   and there are some crates to be installed.
* Optimize `filter_out_installed_crates`: Avoid allocation
   by returning an `Iterator`
* Fix user_agent of `remote::Client`: Let user specify it
* Refactor: Replace `UIThread` with `ui::confirm`
   which is much simpler.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-19 20:00:27 +13:00
Jiahao XU
325cb5cc19
Optimizations plus bug fix (#541)
* Optimize `Download::and_extract`: Avoid dup monomorphization
* Increase buffer size for binstall_crates_v1 to `4096 * 5`
* Optimize `opts::resolve`: Avoid unnecessary `clone`s
* Fix reserve in `opts::resolve`: Do not over-reserve
* Rename field `opts::Options::resolver` => `resolvers`
* Refactor: Extract new type `resolve::PackageInfo`
    - which makes `opts::resolve_inner` easier to understand
    - reduce number of parameters required for `download_extract_and_verify` and
      `collect_bin_files`
    - reducing size of future returned by `opts::resolve_inner` by dropping
      `cargo_toml::{Manifest, Package}` as early as possible since
      `Manifest` is 3000 Bytes large while `Package` is 600 Bytes large.
* Optimize `fetchers::Data`: Use `CompactString` for field name & version
   since they are usually small enough to fit in inlined version of
   `CompactString`.
* Optimize `gh_crate_meta`: Avoid unnecessary allocation
   in `RepositoryHost::get_default_pkg_url_template`.
* Refacator: Use `Itertools::cartesian_product` in `apply_filenames_to_paths`
* Optimize `ops::resolve`: Avoid unnecessary `clone` & reduce future size
   by calling `fetcher.target_meta()` to obtain final metadata after
   downloaded and extracted the binaries.
* Optimize `ops::resolve`: Avoid unnecessary allocation
   in `download_extract_and_verify`: Replace `Itertools::join` with
   `Itertools::format` to avoid allocating the string.
* Fix disabling cargo-install fallback
* Simplify `BinFile::from_product`: Takes `&str` instead of `&Product`
   since we only need `product.name`
* Rename `BinFile::from_product` => `BinFile::new`
* Refactor: Create newtype `ops::resolve::Bin`
   so that we don't need to `unwrap()` on `Product::name`
   and reduce memory usage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 23:59:35 +00:00
Jiahao XU
d9cc3ce219
Refactor: Extract new crate binstalk-types plus other misc refactor and optimization (#535)
* Refactor: Extract new crate binstalk-types
* Optimize: Rm field `CrateInfo::other`
   which also removes dep serde-tuple-vec-map and serde-json from
   binstalk-types.
   
   This also makes `CrateInfo` easier to use, more generic and can be used
   over any `Serializer`, not just `serde_json::Value`.
* Mark all errors in `binstalk-manifests` as non_exhaustive
* Reduce size of `CvsParseError` by using `Box<str>`
   instead of `String` for variant `UnknownSourceType`.
* Reduce size of `CratesTomlParseError` to 16 bytes on 64bit platform
   by boxing variants `TomlWrite` and `CvsParse` as these two fields are
   significantly larger than other variants.
* Unify import style in mod `binstall_crates_v1`
* Replace dep binstalk-manifests with binstalk-types in binstalk-downloader
   to reduce its transitive dependencies and enables binstalk-downloader to
   be built in parallel to binstak-manifests.
* Replace dep binstalk-manifests with binstalk-types in binstalk
   to reduce transitive dependencies and enables binstalk to be built in
   parallel to binstalk-manifests.
   
   This is benefitial because binstalk-manifests pulls in toml_edit, which
   could takes up to 15s to be built on M1 (7-9s for codegen).
* Add dep binstalk-manifests to crates/bin
* Update dependabot and GHA release-pr

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 13:46:27 +13:00
Jiahao XU
c34a2850b3
Optimization: Box variants in Error to reduce size and remove unused variants in BinstallError (#530)
* Box `HttpError` in `binsalk_downloader::remote::Error::Http`
   as `HttpError` contains `Url` which are too big.
* Box `HttpError` in `BinstallError::Http` same as the previous commit.
* Box `TinyTemplateError` in `BinstallError::Template`
  since `TinyTemplateError` is 56 bytes large where most of the other variants are below 40 bytes large.
* Rm unsed variant `BinstallError::VersionUnavailable`
* Box `CratesIoApiError` in `BinstallError::CratesIoApi`
   It is 32 bytes large while other variants are below 40 bytes large.
* Improve err msg for `BinstallError::CrateContext`
* Rm unused variant `BinstallError::VersionReq`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-15 02:46:32 +13:00