Previously it will load the root `Cargo.toml` and treat it as the
manifest for the crate, now it will check its `package.name` and would
search for the workspace if the `package.name` does not match the crate
name.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* perform a bare git cloen to avoid worktree checkouts
This will be way faster on windows
* feat: use the git repository directly to obtain crate information
* Apply code review changes
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Impl `fmt::Display` for `helpers::GitUrl`
and also refactor implementation of `git::Repository::{shallow_clone,
shallow_clone_bare}`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Fix clippy lint warnings
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Fix typo
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
---------
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#885
Now we can take advantage of new argument `--registry` and
env overrides:
- `CARGO_REGISTRIES_DEFAULT` if `--registry` is not specified
- `CARGO_REGISTRIES_{registry_name}_INDEX` for the registry index url
We can also read from `.cargo/config.toml` for:
- default registry and registries configurations
- additional CA bundle `http.cainfo`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
instead of `CompactString` since `target_lexicon::{OperatingSystem,
Architecture, Environment}::into_str()` will return a `&'static str`
most of the time.
Also updated `SUPPORT.md`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#775
- Add dep target-lexicon v0.12.7
- Add `target-{family, arch, libc, vendor}` to
`package.metadata.binstall`.
For `{universal, universal2}-apple-darwin`, the `target-arch` is set to
`universal`.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Same as `cargo-install`'s `--no-track`.
It is also passed to `cargo-install` if it is invoked.
Also fixed `fs::atomic_symlink_file` which on Windows could fallback to
non-atomic install if symlinking failed.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
If `version_req` requests a specific version instead of a range, then
there is no need to pull all versions available from
https://crates.io/api/v1/crates
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#838
- Add new key `subcrate` for rendering `pkg-url`
- Add new release paths in GitHub, GitLab & SourceForge using key `subcrate` for auto-detection
- Add subcrate detection for GitHub and GitLab
- Add `debug!` when using gh api token in `GhApiClient::new`
- Add subcrate testing to `e2e-tests/subcrate.sh`
- Bump cargo-release to 0.24.9 in e2e-tests/live.sh
to fix test failure on MacOS without libssl installed in `/usr/local/`.
- Optimize GhCrateMeta: Detect subcrate and repo-host in `Data::get_repo_info`
to cache the result and avoid duplicate works, this also makes the code
more ergonomic by removing the need to some `unwrap()` plus making it
more efficient since we don't need to clone the url just to modify it.
- Add instrument to `Data::get_repo_info`
- Fix `shellcheck` err in `e2e-tests/*.sh`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
`leon_macros::template!` can parse template at compile-time.
It accepts a utf-8 string literal and uses `leon` internally to parse it, then generate code that evaluates to `Template<'static>`.
- Exclude fuzz from crate leon when publishing
- Impl fn-like proc-macro `leon_macros::template!`
- Add dep `leon-macros` to binstalk
- Use `leon_macros::template!` in `binstalk::fetchers::gh_crate_meta::hosting`
- Add doc for `leon-macros` in `leon`
- Improve `std::fmt::Display` impl for `leon::ParseError`
- Fixed broken infra link in leon
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#851
* Add new dep leon to crate binstalk
* Add new variant `BinstallError::Template{Parse, Render}Error`
* Use `leon::Template` in mod `bins`
* Use `leon::Template` in mod `fetchers::gh_crate_meta`
* Refactor mod `bins`: Rm unused associated fn & fields from `Context`
* Simplify unit testing in mod `fetchers::gh_crate_meta`
* Rm soft-deprecated field `fetchers::gh_crate_meta::Context::format`
and change the `match` to resolve `archive` => `self.archive_format`.
* Make macro_rules `leon::template!` far easier to use
* Construct `leon::Template<'_>` as constant in `gh_crate_meta::hosting`
* Simplify `leon::Values` trait
Change its method `get_value` signature to
```rust
fn get_value(&self, key: &str) -> Option<Cow<'_, str>>;
```
Now, `ValuesFn` also accepts non-`'static` function, but now
`leon::Values` is only implemented for `&ValuesFn<F>` now.
This makes it a little bit more cumbersome to use but I think it's a
reasonable drawback.
* Rm `Send` bound req from `ValuesFn`
* Impl new fn `leon::Template::cast`
for casting `Template<'s>` to `Template<'t>` where `'s: 't`
* Rename `leon::Template::has_keys` => `has_any_of_keys`
* Make checking whether format related keys are present more robust
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Skip checking all fmt ext
if none of the format related keys ("format", "archive-format",
"archive-suffix") are present.
* Only ret `.exe` in `PkgFmt::extensions` on windows
by adding a new param `is_windows: bool`
* Improve debug msg in `GhCrateMeta::fetch_and_extract`
* Add warnings to `GhCrateMeta::find`
* Rm dep tinytemplate
* `impl<'s, 'rhs: 's> ops::AddAssign<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::AddAssign<Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<&Item<'item>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::Add<Template<'rhs>> for Template<'s>` (improved existing `Add` impl)
* `impl<'s, 'rhs: 's> ops::Add<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<&Item<'item>> for Template<'s>`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
Fixed https://github.com/cargo-bins/cargo-quickinstall/issues/195
- Fix `fetchers::QuickInstall`: Stop sending stats for `universal-apple-darwin`
since quickinstall only supports targets officially supports by rust.
- Only send stats report to quickinstall if the `Fetcher::find` is `.await`ed on
This prevents stats report to be sent for cases where the `QuickInstall` fetcher
is actually unused, e.g. resolved to `GhCrateMeta` or other `QuickInstall` fetcher
with different target.
This also reduces amount of http requests created in background.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
- Refactor: Extract new fn `binstalk::helpers::remote::does_url_exist`
- Use new quickinstall release schema in `binstalk::fetchers::QuickInstall`
- Optimize `fetchers::QuickInstall`: Generate url once in `Fetcher::new`
Avoid repeated string allocation plus `Url` parsing.
This also makes changing package_url and stats_url easier.
- Optimize `QuickInstall::report`: Use HEAD instead of GET
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
- Fix zip extraction code: Ensure dir is rwx and file is readable for curr user
- Add more integration test for `ExtractedFiles`
- Fix `bins::infer_bin_dir_template` introduced in #856
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
It also uses `max_stable_version` in the json downloaded from https://crates.io/api/v1/crates/$name if possible, which is equivalent to the version shown on https://crates.io/crates/$name .
- Add new feat `json` to `binstalk-downloader`
- Impl new async fn `Response::json`
- use `Response::json` in `GhApiClient` impl
- Mark all err types in binstalk-downloader as `non_exhaustive`
- Ret `remote::Error` in `remote::Certificate::{from_pem, from_der}` instead of `ReqwestError`.
- Refactor `BinstallError`: Merge variant `Unzip`, `Reqwest` & `Http`
into one variant `Download`.
- Manually download and parse json from httos://crates.io/api/v1
- Remove unused deps `crates_io_api`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#776
- Add new feature gh-api-client to binstalk-downloader
- Impl new type `binstalk_downloader::remote::{RequestBuilder, Response}`
- Impl `binstalk_downloader::gh_api_client::GhApiClient`, exposed if `cfg(feature = "gh-api-client")` and add e2e and unit tests for it
- Use `binstalk_downloader::gh_api_client::GhApiClient` to speedup `cargo-binstall`
- Add new option `--github-token` to supply the token for GitHub restful API, or read from env variable `GITHUB_TOKEN` if not present.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
`bins::BinFile::new` does not include `binary_ext` in the
`BinFile::base_name`.
Also add regression e2e-test.
Fixed#801
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
which does not contain `target`, hence it might report the binary being
available for any target we specified.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#779#791
- Retry request on timeout
- Retry for `StatusCode::{REQUEST_TIMEOUT, GATEWAY_TIMEOUT}`
- Add `DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT` for 503/429
if 503/429 does not give us a header or give us an invalid header on
when to retry, we would default to
`DEFAULT_RETRY_DURATION_FOR_RATE_LIMIT`.
- Fix `Client::get_redirected_final_url`: Retry using `GET` on status code 400..405 + 410
- Rename remote_exists => remote_gettable & support fallback to GET
if HEAD fails due to status code 400..405 + 410.
- Improve `Client::get_stream`: Include url & method in the err of the stream returned
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
futures-util has too many dependencies and it contains a lot of code of
which we only use `futures_util::stream::{FuturesUnordered, StreamExt}`.
We don't even need most of the functionalities in `FuturesUnordered` as
we just need the output of first future that either returns `Err(_)` or
`Ok(Some(_))`.
So we replace it with ou own homebrew solution (~80 loc) and it's easier
to use.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Fix fmt of mod `<GhCrateMeta as Fetcher>::find`
* Add new variant `BinstallError::InvalidPkgFmt`
* Impl new fn `PkgFmt::guess_pkg_format`
* Improve `GhCrateMeta`: Detect cases where `pkg-fmt` is not specified
but `pkg-url` also does not contain format, archive-format or
archive-suffix which is required for automatically deducing the pkg-fmt.
In these cases, we would call `PkgFmt::guess_pkg_format` to try out best
to figure out the pkg-fmt, otherwise we just return an error.
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Fixed#702
* Add new dep windows v0.44.0 for windows only
* Add regression test for #702
* Impl `win::replace_file`
* Use `ReplaceFileW` on windows if `fs::rename` in `fs::atomic_install` fails
* Improve logging and err messages
* Add more regression tests
* Make `BinFile::install_link` atomic: Do not remove file if dst exists
* Fallback to `ReplaceFileW` in `fs::atomic_symlink_file`
* Refactor: Extract new fn `fs::persist`
* Use `fs::persist` instead of `TempFile::persist` in `fs::atomic_install`, which fallbacks to `ReplaceFileW` on windows
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
* Refactor: Extract new mod `ops::resolve::resolution`
* Refactor: Extract new type `ResolutionFetch`
* Refactor: Extract new type `ResolutionSource`
* Improve `Resolution::print`: Provides more details on which crate
is the resolution for.
* Make `Resolution::print` a pub fn
* Refactor: Extract new fn `ResolutionFetch::install`
* Refactor: Extract new async fn `ResolutionSource::install`
* Optimize `ResolutionSource::install` avoiding `OsStr::to_string_lossy`.
* Add new dep command-group v2.0.0 to binstalk with feat with-tokio
* Fix `ResolutionSource::install`: Use `AsyncCommandGroup::group_spawn`
instead of `tokio::process::Command::spawn` to ensure all the processes
spanwed by the `cargo` process can be terminated with one signal sent to
the `cargo` process.
* Fix printing resolution: Make sure they are printed without interleaving
* Refactor `entry::install_crates`
* Improve dry-run output for `ResolutionSource::install`
* Refactor: Extract new fn `ResolutionFetch::print`
* Refactor: Extract new fn `ResolutionSource::print`
* Optimize `Resolution`: Box unit variant `Fetch`
* Improve formatting of `tokio::process::Command`
Prints out sth like `cargo install ...` instead of the `Debug::fmt`.
* Improve dry-run output
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Box fut `Remote::get_redirected_final_url` in `GhCrateMeta::find`
Since the other await point in `GhCrateMeta::find` only needs
`Arc<Self>` and `handles` to be saved, which is much smaller than the
future returned by `Remote::get_redirected_final_url`
* Refactor: Simplify `wait_on_cancellation_signal`
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
- Refactor: Mv fn `utils::asyncify` into mod `utils`
- Improve err msg for task failure in `utils::asyncify`
- Make sure `asyncify` always returns the same annoymous type
that implements `Future` if the `T` is same.
- Rewrite `extract_bin` to avoid `block_in_place`
support cancellation by dropping
- Rm unused dep scopeguard from binstalk-downloader
- Rewrite `extract_tar_based_stream` so that it is cancellable by dropping
- Unbox `extract_future` in `async_extracter::extract_zip`
- Refactor `Download` API: Remove `CancellationFuture` as param
since all futures returned by `Download::and_*` does not call
`block_in_place`, so they can be cancelled by drop instead of using this
cumbersome hack.
- Fix exports from mod `async_tar_visitor`
- Make `signal::{ignore_signals, wait_on_cancellation_signal}` private
- Rm the global variable `CANCELLED` in `wait_on_cancellation_signal`
and rm fn `wait_on_cancellation_signal_inner`
- Optimize `wait_on_cancellation_signal`: Avoid `tokio::select!` on `not(unix)`
- Rm unnecessary `tokio::select!` in `wait_on_cancellation_signal` on unix
Since `unix::wait_on_cancellation_signal_unix` already waits for ctrl + c signal.
- Optimize `extract_bin`: Send `Bytes` to blocking thread for zero-copy
- Optimize `extract_with_blocking_decoder`: Avoid dup monomorphization
- Box fut of `fetch_crate_cratesio` in `PackageInfo::resolve`
- Optimize `extract_zip_entry`: Spawn only one blocking task per fn call
by using a mspc queue for the data to be written to the `outfile`.
This would improve efficiency as using `tokio::fs::File` is expensive:
It spawns a new blocking task, which needs one heap allocation and then
pushed to a mpmc queue, and then wait for it to be done on every loop.
This also fix a race condition where the unix permission is set before
the whole file is written, which might be used by attackers.
- Optimize `extract_zip`: Use one `BytesMut` for entire extraction process
To avoid frequent allocation and deallocation.
- Optimize `extract_zip_entry`: Inc prob of reusing alloc in `BytesMut`
Performs the reserve before sending the buf over mpsc queue to
increase the possibility of reusing the previous allocation.
NOTE: `BytesMut` only reuses the previous allocation if it is the
only one holds the reference to it, which is either on the first
allocation or all the `Bytes` in the mpsc queue has been consumed,
written to the file and dropped.
Since reading from entry would have to wait for external file I/O,
this would give the blocking thread some time to flush `Bytes`
out.
- Disable unused feature fs of dep tokio
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
* Add new dep tokio-tar v0.3.0 to binstalk-downloader
* Add new dep tokio-util v0.7.4 with feat io to binstalk-downloader
* Add dep async-trait v0.1.59 to binstalk-downloader
* Add new dep async-compression v0.3.15 to binstalk-downloader
with features "gzip", "zstd", "xz", "bzip2", "tokio".
* Rewrite `Download::and_visit_tar` to use `tokio-tar`
to avoid the cumbersome `block_in_place`.
* Apply temporary workaround: Rm use of let-else in mod visitor
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
