Commit graph

26 commits

Author SHA1 Message Date
Jiahao XU
0c5a65fb35
Refactor: Extract new crate binstalk-bins (#1294)
To reduce `binstalk` codegen and enable reuse of it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-15 21:48:42 +00:00
Jiahao XU
76c72469eb
Refactor: Extract new crate binstalk-fetchers (#1291)
To reduce `binstalk` codegen and enable better reuse.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-08-14 03:20:34 +00:00
Jiahao XU
d8419ea5a2
feat: Add more variables for template (#1160)
Fixed #775

 - Add dep target-lexicon v0.12.7
 - Add `target-{family, arch, libc, vendor}` to
   `package.metadata.binstall`.

For `{universal, universal2}-apple-darwin`, the `target-arch` is set to
`universal`.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-20 22:05:13 +00:00
Jiahao XU
1432093dcc
feat: Add new cmdline option --no-track (#1111)
Same as `cargo-install`'s `--no-track`.
It is also passed to `cargo-install` if it is invoked.

Also fixed `fs::atomic_symlink_file` which on Windows could fallback to
non-atomic install if symlinking failed.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-06-03 09:15:18 +00:00
Jiahao XU
a27d5aebf6
Use leon for template in binstalk & detect malformed pkg-url/pkg-fmt early (#933)
Fixed #851

* Add new dep leon to crate binstalk
* Add new variant `BinstallError::Template{Parse, Render}Error`
* Use `leon::Template` in mod `bins`
* Use `leon::Template` in mod `fetchers::gh_crate_meta`
* Refactor mod `bins`: Rm unused associated fn & fields from `Context`
* Simplify unit testing in mod `fetchers::gh_crate_meta`
* Rm soft-deprecated field `fetchers::gh_crate_meta::Context::format`
  and change the `match` to resolve `archive` => `self.archive_format`.
* Make macro_rules `leon::template!` far easier to use
* Construct `leon::Template<'_>` as constant in `gh_crate_meta::hosting`
* Simplify `leon::Values` trait
   Change its method `get_value` signature to
   
   ```rust
       fn get_value(&self, key: &str) -> Option<Cow<'_, str>>;
   ```
   
   Now, `ValuesFn` also accepts non-`'static` function, but now
   `leon::Values` is only implemented for `&ValuesFn<F>` now.
   
   This makes it a little bit more cumbersome to use but I think it's a
   reasonable drawback.
* Rm `Send` bound req from `ValuesFn`
* Impl new fn `leon::Template::cast`
   for casting `Template<'s>` to `Template<'t>` where `'s: 't`
* Rename `leon::Template::has_keys` => `has_any_of_keys`
* Make checking whether format related keys are present more robust
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Skip checking all fmt ext
   if none of the format related keys ("format", "archive-format",
   "archive-suffix") are present.
* Only ret `.exe` in `PkgFmt::extensions` on windows
   by adding a new param `is_windows: bool`
* Improve debug msg in `GhCrateMeta::fetch_and_extract`
* Add warnings to `GhCrateMeta::find`
* Rm dep tinytemplate
* `impl<'s, 'rhs: 's> ops::AddAssign<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::AddAssign<Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::AddAssign<&Item<'item>> for Template<'s>`
* `impl<'s, 'rhs: 's> ops::Add<Template<'rhs>> for Template<'s>` (improved existing `Add` impl)
* `impl<'s, 'rhs: 's> ops::Add<&Template<'rhs>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<Item<'item>> for Template<'s>`
* `impl<'s, 'item: 's> ops::Add<&Item<'item>> for Template<'s>`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-03-26 16:11:10 +11:00
Jiahao XU
01c8ecb778
Fix zip extraction and bins::infer_bin_dir_template (#869)
- Fix zip extraction code: Ensure dir is rwx and file is readable for curr user
 - Add more integration test for `ExtractedFiles`
 - Fix `bins::infer_bin_dir_template` introduced in #856

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-06 17:54:31 +11:00
Jiahao XU
9c7da6a179
Return a list of files written to disk in binstalk_downloader::download::Download::and_extract (#856)
to avoid collecting extracted files from disk again in resolution stage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-03-03 23:31:27 +11:00
Jiahao XU
9874dca454
Fix bin names in .crates.toml on win (#802)
`bins::BinFile::new` does not include `binary_ext` in the
`BinFile::base_name`.

Also add regression e2e-test.

Fixed #801

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-02-15 17:24:50 +13:00
Félix Saparelli
6bc1fb4983
Fix #728: fallback to copy when symlinking on windows (#763)
* Fix #728: fallback to copy when symlinking on windows

* Signature hmm?

* Squash warnings on windows

* Just don’t use generics here
2023-02-04 14:14:08 +13:00
Jiahao XU
270bf08a24
Optimize use of tinytemplate (#647)
* Optimize `GhCrateMeta::find`: Cache compiled template `pkg_url`
* Optimize `collect_bin_files`: Cache compiled template `bin_dir`
* Refactor: Improve readability of `gh_crate_meta`
* Optimize `GhCrateMeta::launch_baseline_find_tasks`: Dedup urls before checking
* Optimize `BinFile::new`: Avoid one heap alloc when creating `dest`
* Improve `warn!` msg in `GhCrateMeta::launch_baseline_find_tasks`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2023-01-26 17:33:49 +13:00
Jiahao XU
ab4da7f059
Fix self-update on windows (#712)
Fixed #702


* Add new dep windows v0.44.0 for windows only
* Add regression test for #702
* Impl `win::replace_file`
* Use `ReplaceFileW` on windows if `fs::rename` in `fs::atomic_install` fails
* Improve logging and err messages
* Add more regression tests
* Make `BinFile::install_link` atomic: Do not remove file if dst exists
* Fallback to `ReplaceFileW` in `fs::atomic_symlink_file`
* Refactor: Extract new fn `fs::persist`
* Use `fs::persist` instead of `TempFile::persist` in `fs::atomic_install`, which fallbacks to `ReplaceFileW` on windows

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2023-01-17 18:05:07 +11:00
Jiahao XU
66abf1b8c1
More mionr optimizations (#553)
* Optimize `BinFile::preview_bin`: Use `Path::display` instead of `to_string_lossy` to avoid allocation.
* Refactor: Rm useless `AsRef::as_ref` call
* Optimize `GhCrateMeta::find`: Reduce fut size by converting `Url` to `String`
   `Url` is much larger than `String`.
* Refactor `PackageInfo::resolve`: Destruct `Meta::binstall`
* Optimize `resolve::load_manifest_path`: Avoid allocation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-22 19:35:16 +13:00
Jiahao XU
bdb4b2070d
Minor optimization (#544)
* Optimization: Rm `debug!` in `find_version`
   printing all version iterated obviously doesn't help much in debugging
   in the problem but rather just confusing.
   
   Also this makes it hard for the compiler to optimize the iterators.
* Use let-else in `ManifestVisitor`
* Optimize `BinFile::preview_{bin, link}` for zero-copy
   Return `impl Display` that lazily format instead of allocating a `String`
* Optimize `infer_bin_dir_template`: Generate dir lazily
* Optimize `find_version`: Lazily clone `version_req` only on err
* Refactor `find_version`: Use `bool::then_some`
* Add dep either v1.8.0 to binstalk
* Optimize `GhCrateMeta::find`: Avoid cloning and `Vec` creation
   by using `Either`
* Optimize `ops::install::install_from_package`: Make it a regular fn
   instead of async fn since it does not `.await` on any async fn.
* Optimize `QuickInstall`: Rm field `target`
   since `Arc<Data>` already contains that field.
* Optimize `Fetcher`s: Extract new struct `TargetData`
   so that `Data` can be shared by all fetchers, regardless of the target.
* Optimize `QuickInstall`: Rm unused field `data`
* Optimize `Resolution::print`: Replace branching with conditional move

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-21 10:32:46 +13:00
Jiahao XU
325cb5cc19
Optimizations plus bug fix (#541)
* Optimize `Download::and_extract`: Avoid dup monomorphization
* Increase buffer size for binstall_crates_v1 to `4096 * 5`
* Optimize `opts::resolve`: Avoid unnecessary `clone`s
* Fix reserve in `opts::resolve`: Do not over-reserve
* Rename field `opts::Options::resolver` => `resolvers`
* Refactor: Extract new type `resolve::PackageInfo`
    - which makes `opts::resolve_inner` easier to understand
    - reduce number of parameters required for `download_extract_and_verify` and
      `collect_bin_files`
    - reducing size of future returned by `opts::resolve_inner` by dropping
      `cargo_toml::{Manifest, Package}` as early as possible since
      `Manifest` is 3000 Bytes large while `Package` is 600 Bytes large.
* Optimize `fetchers::Data`: Use `CompactString` for field name & version
   since they are usually small enough to fit in inlined version of
   `CompactString`.
* Optimize `gh_crate_meta`: Avoid unnecessary allocation
   in `RepositoryHost::get_default_pkg_url_template`.
* Refacator: Use `Itertools::cartesian_product` in `apply_filenames_to_paths`
* Optimize `ops::resolve`: Avoid unnecessary `clone` & reduce future size
   by calling `fetcher.target_meta()` to obtain final metadata after
   downloaded and extracted the binaries.
* Optimize `ops::resolve`: Avoid unnecessary allocation
   in `download_extract_and_verify`: Replace `Itertools::join` with
   `Itertools::format` to avoid allocating the string.
* Fix disabling cargo-install fallback
* Simplify `BinFile::from_product`: Takes `&str` instead of `&Product`
   since we only need `product.name`
* Rename `BinFile::from_product` => `BinFile::new`
* Refactor: Create newtype `ops::resolve::Bin`
   so that we don't need to `unwrap()` on `Product::name`
   and reduce memory usage.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-17 23:59:35 +00:00
Jiahao XU
4e875874b6
Replace use of log with tracing (#527)
Resolved #522

* Add dep tracing v0.1.37 to binstalk
* Use `tracing` instead of `log` for logging in binstalk
* Rm dev dep `env_logger` since `log` is no longer used
* Rm unused dep `log` from binstalk

* Replace use of `log` with `tracing` in crates/bin
* Enable feat std of dep log in crates/bin

* Add dep tracing v0.1.37 to binstalk-downloader
* Replace use of `log` with `tracing` in binstalk-downloader
* Rm unused dep `log` from binstalk-downlaoder


* Wrap `ops::{install, resolve}` in `tracing::instrument`
* Wrap `Download::and_{extract, visit_tar}` in `instrument`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-13 21:45:37 +13:00
Jiahao XU
c48bc2a264
Fix crate normalize-path (#508)
* Impl new function `NormalizePath::is_normalized`
* Always normalize and ret `PathBuf` in `NormalizePath::normalize`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-11-03 23:11:43 +13:00
Jiahao XU
9ed04cf89b
Fix clippy error on unit test CI (#503)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-10-28 13:48:16 +13:00
Jiahao XU
34df6176fe
Add more to possible_dirs (#441)
Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 04:03:36 +00:00
Jiahao XU
f482e362ba
Make binaries with required-features optional (#438)
* Make binaries with `required-features` optional so that crates like `sccache` would work as expected.
* Fix `infer_bin_dir_template`: concat with `data.bin_path` introduced in #417 
* Always `chmod +x $bin` on unix in case the archive (e.g. `sccache`) did not set the executable bit of fmt == Bin.
* CI: Install `sccache` but do not run it since it requires cargo env to be ready

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 13:46:55 +10:00
Jiahao XU
4ba1e221ea
Improve auto discover (#433)
* Add new field `gh_crate_meta::Context::archive_suffix`
* Support `archive_suffix = ""` for unix
* Use `archive-suffix` in `FULL_FILENAMES` & `NOVERSION_FILENAMES` to support `PkgFmt::Bin`
* Simplify `PkgFmt::extensions`
* Fix `default_bin_dir_template` in `infer_bin_dir_template`: Fix typo `archive_suffix`
* Test `default_bin_dir_template` in CI
* Fix installation of `PkgFmt::Bin`: `chmod +x $bin` on unix if fmt == Bin
* Add miniserve to `.github/scripts/tests.sh`
* Update `SUPPORT.md`
* Add github-test-Cargo2.toml to test Github pkg-url with bin-dir provided
* Avoid allocation on processing `path_normalized` in `BinFile::from_product`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 02:52:14 +00:00
Jiahao XU
c15903684f
Feature/fix no symlink (#432)
* Refactor `BinFile::from_product`: Simplify logic flow.
* Fix `--no-symlink` behavior

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-28 02:32:22 +00:00
Jiahao XU
2cc12f9b69
Optimize bins::BinFile (#422)
* Extract `infer_bin_dir_template` and call it once in `collect_bin_files`
* Ensure `product.name.is_some()` is true
* Avoid cloning `str` in `BinFile::from_product`
* Optimize: Take reference in `bins::Data`
* Optimize: Construct `CompactString` only when needed

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-26 09:39:41 +13:00
Jiahao XU
3da5cb9d9c
Check source_file_path to ensure it is valid (#421)
* Add new variant `BinstallError::DuplicateSourceFilePath`
* Check `bin.source` in `collect_bin_files`
* Add new variant `BinstallError::InvalidSourceFilePath`
* Make sure generated source_file_path cannot access outside curdir
* Add new variant `BinstallError::EmptySourceFilePath`
* Ensure source_file_path is not empty

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-25 13:32:34 +00:00
Jiahao XU
e034d69e12
Impl try multiple default bin dirs (#417)
From @passcod :

* Make bin-dir an Option
* Use cargo-release as a test case
* WIP: Try multiple default bin dirs
* Update bins.rs
* Update cargo_toml_binstall.rs

From @NobodyXu :
* Optimize & Prepare for support multi `bin-path`s
* Ensure quickinstall bin_dir work as usual.
* Rm dup entries in `FULL_FILENAMES`
* Add second version of `NOVERSION_FILENAMES`
* Impl multiple default `bin-dir`s
* Improve doc for `BinFile::from_product`
* Fix tests.sh

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Félix Saparelli <felix@passcod.name>
2022-09-26 01:15:20 +13:00
Jiahao XU
fa79e7f105
Verify that bin_files exist in resolve stage (#382)
* Refactor: Extract new fn `BinFile::check_source_exists`
* Impl new async fn `AutoAbortJoinHandle::flattened_join`
* Impl new fn `Fetcher::fetcher_name`
* Verify that `bin_files` exist in `resolve` stage
   To ensure that the installation stage won't fail because of missing
   binaries.
* Rm unused `MultiFecther`
* Simplify `Future` impl for `AutoAbortJoinHandle`
* Add new variant `BinstallError::CargoTomlMissingPackage`
* Replace `unwrap` in `resolve_inner` with proper error handling
* Make `Fetcher::new` as a regular function
   instead of an `async` function.
* Ret `Arc<dyn Fetcher>` in trait fn `Fetcher::new`
* Refactor `resolve_inner`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
2022-09-17 09:28:22 +00:00
Félix Saparelli
e25aa50ec9
Rename lib to binstalk (#361) 2022-09-10 16:44:18 +10:00
Renamed from crates/lib/src/bins.rs (Browse further)