Support private github repository (#1690)

* Refactor: Create new crate binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix CI lint warnings

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `just check`: Rm deleted features from `cargo-hack` check

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract  new mod error

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Ret artifact url in `has_release_artifact`

So that we can use it to download from private repositories.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Move `test_graph_ql_error_type` to mod `error`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `cargo test` in `binstalk-git-repo-api``

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unnecessary import in mod `error::test`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename mod `request`` to `release_artifacts`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl draft version of fetching repo info

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Move `HasReleaseArtifacts` failure variants into `GhApiError`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Use `GhRepo` in `GhRelease`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Return `'static` future

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Make sure `'static` Future is returned

To make it easier to create generic function

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add logging to unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract new fn `GhApiClient::do_fetch`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rm unused `percent_encode_http_url_path`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `cargo test` run on CI

`cargo test` run all tests in one process.

As such, `set_global_default` would fail on the second call.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Optimize `GhApiClient::do_fetch`: Avoid unnecessary restful API call

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Rm param `auth_token` for restful API fn

which is always set to `None`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit test for `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor testing: Parameter-ize testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelise `test_get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Create parameter-ised `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelize `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: `gh_api_client::test::create_client` shall not be `async`

as there is no `.await` in it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Return `Url` in `GhApiClient::has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::download_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unused deps added to binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new API `GhApiClient::remote_client`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhApiClient::has_gh_token`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhRepo::try_extract_from_url`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename `ReleaseArtifactUrl` to `GhReleaseArtifactUrl`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new fn `Download::with_data_verifier`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* feature: Support private repository

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add e2e-test/private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `launch_baseline_find_tasks`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix test failure: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Temporarily enable debug output for e2e-test-private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `get_repo_info`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Improve `debug!` logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debugging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Apply suggestions from code review

* Fix compilation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix cargo fmt

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add crate binstalk-git-repo-api to release-pr.yml

* Update crates/binstalk-git-repo-api/Cargo.toml

* Apply suggestions from code review

* Update crates/binstalk/Cargo.toml

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
This commit is contained in:
Jiahao XU 2024-06-10 16:02:12 +10:00 committed by GitHub
parent 48ee0b0e3e
commit 1dbd2460a3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
30 changed files with 1838 additions and 1127 deletions

View file

@ -12,7 +12,6 @@ use itertools::Itertools;
use leon::Template;
use maybe_owned::MaybeOwned;
use semver::{Version, VersionReq};
use tempfile::TempDir;
use tokio::task::spawn_blocking;
use tracing::{debug, error, info, instrument, warn};
@ -21,7 +20,7 @@ use crate::{
errors::{BinstallError, VersionParseError},
fetchers::{Data, Fetcher, TargetData},
helpers::{
self, cargo_toml::Manifest, cargo_toml_workspace::load_manifest_from_workspace,
cargo_toml::Manifest, cargo_toml_workspace::load_manifest_from_workspace,
download::ExtractedFiles, remote::Client, target_triple::TargetTriple,
tasks::AutoAbortJoinHandle,
},
@ -361,7 +360,7 @@ impl PackageInfo {
}
#[cfg(feature = "git")]
Some(Git(git_url)) => {
use helpers::git::{GitCancellationToken, Repository as GitRepository};
use crate::helpers::git::{GitCancellationToken, Repository as GitRepository};
let git_url = git_url.clone();
let name = name.clone();
@ -370,7 +369,7 @@ impl PackageInfo {
let cancel_on_drop = cancellation_token.clone().cancel_on_drop();
let ret = spawn_blocking(move || {
let dir = TempDir::new()?;
let dir = tempfile::TempDir::new()?;
GitRepository::shallow_clone(git_url, dir.as_ref(), Some(cancellation_token))?;
load_manifest_from_workspace(dir.as_ref(), &name).map_err(BinstallError::from)