Support private github repository (#1690)

* Refactor: Create new crate binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix CI lint warnings

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `just check`: Rm deleted features from `cargo-hack` check

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract  new mod error

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Ret artifact url in `has_release_artifact`

So that we can use it to download from private repositories.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Move `test_graph_ql_error_type` to mod `error`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `cargo test` in `binstalk-git-repo-api``

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unnecessary import in mod `error::test`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename mod `request`` to `release_artifacts`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl draft version of fetching repo info

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Move `HasReleaseArtifacts` failure variants into `GhApiError`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Use `GhRepo` in `GhRelease`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Return `'static` future

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Make sure `'static` Future is returned

To make it easier to create generic function

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add logging to unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Extract new fn `GhApiClient::do_fetch`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rm unused `percent_encode_http_url_path`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `cargo test` run on CI

`cargo test` run all tests in one process.

As such, `set_global_default` would fail on the second call.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Optimize `GhApiClient::do_fetch`: Avoid unnecessary restful API call

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Rm param `auth_token` for restful API fn

which is always set to `None`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix unit test for `GhApiClient::get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor testing: Parameter-ize testing

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelise `test_get_repo_info`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: Create parameter-ised `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Parallelize `test_has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor: `gh_api_client::test::create_client` shall not be `async`

as there is no `.await` in it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Return `Url` in `GhApiClient::has_release_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Impl new API `GhApiClient::download_artifact`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Remove unused deps added to binstalk-git-repo-api

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new API `GhApiClient::remote_client`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhApiClient::has_gh_token`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add `GhRepo::try_extract_from_url`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Rename `ReleaseArtifactUrl` to `GhReleaseArtifactUrl`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add new fn `Download::with_data_verifier`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* feature: Support private repository

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add e2e-test/private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix clippy lints

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `launch_baseline_find_tasks`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix test failure: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Temporarily enable debug output for e2e-test-private-github-repo

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `get_repo_info`: Retry on rate limit

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Improve `debug!` logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debugging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add more debug logging

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Apply suggestions from code review

* Fix compilation

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix cargo fmt

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add crate binstalk-git-repo-api to release-pr.yml

* Update crates/binstalk-git-repo-api/Cargo.toml

* Apply suggestions from code review

* Update crates/binstalk/Cargo.toml

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
This commit is contained in:
Jiahao XU 2024-06-10 16:02:12 +10:00 committed by GitHub
parent 48ee0b0e3e
commit 1dbd2460a3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
30 changed files with 1838 additions and 1127 deletions

View file

@ -1,4 +1,4 @@
use std::{fmt, io, marker::PhantomData, path::Path};
use std::{fmt, io, path::Path};
use binstalk_types::cargo_toml_binstall::PkgFmtDecomposed;
use bytes::Bytes;
@ -8,7 +8,7 @@ use tracing::{debug, error, instrument};
pub use binstalk_types::cargo_toml_binstall::{PkgFmt, TarBasedFmt};
use crate::remote::{Client, Error as RemoteError, Url};
use crate::remote::{Client, Error as RemoteError, Response, Url};
mod async_extracter;
use async_extracter::*;
@ -90,38 +90,43 @@ impl DataVerifier for () {
}
}
#[derive(Debug)]
enum DownloadContent {
ToIssue { client: Client, url: Url },
Response(Response),
}
impl DownloadContent {
async fn into_response(self) -> Result<Response, DownloadError> {
Ok(match self {
DownloadContent::ToIssue { client, url } => client.get(url).send(true).await?,
DownloadContent::Response(response) => response,
})
}
}
pub struct Download<'a> {
client: Client,
url: Url,
content: DownloadContent,
data_verifier: Option<&'a mut dyn DataVerifier>,
}
impl fmt::Debug for Download<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
#[allow(dead_code, clippy::type_complexity)]
#[derive(Debug)]
struct Download<'a> {
client: &'a Client,
url: &'a Url,
data_verifier: Option<PhantomData<&'a mut dyn DataVerifier>>,
}
fmt::Debug::fmt(
&Download {
client: &self.client,
url: &self.url,
data_verifier: self.data_verifier.as_ref().map(|_| PhantomData),
},
f,
)
fmt::Debug::fmt(&self.content, f)
}
}
impl Download<'static> {
pub fn new(client: Client, url: Url) -> Self {
Self {
client,
url,
content: DownloadContent::ToIssue { client, url },
data_verifier: None,
}
}
pub fn from_response(response: Response) -> Self {
Self {
content: DownloadContent::Response(response),
data_verifier: None,
}
}
@ -134,8 +139,24 @@ impl<'a> Download<'a> {
data_verifier: &'a mut dyn DataVerifier,
) -> Self {
Self {
client,
url,
content: DownloadContent::ToIssue { client, url },
data_verifier: Some(data_verifier),
}
}
pub fn from_response_with_data_verifier(
response: Response,
data_verifier: &'a mut dyn DataVerifier,
) -> Self {
Self {
content: DownloadContent::Response(response),
data_verifier: Some(data_verifier),
}
}
pub fn with_data_verifier(self, data_verifier: &mut dyn DataVerifier) -> Download<'_> {
Download {
content: self.content,
data_verifier: Some(data_verifier),
}
}
@ -148,9 +169,10 @@ impl<'a> Download<'a> {
> {
let mut data_verifier = self.data_verifier;
Ok(self
.client
.get_stream(self.url)
.content
.into_response()
.await?
.bytes_stream()
.map(move |res| {
let bytes = res?;
@ -257,7 +279,7 @@ impl Download<'_> {
#[instrument]
pub async fn into_bytes(self) -> Result<Bytes, DownloadError> {
let bytes = self.client.get(self.url).send(true).await?.bytes().await?;
let bytes = self.content.into_response().await?.bytes().await?;
if let Some(verifier) = self.data_verifier {
verifier.update(&bytes);
}

View file

@ -1,506 +0,0 @@
use std::{
collections::HashMap,
ops::Deref,
sync::{
atomic::{AtomicBool, Ordering::Relaxed},
Arc, Mutex, RwLock,
},
time::{Duration, Instant},
};
use compact_str::CompactString;
use percent_encoding::{
percent_decode_str, utf8_percent_encode, AsciiSet, PercentEncode, CONTROLS,
};
use tokio::sync::OnceCell;
use crate::remote;
mod request;
pub use request::{GhApiContextError, GhApiError, GhGraphQLErrors};
/// default retry duration if x-ratelimit-reset is not found in response header
const DEFAULT_RETRY_DURATION: Duration = Duration::from_secs(10 * 60);
fn percent_encode_http_url_path(path: &str) -> PercentEncode<'_> {
/// https://url.spec.whatwg.org/#fragment-percent-encode-set
const FRAGMENT: &AsciiSet = &CONTROLS.add(b' ').add(b'"').add(b'<').add(b'>').add(b'`');
/// https://url.spec.whatwg.org/#path-percent-encode-set
const PATH: &AsciiSet = &FRAGMENT.add(b'#').add(b'?').add(b'{').add(b'}');
const PATH_SEGMENT: &AsciiSet = &PATH.add(b'/').add(b'%');
// The backslash (\) character is treated as a path separator in special URLs
// so it needs to be additionally escaped in that case.
//
// http is considered to have special path.
const SPECIAL_PATH_SEGMENT: &AsciiSet = &PATH_SEGMENT.add(b'\\');
utf8_percent_encode(path, SPECIAL_PATH_SEGMENT)
}
fn percent_decode_http_url_path(input: &str) -> CompactString {
if input.contains('%') {
percent_decode_str(input).decode_utf8_lossy().into()
} else {
// No '%', no need to decode.
CompactString::new(input)
}
}
/// The keys required to identify a github release.
#[derive(Clone, Eq, PartialEq, Hash, Debug)]
pub struct GhRelease {
pub owner: CompactString,
pub repo: CompactString,
pub tag: CompactString,
}
/// The Github Release and one of its artifact.
#[derive(Clone, Eq, PartialEq, Hash, Debug)]
pub struct GhReleaseArtifact {
pub release: GhRelease,
pub artifact_name: CompactString,
}
impl GhReleaseArtifact {
/// Create [`GhReleaseArtifact`] from url.
pub fn try_extract_from_url(url: &remote::Url) -> Option<Self> {
if url.domain() != Some("github.com") {
return None;
}
let mut path_segments = url.path_segments()?;
let owner = path_segments.next()?;
let repo = path_segments.next()?;
if (path_segments.next()?, path_segments.next()?) != ("releases", "download") {
return None;
}
let tag = path_segments.next()?;
let artifact_name = path_segments.next()?;
(path_segments.next().is_none() && url.fragment().is_none() && url.query().is_none()).then(
|| Self {
release: GhRelease {
owner: percent_decode_http_url_path(owner),
repo: percent_decode_http_url_path(repo),
tag: percent_decode_http_url_path(tag),
},
artifact_name: percent_decode_http_url_path(artifact_name),
},
)
}
}
#[derive(Debug)]
struct Map<K, V>(RwLock<HashMap<K, Arc<V>>>);
impl<K, V> Default for Map<K, V> {
fn default() -> Self {
Self(Default::default())
}
}
impl<K, V> Map<K, V>
where
K: Eq + std::hash::Hash,
V: Default,
{
fn get(&self, k: K) -> Arc<V> {
let optional_value = self.0.read().unwrap().deref().get(&k).cloned();
optional_value.unwrap_or_else(|| Arc::clone(self.0.write().unwrap().entry(k).or_default()))
}
}
#[derive(Debug)]
struct Inner {
client: remote::Client,
release_artifacts: Map<GhRelease, OnceCell<Option<request::Artifacts>>>,
retry_after: Mutex<Option<Instant>>,
auth_token: Option<CompactString>,
is_auth_token_valid: AtomicBool,
}
/// Github API client for querying whether a release artifact exitsts.
/// Can only handle github.com for now.
#[derive(Clone, Debug)]
pub struct GhApiClient(Arc<Inner>);
impl GhApiClient {
pub fn new(client: remote::Client, auth_token: Option<CompactString>) -> Self {
Self(Arc::new(Inner {
client,
release_artifacts: Default::default(),
retry_after: Default::default(),
auth_token,
is_auth_token_valid: AtomicBool::new(true),
}))
}
}
enum FetchReleaseArtifactError {
Error(GhApiError),
RateLimit { retry_after: Instant },
Unauthorized,
}
impl GhApiClient {
async fn do_fetch_release_artifacts(
&self,
release: &GhRelease,
auth_token: Option<&str>,
) -> Result<Option<request::Artifacts>, FetchReleaseArtifactError> {
use request::FetchReleaseRet::*;
use FetchReleaseArtifactError as Error;
match request::fetch_release_artifacts(&self.0.client, release, auth_token).await {
Ok(ReleaseNotFound) => Ok(None),
Ok(Artifacts(artifacts)) => Ok(Some(artifacts)),
Ok(ReachedRateLimit { retry_after }) => {
let retry_after = retry_after.unwrap_or(DEFAULT_RETRY_DURATION);
let now = Instant::now();
let retry_after = now
.checked_add(retry_after)
.unwrap_or_else(|| now + DEFAULT_RETRY_DURATION);
Err(Error::RateLimit { retry_after })
}
Ok(Unauthorized) => Err(Error::Unauthorized),
Err(err) => Err(Error::Error(err)),
}
}
/// The returned future is guaranteed to be pointer size.
pub async fn has_release_artifact(
&self,
GhReleaseArtifact {
release,
artifact_name,
}: GhReleaseArtifact,
) -> Result<HasReleaseArtifact, GhApiError> {
use FetchReleaseArtifactError as Error;
let once_cell = self.0.release_artifacts.get(release.clone());
let res = once_cell
.get_or_try_init(|| {
Box::pin(async {
{
let mut guard = self.0.retry_after.lock().unwrap();
if let Some(retry_after) = *guard {
if retry_after.elapsed().is_zero() {
return Err(Error::RateLimit { retry_after });
} else {
// Instant retry_after is already reached.
*guard = None;
}
};
}
if self.0.is_auth_token_valid.load(Relaxed) {
match self
.do_fetch_release_artifacts(&release, self.0.auth_token.as_deref())
.await
{
Err(Error::Unauthorized) => {
self.0.is_auth_token_valid.store(false, Relaxed);
}
res => return res,
}
}
self.do_fetch_release_artifacts(&release, None).await
})
})
.await;
match res {
Ok(Some(artifacts)) => {
let has_artifact = artifacts.contains(&artifact_name);
Ok(if has_artifact {
HasReleaseArtifact::Yes
} else {
HasReleaseArtifact::No
})
}
Ok(None) => Ok(HasReleaseArtifact::NoSuchRelease),
Err(Error::Unauthorized) => Ok(HasReleaseArtifact::Unauthorized),
Err(Error::RateLimit { retry_after }) => {
*self.0.retry_after.lock().unwrap() = Some(retry_after);
Ok(HasReleaseArtifact::RateLimit { retry_after })
}
Err(Error::Error(err)) => Err(err),
}
}
}
#[derive(Eq, PartialEq, Copy, Clone, Debug)]
pub enum HasReleaseArtifact {
Yes,
No,
NoSuchRelease,
/// GitHub returns 401 requiring a token.
/// In this case, it makes sense to fallback to HEAD/GET.
Unauthorized,
/// GitHub rate limit is applied per hour, so in case of reaching the rate
/// limit, [`GhApiClient`] will return this variant and let the user decide
/// what to do.
///
/// Usually it is more sensible to fallback to directly HEAD/GET the
/// artifact url than waiting until `retry_after`.
///
/// If you encounter this frequently, then you should consider getting an
/// authentication token (can be personal access or oath access token),
/// which should give you 5000 requests per hour per user.
///
/// Rate limit for unauthorized user is 60 requests per hour per originating
/// IP address, so it is very easy to be rate limited.
RateLimit {
retry_after: Instant,
},
}
#[cfg(test)]
mod test {
use super::*;
use compact_str::{CompactString, ToCompactString};
use std::{env, num::NonZeroU16};
mod cargo_binstall_v0_20_1 {
use super::{CompactString, GhRelease};
pub(super) const RELEASE: GhRelease = GhRelease {
owner: CompactString::new_inline("cargo-bins"),
repo: CompactString::new_inline("cargo-binstall"),
tag: CompactString::new_inline("v0.20.1"),
};
pub(super) const ARTIFACTS: &[&str] = &[
"cargo-binstall-aarch64-apple-darwin.full.zip",
"cargo-binstall-aarch64-apple-darwin.zip",
"cargo-binstall-aarch64-pc-windows-msvc.full.zip",
"cargo-binstall-aarch64-pc-windows-msvc.zip",
"cargo-binstall-aarch64-unknown-linux-gnu.full.tgz",
"cargo-binstall-aarch64-unknown-linux-gnu.tgz",
"cargo-binstall-aarch64-unknown-linux-musl.full.tgz",
"cargo-binstall-aarch64-unknown-linux-musl.tgz",
"cargo-binstall-armv7-unknown-linux-gnueabihf.full.tgz",
"cargo-binstall-armv7-unknown-linux-gnueabihf.tgz",
"cargo-binstall-armv7-unknown-linux-musleabihf.full.tgz",
"cargo-binstall-armv7-unknown-linux-musleabihf.tgz",
"cargo-binstall-universal-apple-darwin.full.zip",
"cargo-binstall-universal-apple-darwin.zip",
"cargo-binstall-x86_64-apple-darwin.full.zip",
"cargo-binstall-x86_64-apple-darwin.zip",
"cargo-binstall-x86_64-pc-windows-msvc.full.zip",
"cargo-binstall-x86_64-pc-windows-msvc.zip",
"cargo-binstall-x86_64-unknown-linux-gnu.full.tgz",
"cargo-binstall-x86_64-unknown-linux-gnu.tgz",
"cargo-binstall-x86_64-unknown-linux-musl.full.tgz",
"cargo-binstall-x86_64-unknown-linux-musl.tgz",
];
}
fn try_extract_artifact_from_str(s: &str) -> Option<GhReleaseArtifact> {
GhReleaseArtifact::try_extract_from_url(&url::Url::parse(s).unwrap())
}
fn assert_extract_gh_release_artifacts_failures(urls: &[&str]) {
for url in urls {
assert_eq!(try_extract_artifact_from_str(url), None);
}
}
#[test]
fn extract_gh_release_artifacts_failure() {
use cargo_binstall_v0_20_1::*;
let GhRelease { owner, repo, tag } = RELEASE;
assert_extract_gh_release_artifacts_failures(&[
"https://examle.com",
"https://github.com",
&format!("https://github.com/{owner}"),
&format!("https://github.com/{owner}/{repo}"),
&format!("https://github.com/{owner}/{repo}/123e"),
&format!("https://github.com/{owner}/{repo}/releases/21343"),
&format!("https://github.com/{owner}/{repo}/releases/download"),
&format!("https://github.com/{owner}/{repo}/releases/download/{tag}"),
&format!("https://github.com/{owner}/{repo}/releases/download/{tag}/a/23"),
&format!("https://github.com/{owner}/{repo}/releases/download/{tag}/a#a=12"),
&format!("https://github.com/{owner}/{repo}/releases/download/{tag}/a?page=3"),
]);
}
#[test]
fn extract_gh_release_artifacts_success() {
use cargo_binstall_v0_20_1::*;
let GhRelease { owner, repo, tag } = RELEASE;
for artifact in ARTIFACTS {
let GhReleaseArtifact {
release,
artifact_name,
} = try_extract_artifact_from_str(&format!(
"https://github.com/{owner}/{repo}/releases/download/{tag}/{artifact}"
))
.unwrap();
assert_eq!(release, RELEASE);
assert_eq!(artifact_name, artifact);
}
}
/// Mark this as an async fn so that you won't accidentally use it in
/// sync context.
async fn create_client() -> Vec<GhApiClient> {
let client = remote::Client::new(
concat!(env!("CARGO_PKG_NAME"), "/", env!("CARGO_PKG_VERSION")),
None,
NonZeroU16::new(10).unwrap(),
1.try_into().unwrap(),
[],
)
.unwrap();
let mut gh_clients = vec![GhApiClient::new(client.clone(), None)];
if let Ok(token) = env::var("GITHUB_TOKEN") {
gh_clients.push(GhApiClient::new(client, Some(token.into())));
}
gh_clients
}
async fn test_specific_release(release: &GhRelease, artifacts: &[&str]) {
for client in create_client().await {
eprintln!("In client {client:?}");
for artifact_name in artifacts {
let ret = client
.has_release_artifact(GhReleaseArtifact {
release: release.clone(),
artifact_name: artifact_name.to_compact_string(),
})
.await
.unwrap();
assert!(
matches!(
ret,
HasReleaseArtifact::Yes | HasReleaseArtifact::RateLimit { .. }
),
"for '{artifact_name}': answer is {:#?}",
ret
);
}
let ret = client
.has_release_artifact(GhReleaseArtifact {
release: release.clone(),
artifact_name: "123z".to_compact_string(),
})
.await
.unwrap();
assert!(
matches!(
ret,
HasReleaseArtifact::No | HasReleaseArtifact::RateLimit { .. }
),
"ret = {:#?}",
ret
);
}
}
#[tokio::test]
async fn test_gh_api_client_cargo_binstall_v0_20_1() {
test_specific_release(
&cargo_binstall_v0_20_1::RELEASE,
cargo_binstall_v0_20_1::ARTIFACTS,
)
.await
}
#[tokio::test]
async fn test_gh_api_client_cargo_binstall_no_such_release() {
for client in create_client().await {
let release = GhRelease {
owner: "cargo-bins".to_compact_string(),
repo: "cargo-binstall".to_compact_string(),
// We are currently at v0.20.1 and we would never release
// anything older than v0.20.1
tag: "v0.18.2".to_compact_string(),
};
let ret = client
.has_release_artifact(GhReleaseArtifact {
release,
artifact_name: "1234".to_compact_string(),
})
.await
.unwrap();
assert!(
matches!(
ret,
HasReleaseArtifact::NoSuchRelease | HasReleaseArtifact::RateLimit { .. }
),
"ret = {:#?}",
ret
);
}
}
mod cargo_audit_v_0_17_6 {
use super::*;
const RELEASE: GhRelease = GhRelease {
owner: CompactString::new_inline("rustsec"),
repo: CompactString::new_inline("rustsec"),
tag: CompactString::new_inline("cargo-audit/v0.17.6"),
};
const ARTIFACTS: &[&str] = &[
"cargo-audit-aarch64-unknown-linux-gnu-v0.17.6.tgz",
"cargo-audit-armv7-unknown-linux-gnueabihf-v0.17.6.tgz",
"cargo-audit-x86_64-apple-darwin-v0.17.6.tgz",
"cargo-audit-x86_64-pc-windows-msvc-v0.17.6.zip",
"cargo-audit-x86_64-unknown-linux-gnu-v0.17.6.tgz",
"cargo-audit-x86_64-unknown-linux-musl-v0.17.6.tgz",
];
#[test]
fn extract_with_escaped_characters() {
let release_artifact = try_extract_artifact_from_str(
"https://github.com/rustsec/rustsec/releases/download/cargo-audit%2Fv0.17.6/cargo-audit-aarch64-unknown-linux-gnu-v0.17.6.tgz"
).unwrap();
assert_eq!(
release_artifact,
GhReleaseArtifact {
release: RELEASE,
artifact_name: CompactString::from(
"cargo-audit-aarch64-unknown-linux-gnu-v0.17.6.tgz",
)
}
);
}
#[tokio::test]
async fn test_gh_api_client_cargo_audit_v_0_17_6() {
test_specific_release(&RELEASE, ARTIFACTS).await
}
}
}

View file

@ -1,447 +0,0 @@
use std::{
borrow::Borrow,
collections::HashSet,
error, fmt,
hash::{Hash, Hasher},
io,
sync::OnceLock,
time::Duration,
};
use compact_str::{CompactString, ToCompactString};
use reqwest::{header::HeaderMap, StatusCode};
use serde::{de::Deserializer, Deserialize, Serialize};
use serde_json::to_string as to_json_string;
use thiserror::Error as ThisError;
use tracing::debug;
use url::Url;
use super::{percent_encode_http_url_path, remote, GhRelease};
#[derive(ThisError, Debug)]
#[error("Context: '{context}', err: '{err}'")]
pub struct GhApiContextError {
context: CompactString,
#[source]
err: GhApiError,
}
#[derive(ThisError, Debug)]
#[non_exhaustive]
pub enum GhApiError {
#[error("IO Error: {0}")]
Io(#[from] io::Error),
#[error("Remote Error: {0}")]
Remote(#[from] remote::Error),
#[error("Failed to parse url: {0}")]
InvalidUrl(#[from] url::ParseError),
/// A wrapped error providing the context the error is about.
#[error(transparent)]
Context(Box<GhApiContextError>),
#[error("Remote failed to process GraphQL query: {0}")]
GraphQLErrors(#[from] GhGraphQLErrors),
}
impl GhApiError {
/// Attach context to [`GhApiError`]
pub fn context(self, context: impl fmt::Display) -> Self {
Self::Context(Box::new(GhApiContextError {
context: context.to_compact_string(),
err: self,
}))
}
}
// Only include fields we do care about
#[derive(Eq, Deserialize, Debug)]
struct Artifact {
name: CompactString,
}
// Manually implement PartialEq and Hash to ensure it will always produce the
// same hash as a str with the same content, and that the comparison will be
// the same to coparing a string.
impl PartialEq for Artifact {
fn eq(&self, other: &Self) -> bool {
self.name.eq(&other.name)
}
}
impl Hash for Artifact {
fn hash<H>(&self, state: &mut H)
where
H: Hasher,
{
let s: &str = self.name.as_str();
s.hash(state)
}
}
// Implement Borrow so that we can use call
// `HashSet::contains::<str>`
impl Borrow<str> for Artifact {
fn borrow(&self) -> &str {
&self.name
}
}
#[derive(Debug, Default, Deserialize)]
pub(super) struct Artifacts {
assets: HashSet<Artifact>,
}
impl Artifacts {
pub(super) fn contains(&self, artifact_name: &str) -> bool {
self.assets.contains(artifact_name)
}
}
pub(super) enum FetchReleaseRet {
ReachedRateLimit { retry_after: Option<Duration> },
ReleaseNotFound,
Artifacts(Artifacts),
Unauthorized,
}
fn check_for_status(status: StatusCode, headers: &HeaderMap) -> Option<FetchReleaseRet> {
match status {
remote::StatusCode::FORBIDDEN
if headers
.get("x-ratelimit-remaining")
.map(|val| val == "0")
.unwrap_or(false) =>
{
Some(FetchReleaseRet::ReachedRateLimit {
retry_after: headers.get("x-ratelimit-reset").and_then(|value| {
let secs = value.to_str().ok()?.parse().ok()?;
Some(Duration::from_secs(secs))
}),
})
}
remote::StatusCode::UNAUTHORIZED => Some(FetchReleaseRet::Unauthorized),
remote::StatusCode::NOT_FOUND => Some(FetchReleaseRet::ReleaseNotFound),
_ => None,
}
}
async fn fetch_release_artifacts_restful_api(
client: &remote::Client,
GhRelease { owner, repo, tag }: &GhRelease,
auth_token: Option<&str>,
) -> Result<FetchReleaseRet, GhApiError> {
let mut request_builder = client
.get(Url::parse(&format!(
"https://api.github.com/repos/{owner}/{repo}/releases/tags/{tag}",
owner = percent_encode_http_url_path(owner),
repo = percent_encode_http_url_path(repo),
tag = percent_encode_http_url_path(tag),
))?)
.header("Accept", "application/vnd.github+json")
.header("X-GitHub-Api-Version", "2022-11-28");
if let Some(auth_token) = auth_token {
request_builder = request_builder.bearer_auth(&auth_token);
}
let response = request_builder.send(false).await?;
if let Some(ret) = check_for_status(response.status(), response.headers()) {
Ok(ret)
} else {
Ok(FetchReleaseRet::Artifacts(response.json().await?))
}
}
#[derive(Deserialize)]
enum GraphQLResponse {
#[serde(rename = "data")]
Data(GraphQLData),
#[serde(rename = "errors")]
Errors(GhGraphQLErrors),
}
#[derive(Debug, Deserialize)]
pub struct GhGraphQLErrors(Box<[GraphQLError]>);
impl GhGraphQLErrors {
fn is_rate_limited(&self) -> bool {
self.0
.iter()
.any(|error| matches!(error.error_type, GraphQLErrorType::RateLimited))
}
}
impl error::Error for GhGraphQLErrors {}
impl fmt::Display for GhGraphQLErrors {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let last_error_index = self.0.len() - 1;
for (i, error) in self.0.iter().enumerate() {
write!(
f,
"type: '{error_type}', msg: '{msg}'",
error_type = error.error_type,
msg = error.message,
)?;
for location in error.locations.as_deref().into_iter().flatten() {
write!(
f,
", occured on query line {line} col {col}",
line = location.line,
col = location.column
)?;
}
for (k, v) in &error.others {
write!(f, ", {k}: {v}")?;
}
if i < last_error_index {
f.write_str("\n")?;
}
}
Ok(())
}
}
#[derive(Debug, Deserialize)]
struct GraphQLError {
message: CompactString,
locations: Option<Box<[GraphQLLocation]>>,
#[serde(rename = "type")]
error_type: GraphQLErrorType,
#[serde(flatten, with = "tuple_vec_map")]
others: Vec<(CompactString, serde_json::Value)>,
}
#[derive(Debug)]
enum GraphQLErrorType {
RateLimited,
Other(CompactString),
}
impl fmt::Display for GraphQLErrorType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(match self {
GraphQLErrorType::RateLimited => "RATE_LIMITED",
GraphQLErrorType::Other(s) => s,
})
}
}
impl<'de> Deserialize<'de> for GraphQLErrorType {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
let s = CompactString::deserialize(deserializer)?;
Ok(match &*s {
"RATE_LIMITED" => GraphQLErrorType::RateLimited,
_ => GraphQLErrorType::Other(s),
})
}
}
#[derive(Debug, Deserialize)]
struct GraphQLLocation {
line: u64,
column: u64,
}
#[derive(Deserialize)]
struct GraphQLData {
repository: Option<GraphQLRepo>,
}
#[derive(Deserialize)]
struct GraphQLRepo {
release: Option<GraphQLRelease>,
}
#[derive(Deserialize)]
struct GraphQLRelease {
#[serde(rename = "releaseAssets")]
assets: GraphQLReleaseAssets,
}
#[derive(Deserialize)]
struct GraphQLReleaseAssets {
nodes: Vec<Artifact>,
#[serde(rename = "pageInfo")]
page_info: GraphQLPageInfo,
}
#[derive(Deserialize)]
struct GraphQLPageInfo {
#[serde(rename = "endCursor")]
end_cursor: Option<CompactString>,
#[serde(rename = "hasNextPage")]
has_next_page: bool,
}
enum FilterCondition {
Init,
After(CompactString),
}
impl fmt::Display for FilterCondition {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
// GitHub imposes a limit of 100 for the value passed to param "first"
FilterCondition::Init => f.write_str("first:100"),
FilterCondition::After(end_cursor) => write!(f, r#"first:100,after:"{end_cursor}""#),
}
}
}
#[derive(Serialize)]
struct GraphQLQuery {
query: String,
}
async fn fetch_release_artifacts_graphql_api(
client: &remote::Client,
GhRelease { owner, repo, tag }: &GhRelease,
auth_token: &str,
) -> Result<FetchReleaseRet, GhApiError> {
static GRAPHQL_ENDPOINT: OnceLock<Url> = OnceLock::new();
let graphql_endpoint = GRAPHQL_ENDPOINT.get_or_init(|| {
Url::parse("https://api.github.com/graphql").expect("Literal provided must be a valid url")
});
let mut artifacts = Artifacts::default();
let mut cond = FilterCondition::Init;
loop {
let query = format!(
r#"
query {{
repository(owner:"{owner}",name:"{repo}") {{
release(tagName:"{tag}") {{
releaseAssets({cond}) {{
nodes {{ name }}
pageInfo {{ endCursor hasNextPage }}
}}
}}
}}
}}"#
);
let graphql_query = to_json_string(&GraphQLQuery { query }).map_err(remote::Error::from)?;
debug!("Sending graphql query to https://api.github.com/graphql: '{graphql_query}'");
let request_builder = client
.post(graphql_endpoint.clone(), graphql_query)
.header("Accept", "application/vnd.github+json")
.bearer_auth(&auth_token);
let response = request_builder.send(false).await?;
if let Some(ret) = check_for_status(response.status(), response.headers()) {
return Ok(ret);
}
let response: GraphQLResponse = response.json().await?;
let data = match response {
GraphQLResponse::Data(data) => data,
GraphQLResponse::Errors(errors) if errors.is_rate_limited() => {
return Ok(FetchReleaseRet::ReachedRateLimit { retry_after: None })
}
GraphQLResponse::Errors(errors) => return Err(errors.into()),
};
let assets = data
.repository
.and_then(|repository| repository.release)
.map(|release| release.assets);
if let Some(assets) = assets {
artifacts.assets.extend(assets.nodes);
match assets.page_info {
GraphQLPageInfo {
end_cursor: Some(end_cursor),
has_next_page: true,
} => {
cond = FilterCondition::After(end_cursor);
}
_ => break Ok(FetchReleaseRet::Artifacts(artifacts)),
}
} else {
break Ok(FetchReleaseRet::ReleaseNotFound);
}
}
}
pub(super) async fn fetch_release_artifacts(
client: &remote::Client,
release: &GhRelease,
auth_token: Option<&str>,
) -> Result<FetchReleaseRet, GhApiError> {
if let Some(auth_token) = auth_token {
let res = fetch_release_artifacts_graphql_api(client, release, auth_token)
.await
.map_err(|err| err.context("GraphQL API"));
match res {
// Fallback to Restful API
Ok(FetchReleaseRet::Unauthorized) => (),
res => return res,
}
}
fetch_release_artifacts_restful_api(client, release, auth_token)
.await
.map_err(|err| err.context("Restful API"))
}
#[cfg(test)]
mod test {
use super::*;
use serde::de::value::{BorrowedStrDeserializer, Error};
macro_rules! assert_matches {
($expression:expr, $pattern:pat $(if $guard:expr)? $(,)?) => {
match $expression {
$pattern $(if $guard)? => true,
expr => {
panic!(
"assertion failed: `{expr:?}` does not match `{}`",
stringify!($pattern $(if $guard)?)
)
}
}
}
}
#[test]
fn test_graph_ql_error_type() {
let deserialize = |input: &str| {
GraphQLErrorType::deserialize(BorrowedStrDeserializer::<'_, Error>::new(input)).unwrap()
};
assert_matches!(deserialize("RATE_LIMITED"), GraphQLErrorType::RateLimited);
assert_matches!(
deserialize("rATE_LIMITED"),
GraphQLErrorType::Other(val) if val == CompactString::new("rATE_LIMITED")
);
}
}

View file

@ -1,15 +1,6 @@
#![cfg_attr(docsrs, feature(doc_auto_cfg))]
pub use bytes;
pub mod download;
/// Github API client.
/// Currently only support github.com and does not support other enterprise
/// github.
#[cfg(feature = "gh-api-client")]
pub mod gh_api_client;
pub mod remote;
mod utils;