Initial signing support (#1345)

* Add CLI options

* Add manifest types

* Thread signature policy through to fetchers

* Thread signing section through from metadata

* Implement signing validation

* Clippy

* Attempt testing

* Yes and

* Why

* fmt

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/gh_crate_meta.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* fixes

* Finish feature

* Document

* Include all fields in the signing.file template

* Readme document

* Review fixes

* Fail on non-utf8 sig

* Thank goodness for tests

* Run test in ci

* Add rsign2 commands

* Log utf8 error

* Update e2e-tests/signing.sh

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests/signing.sh` MacOS CI failure

Move the tls cert creation into `signing.sh` and sleep for 10s to wait
for https server to start.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor e2e-tests-signing files

 - Use a tempdir generated by `mktemp` for all certificates-related
   files
 - Put other checked-in files into `e2e-tests/signing`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` connection err in MacOS CI

Wait for server to start up by trying to connect to it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` passing `-subj` to `openssl` on Windows

Use single quote instead of double quote to avoid automatic expansion
from bash

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` waiting for server to startup

Remove `timeout` since it is not supported on MacOS.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Try to fix windows CI by setting `MSYS_NO_PATHCONV=1` on `openssl` cmds

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` on windows

By using double `//` for the value passed to option `-subj`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed infinite loop in `signing/wait-for-server` on Windows

Pass `--ssl-revoke-best-effort` to prevent schannel from checking ssl
revocation status.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add cap on retry attempt in `signing/wait-for-server.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Let `singing/server.py` print output to stderr

so that we can see the error message there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `signing/server.py` on MacOS CI

use `python3` since macos-latest still has python2 installed and
`python` is a symlink to `python2` there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
This commit is contained in:
Félix Saparelli 2023-09-23 16:02:56 +12:00 committed by GitHub
parent efbd20857b
commit 32beba507b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
29 changed files with 723 additions and 150 deletions

View file

@ -76,14 +76,17 @@ pub trait DataVerifier: Send + Sync {
/// This method can be called repeatedly for use with streaming messages,
/// it will be called in the order of the message received.
fn update(&mut self, data: &Bytes);
/// Finalise the data verification.
///
/// Return false if the data is invalid.
fn validate(&mut self) -> bool;
}
impl<T> DataVerifier for T
where
T: FnMut(&Bytes) + Send + Sync,
{
fn update(&mut self, data: &Bytes) {
(*self)(data)
impl DataVerifier for () {
fn update(&mut self, _: &Bytes) {}
fn validate(&mut self) -> bool {
true
}
}
@ -136,9 +139,7 @@ impl<'a> Download<'a> {
data_verifier: Some(data_verifier),
}
}
}
impl<'a> Download<'a> {
async fn get_stream(
self,
) -> Result<
@ -182,7 +183,7 @@ where
}
impl Download<'_> {
/// Download a file from the provided URL and process them in memory.
/// Download a file from the provided URL and process it in memory.
///
/// This does not support verifying a checksum due to the partial extraction
/// and will ignore one if specified.
@ -216,7 +217,7 @@ impl Download<'_> {
/// Download a file from the provided URL and extract it to the provided path.
///
/// NOTE that this would only extract directory and regular files.
/// NOTE that this will only extract directory and regular files.
#[instrument(skip(path))]
pub async fn and_extract(
self,
@ -257,6 +258,15 @@ impl Download<'_> {
inner(self, fmt, path.as_ref()).await
}
#[instrument]
pub async fn into_bytes(self) -> Result<Bytes, DownloadError> {
let bytes = self.client.get(self.url).send(true).await?.bytes().await?;
if let Some(verifier) = self.data_verifier {
verifier.update(&bytes);
}
Ok(bytes)
}
}
#[cfg(test)]