Initial signing support (#1345)

* Add CLI options

* Add manifest types

* Thread signature policy through to fetchers

* Thread signing section through from metadata

* Implement signing validation

* Clippy

* Attempt testing

* Yes and

* Why

* fmt

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/gh_crate_meta.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/bin/src/args.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Update crates/binstalk-fetchers/src/signing.rs

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* fixes

* Finish feature

* Document

* Include all fields in the signing.file template

* Readme document

* Review fixes

* Fail on non-utf8 sig

* Thank goodness for tests

* Run test in ci

* Add rsign2 commands

* Log utf8 error

* Update e2e-tests/signing.sh

Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests/signing.sh` MacOS CI failure

Move the tls cert creation into `signing.sh` and sleep for 10s to wait
for https server to start.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Refactor e2e-tests-signing files

 - Use a tempdir generated by `mktemp` for all certificates-related
   files
 - Put other checked-in files into `e2e-tests/signing`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` connection err in MacOS CI

Wait for server to start up by trying to connect to it.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` passing `-subj` to `openssl` on Windows

Use single quote instead of double quote to avoid automatic expansion
from bash

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix `e2e-tests-signing` waiting for server to startup

Remove `timeout` since it is not supported on MacOS.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Try to fix windows CI by setting `MSYS_NO_PATHCONV=1` on `openssl` cmds

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed `e2e-tests-signing` on windows

By using double `//` for the value passed to option `-subj`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fixed infinite loop in `signing/wait-for-server` on Windows

Pass `--ssl-revoke-best-effort` to prevent schannel from checking ssl
revocation status.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Add cap on retry attempt in `signing/wait-for-server.sh`

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Let `singing/server.py` print output to stderr

so that we can see the error message there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

* Fix running `signing/server.py` on MacOS CI

use `python3` since macos-latest still has python2 installed and
`python` is a symlink to `python2` there.

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>

---------

Signed-off-by: Jiahao XU <Jiahao_XU@outlook.com>
Co-authored-by: Jiahao XU <Jiahao_XU@outlook.com>
This commit is contained in:
Félix Saparelli 2023-09-23 16:02:56 +12:00 committed by GitHub
parent efbd20857b
commit 32beba507b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
29 changed files with 723 additions and 150 deletions

View file

@ -23,17 +23,35 @@ pub(super) struct RegistryConfig {
pub(super) dl: CompactString,
}
struct Sha256Digest(Sha256);
struct Sha256Digest {
expected: Vec<u8>,
actual: Option<Vec<u8>>,
state: Option<Sha256>,
}
impl Default for Sha256Digest {
fn default() -> Self {
Sha256Digest(Sha256::new())
impl Sha256Digest {
fn new(checksum: Vec<u8>) -> Self {
Self {
expected: checksum,
actual: None,
state: Some(Sha256::new()),
}
}
}
impl DataVerifier for Sha256Digest {
fn update(&mut self, data: &Bytes) {
self.0.update(data);
if let Some(ref mut state) = &mut self.state {
state.update(data);
}
}
fn validate(&mut self) -> bool {
if let Some(state) = self.state.take() {
self.actual = Some(state.finalize().to_vec());
}
self.actual.as_ref().unwrap() == &self.expected
}
}
@ -49,18 +67,16 @@ pub(super) async fn parse_manifest(
let mut manifest_visitor = ManifestVisitor::new(format!("{crate_name}-{version}").into());
let checksum = decode_base16(cksum.as_bytes()).map_err(RegistryError::from)?;
let mut sha256_digest = Sha256Digest::default();
let mut digest = Sha256Digest::new(checksum);
Download::new_with_data_verifier(client, crate_url, &mut sha256_digest)
Download::new_with_data_verifier(client, crate_url, &mut digest)
.and_visit_tar(TarBasedFmt::Tgz, &mut manifest_visitor)
.await?;
let digest_checksum = sha256_digest.0.finalize();
if digest_checksum.as_slice() != checksum.as_slice() {
if !digest.validate() {
Err(RegistryError::UnmatchedChecksum {
expected: cksum.into(),
actual: encode_base16(digest_checksum.as_slice()).into(),
expected: encode_base16(digest.expected.as_slice()).into(),
actual: encode_base16(digest.actual.unwrap().as_slice()).into(),
})
} else {
manifest_visitor.load_manifest()