Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal file format changes #5

Closed
bheisler opened this issue Jun 29, 2020 · 2 comments · Fixed by #6
Closed

Internal file format changes #5

bheisler opened this issue Jun 29, 2020 · 2 comments · Fixed by #6

Comments

@bheisler
Copy link

Hey, this is just a heads-up that the internal file formats used by Criterion.rs will be changing in the next release. Specifically, sample.json and estimates.json. Since they are internal files I don't consider this a breaking change, but I thought I'd let you know so you can update critcmp.

New sample.json format contains one SavedSample:

pub(crate) enum ActualSamplingMode {
    Linear,
    Flat,
}
pub(crate) struct SavedSample {
    sampling_mode: ActualSamplingMode,
    iters: Vec<f64>,
    times: Vec<f64>,
}

New estimates.json format contains one Estimates:

pub struct ConfidenceInterval {
    pub confidence_level: f64,
    pub lower_bound: f64,
    pub upper_bound: f64,
}
pub struct Estimate {
    pub confidence_interval: ConfidenceInterval,
    pub point_estimate: f64,
    pub standard_error: f64,
}
pub struct Estimates {
    pub mean: Estimate,
    pub median: Estimate,
    pub median_abs_dev: Estimate,
    pub slope: Option<Estimate>,
    pub std_dev: Estimate,
}

I should also mention that cargo-criterion has its own completely separate file structure using CBOR instead of JSON. It's getting pretty late here so I'll let you get the structure for that from the code.

zeeshanlakhani pushed a commit to capsule-rs/capsule that referenced this issue Jul 3, 2020
- remove need for codecov token in codecov action
  * action doesn't need it in public repos and will cause issues w/ forks where secrets don't persist
- comment out coverage job until xd009642/tarpaulin#461 is solved.
- bench-pr will fail on comparison due to criterion change upstream in format
  * related to BurntSushi/critcmp#5.  Future comparisons should work fine.
@zeeshanlakhani
Copy link

zeeshanlakhani commented Jul 3, 2020

@bheisler would this be related to this new error I'm seeing when running critcmp --export changes > bench-runs/changes.json via criterion 0.3.3 (versioned as 0.3 in our code):

/home/capsule/bench/target/criterion/mbuf__alloc_vs_alloc_bulk/mbuf__alloc_bulk/changes: estimates.json: missing field `Mean` at line 1 column 982
root@buster:/home/capsule/bench

Thanks.

@bheisler
Copy link
Author

bheisler commented Jul 3, 2020

Probably related, yes.

zeeshanlakhani pushed a commit to capsule-rs/capsule that referenced this issue Jul 6, 2020
- remove need for codecov token in codecov action
  * action doesn't need it in public repos and will cause issues w/ forks where secrets don't persist
- comment out coverage job until xd009642/tarpaulin#461 is solved.
- allow critcmp comparisons to fail until BurntSushi/critcmp#5 is handled
zeeshanlakhani added a commit to capsule-rs/capsule that referenced this issue Jul 6, 2020
- remove need for codecov token in codecov action
  * action doesn't need it in public repos and will cause issues w/ forks where secrets don't persist
- comment out coverage job until xd009642/tarpaulin#461 is solved.
- allow critcmp comparisons to fail until BurntSushi/critcmp#5 is handled
yamafaktory added a commit to yamafaktory/jql that referenced this issue Aug 6, 2020
yamafaktory added a commit to yamafaktory/jql that referenced this issue Aug 6, 2020
* chore: use license instead of license-file

Bump up deps too.

* fix: downgrade criterion version

BurntSushi/critcmp#5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants