Extension of the metric reported by the benchmark runner #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pull Request Test Coverage Report for Build 11065473974Details
💛 - Coveralls |
rubensworks
reviewed
Sep 25, 2024
|
@surilindur could you also have a look? |
|
I am not sure I understand what the 'raw aggregate result' means or how it differs from the aggregated results or the raw results, but if it works then I think it is great. 🙂 The changes do not seem to modify a lot of existing things. |
It just add all the time measurement with the aggregate results. Perhaps, I could find a better name. |
surilindur
approved these changes
Sep 25, 2024
rubensworks
requested changes
Sep 26, 2024
Co-authored-by: Ruben Taelman <[email protected]>
rubensworks
reviewed
Sep 26, 2024
…egateResult, and optional iteration results made mandatory.
…aton/sparql-benchmark-runner.js into feature/extended_results
|
Great! Released as 4.3.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Main changes
This PR adds new metrics to the benchmark runner for better statistical analysis. There are two major changes.
The first one is that for the timestamps, time, and number of HTTP requests, the standard deviation is reported. This change is not configurable. The standard deviation was not added to the result field, but I do not expect it to be used. I believe that when it comes to results, the user would want to know if they are the same or not and do not want to do more analysis on it.
The second is an option via the
outputRawResultsparameter inISparqlBenchmarkRunnerArgsto generate in the output the raw data. I only implemented it for the time being because I expected it to be the only measurement where a user would want to do a more extensive statistical analysis, such as statistical significance and analysis of the distribution of the results. This field is optional and generates a newIAggregateResultwith the extra information. I opted not to generate a separate file so that I do not change the current interface of the runner.Minor changes
Some light documentation has been added.