Factcheck Benchmark

FactCheck is an algorithm that uses evidences found on the web to confirm or deny a particular query. FactBench is a collection of RDF models that are bounded by time spans and are group in train and test sets. This repository implements the components required by Hobbit benchmarking platform to facilitate the evaluation of FactCheck using the models provided by FactBench.

Running the Benchmark

1. Follow the steps outlined for setting up the SDK benchmark environment.
2. Clone the the FactBench repository and copy the test and train directories to the data/factbench\ directory. The resources/factbench.ini should be updated with full path on the host system when running the benchmark. On the other hand, data/factbench.ini should remain unchanged as it is used to create the container.
3. Obtain the dbpedia_metrics.sql file from FactCheck repository and place it in the factcheck-data/db directory. This file will be used to initialize the FactCheck database Docker container, when it is being created.
4. In order to create the FactCheck Docker container, the following should be present in the factcheck-data/api directory:
   • the compiled factcheck-api-*.jar file from the factcheck-api project.
   • the /machinelearning directory from the FactCheck repository. The path would be factcheck-data/api/machinelearning.
   • the /wordnet directory from the FactCheck repository. The path would be factcheck-data/api/wordnet.
   • the defacto.ini file stores the configuration that will be use when the application runs in its Docker container.

Deploying the benchmark

1. Follow the steps to build the docker images that can be uploaded to the remote respositories. Please find the basic benchmark component implementations in the sources folder. You may extend the components with logic of your benchmark and debug the components as pure java codes by running the checkHealth() method from ExampleBenchmarkTest). You may specify input parameters models for benchmark and system you are running.
2. The push.sh can be execute after logging into the repository to push the images.