From 8cc5730aa83c49b4e40791c558b8cc7e7406daf2 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Thu, 3 Aug 2023 09:24:29 -0400 Subject: [PATCH 01/13] docs: rename experimentalists in list included with index.md --- docs/experimentalist/index.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index 6382c8227..f3f5563bb 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -28,13 +28,12 @@ experimental conditions that have already been probed $\vec{x}' \in X'$, or respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented in AutoRA. -| Experimentalist | Function | Arguments | -|------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------| -| Random | $\vec{x_i} \sim U[a_i,b_i]$ | | -| Novelty | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | -| Least Confident | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | -| Model Comparison | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | -| Falsification | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | - +| Experimentalist | Function | Arguments | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------| +| Random | $\vec{x_i} \sim U[a_i,b_i]$ | | +| Novelty | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | +| Uncertainty | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | +| Model Disagreement | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | +| Falsification | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | From a07c9a7981dd99f90a28e9e82189ee8557a7ef3a Mon Sep 17 00:00:00 2001 From: benwandrew Date: Thu, 3 Aug 2023 09:50:03 -0400 Subject: [PATCH 02/13] docs: add additional experimentalists and link to packages --- docs/experimentalist/index.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index f3f5563bb..8ea0fa2c7 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -12,7 +12,7 @@ As an example, consider a visual discrimination tasks in which participants are with two lines of different lengths, and are asked to indicate which line is longer. There are two factors in this experiment: the length of the first line and the length of the second line. Instances of the two line lengths -(e.g., 2.0 cm for the first line and 2.1 cm for the sceond line) +(e.g., 2.0 cm for the first line and 2.1 cm for the second line) can be considered levels of the two factors, respectively. Thus, *an experimental condition is a vector of values that corresponds to a specific combination of experiment levels $x_i$, each of which is an instance of an experiment factor.* @@ -28,12 +28,15 @@ experimental conditions that have already been probed $\vec{x}' \in X'$, or respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented in AutoRA. -| Experimentalist | Function | Arguments | -|--------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------| -| Random | $\vec{x_i} \sim U[a_i,b_i]$ | | -| Novelty | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | -| Uncertainty | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | -| Model Disagreement | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | -| Falsification | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | - - +| Name | Docs | Function | Arguments | +|---------------------------------------------------------------------------------------------------|------|-------------------------------------------------------------------------------------------------------------------------------|-------------| +| [Random](https://pypi.org/project/autora-core/) | | $\vec{x_i} \sim U[a_i,b_i]$ | | +| [Novelty](https://pypi.org/project/autora-experimentalist-sampler-novelty/) | | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | +| [Uncertainty](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/) | | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | +| [Model Disagreement](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/) | | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | +| [Falsification](https://pypi.org/project/autora-experimentalist-falsification/) | | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | +| [Mixture](https://pypi.org/project/mixture-experimentalist/) | | [ ] | [ ] | +| [Nearest Value](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/) | | [ ] | [ ] | +| Leverage | | [ ] | [ ] | +| [Inequality](https://pypi.org/project/autora-experimentalist-sampler-inequality/) | | [ ] | [ ] | +| Assumption | | [ ] | [ ] | From cb6871f92fdd737046b658b3a0c86a94c43b0fc8 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Thu, 3 Aug 2023 10:07:15 -0400 Subject: [PATCH 03/13] docs: move links to a new column --- docs/experimentalist/index.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index 8ea0fa2c7..4a25652fa 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -28,15 +28,15 @@ experimental conditions that have already been probed $\vec{x}' \in X'$, or respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented in AutoRA. -| Name | Docs | Function | Arguments | -|---------------------------------------------------------------------------------------------------|------|-------------------------------------------------------------------------------------------------------------------------------|-------------| -| [Random](https://pypi.org/project/autora-core/) | | $\vec{x_i} \sim U[a_i,b_i]$ | | -| [Novelty](https://pypi.org/project/autora-experimentalist-sampler-novelty/) | | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | -| [Uncertainty](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/) | | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | -| [Model Disagreement](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/) | | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | -| [Falsification](https://pypi.org/project/autora-experimentalist-falsification/) | | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | -| [Mixture](https://pypi.org/project/mixture-experimentalist/) | | [ ] | [ ] | -| [Nearest Value](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/) | | [ ] | [ ] | -| Leverage | | [ ] | [ ] | -| [Inequality](https://pypi.org/project/autora-experimentalist-sampler-inequality/) | | [ ] | [ ] | -| Assumption | | [ ] | [ ] | +| Name | Links | Function | Arguments | +|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------| +| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | $\vec{x_i} \sim U[a_i,b_i]$ | | +| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | +| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | +| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | +| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | +| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | [ ] | [ ] | +| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | [ ] | [ ] | +| Leverage | | [ ] | [ ] | +| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | [ ] | [ ] | +| Assumption | | [ ] | [ ] | From 2dc8d6f147c2a0f4629c689c45a0a3fe5552fb89 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Thu, 3 Aug 2023 11:07:47 -0400 Subject: [PATCH 04/13] docs: add descriptions and arguments --- docs/experimentalist/index.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index 4a25652fa..b4e0bec3c 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -28,15 +28,15 @@ experimental conditions that have already been probed $\vec{x}' \in X'$, or respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented in AutoRA. -| Name | Links | Function | Arguments | -|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------| -| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | $\vec{x_i} \sim U[a_i,b_i]$ | | -| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ | -| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ | -| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ | -| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ | -| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | [ ] | [ ] | -| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | [ ] | [ ] | -| Leverage | | [ ] | [ ] | -| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | [ ] | [ ] | -| Assumption | | [ ] | [ ] | +| Name | Links | Description | Arguments | +|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------| +| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | | +| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ | +| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ | +| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ | +| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ | +| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ | +| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ | +| Leverage | | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ | +| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ | +| Assumption | | [ ] | [ ] | From 3a2d44bc8e36ba62ba581e07f56709e6712546f9 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Fri, 4 Aug 2023 07:37:01 -0400 Subject: [PATCH 05/13] docs: add package link for leverage sampler --- docs/experimentalist/index.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index b4e0bec3c..9cf0c9319 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -37,6 +37,5 @@ respective dependent measures $\vec{y}' \in Y'$. The following table includes th | Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ | | Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ | | Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ | -| Leverage | | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ | +| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ | | Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ | -| Assumption | | [ ] | [ ] | From fc2806757611f217cb85e3691e29ac31afe2b7b8 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Fri, 4 Aug 2023 07:49:17 -0400 Subject: [PATCH 06/13] docs: add docs link for leverage sampler --- docs/experimentalist/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/experimentalist/index.md b/docs/experimentalist/index.md index 9cf0c9319..52183c1a6 100644 --- a/docs/experimentalist/index.md +++ b/docs/experimentalist/index.md @@ -37,5 +37,5 @@ respective dependent measures $\vec{y}' \in Y'$. The following table includes th | Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ | | Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ | | Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ | -| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ | +| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ | | Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ | From c14c3257e1bc0f8fe3fec9e1abb80aa4ac87d1d7 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Mon, 7 Aug 2023 08:09:10 -0400 Subject: [PATCH 07/13] docs: update index.md for theorists to include overview image and table of implementations. --- docs/theorist/index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/theorist/index.md b/docs/theorist/index.md index 1afc57ca5..073af8715 100644 --- a/docs/theorist/index.md +++ b/docs/theorist/index.md @@ -1,3 +1,14 @@ # Theorist Overview AutoRA consists of a set of techniques designed to automate the construction of interpretable models from data. To approach this problem, we can consider computational models as small, interpretable [computation graphs](https://medium.com/tebs-lab/deep-neural-networks-as-computational-graphs-867fcaa56c9) (see also [Musslick, 2021](https://arxiv.org/abs/2103.13939)). A computation graph can take experiment parameters as input (e.g. the brightness of a visual stimulus) and can transform this input through a combination of functions to produce observable dependent measures as output (e.g. the probability that a participant can detect the stimulus). + +![Overview](../img/theorist.png) + +The following table includes the theorists currently implemented in AutoRA. + +| Name | Links | Description | Arguments | +|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| +| Differentiable Architecture Search (DARTS) | [Package](https://pypi.org/project/autora-theorist-darts/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | A theorist that automates the discovery of neural network architectures by making architecture search amenable to gradient descent. | $X', Y'$ | +| Bayesian Machine Scientist (BMS) | [Package](https://pypi.org/project/autora-theorist-bms/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bms/) | A theorist that uses a Bayesian approach to symbolic regression to find interpretable expressions which capture relationships within data. | $X', Y'$ | +| Bayesian Symbolic Regression (BSR) | [Package](https://pypi.org/project/autora-theorist-bsr/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bsr/) | A theorist that uses a Bayesian approach to symbolic regression to find compositions of simple functions that accurately map independent variables to dependent variables within a given dataset. | $X', Y'$ | + From 02cbb5897f19640ec2a31dce90f7897eb8bb4b5a Mon Sep 17 00:00:00 2001 From: benwandrew Date: Mon, 7 Aug 2023 08:31:24 -0400 Subject: [PATCH 08/13] docs: add language about arguments --- docs/theorist/index.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/theorist/index.md b/docs/theorist/index.md index 073af8715..9d2fa6d44 100644 --- a/docs/theorist/index.md +++ b/docs/theorist/index.md @@ -4,7 +4,8 @@ AutoRA consists of a set of techniques designed to automate the construction of ![Overview](../img/theorist.png) -The following table includes the theorists currently implemented in AutoRA. +Theorist use information about experimental conditions that have already been probed $\vec{x}' \in X'$ and +respective dependent measures $\vec{y}' \in Y'$. The following table includes the theorists currently implemented in AutoRA. | Name | Links | Description | Arguments | |--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| From 83a84dad822aea7e950daf2b6f8ea07fe8720df9 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Mon, 7 Aug 2023 08:47:29 -0400 Subject: [PATCH 09/13] docs: add overview image and table of implementations for experiment runner index.md --- docs/experiment-runner/index.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/experiment-runner/index.md b/docs/experiment-runner/index.md index 9b0b52983..bbefa4f9a 100644 --- a/docs/experiment-runner/index.md +++ b/docs/experiment-runner/index.md @@ -1,3 +1,17 @@ # Experiment Runner Overview -AutoRA includes tools for running synthetic and real experiments. +AutoRA includes tools for running synthetic and real-world experiments. + +![Overview](../img/experiment_runner.png) + +For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted (e.g., Firebase). Recruitment managers facilitate recruitment and coordination of study participants. + +Experiment runner tools use information about new experimental conditions $\vec{x}' \in X$. The following table includes the various experiment runner tools currently implemented in AutoRA. + +| Name | Links | Description | Arguments | +|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| +| Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. | | +| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. | $X$ | +| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. | | +| Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. | $X$ | + From f9a3fb874c5356c30f2a9d3558ea98de165c01d1 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Mon, 7 Aug 2023 08:56:01 -0400 Subject: [PATCH 10/13] docs: update language in theorist table --- docs/theorist/index.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/theorist/index.md b/docs/theorist/index.md index 9d2fa6d44..d34179db2 100644 --- a/docs/theorist/index.md +++ b/docs/theorist/index.md @@ -7,9 +7,9 @@ AutoRA consists of a set of techniques designed to automate the construction of Theorist use information about experimental conditions that have already been probed $\vec{x}' \in X'$ and respective dependent measures $\vec{y}' \in Y'$. The following table includes the theorists currently implemented in AutoRA. -| Name | Links | Description | Arguments | -|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| -| Differentiable Architecture Search (DARTS) | [Package](https://pypi.org/project/autora-theorist-darts/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | A theorist that automates the discovery of neural network architectures by making architecture search amenable to gradient descent. | $X', Y'$ | -| Bayesian Machine Scientist (BMS) | [Package](https://pypi.org/project/autora-theorist-bms/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bms/) | A theorist that uses a Bayesian approach to symbolic regression to find interpretable expressions which capture relationships within data. | $X', Y'$ | -| Bayesian Symbolic Regression (BSR) | [Package](https://pypi.org/project/autora-theorist-bsr/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bsr/) | A theorist that uses a Bayesian approach to symbolic regression to find compositions of simple functions that accurately map independent variables to dependent variables within a given dataset. | $X', Y'$ | +| Name | Links | Description | Arguments | +|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| +| Differentiable Architecture Search (DARTS) | [Package](https://pypi.org/project/autora-theorist-darts/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | A theorist that automates the discovery of neural network architectures by making architecture search amenable to gradient descent. | $X', Y'$ | +| Bayesian Machine Scientist (BMS) | [Package](https://pypi.org/project/autora-theorist-bms/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bms/) | A theorist that uses one algorithmic Bayesian approach to symbolic regression, with the aim of discovering interpretable expressions which capture relationships within data. | $X', Y'$ | +| Bayesian Symbolic Regression (BSR) | [Package](https://pypi.org/project/autora-theorist-bsr/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bsr/) | A theorist that uses another algorithmic Bayesian approach to symbolic regression, with the aim of discovering interpretable expressions which capture relationships within data. | $X', Y'$ | From a1a12cf1042e13cb632bddef789ff9a20a356f1e Mon Sep 17 00:00:00 2001 From: benwandrew Date: Mon, 7 Aug 2023 08:59:06 -0400 Subject: [PATCH 11/13] docs: language for experiment runner overview --- docs/experiment-runner/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/experiment-runner/index.md b/docs/experiment-runner/index.md index bbefa4f9a..6e32c8d05 100644 --- a/docs/experiment-runner/index.md +++ b/docs/experiment-runner/index.md @@ -4,9 +4,9 @@ AutoRA includes tools for running synthetic and real-world experiments. ![Overview](../img/experiment_runner.png) -For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted (e.g., Firebase). Recruitment managers facilitate recruitment and coordination of study participants. +For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants. -Experiment runner tools use information about new experimental conditions $\vec{x}' \in X$. The following table includes the various experiment runner tools currently implemented in AutoRA. +Experiment runner tools may take in information about new experimental conditions $\vec{x}' \in X$. The following table includes the various experiment runner tools currently implemented in AutoRA. | Name | Links | Description | Arguments | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| From 2b98f12513fd024ed66d20d109e4dc969586eb51 Mon Sep 17 00:00:00 2001 From: benwandrew Date: Tue, 8 Aug 2023 05:30:39 -0400 Subject: [PATCH 12/13] docs: remove arguments from experiment runner table --- docs/experiment-runner/index.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/experiment-runner/index.md b/docs/experiment-runner/index.md index 6e32c8d05..6f520f382 100644 --- a/docs/experiment-runner/index.md +++ b/docs/experiment-runner/index.md @@ -8,10 +8,10 @@ For synthetic experiments, these tools consist of ground-truth models from vario Experiment runner tools may take in information about new experimental conditions $\vec{x}' \in X$. The following table includes the various experiment runner tools currently implemented in AutoRA. -| Name | Links | Description | Arguments | -|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------| -| Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. | | -| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. | $X$ | -| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. | | -| Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. | $X$ | +| Name | Links | Description | +|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. | +| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. | +| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. | +| Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. | From 0d3ac432fd12c95d222872a1d5ca66937f38f3ba Mon Sep 17 00:00:00 2001 From: benwandrew Date: Tue, 8 Aug 2023 05:34:27 -0400 Subject: [PATCH 13/13] docs: change language about what experiment runners can take in Co-authored-by: Sebastian Musslick --- docs/experiment-runner/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/experiment-runner/index.md b/docs/experiment-runner/index.md index 6f520f382..8f2e0d25a 100644 --- a/docs/experiment-runner/index.md +++ b/docs/experiment-runner/index.md @@ -6,7 +6,7 @@ AutoRA includes tools for running synthetic and real-world experiments. For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants. -Experiment runner tools may take in information about new experimental conditions $\vec{x}' \in X$. The following table includes the various experiment runner tools currently implemented in AutoRA. +Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in AutoRA. | Name | Links | Description | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|