diff --git a/docs/src/api-dagger/types.md b/docs/src/api-dagger/types.md index a36ee3a0f..4e6d0460e 100644 --- a/docs/src/api-dagger/types.md +++ b/docs/src/api-dagger/types.md @@ -16,7 +16,6 @@ DTask ## Task Options Types ```@docs Options -Sch.ThunkOptions Sch.SchedulerOptions ``` diff --git a/docs/src/checkpointing.md b/docs/src/checkpointing.md index 97dd8e8ed..fa798185d 100644 --- a/docs/src/checkpointing.md +++ b/docs/src/checkpointing.md @@ -71,7 +71,7 @@ z = collect(Z) ``` Two changes were made: first, we `enumerate(X.chunks)` so that we can get a -unique index to identify each `chunk`; second, we specify a `ThunkOptions` to +unique index to identify each `chunk`; second, we specify options to `delayed` with a `checkpoint` and `restore` function that is specialized to write or read the given chunk to or from a file on disk, respectively. Notice the usage of `collect` in the `checkpoint` function, and the use of diff --git a/docs/src/task-spawning.md b/docs/src/task-spawning.md index d1d3466e3..8bedc6a65 100644 --- a/docs/src/task-spawning.md +++ b/docs/src/task-spawning.md @@ -12,9 +12,9 @@ or `spawn` if it's more convenient: `Dagger.spawn(f, Dagger.Options(options), args...; kwargs...)` -When called, it creates an [`DTask`](@ref) (also known as a "thunk" or -"task") object representing a call to function `f` with the arguments `args` and -keyword arguments `kwargs`. If it is called with other thunks as args/kwargs, +When called, it creates an [`DTask`](@ref) (also known as a "task" or +"thunk") object representing a call to function `f` with the arguments `args` and +keyword arguments `kwargs`. If it is called with other tasks as args/kwargs, such as in `Dagger.@spawn f(Dagger.@spawn g())`, then, in this example, the function `f` gets passed the results of executing `g()`, once that result is available. If `g()` isn't yet finished executing, then the execution of `f` @@ -29,7 +29,7 @@ it'll be passed as-is to the function `f` (with some exceptions). !!! note "Task / thread occupancy" By default, `Dagger` assumes that tasks saturate the thread they are running on and does not try to schedule other tasks on the thread. - This default can be controlled by specifying [`Sch.ThunkOptions`](@ref) (more details can be found under [Scheduler and Thunk options](@ref)). + This default can be controlled by specifying [`Options`](@ref) (more details can be found under [Task and Scheduler options](@ref)). The section [Changing the thread occupancy](@ref) shows a runnable example of how to achieve this. ## Options @@ -37,14 +37,8 @@ it'll be passed as-is to the function `f` (with some exceptions). The [`Options`](@ref Dagger.Options) struct in the second argument position is optional; if provided, it is passed to the scheduler to control its behavior. [`Options`](@ref Dagger.Options) contains a `NamedTuple` of option -key-value pairs, which can be any of: -- Any field in [`Sch.ThunkOptions`](@ref) (see [Scheduler and Thunk options](@ref)) -- `meta::Bool` -- Pass the input [`Chunk`](@ref) objects themselves to `f` and - not the value contained in them. - -There are also some extra options that can be passed, although they're considered advanced options to be used only by developers or library authors: -- `get_result::Bool` -- return the actual result to the scheduler instead of [`Chunk`](@ref) objects. Used when `f` explicitly constructs a [`Chunk`](@ref) or when return value is small (e.g. in case of reduce) -- `cache::Bool` -- cache the result of this Thunk such that if the thunk is evaluated again, one can just reuse the cached value. If it’s been removed from cache, recompute the value. +key-value pairs, which can be any field in [`Options`](@ref) +(see [Task and Scheduler options](@ref)). ## Simple example @@ -65,7 +59,7 @@ s = Dagger.@spawn combine(p, q, r) @assert fetch(s) == 16 ``` -The thunks `p`, `q`, `r`, and `s` have the following structure: +The tasks `p`, `q`, `r`, and `s` have the following structure: ![graph](https://user-images.githubusercontent.com/25916/26920104-7b9b5fa4-4c55-11e7-97fb-fe5b9e73cae6.png) @@ -122,23 +116,24 @@ x::DTask @assert fetch(x) == 3 # fetch the result of `@spawn` ``` -This is useful for nested execution, where an `@spawn`'d thunk calls `@spawn`. This is detailed further in [Dynamic Scheduler Control](@ref). +This is useful for nested execution, where an `@spawn`'d task calls `@spawn`. +This is detailed further in [Dynamic Scheduler Control](@ref). ## Errors -If a thunk errors while running under the eager scheduler, it will be marked as -having failed, all dependent (downstream) thunks will be marked as failed, and -any future thunks that use a failed thunk as input will fail. Failure can be +If a task errors while running under the eager scheduler, it will be marked as +having failed, all dependent (downstream) tasks will be marked as failed, and +any future tasks that use a failed task as input will fail. Failure can be determined with `fetch`, which will re-throw the error that the -originally-failing thunk threw. `wait` and `isready` will *not* check whether a -thunk or its upstream failed; they only check if the thunk has completed, error +originally-failing task threw. `wait` and `isready` will *not* check whether a +task or its upstream failed; they only check if the task has completed, error or not. This failure behavior is not the default for lazy scheduling ([Lazy API](@ref)), -but can be enabled by setting the scheduler/thunk option ([Scheduler and Thunk options](@ref)) +but can be enabled by setting the scheduler/task option ([Task and Scheduler options](@ref)) `allow_error` to `true`. However, this option isn't terribly useful for -non-dynamic usecases, since any thunk failure will propagate down to the output -thunk regardless of where it occurs. +non-dynamic usecases, since any task failure will propagate down to the output +task regardless of where it occurs. ## Cancellation @@ -197,7 +192,7 @@ end ``` Alternatively, if you want to compute but not fetch the result of a lazy -operation, you can call `compute` on the thunk. This will return a `Chunk` +operation, you can call `compute` on the task. This will return a `Chunk` object which references the result (see [Chunks](@ref) for more details): ```julia @@ -214,16 +209,15 @@ Note that, as a legacy API, usage of the lazy API is generally discouraged for m - Distinct schedulers don't share runtime metrics or learned parameters, thus causing the scheduler to act less intelligently - Distinct schedulers can't share work or data directly -## Scheduler and Thunk options +## Task and Scheduler options While Dagger generally "just works", sometimes one needs to exert some more fine-grained control over how the scheduler allocates work. There are two -parallel mechanisms to achieve this: Scheduler options (from -[`Sch.SchedulerOptions`](@ref)) and Thunk options (from -[`Sch.ThunkOptions`](@ref)). These two options structs contain many shared -options, with the difference being that Scheduler options operate -globally across an entire DAG, and Thunk options operate on a thunk-by-thunk -basis. +parallel mechanisms to achieve this: Task options (from [`Options`](@ref)) and +Scheduler options (from [`Sch.SchedulerOptions`](@ref)). These two options +structs contain many shared options, with the difference being that Scheduler +options operate globally across an entire DAG, and Task options operate on a +task-by-task basis. Scheduler options can be constructed and passed to `collect()` or `compute()` as the keyword argument `options` for lazy API usage: @@ -237,7 +231,7 @@ compute(t; options=opts) collect(t; options=opts) ``` -Thunk options can be passed to `@spawn/spawn`, `@par`, and `delayed` similarly: +Task options can be passed to `@spawn/spawn`, `@par`, and `delayed` similarly: ```julia # Execute on worker 1 @@ -250,8 +244,9 @@ delayed(+; single=1)(1, 2) ## Changing the thread occupancy -One of the supported [`Sch.ThunkOptions`](@ref) is the `occupancy` keyword. -This keyword can be used to communicate that a task is not expected to fully saturate a CPU core (e.g. due to being IO-bound). +One of the supported [`Options`](@ref) is the `occupancy` keyword. +This keyword can be used to communicate that a task is not expected to fully +saturate a CPU core (e.g. due to being IO-bound). The basic usage looks like this: ```julia diff --git a/test/options.jl b/test/options.jl index e832a0827..dd32ce1d6 100644 --- a/test/options.jl +++ b/test/options.jl @@ -29,9 +29,8 @@ end # Special handling (:scope, AnyScope(), ProcessScope(first_wid), ProcessScope(last_wid)), (:processor, OSProc(), Dagger.ThreadProc(first_wid, 1), Dagger.ThreadProc(last_wid, 1)), - # ThunkOptions field + # Options field (:single, 0, first_wid, last_wid), - # Thunk field (:meta, false, true, false) ] # Test local and remote default values