dvc exp gc vs dvc exp remove behavior

[dberenbaum]

See background in #5676 (comment).

Both dvc exp gc and dvc exp remove may be confusing, and the difference between them is unclear. Do we need both, and if so, what purpose does each serve?

There are similarities between them:

• They both are about getting rid of experiments.
• AFAIK they both take the same action on those experiments (removing the git ref).

I think the difference is that dvc exp gc is about "cleaning up" experiments that dvc already knows may be no longer useful, while dvc exp remove is about telling dvc which experiments are no longer useful. Therefore, they have different syntax for identifying which experiments to drop. The commands could also take different actions on those experiments (maybe dvc exp gc forces git garbage collection), although that's not true today.

Open questions:

• Do we need both?
• What is the purpose of each?
• What should be the syntax to specify experiments to drop (for each or combined)?
• What should dvc do with the specified experiments (for each or combined)?

@karajan1001 @pmrowla

[iesahin]

1. I don't think we need both. dvc exp gc can be merged with dvc gc. Garbage collection can work on the experiments with a single command.

2. dvc exp gc seems like an analog of dvc gc for the experiments, but without dvc gc, it's confusing. It doesn't actually garbage collect the artifacts (in the cache), it just removes references in .git/refs. dvc exp remove can also be confusing because of dvc remove. I'd go for a single experiment deletion command: dvc exp delete or change dvc remove to dvc delete and merge dvc exp gc and dvc gc, into a single dvc gc.

3. The syntax can be dvc exp remove --keep all --delete exp-12345. Or dvc exp remove --include all --exclude exp-12345 to delete all experiments except exp-12345. The --keep and --delete options seem more clear to me. They can have special options: all, workspace, all-branches, all-tags, all-commits, before:2021-08-18, or experiment ids, tag, branch names.

• --delete option is clear when used solely.
• --keep option doesn't do anything if used solely. But when it's used with --delete, it subtracts the specified elements from the deletion set. So dvc exp remove --delete all --keep workspace deletes all non-workspace experiments (current dvc exp gc --workspace behavior), dvc exp remove --delete exp-12345 exp-12930 exp-48938 --keep workspace deletes the listed experiments if they are not in the workspace. dvc exp remove --delete workspace --keep after:2021-08-10 deletes experiments in the workspace before 2021-08-10. I can detail this if needed. (--keep can be called --except if it sounds better.)

In general, I believe this provides a cleaner and more expressive interface for deleting experiments.

[iesahin]

--delete as an option may not be necessary. dvc exp remove all --except workspace or dvc exp remove workspace --except after:2021-08-08 looks better. Special modifiers all, workspace..., etc., can be --all --workspace... but their meaning will be the inverse of dvc exp gc options. dvc exp remove --workspace --except exp-1234 will remove all workspace experiments except exp-1234.

[dberenbaum]

1. I don't think we need both. `dvc exp gc` can be merged with `dvc gc`. Garbage collection can work on the experiments with a single command.

Makes sense at first glance. If someone is no longer interested in some old commit, it seems unlikely they are interested in the experiments derived from it. Does dvc gc currently check and keep data associated with experiment commits @pmrowla?

Would dvc gc also remove the experiment refs, or would that exclusively be done by dvc exp remove?

2\. `dvc exp remove` can also be confusing because of `dvc remove`. I'd go for a single experiment deletion command: `dvc exp delete` or change `dvc remove` to `dvc delete` and merge `dvc exp gc` and `dvc gc`, into a single `dvc gc`.

This makes sense, although I'm not sure it's confusing enough to make a breaking change at this point. We do have this potential confusion with other commands (dvc add/dvc stage add, dvc run/dvc exp run), and I haven't noticed it as a major issue for users.

[daavoo]

I agree with @iesahin , we could get rid of dvc exp gc.

I also like the --except syntax with special selectors (all, workspace, etc.). The following feels more intuitive to me than the current behavior:

dvc exp remove --workspace --except exp-1234 will remove all workspace experiments except exp-1234.

As an alternative (or maybe as an extension) I really like using glob patterns as targets for some dvc commands. I think that having support for that would be useful specially when --name is used as a way to organize experiments (i.e. dvc exp remove resnet-50-*)

[dberenbaum]

These additional options are good ideas, although I'm not sure what options are highest priority or what syntax will be most useful. Maybe we can wait and see how much granularity/complexity users need for dropping experiments and revisit when we have more feedback.

[karajan1001]

exp remove is used locally while exp gc can remove remote cache. Quite different in a collaborating project.

[daavoo]

Did I missed it or is that info not present the docs?

[karajan1001]

Did I missed it or is that info not present the docs?

Sorry I used to believe exp gc can do what gc does removing exps in clouds.

[dberenbaum]

Users might accidentally generate and push large experiments commits that they want to delete (not only the refs, but the actual git data). How can these be removed/cleaned?