cargo package --no-verify fails if a package's version req is too high for the registry (but works locally)

[weihanglo]

Problem

#13447 dd698ff has some unintentional consequences:

cargo package after than generates a lockfile also when packaging libraries.
However, if the crate being packaged depends on some unpublished dependencies. When generating the lockfile Cargo then can't find unpublished deps on registry index, so it bailed out.

Steps

This could be simplly reproduced in rust-lang/cargo as of today (2025-01-13)

1. Checkout to 54df3c7
2. Bump cargo-test-support to an unpublished version such as 99.99.99
3. cargo +1.83.0 package -p cargo-test-support --no-verify — expect to succeed
4. cargo +1.84.0 package -p cargo-test-support --no-verify — expect to fail

Possible Solution(s)

Assume if we have passed the local registry unconditionally, we should have the issue today.

cargo/src/cargo/ops/cargo_package/mod.rs

Lines 219 to 225 in f15df8f

	let mut local_reg = if ws.gctx().cli_unstable().package_workspace {
	let reg_dir = ws.target_dir().join("package").join("tmp-registry");
	sid.map(|sid| TmpRegistry::new(ws.gctx(), reg_dir, sid))
	.transpose()?
	} else {
	None
	};

Version

• cargo 1.84.0 (66221abde 2024-11-19)
• cargo 1.86.0-nightly (088d49608 2025-01-10)

[epage]

So this is a problem that existed before 54df3c7 but only for some crates. Now all see it, right?

[weihanglo]

Yeah I believe it was only bugging for packages having [[bin]] crates, but now every package.

[epage]

@Eh2406 our resolve call is

cargo/src/cargo/ops/cargo_package/mod.rs

Lines 655 to 664 in 54df3c7

	let mut new_resolve = ops::resolve_with_previous(
	&mut tmp_reg,
	&tmp_ws,
	&CliFeatures::new_all(true),
	HasDevUnits::Yes,
	orig_resolve.as_ref(),
	None,
	&[],
	true,
	)?;

Is there a way to say "prefer existing but don't lock them? Would keep = |_| false do that? Seems like what we'd want (have to double check yanked) and then cargo package already has logic to warn in these situations which imo is good because its easy to accidentally release a package and not realize you forgot to release a dependency.

[weihanglo]

In terms of a swift fix: How stable is TmpRegistry?
It might be helpful if we could use it internally without stabilizing the entire -Zpackage-workspace.

[epage]

I'm not sure how that would fix it. We wouldn't be putting the package into TmpRegistry to be able to be used unless the user requests it at which point it will work anyways.

[weihanglo]

More like Cargo automatically publishes unpublished dependencies crates when needed to TmpRegistry before building the lock?

[epage]

I suspect getting it right to sometimes publish to TmpRegistry can cause us other problems and would get in the way of stabilizing workspace packaging that I feel like that less likely to be our best option for a swift fix.

if we feel the need to backport to beta, a revert is likely the best route. If not, we should at least understand what it would take on the resolver side.

[sehz]

We are running into same issue. Can this is be reverted?

[epage]

I cannot reproduce this if I bump the package's version but don't bump the version requirement. I also missed the detail that --no-verify is required..

This means that cargo is able to adjust the lockfile correctly as needed and the problem is that we moved "dependency exists" checks from the verify step to the packaging step, forcing it to always happen.

The documentation for cargo publish --no-verify just says

Don’t verify the contents by building them.

I think this still leaves room for other verification. Historically, we've done some registry and dependency verification and we recently added "is this published?" (#14448).

cargo/src/cargo/ops/registry/publish.rs

Lines 96 to 139 in 54df3c7

	let just_pkgs: Vec<_> = pkgs.iter().map(|p| p.0).collect();
	let reg_or_index = match opts.reg_or_index.clone() {
	Some(r) => {
	validate_registry(&just_pkgs, Some(&r))?;
	Some(r)
	}
	None => {
	let reg = super::infer_registry(&just_pkgs)?;
	validate_registry(&just_pkgs, reg.as_ref())?;
	if let Some(RegistryOrIndex::Registry(ref registry)) = &reg {
	if registry != CRATES_IO_REGISTRY {
	// Don't warn for crates.io.
	opts.gctx.shell().note(&format!(
	"found `{}` as only allowed registry. Publishing to it automatically.",
	registry
	))?;
	}
	}
	reg
	}
	};
	// This is only used to confirm that we can create a token before we build the package.
	// This causes the credential provider to be called an extra time, but keeps the same order of errors.
	let source_ids = super::get_source_id(opts.gctx, reg_or_index.as_ref())?;
	let (mut registry, mut source) = super::registry(
	opts.gctx,
	&source_ids,
	opts.token.as_ref().map(Secret::as_deref),
	reg_or_index.as_ref(),
	true,
	Some(Operation::Read).filter(|_| !opts.dry_run),
	)?;
	{
	let _lock = opts
	.gctx
	.acquire_package_cache_lock(CacheLockMode::DownloadExclusive)?;
	for (pkg, _) in &pkgs {
	verify_unpublished(pkg, &mut source, &source_ids, opts.dry_run, opts.gctx)?;
	verify_dependencies(pkg, &registry, source_ids.original)?;
	}
	}

Its also not clear to me why you would be publishing packages without dependencies present. Is this a workaround for #4242? We already have a method for this for some cases (leaving off versions for dev-dependencies). If there is a case that doesn't cover, it would be good to make sure its reported in #4242.

[weihanglo]

Note that this is about cargo package not publish.

There are workflows packaging source code into .crate files before any publishment. Not going to judge in this comment whether those workflows should be supported by Cargo, but the behavior did change in 1.84.

This Cargo-flavored test works in 1.83 but fails in 1.84.

#[cargo_test]
fn unpublished_dependency() {
    Package::new("dep", "0.1.0").publish();

    let p = project()
        .file(
            "Cargo.toml",
            r#"
            [package]
            name = "main"
            version = "0.0.1"
            edition = "2015"

            [dependencies]
            dep = { path = "dep", version = "0.1.1" }
        "#,
        )
        .file("src/lib.rs", "")
        .file(
            "dep/Cargo.toml",
            r#"
            [package]
            name = "dep"
            version = "0.1.1"
            edition = "2015"
        "#,
        )
        .file("dep/src/lib.rs", "")
        .build();

    p.cargo("package --package main --no-verify --no-metadata")
        .with_status(101)
        .with_stderr_data(str![[r#"
[PACKAGING] main v0.0.1 ([ROOT]/foo)
[UPDATING] `dummy-registry` index
[ERROR] failed to prepare local package for uploading

Caused by:
  failed to select a version for the requirement `dep = "^0.1.1"`
  candidate versions found which didn't match: 0.1.0
  location searched: `dummy-registry` index (which is replacing registry `crates-io`)
  required by package `main v0.0.1 ([ROOT]/foo)`
  perhaps a crate was updated and forgotten to be re-vendored?

"#]])
        .run();
}

[epage]

That seems like a very limited workflow as it only worked if the crates had no [[bin]]s and no [[example]]s in them. Once you add one, it failed.

[epage]

@sehz could you help us understand more about your use case? I feel like there are reasons to keep the new behavior but I want to also better understand your situation to better understand how compelling the old behavior is.

[sehz]

We are building development version of binary which depends on unpublished version of crates for obvious reason. The binary is not in the same repo as other crates. So we packaged unpublished version of crates so it can be refer by binary crate.

[sehz]

Would be ok to have previous behavior with flag.

[epage]

Could you go into more detail. I feel I'm not quite getting why cargo package is being used and particularly why you need to package when dependencies are unavailable.