Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix soroban-test integration tests to make sure the full suite is being run in CI/CD #1682

Open
elizabethengelman opened this issue Oct 21, 2024 · 6 comments · May be fixed by #1734
Open

Fix soroban-test integration tests to make sure the full suite is being run in CI/CD #1682

elizabethengelman opened this issue Oct 21, 2024 · 6 comments · May be fixed by #1734
Assignees
@Ifropc

Comments

@elizabethengelman
Copy link
Contributor

elizabethengelman commented Oct 21, 2024 •
edited
Loading

I noticed that when running cargo test --package soroban-test --test it locally, or in a GH Action, not all of the tests seem to run. The output looks like this:

running 50 tests
test arg_parsing::parse_bool ... ok
test arg_parsing::parse_i32 ... ok
...
test arg_parsing::parse_u32 ... ok
test arg_parsing::parse_u64 ... ok
test config::generate_key_on_testnet ... ok
test arg_parsing::parse_enum_const ... ok
test arg_parsing::parse_enum ... ok
test arg_parsing::parse_i256 ... ok
test arg_parsing::parse_obj ... ok
test config::seed_phrase ... ok
test config::read_key ... ok
Example contract method which takes a struct
Usage Notes:
Each arg has a corresponding --<arg_name>-file-path which is a path to a file containing the corresponding JSON argument.
Note: The only types which aren't JSON are Bytes and BytesN, which are raw bytes

Usage: strukt_hel [OPTIONS]

Options:
      --strukt <{ a: u32, b: bool, c: Symbol }>
          This is from the rust doc above the struct Test
          Example:
            --strukt '{ "a": 1, "b": true, "c": "hello" }'

  -h, --help
          Print help (see a summary with '-h')

The output doesn't include the final test result including the number of failed and passed tests (i.e. test result: FAILED, 31 passed; 10 failed;...), nor does it include the status of all of the config tests, or any of the build, init, plugin, or version tests.

After digging a little bit, it seems like something in the help.rs file is causing this, though I'm not 100% sure what is going on. It looks like help.rs is potentially capturing the stdout (and maybe stderr too), so that the rest of the test output doesn't make it to the command line.

I discovered this because there are a couple of tests that should be failing now, but they aren't.

Repo steps

To see this, comment out the tests in help.rs and you'll see that there are some failing tests. I'm not sure if all of them are valid failures, but the error inversion.rs looks legit.

➜  stellar-cli git:(main) cargo test --package soroban-test --test it
    Finished `test` profile [unoptimized + debuginfo] target(s) in 17.30s
     Running tests/it/main.rs (target/debug/deps/it-d93d586aa933d606)

running 41 tests
test arg_parsing::parse_bool ... ok
test arg_parsing::parse_i128 ... ok
test arg_parsing::parse_i32 ... ok
test arg_parsing::parse_null ... ok
test arg_parsing::parse_bytesn ... ok
test arg_parsing::parse_bytes ... ok
test arg_parsing::parse_bytesn_when_hex_is_all_numbers ... ok
test arg_parsing::parse_bytes_when_hex_is_all_numbers ... ok
test arg_parsing::parse_optional_bool_with_no_quotation_marks ... ok
test arg_parsing::parse_symbol ... ok
test arg_parsing::parse_symbol_with_no_quotation_marks ... ok
test arg_parsing::parse_optional_symbol_with_no_quotation_marks ... ok
test arg_parsing::parse_u128 ... ok
test arg_parsing::parse_u32 ... ok
test arg_parsing::parse_u64 ... ok
test config::generate_key_on_testnet ... ok
test arg_parsing::parse_enum_const ... ok
test arg_parsing::parse_obj ... ok
test arg_parsing::parse_enum ... ok
test arg_parsing::parse_i256 ... ok
test config::seed_phrase ... ok
test config::read_key ... ok
test init::init ... ok
test plugin::list ... FAILED
test plugin::soroban_hello ... FAILED
test build::build_no_package_found ... ok
test build::build_default_members ... ok
test build::build_all ... ok
test build::build_package_by_name ... ok
test build::build_all_when_in_non_package_directory ... FAILED
test build::build_package_by_current_dir ... ok
test config::set_and_remove_global_network ... FAILED
test config::generate_key ... ok
test version::version ... FAILED
test config::use_env ... ok
test plugin::has_no_path_failure ... ok
test config::multiple_networks ... FAILED
test config::set_and_remove_network ... FAILED
test config::use_default_futurenet ... FAILED
test config::use_default_testnet ... FAILED
test init::init_and_deploy ... FAILED

failures:
...

---- version::version stdout ----
thread 'version::version' panicked at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5:
Unexpected stdout, failed diff original var
├── original: soroban 21.5.0 (v20.0.0-329-g3f06e1dd23fd1b892dc1d1fac2cd406dee4fabf4)
│   stellar-xdr 22.0.0-rc.1.1 (72e523004b5906eb1829990f9b14d2f0fa3018f0)
│   xdr curr (529d5176f24c73eeccfa5eba481d4e89c19b1181)
├── diff:
│   --- 	orig
│   +++ 	var
│   @@ -1 +1 @@
│   -soroban 21.5.0 (v20.0.0-329-g3f06e1dd23fd1b892dc1d1fac2cd406dee4fabf4)
│   +stellar 21.5.0 (v20.0.0-359-gfdbd9b847d73ae932afeb409f32c86d922194844-dirty)
└── var as str: stellar 21.5.0 (v20.0.0-359-gfdbd9b847d73ae932afeb409f32c86d922194844-dirty)
    stellar-xdr 22.0.0-rc.1.1 (72e523004b5906eb1829990f9b14d2f0fa3018f0)
    xdr curr (529d5176f24c73eeccfa5eba481d4e89c19b1181)

...

failures:
    build::build_all_when_in_non_package_directory
    config::multiple_networks
    config::set_and_remove_global_network
    config::set_and_remove_network
    config::use_default_futurenet
    config::use_default_testnet
    init::init_and_deploy
    plugin::list
    plugin::soroban_hello
    version::version

test result: FAILED. 31 passed; 10 failed; 0 ignored; 0 measured; 0 filtered out; finished in 50.61s
@github-project-automation github-project-automation bot moved this to Backlog (Not Ready) in DevX Oct 21, 2024
@juandiegocv27
Copy link

Hi @elizabethengelman can i work in this issue? I have experience with rust and testing👋🏻

@elizabethengelman elizabethengelman mentioned this issue Oct 25, 2024
Closed
@janewang janewang moved this from Backlog (Not Ready) to Todo (Ready for Dev) in DevX Oct 29, 2024
This was referenced Nov 18, 2024
Open
Merged
Merged
Merged
@Ifropc
Copy link
Contributor

Ifropc commented Nov 18, 2024

Integration tests were not running properly, as mentioned in this issue. It was caused by help tests exiting suite early with exit code 0, therefore effectively ignoring all other tests.
Help tests are fixed in #1734 and as the result failures are now visible:

failures:
    build::build_all_when_in_non_package_directory
    config::generate_key
    config::multiple_networks
    config::set_and_remove_global_network
    config::set_and_remove_network
    config::set_default_identity
    config::set_default_network
    config::use_default_futurenet
    config::use_default_testnet
    init::init
    init::init_and_deploy
    version::version

All failures are addressed and fixed in different PRs:

@Ifropc Ifropc moved this from Todo (Ready for Dev) to Needs Review in DevX Nov 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been assigned for 30 days with no activity. It will be closed in 30 days unless the stale label is removed, and the assignee is removed or updated.

@github-actions github-actions bot added the stale label Dec 19, 2024
@Ifropc Ifropc removed the stale label Dec 19, 2024
Ifropc added a commit that referenced this issue Dec 20, 2024
@Ifropc
#1682: fix build tests (#1736)
2c215cd 
* Fix build tests

* revert implementation
@sagpatil
Copy link
Contributor

@Ifropc is this good to close with your PR merge ?

@janewang
Copy link
Contributor

This is closed as integration tests PRs are merged. Unit test PR is pending

@github-project-automation github-project-automation bot moved this from Needs Review to Done in DevX Jan 10, 2025
@Ifropc
Copy link
Contributor

Ifropc commented Jan 10, 2025 •
edited
Loading

This is the PR for this issue #1734
It's fixes for both unit and integration tests (it)

@Ifropc Ifropc reopened this Jan 10, 2025
@Ifropc Ifropc linked a pull request Jan 10, 2025 that will close this issue
#1682: fix unit tests #1734
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ifropc Ifropc

Labels
None yet
Projects
DevX
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#1682: fix unit tests stellar/stellar-cli
5 participants
@janewang @sagpatil @elizabethengelman @Ifropc @juandiegocv27