Feat/stream utils #894

Tabaie · 2023-10-27T18:01:48Z

Description

Just a few utility functions for the compress.Stream class. Makes it easy to play with non-byte-aligned compression schemes.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How has this been tested?

Test A
Test B

How has this been benchmarked?

Benchmark A, on Macbook pro M1, 32GB RAM
Benchmark B, on x86 Intel xxx, 16GB RAM

Checklist:

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
I did not modify files generated from templates
golangci-lint does not output errors locally
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream modules

github-actions · 2023-10-27T18:12:49Z

📦 github.com/consensys/gnark/std/compress/lzss_v1
❌ Test1ZeroSnark 3.8s

compilation at 0 out of 1 ; 0 %
constructing witness
proving
time taken: 476 ms
compilation at 0 out of 1 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:22
        	Error:      	Received unexpected error:
        	            	constraint #20 is not satisfied: lookup query too large
        	Test:       	Test1ZeroSnark

❌ Test257ZerosAfterNonzeroSnark 10.13s

compilation at 0 out of 258 ; 0 %
constructing witness
proving
time taken: 487 ms
compilation at 0 out of 258 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:74
        	Error:      	Received unexpected error:
        	            	constraint #56 is not satisfied: lookup query too large
        	Test:       	Test257ZerosAfterNonzeroSnark

❌ Test2ZeroSnark 3.25s

compilation at 0 out of 2 ; 0 %
constructing witness
proving
time taken: 427 ms
compilation at 0 out of 2 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:27
        	Error:      	Received unexpected error:
        	            	constraint #20 is not satisfied: lookup query too large
        	Test:       	Test2ZeroSnark

❌ Test300ZerosAfterNonzeroSnark 10.16s

compilation at 0 out of 302 ; 0 %
constructing witness
proving
time taken: 1139 ms
compilation at 0 out of 302 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:79
        	Error:      	Received unexpected error:
        	            	constraint #84 is not satisfied: lookup query too large
        	Test:       	Test300ZerosAfterNonzeroSnark

❌ Test300ZerosSnark 8.17s

compilation at 0 out of 300 ; 0 %
constructing witness
proving
time taken: 684 ms
compilation at 0 out of 300 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:40
        	Error:      	Received unexpected error:
        	            	constraint #32 is not satisfied: lookup query too large
        	Test:       	Test300ZerosSnark

❌ Test8ZerosAfterNonzeroSnark 3.66s

compilation at 0 out of 9 ; 0 %
constructing witness
proving
time taken: 555 ms
compilation at 0 out of 9 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:65
        	Error:      	Received unexpected error:
        	            	constraint #44 is not satisfied: lookup query too large
        	Test:       	Test8ZerosAfterNonzeroSnark

❌ Test8ZerosSnark 3.02s

compilation at 0 out of 8 ; 0 %
constructing witness
proving
time taken: 380 ms
compilation at 0 out of 8 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:32
        	Error:      	Received unexpected error:
        	            	constraint #20 is not satisfied: lookup query too large
        	Test:       	Test8ZerosSnark

❌ TestCalldataSnark 6m6.48s

❌ TestCalldataSnark/3c2943 6m6.48s

    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:96
        	Error:      	Received unexpected error:
        	            	constraint #22874 is not satisfied: lookup query too large
        	Test:       	TestCalldataSnark/3c2943

🚧 TestCompareCoverage 0s

❌ TestTwoZerosAfterNonzeroSnark 3.43s

compilation at 0 out of 3 ; 0 %
constructing witness
proving
time taken: 562 ms
compilation at 0 out of 3 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:60
        	Error:      	Received unexpected error:
        	            	constraint #44 is not satisfied: lookup query too large
        	Test:       	TestTwoZerosAfterNonzeroSnark

❌ TestZeroAfterNonzeroSnark 2.1s

compilation at 0 out of 2 ; 0 %
constructing witness
proving
time taken: 166 ms
compilation at 0 out of 2 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:55
        	Error:      	Received unexpected error:
        	            	constraint #44 is not satisfied: lookup query too large
        	Test:       	TestZeroAfterNonzeroSnark

github-actions · 2023-10-27T19:34:18Z

📦 github.com/consensys/gnark/std/compress/lzss_v1
❌ Test1ZeroSnark 4.25s

compilation at 0 out of 1 ; 0 %
constructing witness
proving
time taken: 576 ms
compilation at 0 out of 1 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:22
        	Error:      	Received unexpected error:
        	            	constraint #20 is not satisfied: lookup query too large
        	Test:       	Test1ZeroSnark

❌ Test257ZerosAfterNonzeroSnark 8.38s

compilation at 0 out of 258 ; 0 %
constructing witness
proving
time taken: 475 ms
compilation at 0 out of 258 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:74
        	Error:      	Received unexpected error:
        	            	constraint #56 is not satisfied: lookup query too large
        	Test:       	Test257ZerosAfterNonzeroSnark

❌ Test2ZeroSnark 3.42s

compilation at 0 out of 2 ; 0 %
constructing witness
proving
time taken: 486 ms
compilation at 0 out of 2 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:27
        	Error:      	Received unexpected error:
        	            	constraint #20 is not satisfied: lookup query too large
        	Test:       	Test2ZeroSnark

❌ Test300ZerosAfterNonzeroSnark 11.88s

compilation at 0 out of 302 ; 0 %
constructing witness
proving
time taken: 980 ms
compilation at 0 out of 302 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:79
        	Error:      	Received unexpected error:
        	            	constraint #84 is not satisfied: lookup query too large
        	Test:       	Test300ZerosAfterNonzeroSnark

❌ Test300ZerosSnark 8.42s

compilation at 0 out of 300 ; 0 %
constructing witness
proving
time taken: 735 ms
compilation at 0 out of 300 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:40
        	Error:      	Received unexpected error:
        	            	constraint #32 is not satisfied: lookup query too large
        	Test:       	Test300ZerosSnark

❌ Test8ZerosAfterNonzeroSnark 2.96s

compilation at 0 out of 9 ; 0 %
constructing witness
proving
time taken: 446 ms
compilation at 0 out of 9 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:65
        	Error:      	Received unexpected error:
        	            	constraint #44 is not satisfied: lookup query too large
        	Test:       	Test8ZerosAfterNonzeroSnark

❌ Test8ZerosSnark 2.78s

compilation at 0 out of 8 ; 0 %
constructing witness
proving
time taken: 353 ms
compilation at 0 out of 8 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:32
        	Error:      	Received unexpected error:
        	            	constraint #20 is not satisfied: lookup query too large
        	Test:       	Test8ZerosSnark

❌ TestCalldataSnark 6m7.48s

❌ TestCalldataSnark/3c2943 6m7.48s

Size Compression ratio: 3.3586985043295723
Estimated Compression ratio (with Huffman): 3.6711061716887445
Compressed size: 7KB
Compressed size (with Huffman): 6KB
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:96
        	Error:      	Received unexpected error:
        	            	constraint #22874 is not satisfied: lookup query too large
        	Test:       	TestCalldataSnark/3c2943

🚧 TestCompareCoverage 0s

❌ TestTwoZerosAfterNonzeroSnark 4s

compilation at 0 out of 3 ; 0 %
constructing witness
proving
time taken: 617 ms
compilation at 0 out of 3 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:60
        	Error:      	Received unexpected error:
        	            	constraint #44 is not satisfied: lookup query too large
        	Test:       	TestTwoZerosAfterNonzeroSnark

❌ TestZeroAfterNonzeroSnark 2.42s

compilation at 0 out of 2 ; 0 %
constructing witness
proving
time taken: 179 ms
compilation at 0 out of 2 ; 0 %
constructing witness
proving
    snark_test.go:311: 
        	Error Trace:	/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:311
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:280
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:251
        	            				/home/runner/work/gnark/gnark/std/compress/lzss_v1/snark_test.go:55
        	Error:      	Received unexpected error:
        	            	constraint #44 is not satisfied: lookup query too large
        	Test:       	TestZeroAfterNonzeroSnark

github-actions · 2023-11-01T19:24:55Z

📦 github.com/consensys/gnark/backend/plonk
❌ TestProver 0s

❌ TestProver/bn254 0s

📦 github.com/consensys/gnark/internal/stats
❌ TestCircuitStatistics 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls12_377/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls12_377/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls12_377/plonkFRI 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls12_381/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls12_381/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls12_381/plonkFRI 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls24_315/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls24_315/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls24_315/plonkFRI 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls24_317/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls24_317/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bls24_317/plonkFRI 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bn254/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bn254/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bn254/plonkFRI 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bw6_633/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bw6_633/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bw6_633/plonkFRI 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bw6_761/groth16 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bw6_761/plonk 0s

❌ TestCircuitStatistics/math/bits.ToNAF/unconstrained/bw6_761/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls12_377/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls12_377/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls12_377/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls12_381/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls12_381/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls12_381/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls24_315/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls24_315/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls24_315/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls24_317/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls24_317/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bls24_317/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bn254/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bn254/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bn254/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bw6_633/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bw6_633/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bw6_633/plonkFRI 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bw6_761/groth16 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bw6_761/plonk 0s

❌ TestCircuitStatistics/math/emulated/secp256k1_64/bw6_761/plonkFRI 0s

❌ TestCircuitStatistics/pairing_bls12377/bw6_761/groth16 0s

❌ TestCircuitStatistics/pairing_bls12377/bw6_761/plonk 0s

❌ TestCircuitStatistics/pairing_bls12377/bw6_761/plonkFRI 0s

❌ TestCircuitStatistics/pairing_bls24315/bw6_633/groth16 0s

📦 github.com/consensys/gnark/std/algebra/emulated/fields_bls12381
❌ TestFp12ExptTorus 0s

📦 github.com/consensys/gnark/std/algebra/emulated/fields_bn254
❌ TestFp12ExptTorus 0s

📦 github.com/consensys/gnark/std/algebra/emulated/fields_bw6761
❌ TestExptFp6 0s

📦 github.com/consensys/gnark/std/algebra/emulated/sw_bls12381
❌ TestScalarMulG2BySeedTestSolve 0s

📦 github.com/consensys/gnark/std/algebra/emulated/sw_bn254
❌ TestScalarMulG2BySeedTestSolve 0s

📦 github.com/consensys/gnark/std/algebra/emulated/sw_bw6761
❌ TestFinalExponentiationTestSolve 0s

📦 github.com/consensys/gnark/std/algebra/native/sw_bls12377
❌ TestConstantScalarMulG2 0s

❌ TestConstantScalarMulG2/bw6_761 0s

📦 github.com/consensys/gnark/std/algebra/native/sw_bls24315
❌ TestVarScalarMulG1 0s

❌ TestVarScalarMulG1/bw6_633 0s

📦 github.com/consensys/gnark/std/algebra/native/twistededwards
❌ TestCurve 0s

❌ TestCurve/bls12_377 0s

❌ TestCurve/bn254 0s

Tabaie added 2 commits October 27, 2023 12:02

feat: r/w num, (un)marshal for stream

5b18645

test: marshal test

2011fd7

Tabaie requested a review from gbotrel October 27, 2023 18:01

chore make staticcheck happy

34b10f4

feat: new stream

fd0fa11

gbotrel closed this Nov 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/stream utils #894

Feat/stream utils #894

Tabaie commented Oct 27, 2023

github-actions bot commented Oct 27, 2023

github-actions bot commented Oct 27, 2023

github-actions bot commented Nov 1, 2023

Feat/stream utils #894

Feat/stream utils #894

Conversation

Tabaie commented Oct 27, 2023

Description

How has this been tested?

How has this been benchmarked?

Checklist:

github-actions bot commented Oct 27, 2023

github-actions bot commented Oct 27, 2023

github-actions bot commented Nov 1, 2023