Skip to content
This repository has been archived by the owner on Nov 19, 2024. It is now read-only.

Commit

Permalink
chore: rename to flat-head
Browse files Browse the repository at this point in the history
  • Loading branch information
suchapalaver committed Aug 16, 2024
1 parent 929dae3 commit 09544c8
Show file tree
Hide file tree
Showing 3 changed files with 11 additions and 11 deletions.
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "flat_head"
name = "flat-head"
version = "0.1.0"
edition = "2021"

Expand Down
18 changes: 9 additions & 9 deletions Readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ Flat head is developed to be a crate that verifies flat files generated from Fir

There are a few different binaries to run, depending on the desired functionality:

`flat_head` is for general usage on flat files in a local folder. Run
`cargo run --bin flat_head help` for commands and options.
`flat-head` is for general usage on flat files in a local folder. Run
`cargo run --bin flat-head help` for commands and options.

`fetch-s3` is supposed to be used to fetch flat files from an s3-like object storage.

Expand All @@ -27,22 +27,22 @@ Here are some examples of how to use the commands:
1. To validate flat files in a folder, a start epoch and a end epoch must be provided. `-d` flag can be used for debugging or log information.

```
cargo run --bin flat_head -- era-validate --store-url file:///<full-path-to-folder> -s 0
cargo run --bin flat-head -- era-validate --store-url file:///<full-path-to-folder> -s 0
```


2. To fetch flat files from a s3 bucket and validate each epoch as they arrive:

```
cargo run --bin flat_head -- era-validate --store-url s3:///<full-path-to-folder> -s 0
cargo run --bin flat-head -- era-validate --store-url s3:///<full-path-to-folder> -s 0
```

3. To fetch files from a seaweed-fs s3 compatible bucket and validate epochs:


```
cargo run --bin flat_head -- era-validate --store-url http://localhost:8333/newbucket3 -s 0 -e 1 --compatible s3
cargo run --bin flat-head -- era-validate --store-url http://localhost:8333/newbucket3 -s 0 -e 1 --compatible s3
```

Expand All @@ -51,22 +51,22 @@ Note that in this case it is using seaweed-fs s3 compatible API.
4. To fetch flat files from a Webdav server and validate each file as they arrive:

```
cargo run --bin flat_head -- era-validate --store-url http:///<full-path-to-folder> -s 0
cargo run --bin flat-head -- era-validate --store-url http:///<full-path-to-folder> -s 0
```


5. To fetch flat files from a gcloud bucket, and validate each epoch as they arrive:

```
cargo run --bin flat_head -- era-validate --store-url gs:///<full-path-to-folder> -s 0
cargo run --bin flat-head -- era-validate --store-url gs:///<full-path-to-folder> -s 0
```


### notice about usage

Flat files should come compressed with Zstandard (zstd) from Firehose. Flat_head handles decompression by default, but if it is necessary to disable it pass to the args: `-c false`. This is the same for all other binaries.
Flat files should come compressed with Zstandard (zstd) from Firehose. Flat head handles decompression by default, but if it is necessary to disable it pass to the args: `-c false`. This is the same for all other binaries.

Passing `--end-epoch` is not necessary, although without it, `flat_head` will only validate the start epoch passed as param.
Passing `--end-epoch` is not necessary, although without it, `flat-head` will only validate the start epoch passed as param.

`era-validate` will skip the files that were already verified and written into `lockfile.json`.
It stops abruptly if verification of any file fails. If files are compressed as `.zst` it is also capable
Expand Down

0 comments on commit 09544c8

Please sign in to comment.