diff --git a/README.md b/README.md index da6c8620..b2a3b0cb 100644 --- a/README.md +++ b/README.md @@ -80,14 +80,13 @@ Sparsify empowers you to compress models through two components: - **Sparsify CLI/API** - a Python package and GitHub repository that allows you to run Sparsify Experiments locally, sync with the Sparsify Cloud, and integrate them into your workflows. ## Table of Contents - - [Quickstart Guide](#quickstart-guide) - - [Install and Setup](#1-install-and-setup) - - [Run an Experiment](#2-run-an-experiment) - - [Compare Results](#3-compare-results) - - [Deploy a Model](#4-deploy-a-model) -- [Companion Guides](#companion-guides) -- [Resources](#resources) + - [1. Install and Setup](#1-install-and-setup) + - [2. Run an Experiment](#2-run-an-experiment) + - [3. Compare Results](#3-compare-results) + - [4. Deploy a Model](#4-deploy-a-model) +- [Companion Guides](#companion-guides) +- [Resources](#resources) ## Quickstart Guide @@ -116,12 +115,14 @@ First, verify that you have the correct software and hardware to run the Sparsif Sparsify is tested on Python 3.8 and 3.10, ONNX 1.5.0-1.12.0, ONNX opset version 11+, and manylinux compliant systems. Sparsify is not supported natively on Windows and MAC OS. + +Additionally, for installation from PyPi, pip 20.3+ is required.
Hardware -Sparsify requires a GPU with CUDA + CuDNN in order to sparsify neural networks. +Sparsify requires a GPU with CUDA + CuDNN in order to sparsify neural networks. We recommend you use a Linux system with a GPU that has a minimum of 16GB of GPU Memory, 128GB of RAM, 4 CPU cores, and is CUDA-enabled. If you are sparsifying a very large model, you may need more RAM than the recommended 128GB. If you encounter issues setting up your training environment, [file a GitHub issue](https://github.com/neuralmagic/sparsify/issues). @@ -129,8 +130,8 @@ If you encounter issues setting up your training environment, [file a GitHub iss #### 1.2 Create an Account -Creating a new one-time account is simple and free. -An account is required to manage your Experiments and API keys. +Creating a new one-time account is simple and free. +An account is required to manage your Experiments and API keys. Visit the [Neural Magic's Web App Platform](https://account.neuralmagic.com/signup) and create an account by entering your email, name, and unique password. If you already have a Neural Magic Account, [sign in](https://account.neuralmagic.com/signin) with your email. @@ -146,7 +147,7 @@ Install with pip using: pip install sparsify-nightly ``` -#### 1.4 Login via CLI +#### 1.4 Log in via CLI Next, with Sparsify installed on your training hardware: 1. Authorize the local CLI to access your account by running the sparsify.login command and providing your API key. @@ -182,14 +183,14 @@ To run a One-Shot Experiment for your model, dataset, and use case, use the foll sparsify.run one-shot --use-case USE_CASE --model MODEL --data DATASET --optim-level OPTIM_LEVEL ``` -For example, to sparsify a ResNet50 model on the ImageNet dataset for image classification, run the following commands: +For example, to sparsify a ResNet-50 model on the ImageNet dataset for image classification, run the following commands: ```bash wget https://public.neuralmagic.com/datasets/cv/classification/imagenet_calibration.tar.gz tar -xzf imagenet_calibration.tar.gz sparsify.run one-shot --use-case image_classification --model "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/base-none" --data ./imagenet_calibration --optim-level 0.5 ``` -Or, to sparsify a BERT model on the SST-2 dataset for sentiment analysis, run the following commands: +Or, to sparsify a BERT model on the SST2 dataset for sentiment analysis, run the following commands: ```bash wget https://public.neuralmagic.com/datasets/nlp/text_classification/sst2_calibration.tar.gz tar -xzf sst2_calibration.tar.gz @@ -199,7 +200,7 @@ sparsify.run one-shot --use-case text_classification --model "zoo:nlp/sentiment_ To dive deeper into One-Shot Experiments, read through the [One-Shot Experiment Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/one-shot_experiment-guide.md). -Note, One-Shot Experiments currently require the model to be in an ONNX format and the dataset to be in a Numpy format. +Note, One-Shot Experiments currently require the model to be in an ONNX format and the dataset to be in a NumPy format. More details are provided in the One-Shot Experiment Guide. @@ -209,19 +210,19 @@ More details are provided in the One-Shot Experiment Guide. |----------|----------------------|-----------| | **++++** | **++++** | **+++++** | -Sparse-Transfer Experiments quickly create a smaller and faster model for your dataset by transferring from a [SparseZoo](https://sparsezoo.neuralmagic.com/) pre-sparsified foundational model o, providing a 5-10x speedup with minimal accuracy loss, ideal for quick model optimization without retraining your model. +Sparse-Transfer Experiments quickly create a smaller and faster model for your dataset by transferring from a [SparseZoo](https://sparsezoo.neuralmagic.com/) pre-sparsified foundational model, providing a 5-10x speedup with minimal accuracy loss, ideal for quick model optimization without retraining your model. To run a Sparse-Transfer Experiment for your model (optional), dataset, and use case, run the following command: ```bash sparsify.run sparse-transfer --use-case USE_CASE --model OPTIONAL_MODEL --data DATASET --optim-level OPTIM_LEVEL ``` -For example, to sparse transfer a SparseZoo model to the ImageNette dataset for image classification, run the following command: +For example, to sparse transfer a SparseZoo model to the Imagenette dataset for image classification, run the following command: ```bash sparsify.run sparse-transfer --use-case image_classification --data imagenette --optim-level 0.5 ``` -Or, to sparse transfer a SparseZoo model to the SST-2 dataset for sentiment analysis, run the following command: +Or, to sparse transfer a SparseZoo model to the SST2 dataset for sentiment analysis, run the following command: ```bash sparsify.run sparse-transfer --use-case text_classification --data sst2 --optim-level 0.5 ``` @@ -229,7 +230,7 @@ sparsify.run sparse-transfer --use-case text_classification --data sst2 --optim- To dive deeper into Sparse-Transfer Experiments, read through the [Sparse-Transfer Experiment Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/sparse-transfer_experiment-guide.md). -Note, Sparse-Transfer Experiments require the model to be saved in a PyTorch format corresponding to the underlying integration such as Ultralytics YOLOv5 or HuggingFace Transformers. +Note, Sparse-Transfer Experiments require the model to be saved in a PyTorch format corresponding to the underlying integration such as Ultralytics YOLOv5 or Hugging Face Transformers. Datasets must additionally match the expected format of the underlying integration. More details and exact formats are provided in the Sparse-Transfer Experiment Guide. @@ -247,12 +248,12 @@ To run a Training-Aware Experiment for your model, dataset, and use case, run th sparsify.run training-aware --use-case USE_CASE --model OPTIONAL_MODEL --data DATASET --optim-level OPTIM_LEVEL ``` -For example, to sparsify a ResNet50 model on the ImageNette dataset for image classification, run the following command: +For example, to sparsify a ResNet-50 model on the Imagenette dataset for image classification, run the following command: ```bash sparsify.run training-aware --use-case image_classification --model "zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenette/base-none" --data imagenette --optim-level 0.5 ``` -Or, to sparsify a BERT model on the SST-2 dataset for sentiment analysis, run the following command: +Or, to sparsify a BERT model on the SST2 dataset for sentiment analysis, run the following command: ```bash sparsify.run training-aware --use-case text_classification --model "zoo:nlp/sentiment_analysis/bert-base/pytorch/huggingface/sst2/base-none" --data sst2 --optim-level 0.5 ``` @@ -260,7 +261,7 @@ sparsify.run training-aware --use-case text_classification --model "zoo:nlp/sent To dive deeper into Training-Aware Experiments, read through the [Training-Aware Experiment Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/training-aware_experiment-guide.md). -Note, Training-Aware Experiments require the model to be saved in a PyTorch format corresponding to the underlying integration such as Ultralytics YOLOv5 or HuggingFace Transformers. +Note that Training-Aware Experiments require the model to be saved in a PyTorch format corresponding to the underlying integration such as Ultralytics YOLOv5 or Hugging Face Transformers. Datasets must additionally match the expected format of the underlying integration. More details and exact formats are provided in the Training-Aware Experiment Guide. @@ -284,7 +285,7 @@ You can compare the accuracy by looking through the metrics printed out to the c Additionally, you can use [DeepSparse](https://github.com/neuralmagic/deepsparse) to compare the inference performance on your CPU deployment hardware. -Note: In the near future, you will be able to visualize the results in the Cloud, simulate other scenarios and hyperparameters, compare the results to other Experiments, and package for your deployment scenario. +Note: In the near future, you will be able to visualize the results in Sparsify Cloud, simulate other scenarios and hyperparameters, compare the results to other Experiments, and package for your deployment scenario. To run a benchmark on your deployment hardware, use the `deepsparse.benchmark` command with your original model and the new optimized model. @@ -354,9 +355,11 @@ If you're not ready for deploying, congratulations on completing the quickstart! ## Companion Guides - [Sparsify Cloud User Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/cloud-user-guide.md) -- [Sparsify Use Cases Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/use-cases-guide.md) -- [Sparsify Models Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/models-guide.md) - [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md) +- [Sparsify Models Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/models-guide.md) +- [One-Shot Experiments Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/one-shot-experiment-guide.md) +- [Sparse-Transfer Experiments Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/sparse-transfer-experiment-guide.md) +- [Training-Aware Experiments Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/training-aware-experiment-guide.md) ## Resources @@ -364,7 +367,7 @@ Now that you have explored Sparsify [Alpha], here are other related resources. ### Feedback and Support -Report UI issues and CLI errors, submit bug reports, and provide general feedback about the product to the team via the [nm-sparsify Slack Channel](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-1xkdlzwv9-2rvS6yQcCs7VDNUcWxctnw), or via [GitHub Issues](https://github.com/neuralmagic/sparsify/issues). Alpha support is provided through those channels. +Report UI issues and CLI errors, submit bug reports, and provide general feedback about the product to the Sparsify team via the [nm-sparsify Slack Channel](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-1xkdlzwv9-2rvS6yQcCs7VDNUcWxctnw), or via [GitHub Issues](https://github.com/neuralmagic/sparsify/issues). Alpha support is provided through those channels. ### Terms and Conditions @@ -380,7 +383,7 @@ Thank you in advance for your feedback and interest! ### Learning More -- Documentation: [SparseML,](https://docs.neuralmagic.com/sparseml/) [SparseZoo,](https://docs.neuralmagic.com/sparsezoo/) [Sparsify (1st Generation),](https://docs.neuralmagic.com/sparsify/) [DeepSparse](https://docs.neuralmagic.com/deepsparse/) +- Documentation: [SparseML](https://docs.neuralmagic.com/sparseml/), [SparseZoo](https://docs.neuralmagic.com/sparsezoo/), [Sparsify](https://docs.neuralmagic.com/sparsify/), [DeepSparse](https://docs.neuralmagic.com/deepsparse/) - Neural Magic: [Blog,](https://www.neuralmagic.com/blog/) [Resources](https://www.neuralmagic.com/resources/) ### Release History diff --git a/docs/datasets-guide.md b/docs/datasets-guide.md index a1f1f466..b95d0b9d 100644 --- a/docs/datasets-guide.md +++ b/docs/datasets-guide.md @@ -20,171 +20,245 @@ For all Sparsify Experiments, you will need to provide a dataset to create a spa Due to the varied ML pipelines and implementations, Sparsify standardizes on a few popular formats for datasets. You will need to make sure that your data is formatted properly according to the standards listed below. -## Predefined Use Cases +## Table of Contents -### Training-Aware and Sparse-Transfer +1. [Image Classification](#image-classification) +2. [Object Detection](#object-detection) +3. [Image Segmentation](#image-segmentation) +4. [NLP](#nlp) +5. [NPZ](#npz) +6. [Custom](#custom) -Training-Aware and Sparse-Transfer utilize specific dataset standards depending on the use case. -Each one is listed below with an example. +## Image Classification -#### Image Classification +For image classification tasks, Sparsify relies on the standard `SPLIT/CLASS/IMAGE` format used by the PyTorch ImageFolder class. -For image classification tasks, Sparsify relies on the dataset format standard used by the PyTorch ImageFolder class. -This format is fairly simple and intuitive, and it is also widely used in the machine learning community. - -##### Specifications - -- The root folder should contain subdirectories, each representing a single class of images. +### Specifications +- The root folder should contain `train` and `val` subdirectories, each representing the training and validation splits of the dataset. +- Each split should contain subdirectories, each representing a single class of images. - Images of a particular class/category should be placed inside the corresponding subdirectory. - The subdirectory name is used as the class label and should be unique for each class. - The images should be in a format readable by the Python Imaging Library (PIL), which includes formats such as .jpeg, .png, .bmp, etc. - Images do not need to be of the same size. -The PyTorch ImageFolder class automatically assigns numerical class labels to the images based on the lexicographical order of their class directories. -Therefore, it is crucial to ensure the directories are properly named to avoid any confusion or mislabeling. - -##### Example - -For an image classification task involving dogs and cats, the dataset directory should be structured as follows: +The root directory containing the splits data samples should be passed to the CLI as the `--data` argument. +### Structure +```text +data +├── train +│ ├── class_1 +│ │ ├── image_1.png +│ │ ├── image_2.png +│ │ └── ... +│ ├── class_2 +│ │ ├── image_1.png +│ │ ├── image_2.png +│ │ └── ... +│ └── ... +└── val + ├── class_1 + │ ├── image_1.png + │ ├── image_2.png + │ └── ... + ├── class_2 + │ ├── image_1.png + │ ├── image_2.png + │ └── ... + └── ... ``` -root/dog/xxx.png -root/dog/xxy.png -root/dog/xxz.png -root/cat/123.png -root/cat/nsa.png -root/cat/asd.png -``` +For more details and examples on creating image classification datasets for Sparsify, read the [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md). -In this example, all images within the 'dog' subdirectory will be labeled as 'dog', and all images within the 'cat' subdirectory will be labeled as 'cat'. -The exact filenames ('xxx.png', 'xxy.png', etc.) do not matter; what matters is the directory structure and the directory names. +### Example -By organizing the data in this way, it can be easily read and labeled by the PyTorch ImageFolder class, and thus easily used for training image classification models in Sparsify. -Note, the class labels ('dog', 'cat') are case-sensitive and the order of the classes would be sorted lexicographically. -Here, 'cat' will be considered class 0, and 'dog' will be class 1, due to alphabetical order. -#### Object Detection / Image Segmentation +## Object Detection -For object detection and image segmentation tasks, Sparsify supports the dataset format used by YOLOv5. -This format is specifically designed for tasks involving bounding boxes and segmentation masks and is widely adopted in the community. +For object detection tasks, Sparsify utilizes the YOLO format for datasets. +This is the same format used by Ultralytics [YOLOv5/YOLOv8](https://docs.ultralytics.com/datasets/detect/) +The format is made up of a YAML file containing the root dataset location, the classes, and the training and validation split locations. -##### Specifications +If a directory is supplied instead and there is no YAML file within the directory, Sparsify will automatically create one for you. +To auto create a YAML file, the directory structure must be the same as listed below in addition to containing a classes.txt file which contains the class names with one per line. -- Images should be stored in a common directory, generally named `images`. -- Annotations for the images should be stored in a separate directory, often named `labels`. -- Images can be in formats readable by OpenCV (e.g. .jpg, .png). -- Each image should have a corresponding annotation file. The annotation files should be in plain text format (.txt). -- The name of the annotation file should be the same as the corresponding image file, except with a .txt extension. -- Annotation files for object detection should contain one line for each object in the image. Each line should be in the format: ` `, where the values are normalized relative to the size of the image. -- Annotation files for image segmentation should contain information about the segmentation masks. +### Specifications +- The root folder should contain `labels` and `images` subdirectories. +- Underneath both the `labels` and `images` directories, there should be `train` and `val` subdirectories, each representing the training and validation splits of the dataset. +- The split directories under `labels` should contain the YOLO format label files with a single `.txt` file per image. +- The text files underneath the `labels` directories should contain a single line per object of the format `class_index x_center y_center width height` where the coordinates are normalized between 0 and 1 and the class numbers are zero-indexed. +- The split directories under `images` should contain the images of any size in a format readable by the Python Imaging Library (PIL), which includes formats such as .jpeg, .png, .bmp, etc. +- Each image file must have a corresponding label file with the same name in the `labels` directory. +- If supplying a directory without a YAML file, the directory must also contain a `classes.txt` file with one class name per line in the same order as the class numbers in the label files. -##### Example +### Structure +```text +data +├── images +│ ├── train +│ │ ├── image_1.png +│ │ ├── image_2.png +│ │ └── ... +│ ├── val +│ │ ├── image_1.png +│ │ ├── image_2.png +│ │ └── ... +│ └── ... +├── labels +│ ├── train +│ │ ├── image_1.txt +│ │ ├── image_2.txt +│ │ └── ... +│ ├── val +│ │ ├── image_1.txt +│ │ ├── image_2.txt +│ │ └── ... +│ └── ... +├── classes.txt +└── dataset.yaml +``` -For an object detection task involving detecting cars and pedestrians, the dataset directory should be structured as follows: +For more details and examples on creating object detection datasets for Sparsify, read the [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md). -``` -dataset/ -├── images/ -│ ├── image1.jpg -│ └── image2.jpg -└── labels/ - ├── image1.txt - └── image2.txt -``` +### Example -For `image1.jpg`, if there's a car and a pedestrian in the image, the corresponding `image1.txt` file could look like this: -``` -0 0.5 0.6 0.2 0.3 -1 0.7 0.8 0.1 0.2 -``` +## Image Segmentation -This would mean that there is an object of class 0 (car) centered at (50% of image width, 60% of image height) and having a width of 20% of the image width and height 30% of the image height. -The second line is similar but for an object of class 1 (pedestrian). +For image segmentation tasks, Sparsify utilizes the YOLO format for datasets. +This is the same format used by Ultralytics [YOLOv5/YOLOv8](https://docs.ultralytics.com/datasets/segment/) +The format is made up of a YAML file containing the root dataset location, the classes, and the training and validation split locations. -For image segmentation, the labels might be more complex, including segmentation masks that indicate which pixels belong to which object category. +If a directory is supplied instead and there is no YAML file within the directory, Sparsify will automatically create one for you. +To auto create a YAML file, the directory structure must be the same as listed below in addition to containing a classes.txt file which contains the class names with one per line. -Make sure the class labels are consistent with what is expected by the YOLOv5 configuration you are using, and that the bounding box coordinates are normalized as described above. +### Specifications +- The root folder should contain `annotations` and `images` subdirectories. +- Underneath both the `annotations` and `images` directories, there should be `train` and `val` subdirectories, each representing the training and validation splits of the dataset. +- The split directories under `annotations` should contain the YOLO format annotation files with a single `.txt` file per image. +- The text files underneath the `annotations` directories should contain a single line per object of the format `class_index x_1 y_1 x_2 y_2 x_3 y_3` where the coordinates that bound the object are normalized between 0 and 1 and the class numbers are zero-indexed. +- The split directories under `images` should contain the images of any size in a format readable by the Python Imaging Library (PIL), which includes formats such as .jpeg, .png, .bmp, etc. +- Each image file must have a corresponding annotation file with the same name in the `annotations` directory. +- If supplying a directory without a YAML file, the directory must also contain a `classes.txt` file with one class name per line in the same order as the class numbers in the annotation files. -#### Natural Language (NLP/NLG) +### Structure +```text +data +├── images +│ ├── train +│ │ ├── image_1.png +│ │ ├── image_2.png +│ │ └── ... +│ ├── val +│ │ ├── image_1.png +│ │ ├── image_2.png +│ │ └── ... +│ └── ... +├── annotations +│ ├── train +│ │ ├── image_1.txt +│ │ ├── image_2.txt +│ │ └── ... +│ ├── val +│ │ ├── image_1.txt +│ │ ├── image_2.txt +│ │ └── ... +│ └── ... +├── classes.txt +└── dataset.yaml +``` -For natural language processing (NLP) and natural language generation (NLG) tasks, Sparsify supports the dataset formats used by the Hugging Face library. -Hugging Face datasets can be represented in various file formats including JSON, CSV, and JSON lines format (.json). +For more details and examples on creating segmentation datasets for Sparsify, read the [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md). -##### Specifications +### Example -- Each row or line in your data file should represent a single example. -- The data must include the features necessary for your task. For example, a dataset for text classification might include 'text' and 'label' fields. -- For JSON files, each line should be a separate, self-contained JSON object. -- For CSV files, the first row should include the column names, and each subsequent row should include the fields for a single example. -- The file should be UTF-8 encoded to support a wide range of text inputs. -##### Example +## NLP -Here is an example of how you might structure a dataset for a sentiment analysis task: +For NLP tasks, Sparsify utilizes the HuggingFace [Datasets](https://huggingface.co/docs/datasets/) format and expectations. +Hugging Face datasets can be represented in various file formats, including CSV, and JSON lines format (.jsonl). -If you're using a JSON lines (.json) format, your file could look like this: +Specifications: +- The root folder should contain JSON or CSV files associated with each split of the dataset. +- The JSON or CSV files must be named such that the training data contains the word `train`, validation data contains the word `val`, and any optional test data contains the word `test`. +- For JSON files, each line must be a JSON object representing a single data sample. +- For CSV files, the first row must be a header row containing the column names. +- The label column must be named `label`. +- The features column will be dynamically determined based on the column names and the rules below + - If both `setence1` and `sentence2` are present, these columns will be taken as the features. + - Otherwise the first non label columns will be used for the features with sentence1 being set to the first column and setence2 being set to the second if present. +- The files should be UTF-8 encoded. +### Structure + +#### JSON +```text +data +├── train.json +├── val.json +└── test.json ``` + +Where the contents of each JSON file would look like the following: +```text {"text": "I love this movie!", "label": "positive"} {"text": "This movie was awful.", "label": "negative"} {"text": "I have mixed feelings about this film.", "label": "neutral"} ``` -Each line is a separate JSON object, representing a single example. - -If you're using a CSV format, your file could look like this: - +#### CSV +```text +data +├── train.csv +├── val.csv +└── test.csv ``` + +Where the contents of each CSV file would look like the following: +```text text,label "I love this movie!","positive" "This movie was awful.","negative" "I have mixed feelings about this film.","neutral" ``` -The first row contains the column names, and each subsequent row represents a single example. +### Example -Whether you choose to use JSON lines or CSV will depend on your specific needs and preferences, but either format will work well with Hugging Face and Sparsify. -Make sure your data is formatted correctly according to these specifications to ensure it can be used in your experiments. -### One-Shot -For One-Shot Experiments, Sparsify utilizes the `.npz` format for data storage, which is a file format based on the popular NumPy library. -This format is efficient and versatile. -In the near future, more functionality will be landed such that the definitions given above for Training-Aware and Sparse-Transfer will work as well. +## NPZ -#### Specifications +For One-Shot Experiments, Sparsify utilizes the `.npz` format for data storage, which is a file format based on the popular NumPy library. +In the future, more formats will be added for support with One-Shot Experiments. -- Each `.npz` file should contain a single data sample, with no batch dimension. This data sample will be run through the ONNX model. +### Specifications +- Each `.npz` file should contain a single data sample, with no batch dimension. + This data sample will be run through the ONNX model. - The `.npz` file should be structured as a dictionary, mapping the input name in the ONNX specification to a numpy array containing the data. -- All data samples should be stored under the same directory, typically named `data`. +- All data samples should be stored under the same directory, typically named `data`. -The local file structure should look like the following: +The root directory containing the data samples should be passed to the CLI as the `--data` argument. +### Structure ```text data - -- input1.npz - -- input2.npz - -- input3.npz +├── input1.npz +├── input2.npz +├── input3.npz ``` -#### Example - -For example, if you have a BERT-style model with a sequence length of 128, each `.npz` file should contain a dictionary mapping input names ("input_ids", "attention_mask", "token_type_ids") to numpy arrays of the appropriate size: - +Where each `input#.npz` file contains a single data sample, and the data sample is structured as a dictionary mapping the input name in the ONNX specification to a numpy array containing the data that matches the input shapes without the batch dimension. +For example, a BERT-style model running with a sequence length of 128 would have the following data sample: ```text { - "input_ids": ndarray(128,), - "attention_mask": ndarray(128,), - "token_type_ids": ndarray(128,) + "input_ids": ndarray(128,), + "attention_mask": ndarray(128,), + "token_type_ids": ndarray(128,) } ``` -The dictionary keys should match the names of the inputs in the ONNX model specification, and the shapes of the arrays should match the expected input shapes of the model. - -#### Generating NPZ Files +### Example Below is an example script for generating this file structure from a PyTorch module before the ONNX export: @@ -240,7 +314,7 @@ model.save() Note: Replace `YOUR_MODEL` and `YOUR_DATA_LOADER` with your PyTorch model and data loader, respectively. -## Custom Use Cases +## Custom Currently, custom use cases are not supported for dataset representation and datasets must conform to the definitions above. In the near future, these will be supported through plugin specifications. diff --git a/docs/models-guide.md b/docs/models-guide.md index 1f8b9aff..6610ca4c 100644 --- a/docs/models-guide.md +++ b/docs/models-guide.md @@ -16,19 +16,80 @@ limitations under the License. # Sparsify Models Guide -For any Sparsify Experiments, a dense model can be supplied for sparsification. -One-Shot is the only experiment type that requires a model to be passed in. -For others, a default model will be chosen to best fit the given use case. +For most Sparsify Experiments, you will need to provide a base model to create a sparse model from. Due to the varied ML pipelines and implementations, Sparsify standardizes on a few popular formats for models. -You will need to make sure that your models are formatted properly according to the standards listed below. +You will need to make sure that your model is formatted properly according to the standards listed below. -## One-Shot +## Table of Contents -The ONNX model format is the only currently supported format for One-Shot. -See the [SparseML documentation](https://docs.neuralmagic.com) for exporting to ONNX formats. -In the near future, more formats will be added for support with One-Shot. +1. [Image Classification](#image-classification) +2. [Object Detection](#object-detection) +3. [Image Segmentation](#image-segmentation) +4. [NLP](#nlp) +5. [ONNX](#onnx) +6. [Custom](#custom) -## Training-Aware and Sparse-Transfer +## Image Classification -The PyTorch model format is the only currently supported format for Training-Aware and Sparse-Transfer Experiments. -The exact format will depend on the pipeline, and therefore the use case, for the experiment. +For image classification tasks, Sparsify relies on the PTH format generated from SparseML. +Specifically, the PTH format generated from the `ModuleExporter` class in SparseML. +This will save a model in the PTH format with the following structure: + +### Structure +```text +{ + "state_dict": model.state_dict(), + "optimizer": optimizer.state_dict(), + "recipe": recipe, + "epoch": epoch, + "arch_key": arch_key, +} +``` + +### Example +```python +from sparseml.pytorch.image_classification.utils import ModuleExporter +from torchvision.models import resnet18 + +model = resnet18() +exporter = ModuleExporter(model, "./") +exporter.export_pytorch( + optimizer=None, + epoch=-1, + recipe=None, + name=f"{model}.pth", + arch_key="resnet18", +) +``` + +## Object Detection + +For object detection tasks, Sparsify utilizes the YOLO format for models. +This is the same format used by Ultralytics [YOLOv5/YOLOv8](https://docs.ultralytics.com/) +This is the default format that is saved from training within the YOLOv5 or YOLOv8 repos. + +More information on the YOLO format can be found [here](https://docs.ultralytics.com/tasks/detect/#models). + +## Image Segmentation + +For image segmentation tasks, Sparsify utilizes the YOLO format for models. +This is the same format used by Ultralytics [YOLOv5/YOLOv8](https://docs.ultralytics.com/) +This is the default format that is saved from training within the YOLOv5 or YOLOv8 repos. + +More information on the YOLO format can be found [here](https://docs.ultralytics.com/tasks/segment/#models). + +## NLP + +For NLP tasks, Sparsify utilizes the HuggingFace Models format and expectations. +This includes the standard tokenizer.json, config.json, and bin files. +If using any of the standard transformers pathways externally or through SparseML, then this is the default format models are saved in. + +More information on the HuggingFace Models format can be found [here](https://huggingface.co/transformers/model_sharing.html). + +## ONNX + +For One-Shot Experiments, Sparsify utilizes the `.ONNX` format for models. +In the future, more formats will be added for support with One-Shot Experiments. + +For more information on the ONNX format, see the [ONNX website](https://onnx.ai/). +For more information on exporting to the ONNX format, see our docs page [here](https://docs.neuralmagic.com/user-guides/onnx-export). diff --git a/docs/one-shot-experiment-guide.md b/docs/one-shot-experiment-guide.md new file mode 100644 index 00000000..49b0159b --- /dev/null +++ b/docs/one-shot-experiment-guide.md @@ -0,0 +1,146 @@ + + +# Sparsify One-Shot Experiment Guide + +If you're just getting started with Sparsify, we recommend you try out this One-Shot Experiment pathway first. +We also have Sparse-Transfer and Training-Aware Experiments, which you can explore in the [Next Steps](#next-steps) section of this guide. + +## Table of Contents + +1. [Experiment Overview](#experiment-overview) +2. [CLI Quickstart](#cli-quickstart) +4. [Examples](#examples) +5. [Next Steps](#next-steps) +6. [Resources](#resources) + + +## Experiment Overview + +| Sparsity | Sparsification Speed | Accuracy | +|----------|----------------------|----------| +| **++** | **+++++** | **+++** | + +One-Shot Experiments are the quickest way to create a faster and smaller version of your model. +The algorithms are applied to the model post-training, utilizing a calibration dataset, so they result in no further training time and much faster sparsification times compared with Training-Aware Experiments. + +Generally, One-Shot Experiments result in a 3-5x speedup with minimal accuracy loss. +They are ideal for when you want to quickly sparsify your model and have limited time to spend on the sparsification process. + +The CLI Quickstart below will walk you through the steps to run a One-Shot Experiment on your model. +To utilize the cloud pathways for One-Shot Experiments, review the [Cloud User Guide](./cloud-user-guide.md). + +## CLI Quickstart + +Now that you understand what a One-Shot Experiment is and the benefits, including short optimization time due to post-training algorithms, you can now use the CLI to effectively run a One-Shot Experiment. + +Before you run a One-Shot Experiment, confirm you are logged into the Sparsify CLI. +For installation and setup instructions, review the [Install and Setup Section](../README.md#1-install-and-setup) in the Sparsify README. + +One-Shot Experiments use the following general command: + +```bash +sparsify.run one-shot --use-case USE_CASE --model MODEL --data DATA --optim-level OPTIM_LEVEL* +``` + +* optional arguments + +The description, rules, and possible values for each of the arguments are described below: +- [USE_CASE](#use_case) +- [MODEL](#model) +- [DATA](#data) +- [OPTIM_LEVEL](#optim_level) (Optional) + +### USE_CASE + +The generally supported use cases for Sparsify are: +- `cv-classification` +- `cv-detection` +- `cv-segmentation` +- `nlp-question_answering` +- `nlp-text_classification` +- `nlp-sentiment_analysis` +- `nlp-token_classification` +- `nlp-named_entity_recognition` + +Note that other aliases are recognized for these use cases, such as image-classification for cv-classification. +Sparsify will automatically recognize these aliases and apply the correct use case. + +For One-Shot Experiments, both the CLIs and APIs always support custom use cases. +To utilize, run a One-Shot Experiment with `--use-case` set to the desired custom use case. +This custom use case can be any ASCII string. + +### MODEL + +One-Shot requires the model provided to be in an [ONNX format](https://onnx.ai/). +The ONNX model must be exported with static input shapes and not contain custom ONNX operators. +For guidance on how to convert a PyTorch model to ONNX, read our [ONNX Export User Guide](https://docs.neuralmagic.com/user-guides/onnx-export). + +In the near future, more formats including PyTorch will be added for support with One-Shot Experiments. + +### DATA + +For One-Shot Experiments, Sparsify utilizes the `.npz` format for data storage, which is a file format based on the popular NumPy library. +In the future, more formats will be added for support with One-Shot Experiments. + +Specifically, the following structure is expected for the dataset: +```text +data +├── input1.npz +├── input2.npz +├── input3.npz +``` + +Where each `input#.npz` file contains a single data sample, and the data sample is structured as a dictionary mapping the input name in the ONNX specification to a numpy array containing the data that matches the input shapes without the batch dimension. +For example, a BERT-style model running with a sequence length of 128 would have the following data sample: +```text +{ + "input_ids": ndarray(128,), + "attention_mask": ndarray(128,), + "token_type_ids": ndarray(128,) +} +``` + +For more information on the specs and guides for creating the NPZ format, read the [NPZ Dataset Guide](./datasets-guide.md#npz). + +#### OPTIM_LEVEL + +When using Sparsify, the optim (sparsification) level is one of the top arguments you should decide on. +Specifically, it controls how much sparsification is applied to your model, with higher values resulting in faster and more compressed models. +At the max range, though, you may see a drop in accuracy. + +Given that One-Shot is applied in post-training, the sparsity ranges are lowered to avoid accuracy drops as compared with Sparse-Transfer or Training-Aware. +The current ranges are the following (subject to change): +- optim-level == 0.0: no sparsification is applied and the input model is returned as a baseline test case. +- optim-level < 0.3: INT8 quantization of the model (activations and weights) is applied. +- optim-level >= 0.3: unstructured pruning (sparsity) is applied to the weights of the model from 40% for 0.3 to 80% for 1.0 with linear scaling between. + Additionally, INT8 quantization of the model is applied. + +The default of 0.5 will result in a ~50% sparse model with INT8 quantization. + +## Examples + +Check back in soon for walkthroughs and examples of One-Shot Experiments applied to various popular models and use cases. + +### Next Steps + +Now that you have successfully run a One-Shot Experiment, check out the following guides to continue your Sparsify journey: +- [Sparse Transfer Experiment Guide](./sparse-transfer-experiment-guide.md) +- [Training Aware Experiment Guide](./training-aware-experiment-guide.md) + +### Resources + +To learn more about Sparsify and the available pathways other than One-Shot Experiments, refer to the [Sparsify README](../README.md). diff --git a/docs/one-shot_experiment-guide.md b/docs/one-shot_experiment-guide.md deleted file mode 100644 index 9a60a022..00000000 --- a/docs/one-shot_experiment-guide.md +++ /dev/null @@ -1,251 +0,0 @@ - - - - -# Sparsify One-Shot Experiment Guide - -If you're just getting started with Sparsify, we recommend you try out this One-Shot Experiment pathway first. We also have Sparse-Transfer and Training-Aware Experiments, which you can explore in the [Next Steps](#next-steps) section of this guide. - -## Overview -1. One-Shot Experiment Overview -2. One-Shot CLI Quickstart -3. One-Shot Cloud Quickstart -4. Next Steps -5. Resources - - -### One-Shot Experiment Overview - -| Sparsity | Sparsification Speed | Accuracy | -|----------|----------------------|----------| -| **++** | **+++++** | **+++** | - -One-Shot Experiments are the quickest way to create a faster and smaller version of your model. -The algorithms are applied to the model post-training, utilizing a calibration dataset, so they result in no further training time and much faster sparsification times compared with Training-Aware Experiments. - -Generally, One-Shot Experiments result in a 3-5x speedup with minimal accuracy loss. -They are ideal for when you want to quickly sparsify your model and have limited time to spend on the sparsification process. - - -### One-Shot CLI Quickstart - -Now that you understand what a One-Shot Experiment is and the benefits, including short optimization time due to post-training algorithms, you can now use the CLI to effectively run a One-Shot Experiment. - -Before you run a One-Shot Experiment, confirm you are logged into the Sparsify CLI. For installation and setup instructions, review the [Sparsify Install and Setup Section](README section.com) in the Sparsify README. - -One-Shot Experiments use the following general command: - -```bash -sparsify.run one-shot --use-case USE_CASE --model MODEL --data DATA --optim-level OPTIM_LEVEL -``` - -The values for each of the arguments follow these general rules: -- [**`USE_CASE`** ](#use_case) -- [**`MODEL`**](#model) -- [**`DATA`**](#data) -- [**`OPTIM_LEVEL`**](#optim_level) - -#### USE_CASE - -The generally supported use cases for Sparsify are: - -- CV - classification: `cv-classification` -- CV - detection: `cv-detection` -- CV - segmentation: `cv-segmentation` -- NLP - question answering: `nlp-question_answering` -- NLP - text classification: `nlp-text_classification` -- NLP - sentiment analysis: `nlp-sentiment_analysis` -- NLP - token classification: `nlp-token_classification` -- NLP - named entity recognition: `nlp-named_entity_recognition` - -Note that other aliases are recognized for these use cases, such as image-classification for cv-classification. Sparsify will automatically recognize these aliases and apply the correct use case. - -For One-Shot Experiments, both the CLIs and APIs always support custom use cases. To utilize, run a One-Shot Experiment with `--use-case` set to the desired custom use case. This custom use case can be any string as long as it does not contain ASCII characters. - -For full details on Sparsify use cases, read the [Sparsify Use Cases Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/use-cases-guide.md). - -#### MODEL - -One-Shot requires the model provided to be in an [ONNX format](https://onnx.ai/). For guidance on how to convert a PyTorch model to ONNX, read our [ONNX Export User Guide](https://docs.neuralmagic.com/user-guides/onnx-export). - -In the near future, more formats including PyTorch will be added for support with One-Shot Experiments. - -#### DATA - -For One-Shot Experiments, Sparsify utilizes the `.npz` format for data storage, which is a file format based on the popular NumPy library. This format is efficient and versatile. - -##### Dataset Specifications - -- Each `.npz` file should contain a single data sample, with no batch dimension. This data sample will be run through the ONNX model. -- The `.npz` file should be structured as a dictionary, mapping the input name in the ONNX specification to a numpy array containing the data. -- All data samples should be stored under the same directory, typically named `data`. - -The local file structure should look like the following: - -```text -data - -- input1.npz - -- input2.npz - -- input3.npz -``` - -##### Example - -For example, if you have a BERT-style model with a sequence length of 128, each `.npz` file should contain a dictionary mapping input names ("input_ids", "attention_mask", "token_type_ids") to numpy arrays of the appropriate size: - -```text -{ - "input_ids": ndarray(128,), - "attention_mask": ndarray(128,), - "token_type_ids": ndarray(128,) -} -``` - -The dictionary keys should match the names of the inputs in the ONNX model specification, and the shapes of the arrays should match the expected input shapes of the model. - -##### Generating NPZ Files - -Below is an example script for generating this file structure from a PyTorch module **before the ONNX export**: - -```python -import numpy as np -import torch -from torch import Tensor - -class NumpyExportWrapper(torch.nn.Module): - def __init__(self, model): - super(NumpyExportWrapper, self).__init__() - self.model = model - self.model.eval() # Set model to evaluation mode - self.numpy_data = [] - - def forward(self, *args, **kwargs): - with torch.no_grad(): - inputs = {} - batch_size = 0 - - for index, arg in enumerate(args): - if isinstance(arg, Tensor): - inputs[f"input_{index}"] = arg - batch_size = arg.size[0] - - for key, val in kwargs.items(): - if isinstance(val, Tensor): - inputs[key] = val - batch_size = val.shape[0] - - start_index = len(self.numpy_data) - for _ in range(batch_size): - self.numpy_data.append({}) - - for input_key in iter(inputs): - for idx, input in enumerate(inputs[input_key]): - self.numpy_data[start_index+idx][input_key] = input - - return self.model(*args, **kwargs) - - def save(self, path: str = "data"): - for index, item in enumerate(self.numpy_data): - npz_file_path = f'{path}/input{str(index).zfill(4)}.npz' - np.savez(npz_file_path, **item) - - print(f'Saved {len(self.numpy_data)} npz files to {path}') - -model = NumpyExportWrapper(YOUR_MODEL) -for data in YOUR_DATA_LOADER: - model(data[0]) -model.save() -``` - -Note: Replace `YOUR_MODEL` and `YOUR_DATA_LOADER` with your PyTorch model and data loader, respectively. - -For full details on Sparsify datasets, read the [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md#sparsify-datasets-guide). - -#### OPTIM_LEVEL - -When using Sparsify, the optim (sparsification) level is one of the top arguments you should decide on. Specifically, it controls how much sparsification is applied to your model, with higher values resulting in faster and more compressed models. At the max range, though, you may see a drop in accuracy. - -The optim level can be set anywhere from 0.0 to 1.0, where 0.0 is for no sparsification and 1.0 is for maximum sparsification. -0.5 is the default optim level and is a good starting point for most use cases. - -##### One-Shot Optim Levels - -Given that One-Shot is applied in post-training, the sparsity ranges are lowered to avoid accuracy drops as compared with Sparse-Transfer or Training-Aware. -The specific ranges are the following: - -- optim-level == 0.0: no sparsification is applied and the input model is returned as a baseline test case. -- optim-level < 0.3: INT8 quantization of the model (activations and weights) is applied. -- optim-level >= 0.3: unstructured pruning (sparsity) is applied to the weights of the model from 40% for 0.3 to 80% for 1.0 with linear scaling between. - Additionally, INT8 quantization of the model is applied. - -The default of 0.5 will result in a ~50% sparse model with INT8 quantization. - - -### Example One-Shot Experiment CLI Commands - -Here are code examples of One-Shot Experiments you may wish to run; pick your use case and start sparsifying with One-Shot! - -#### Running One-Shot Experiments - -##### Computer Vision Use Case: - -You have an image classification use case and want to run a One-Shot Experiment on a dense ResNet-50 model using the imagenette dataset. You want to quickly and cheaply generate a sparse model so that you can build a prototype of the ResNet-50 model inferencing on a CPU server in the cloud with DeepSparse. Getting a working model that meets your deployment requirements on the imagenette dataset will give you the confidence to continue on your initiative knowing you can hit the metrics required for the business. - -You are targeting a balanced model in terms of wanting to get a 3-5x performance boost in latency while also maintaining the high accuracy of the model so that you can confidently deploy the model in production to solve your business case. - -You can use a Sparsify One-Shot Experiment to try and reach your goal. You have a standard ResNet-50 model as your dense baseline on imagenette which Sparsify already has as an alias model and npz formatted dataset hosted for you to use out of the box. Since you want to very quickly achieve a 3-5x speedup in latency performance with minimal training costs, a One-Shot Experiment makes the most sense for you for its fast optimization and lower, moderately performant sparsity profile. - -With all of these considerations in mind, run the following One-Shot Experiment command to achieve this use case goal: -```bash -sparsify.run one-shot --use-case image_classification --model resnet50 --data imagenette --optim-level 0.5 -``` -The output is as follows: - -MARK - -##### NLP Use Case: -You are working on a text classification use case to help classify text reviews received from your customers through your e-commerce website. You have been having slow inference times using the BERT-base model and want to improve the performance to save costs. - -You want to quickly and cheaply generate a sparse BERT-base model so that you can use it to classify our customer reviews at a lower cost due to the improved performance and speed of the model. You are focused on improving the throughput of the model to process more requests, faster. - -You are targeting a balanced model in terms of wanting to get a 3-5x performance boost in throughput while having a high accuracy so your classifications are actionable. - -You can use a Sparsify One-Shot Experiment to try and reach your goal. You have a standard BERT-base model as our dense baseline on the SST2 dataset which Sparsify already has as an alias model and npz formatted dataset hosted for you to use out of the box. You want to try and reduce your costs by improving the throughput performance of your model and you are limited by our compute spend and team size. A One-Shot Experiment makes the most sense for you for its fast optimization and lower cost pathway as opposed to fully retraining the model to optimize it. - -With all of these considerations in mind, run the following One-Shot Experiment command to achieve your goal this use case goal: - -```bash -sparsify.run one-shot --use-case text_classification --model bert-base --data sst2 --optim-level 0.5 -``` -The output is as follows: -MARK - - -### One-Shot Cloud Quickstart - -In addition to manually creating commands, you can use the Sparsify Cloud to generate Sparsify One-Shot Experiment commands. - -To get started, read the [Sparsify Cloud User Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/cloud-user-guide.md). - - -### Next Steps - -Now that you have successfully run a One-Shot Experiment, check out the [Sparse-Transfer](LINK.com) and [Training-Aware](LINK.com) Experiments to target different sparsity profiles. - - -### Resources -To learn more about Sparsify and all of the available pathways outside of One-Shot Experiments, refer to the [Sparsify README](https://github.com/neuralmagic/sparsify). diff --git a/docs/sparse-transfer-experiment-guide.md b/docs/sparse-transfer-experiment-guide.md index 830fb61b..ba108b19 100644 --- a/docs/sparse-transfer-experiment-guide.md +++ b/docs/sparse-transfer-experiment-guide.md @@ -1,6 +1,3 @@ - - - + +# Sparsify Training-Aware Experiment Guide + +The Sparsify Training-Aware Experiment Guide is a guide for running Training-Aware Experiments with the Sparsify CLI. +We also have One-Shot and Sparse-Transfer Experiments, which you can explore in the [Next Steps](#next-steps) section of this guide. + +## Table of Contents + +1. [Experiment Overview](#experiment-overview) +2. [CLI Quickstart](#cli-quickstart) +4. [Examples](#examples) +5. [Next Steps](#next-steps) +6. [Resources](#resources) + +## Experiment Overview + +| Sparsity | Sparsification Speed | Accuracy | +|-----------|-----------------------|-----------| +| **+++++** | **++** | **+++++** | + +Training-Aware Experiments are the most accurate way to create a faster and smaller model for your dataset. +The algorithms are applied to the model during training, so they offer the best possible recovery of accuracy. +However, they do require additional training time and hyperparameter tuning to achieve the best results. + +Generally, Training-Aware Experiments result in a 6–12x speedup with minimal accuracy loss. +They are ideal when you have the time to train a model, have a custom model, or want to achieve the best possible accuracy. + +The CLI Quickstart below will walk you through the steps to run a Training-Aware Experiment on your model. +To utilize the cloud pathways for Training-Aware Experiments, review the [Cloud User Guide](./cloud-user-guide.md). + +## CLI Quickstart + +Now that you understand what a Training-Aware Experiment is and the benefits, including the best possible recovery of accuracy for an optimized model, you're ready to use the CLI to effectively run a Training-Aware Experiment. + +Before you run a Training-Aware Experiment, confirm you are logged in to the Sparsify CLI. +For instructions on Installation and Setup, review the [Sparsify Install and Setup Section](READMEsection.com) in the Sparsify README. + +Training-Aware Experiments use the following general command: + +```bash +sparsify.run training-aware --use-case USE_CASE --model MODEL --data DATA --optim-level OPTIM_LEVEL* +``` + +* optional arguments + +The values for each of the arguments follow these general rules: +- [USE_CASE](#use_case) +- [MODEL](#model) +- [DATA](#data) +- [OPTIM_LEVEL](#optim_level) (Optional) + +### USE_CASE + +The generally supported use cases for Sparsify are: +- `cv-classification` +- `cv-detection` +- `cv-segmentation` +- `nlp-question_answering` +- `nlp-text_classification` +- `nlp-sentiment_analysis` +- `nlp-token_classification` +- `nlp-named_entity_recognition` + +Note that other aliases are recognized for these use cases, such as image-classification for cv-classification. +Sparsify will automatically recognize these aliases and apply the correct use case. + +Currently, custom use cases are not supported for Training-Aware Experiments. + +#### MODEL + +Models are optional for the Sparse-Transfer pathway. +If no model is provided, a performance and accuracy balanced base model for the use case will be chosen. + +If you choose to override the model, it is expected to be a pre-sparsified model and adhere to the following formats depending on the use case: +- `cv-classification`: SparseML PTH Format + - [Image Classification Models Guide](./models-guide#image-classification) +- `cv-detection` - YOLOv5/YOLOv8 Format + - [Object Detection Models Guide](./models-guide#object-detection) + - Example structure: data/classes.txt; data/images/{SPLIT}/{IMAGE.EXT}; data/labels/{SPLIT}/{IMAGE.EXT}) +- `cv-segmentation` - YOLOv5/YOLOv8 Format + - [Image Segmentation Models Guide](./models-guide#image-segmentation) +- `nlp-*`: Hugging Face Format + - [NLP Models Guide](./models-guide#nlp) + +Currently, custom use cases are not supported for model representation and models must conform to the definitions above. +In the near future, these will be supported through plugin specifications. + +For full details on Sparsify models, read the [Sparsify Models Guide](./models-guide.md). + +#### DATA + +For all Sparsify Experiments, you will need to provide a dataset to create a sparse model. +Due to the varied ML pipelines and implementations, Sparsify standardizes on a few popular formats for datasets. +Confirm that your data is formatted properly according to the standards listed below. + +Different use cases may require different input formats depending on what is considered standard for that use case. +Specifically, the following are the supported formats as well as links to specs and guides for creating datasets for each format: +- `cv-classification`: Image Folder Format + - [Image Classification Dataset Guide](./datasets-guide#image-classification) + - Example structure: data/{SPLIT}/{CLASS}/{IMAGE.EXT}) +- `cv-detection` - YOLO Format + - [Object Detection Dataset Guide](./datasets-guide#object-detection) + - Example structure: data/classes.txt; data/images/{SPLIT}/{IMAGE.EXT}; data/labels/{SPLIT}/{IMAGE.EXT}) +- `cv-segmentation` - YOLO Format + - [Image Segmentation Dataset Guide](./datasets-guide#image-segmentation) + - Example structure: data/classes.txt; data/images/{SPLIT}/{IMAGE.EXT}; data/annotations/{SPLIT}/{IMAGE.EXT}) +- `nlp-*`: Hugging Face CSV or JSONW Format + - [NLP Dataset Guide](./datasets-guide#nlp) + - Example structure: data/{SPLIT}.csv or data/{SPLIT}.jsonl or data/{SPLIT}.json + +Currently, custom use cases are not supported for dataset representation and datasets must conform to the definitions above. +In the near future, these will be supported through plugin specifications. + +For full details on Sparsify datasets, read the [Sparsify Datasets Guide](./datasets-guide.md). + +#### OPTIM_LEVEL + +When using Sparsify, the optim (sparsification) level is one of the top arguments you should decide on. +Specifically, it controls how much sparsification is applied to your model with higher values resulting in faster and more compressed models. +At the max range, though, you may see a drop in accuracy. + +Given that Training-Aware is applied while training, the sparsity ranges are increased as compared to one shot since accuracy recovery is easier at higher sparsities. +The specific ranges are the following: +- optim-level == 0.0: no sparsification is applied and the input model is returned as a baseline test case. +- optim-level < 0.3: INT8 quantization of the model (activations and weights) is applied. +- optim-level >= 0.3: unstructured pruning (sparsity) is applied to the weights of the model from 60% for 0.3 to 95% for 1.0 with linear scaling between. + Additionally, INT8 quantization of the model is applied. + +The default of 0.5 will result in a ~70% sparse model with INT8 quantization, and is a good default to start with. + +## Examples + +Check back in soon for walkthroughs and examples of One-Shot Experiments applied to various popular models and use cases. + +### Next Steps + +Now that you have successfully run a Training-Aware Experiment, check out the following guides to continue your Sparsify journey: +- [One-Shot Experiment Guide](./one-shot-experiment-guide.md) +- [Sparse Transfer Experiment Guide](./sparse-transfer-experiment-guide.md) + +### Resources + +To learn more about Sparsify and the available pathways other than Training-Aware Experiments, refer to the [Sparsify README](../README.md). diff --git a/docs/training-aware_experiment-guide.md b/docs/training-aware_experiment-guide.md deleted file mode 100644 index 1225547a..00000000 --- a/docs/training-aware_experiment-guide.md +++ /dev/null @@ -1,302 +0,0 @@ - - - - - - -# Sparsify Training-Aware Experiment Guide - -## Overview -1. Training-Aware Experiment Overview -2. Training-Aware CLI Quickstart -3. Training-Aware Cloud Quickstart -4. Next Steps -5. Resources - - - -#### Training-Aware Experiments - -| Sparsity | Sparsification Speed | Accuracy | -|-----------|-----------------------|-----------| -| **+++++** | **++** | **+++++** | - -Training-Aware Experiments are the most accurate way to create a faster and smaller model for your dataset. -The algorithms are applied to the model during training, so they offer the best possible recovery of accuracy. -However, they do require additional training time and hyperparameter tuning to achieve the best results. - -Generally, Training-Aware Experiments result in a 6–12x speedup with minimal accuracy loss. They are ideal when you have the time to train a model, have a custom model, or want to achieve the best possible accuracy. - - -### Training-Aware CLI Quickstart - -Now that you understand what a Training-Aware Experiment is and the benefits, including the best possible recovery of accuracy for an optimized model, you can now use the CLI to effectively run a Training-Aware Experiment. - -Before you run a Training-Aware Experiment, confirm you are logged in to the Sparsify CLI. For instructions on Installation and Setup, review the [Sparsify Install and Setup Section](READMEsection.com) in the Sparsify README. - -Training-Aware Experiments use the following general command: - -```bash -sparsify.run training-aware --use-case USE_CASE --model MODEL --data DATA --optim-level OPTIM_LEVEL -``` - -The values for each of the arguments follow these general rules: -- [**`USE_CASE`** ](#use_case) -- [**`MODEL`**](#model) -- [**`DATA`**](#data) -- [**`OPTIM_LEVEL`**](#optim_level) - -#### USE_CASE - -The generally supported use cases for Sparsify are: - -- CV - classification: `cv-classification` -- CV - detection: `cv-detection` -- CV - segmentation: `cv-segmentation` -- NLP - question answering: `nlp-question_answering` -- NLP - text classification: `nlp-text_classification` -- NLP - sentiment analysis: `nlp-sentiment_analysis` -- NLP - token classification: `nlp-token_classification` -- NLP - named entity recognition: `nlp-named_entity_recognition` - -Note that other aliases are recognized for these use cases, such as image-classification for cv-classification. Sparsify will automatically recognize these aliases and apply the correct use case. - -For Training-Aware Experiments, custom use cases are only supported with the APIs for custom integrations. This is because non-custom integrations utilize plugins that correspond to the appropriate use case for training pipelines. To utilize this, ensure that you have a training pipeline ready to go and inject the Sparsify API into the training pipeline with the desired use case passed in as an argument. More information on this specific pathway will be available in the near future as Sparsify's development progresses. - -For full details on Sparsify use cases, read the [Sparsify Use Cases Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/use-cases-guide.md). - -#### MODEL - - -The PyTorch model format is the supported model format for Training-Aware Experiments. The exact format will depend on the pipeline, and therefore the use case, for the Training-Aware Experiment. - -#### DATA - -For all Sparsify Experiments, you will need to provide a dataset to create a sparse model. -Due to the varied ML pipelines and implementations, Sparsify standardizes on a few, popular formats for datasets. -Confirm that your data is formatted properly according to the standards listed below. - -##### Predefined Use Cases - -Training-Aware Experiments utilize specific dataset standards depending on the use case. -Each one is listed below with an example. - -##### Image Classification - -For image classification tasks, Sparsify relies on the dataset format standard used by the PyTorch ImageFolder class. -This format is fairly simple and intuitive, and it is also widely used in the machine-learning community. - -##### Specifications - -- The root folder should contain subdirectories, each representing a single class of images. -- Images of a particular class/category should be placed inside the corresponding subdirectory. -- The subdirectory name is used as the class label and should be unique for each class. -- The images should be in a format readable by the Python Imaging Library (PIL), which includes formats such as .jpeg, .png, .bmp, etc. -- Images do not need to be of the same size. - -The PyTorch ImageFolder class automatically assigns numerical class labels to the images based on the lexicographical order of their class directories. -Therefore, it is crucial to ensure the directories are properly named to avoid any confusion or mislabeling. - -##### Image Classification Example - -For an image classification task involving dogs and cats, the dataset directory should be structured as follows: - -``` -root/dog/xxx.png -root/dog/xxy.png -root/dog/xxz.png - -root/cat/123.png -root/cat/nsa.png -root/cat/asd.png -``` - -In this example, all images within the 'dog' subdirectory will be labeled as 'dog', and all images within the 'cat' subdirectory will be labeled as 'cat'. -The exact filenames ('xxx.png', 'xxy.png', etc.) do not matter; what matters is the directory structure and the directory names. - -By organizing the data in this way, it can be easily read and labeled by the PyTorch ImageFolder class, and thus easily used for training image classification models in Sparsify. - -Note that the class labels ('dog', 'cat') are case-sensitive and the order of the classes would be sorted lexicographically. -Here, 'cat' will be considered class 0, and 'dog' will be class 1, due to alphabetical order. - -##### Object Detection / Image Segmentation - -For object detection and image segmentation tasks, Sparsify supports the dataset format used by YOLOv5. -This format is specifically designed for tasks involving bounding boxes and segmentation masks and is widely adopted in the community. - -##### Specifications - -- Images should be stored in a common directory, generally named `images`. -- Annotations for the images should be stored in a separate directory, often named `labels`. -- Images can be in formats readable by OpenCV (e.g. .jpg, .png). -- Each image should have a corresponding annotation file. The annotation files should be in plain text format (.txt). -- The name of the annotation file should be the same as the corresponding image file, except with a .txt extension. -- Annotation files for object detection should contain one line for each object in the image. Each line should be in the format: ` `, where the values are normalized relative to the size of the image. -- Annotation files for image segmentation should contain information about the segmentation masks. - -##### Object Detection / Image Segmentation Example - -For an object detection task involving detecting cars and pedestrians, the dataset directory should be structured as follows: - -``` -dataset/ -├── images/ -│ ├── image1.jpg -│ └── image2.jpg -└── labels/ - ├── image1.txt - └── image2.txt -``` - -For `image1.jpg`, if there's a car and a pedestrian in the image, the corresponding `image1.txt` file could look like this: - -``` -0 0.5 0.6 0.2 0.3 -1 0.7 0.8 0.1 0.2 -``` - -This would mean that there is an object of class 0 (car) centered at (50% of the image width, 60% of the image height) and having a width of 20% of the image width and a height of 30% of the image height. -The second line is similar but for an object of class 1 (pedestrian). - -For image segmentation, the labels might be more complex, including segmentation masks that indicate which pixels belong to which object category. - -Make sure the class labels are consistent with what is expected by the YOLOv5 configuration you are using, and that the bounding box coordinates are normalized as described above. - -##### Natural Language (NLP/NLG) - -For natural language processing (NLP) and natural language generation (NLG) tasks, Sparsify supports the dataset formats used by the Hugging Face library. -Hugging Face datasets can be represented in various file formats including JSON, CSV, and JSON lines format (.jsonl). - -##### Specifications - -- Each row or line in your data file should represent a single example. -- The data must include the features necessary for your task. For example, a dataset for text classification might include 'text' and 'label' fields. -- For JSON files, each line should be a separate, self-contained JSON object. -- For CSV files, the first row should include the column names, and each subsequent row should include the fields for a single example. -- The file should be UTF-8 encoded to support a wide range of text inputs. - -##### Natural Language (NLP/NLG) Example - -Here is an example of how you might structure a dataset for a sentiment analysis task: - -If you're using a JSON lines (.jsonl) format, your file could look like this: - -``` -{"text": "I love this movie!", "label": "positive"} -{"text": "This movie was awful.", "label": "negative"} -{"text": "I have mixed feelings about this film.", "label": "neutral"} -``` - -Each line is a separate JSON object, representing a single example. - -If you are using a CSV format, your file could look like this: - -``` -text,label -"I love this movie!","positive" -"This movie was awful.","negative" -"I have mixed feelings about this film.","neutral" -``` - -The first row contains the column names, and each subsequent row represents a single example. - -Whether you choose to use JSON lines or CSV will depend on your specific needs and preferences, but either format will work well with Hugging Face and Sparsify. -Make sure your data is formatted correctly according to these specifications to ensure it can be used in your experiments. - -##### Custom Use Cases -Currently, custom use cases are not supported for dataset representation and datasets must conform to the definitions above. In the near future, these will be supported through plugin specifications. - -For full details on Sparsify datasets, read the [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md#sparsify-datasets-guide). - -#### OPTIM_LEVEL - -When using Sparsify, the optim (sparsification) level is one of the top arguments you should decide on. Specifically, it controls how much sparsification is applied to your model with higher values resulting in faster and more compressed models. At the max range, though, you may see a drop in accuracy. - -The optim level can be set anywhere from 0.0 to 1.0, where 0.0 is for no sparsification and 1.0 is for maximum sparsification. -0.5 is the default optim level and is a good starting point for most use cases. - -##### Training-Aware Optim Levels - - -Given that Training-Aware is applied while training, the sparsity ranges are increased as compared to one shot since accuracy recovery is easier at higher sparsities. - -The specific ranges are the following: - -- optim-level == 0.0: no sparsification is applied and the input model is returned as a baseline test case. -- optim-level < 0.3: INT8 quantization of the model (activations and weights) is applied. -- optim-level >= 0.3: unstructured pruning (sparsity) is applied to the weights of the model from 60% for 0.3 to 95% for 1.0 with linear scaling between. - Additionally, INT8 quantization of the model is applied. - -The default of 0.5 will result in a ~70% sparse model with INT8 quantization. - - -### Example Training-Aware Experiment CLI Commands - -Here are code examples of Training-Aware Experiments you may wish to run; pick your use case and start sparsifying with Training-Aware! - -#### Running Training-Aware Experiments - -##### Computer Vision Use Case: - -You have an image classification use case and want to run a Training-Aware Experiment on a dense ResNet-50 model using the imagenette dataset. You want to ensure you have the most sparsity to get the best possible performance and maintain a high level of accuracy. - -You are targeting a balanced model in terms of wanting to get a 6-12x performance boost in latency while also maintaining the high accuracy of the model so that you can confidently deploy the model in production to solve your business case. - -You can use a Sparsify Training-Aware Experiment to try and reach your goal. Training-Aware Experiments apply SOTA optimization techniques during training to generate a highly optimized sparse model with very little to no impact on accuracy. Since you want to get the most possible performance speedup in latency and need a high level of accuracy, a Training-Aware Experiment makes the most sense for you for its highly optimized, performant sparsity profile as well as high accuracy profile. - -With all of these considerations in mind, run the following Training-Aware Experiment command to achieve your use case goal: -```bash -sparsify.run training-aware --use-case image_classification --model resnet50 --data imagenette --optim-level 0.5 -``` - - -The output is as follows: - -MARK - -##### NLP Use Case: -You are working on a text classification use case to help classify text reviews received from your customers through your e-commerce website. You have been having slow inference times using the BERT-base model, but have an accurate model that you want to ensure does not take a large hit. - -You are targeting a balanced model, but are targeting a significant 6-12x performance boost in text classification throughput while also maintaining the highest level of accuracy with the model so that you can confidently deploy the model in production to solve your business case. You are focused on improving the throughput of the model to process more requests, but sacrifice as few points in accuracy as possible. - -You are targeting a balanced model in terms of wanting to get a 6-12x performance boost in throughput while losing little to no accuracy so your classifications are actionable. - -You can use a Sparsify Training-Aware Experiment to try and reach your goal. Since you want to use the SST2 dataset on BERT-base to get the highest performing model with the lowest accuracy hit, a Training-Aware Experiment makes the most sense for you for its highly optimized, performant sparsity profile as well as high accuracy profile. - -With all of these considerations in mind, run the following Training-Aware Experiment command to achieve your use case goal: - -```bash -sparsify.run training-aware --use-case text_classification --model bert-base --data sst2 --optim-level 0.5 -``` -The output is as follows: -MARK - - -### Training-Aware Cloud Quickstart - -In addition to manually creating commands, you use Sparsify Cloud to generate Sparsify Training-Aware Experiment commands. - -To get started, read the [Sparsify Cloud User Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/cloud-user-guide.md). - - -### Next Steps - -Now that you have successfully run a Training-Aware Experiment, check out the [One-Shot](https://github.com/neuralmagic/sparsify/blob/main/docs/one-shot_experiment-guide.md) and [Sparse-Transfer](https://github.com/neuralmagic/sparsify/blob/main/docs/sparse-transfer_experiment-guide.md) Experiments to target different sparsity profiles. - - -### Resources -To learn more about Sparsify and all of the available pathways outside of Training-Aware Experiments, refer to the [Sparsify README](https://github.com/neuralmagic/sparsify). diff --git a/docs/use-cases-guide.md b/docs/use-cases-guide.md deleted file mode 100644 index b234e039..00000000 --- a/docs/use-cases-guide.md +++ /dev/null @@ -1,57 +0,0 @@ - - -# Sparsify Use Cases Guide - -To use Sparsify, you must specify a use case for all experiments to run. -A use case is the specific task or domain/sub-domain you wish to sparsify a model for, such as image classification, object detection, or text classification. -It is used to enable Sparsify to apply the best sparsification techniques for your use case, to automatically package the model for deployment, and, depending on what is run, to load specific pipelines for data loading and training. - -## Use Cases - -The generally supported use cases for Sparsify currently are: -- CV - classification: `cv-classification` -- CV - detection: `cv-detection` -- CV - segmentation: `cv-segmentation` -- NLP - question answering: `nlp-question_answering` -- NLP - text classification: `nlp-text_classification` -- NLP - sentiment analysis: `nlp-sentiment_analysis` -- NLP - token classification: `nlp-token_classification` -- NLP - named entity recognition: `nlp-named_entity_recognition` - -Note, other aliases are recognized for these use cases, such as image-classification for cv-classification. -Sparsify will automatically recognize these aliases and apply the correct use case. - -### Custom Use Cases - -If you wish to use Sparsify for a use case that is not in the list of currently supported use cases, you can use a custom use case for some pathways in Sparsify. -The custom use cases will be saved in Sparsify Cloud for future reuse when run through a supported pathway. -The pathways that support custom use cases are listed below. - -Note that custom use cases will prevent Sparsify from applying known, domain-specific knowledge for the sparsification of your model. -Additionally, it will prevent auto-filling of the pre- and post-processing functions when creating a deployment package. - -#### One-Shot - -For One-Shot Experiments, both the CLIs and APIs always will support custom use cases. -To utilize, run a One-Shot Experiment with `--use-case` set to the desired custom use case. - -### Training-Aware - -For Training-Aware Experiments, custom use cases are only supported with the APIs for custom integrations. -This is because non-custom integrations utilize plugins that correspond to the appropriate use case for training pipelines. -To utilize this, ensure that you have a training pipeline ready to go and inject the Sparsify API into the training pipeline with the desired use case passed in as an argument. -More information on this specific pathway will be available in the near future as Sparsify's development progresses.