Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds Olive Robotics information and a basic introduction to the project. #64

Merged
merged 4 commits into from
Oct 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 51 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Fruit detection

This repository contains the code of a case study where we have combined multiple technologies to study the how we can leverage NVidia Omniverse™, ROS 2, PyTorch and the great olixVision™ Camera to simulate and run a system that detects fruit. There are multiple applications for a system like this, ranging from agriculture to industrial automation applications. This case study shows how the simulation and systhetic data generation processes can be used to model a camera sensor, generate datasets for ML model training and evaluation as well as the real life integration into a real and functional system.

The setup built for this system can be seen in the following picture:

<img src="./doc/hardware_setup.png" alt="Hardware setup" width="720"/>

Connected to the camera, there is a Lenovo Legion Pro 7i 16, with an Intel® Core™ i9-14900HX, 32GB of RAM, 2TB of disk, NVidia GeForce RTX 4090 GPU, which was used for development. Requirements for running the system are less than for development and training. See the following sections for more information.

# Requisites

- [Docker](https://docs.docker.com/engine/install/ubuntu/)
Expand Down Expand Up @@ -31,9 +39,43 @@ pre-commit run --all-files

# Documentation


## olixVision™ Camera by Olive Robotics

Please, consider visiting [Olive Robotics web page](https://olive-robotics.com/) to learn more about their portfolio. In particular, you can refer to the [olixVision™ Camera](https://olive-robotics.com/?page_id=613) for further details and the specifications of the camera.

The olixVision™ Camera provides a ROS 2 native interface which allows to quickly integrate it to any ROS 2 enabled application. The necessary steps to connect to it are described in the [documentation page](https://olive-robotics.com/?page_id=592). Navigate through the Hardware > Camera menu titles to access all the details about it. For a quick setup, we recommend following these steps:

1. Connect the USB cable to the camera

<img src="./doc/usb_camera.jpeg" alt="USB connection to the camera" width="720"/>

2. Connect the USB cable to your computer

<img src="./doc/usb_computer.jpeg" alt="USB connection to the computer" width="720"/>

3. Configure the network interface

The camera will create an Ethernet over USB driver. It comes with a static IP configured, which it is 192.168.7.2 in our case, and then one needs to validate the new network interface appears in the network manager and configure the computer interface accordingly. For us, that is:

- IP: 192.168.7.100
- Netmask: 255.255.255.0
- Gateway: N/A
- Static IP configuration

<img src="./doc/network_configuration.png" alt="Networkd configuration" width="720"/>

4. Verify you can access the web dashboard

Head to your favourite browser and type the IP of the camera in the URL. You should be able to see the camera is up when it displays a dashboard similar to the following:

<img src="./doc/olive_dashboard.png" alt="Olive dashboard" width="720"/>

Please look into the oficial documentation to learn more about other ways in you can use the camera.

## Architecture

Within the docker directory you'll find a docker compose file, the dockerfiles for each image and some custom configuration files.
Within the docker directory you will find a docker compose file, the dockerfiles for each image and some custom configuration files.
The system relies on using profiles to select which set of services build and run depending on the workflow. The following sections explain how to deal with them.

## Profiles
Expand All @@ -43,19 +85,19 @@ The available profiles are:
- `training`: trains a fasterrcnn_resnet50_fpn model based on a synthetic dataset.
- `detection`: loads the detection stack.
- `visualization`: loads RQt to visualize the input and output image processing.
- `webcam`: loads the usb_cam driver that makes a connected webcam to publish. Useful when the Olive Camera is not available.
- `webcam`: loads the usb_cam driver that makes a connected webcam to publish. Useful when the olixVision™ Camera is not available.
- `simulation`: loads the simulation NVIDIA Omniverse™.
- `dataset_gen`: generates a training dataset using NVIDIA Omniverse™.

Compound profiles are:

- `olive_pipeline`: loads `visualization` and `detection`, expects the Olive Camera to be connected.
- `olive_pipeline`: loads `visualization` and `detection`, expects the olixVision™ Camera to be connected.

```mermaid
graph TD
A[Olive Camera] --> B[Fruit Detection Node]
A[olixVision™ Camera] --> B[Fruit Detection Node]
B[Fruit Detection Node] --> C[RQt Visualization]
A[Olive Camera] --> C[RQt Visualization]
A[olixVision™ Camera] --> C[RQt Visualization]
```

- `webcam_pipeline`: loads `webcam`,`visualization` and `detection`.
Expand Down Expand Up @@ -152,7 +194,7 @@ To run the system you need to define which profile(s) to run. You can pile profi

### Running olive_pipeline

To load the system with the Olive Camera, detection and the visualization in RQt, you can do the following:
To load the system with the olixVision™ Camera, detection and the visualization in RQt, you can do the following:

1. Connect the camera to the USB port of your computer.

Expand All @@ -170,6 +212,8 @@ docker compose -f docker/docker-compose.yml --profile olive_pipeline up
docker compose -f docker/docker-compose.yml --profile olive_pipeline down
```

![olixVision™ Camera](./doc/olive_pipeline.gif)

### Running webcam_pipeline

To load the system with a webcam, detection and the visualization in RQt, you can do the following:
Expand Down Expand Up @@ -256,7 +300,7 @@ docker exec -it rosbag bash
source /opt/ros/humble/setup.bash
```

4. With a running system (see any of the aforementioned run ways), record the Olive Camera:
4. With a running system (see any of the aforementioned run ways), record the olixVision™ Camera:

```bash
ros2 bag record -s mcap -o my_rosbag /olive/camera/id01/image/compressed
Expand Down
Binary file added doc/hardware_setup.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/network_configuration.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/olive_dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/olive_pipeline.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/usb_camera.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added doc/usb_computer.jpeg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions isaac_ws/simulation_ws/assets/olive_camera/README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Olive camera
# olixVision™ Camera

## Sensor and lens configuration information

- Olive Camera documentation: https://docs.olive-robotics.com/hardware/camera/camera_01_tp.html
- Sensor OV2710 datasheet: https://static6.arrow.com/aropdfconversion/20a4701909ea0c3602c8573c489a64272f1c28d6/ov2710pbv1.1web.pdf
- Olive Camera compatible lenses: https://docs.olive-robotics.com/hardware/camera/camera_01_tp.html#supported-camera-lenses
- olixVision™ Camera [documentation](https://docs.olive-robotics.com/hardware/camera/camera_01_tp.html)
- Sensor OV2710 [datasheet](https://static6.arrow.com/aropdfconversion/20a4701909ea0c3602c8573c489a64272f1c28d6/ov2710pbv1.1web.pdf)
- olixVision™ Camera [compatible lenses](https://docs.olive-robotics.com/hardware/camera/camera_01_tp.html#supported-camera-lenses)

For this project, we have worked with the "2.8-12mm 1:1.4 IR" and "Far-view" lenses.

Expand Down
Loading