Merge branch 'topic-recognition' into topic-recognition

.gitignore

@@ -6,7 +6,7 @@ __pycache__/
 
 # C extensions
 *.so
-
+.idea
 # Distribution / packaging
 .Python
 build/
@@ -128,22 +128,8 @@ dmypy.json
 # vscode config file
 .vscode/
 
+# pycharm project settings
+.idea/
+
 # no tracking mypy config file
-mypy.ini
-*.png
-recognition/s4641500/pirate/driver_script.ipynb
-recognition/ISICs_UNet/README.md
-recognition/XUE4645768/Readme.md
-recognition/s4641500/pirate/yes.py
-recognition/s4641500/pirate/train.py
-recognition/s4641500/pirate/predict.py
-recognition/s4641500/pirate/modules.py
-recognition/s4641500/pirate/dataset.py
-recognition/ISICs_UNet/README.md
-recognition/s4641500/keras_metadata.pb
-recognition/s4641500/saved model/model/keras_metadata.pb
-recognition/s4641500/saved model/model/saved_model.pb
-recognition/s4641500/saved model/model/variables/variables.data-00000-of-00001
-recognition/s4641500/pirate/VQVAE.py
-recognition/s4641500/pirate/pixelcnn.py
-recognition/ISICs_Unet/README.md
+mypy.ini

.gitmodules

@@ -0,0 +1,9 @@
+[submodule "recognition/46616780_YOLO_Hannah_Scholz/yolov5"]
+	path = recognition/46616780_YOLO_Hannah_Scholz/yolov5
+	url = https://github.com/hannahscholz/yolov5.git
+[submodule "recognition/46616780_YOLO_Hannah_Scholz/--force"]
+	path = recognition/46616780_YOLO_Hannah_Scholz/--force
+	url = https://github.com/hannahscholz/yolov5
+[submodule "recognition/46616780_YOLO_Hannah_Scholz/--yolov5"]
+	path = recognition/46616780_YOLO_Hannah_Scholz/--yolov5
+	url = https://github.com/hannahscholz/yolov5

recognition/45183735/README.md

@@ -0,0 +1,84 @@
+# Style-Based Generative Adversarial Networks (StyleGAN) - 45183735
+This StyleGAN model is implemented by using tensorflow for the OASIS brain dataset which provide a "reasonably clear image". 
+The structure of this model is from the original StyleGan paper. [here](https://arxiv.org/pdf/1812.04948.pdf)
+## Description of the Algorithm
+StyleGAN is one of the state-of-the-art extension for the Generative Adversarial Networks (GAN) which use for generating
+high-quality images. it also contains two model generator model and discriminator model, generator model will learn 
+to create a fake image cheat the discriminator, and the discriminator is a classifier to identify the real and fake 
+images. StyleGAN make an improvement for the generator model and does not modify the discriminator model.
+
+![styleGan generator structure](./readme_png/styleGAN_generator_structure.PNG)
+
+![AdaIN formula](./readme_png/AdaIN.png)
+
+### Generator model
+The generator model for StyleGAN include two part, Mapping network and Synthesis network,
+Compare to the traditional generator, Sytle-based generator map the Latent input z through the mapping network to an 
+intermediate latent space w instead of feeds directly into the input layer only. 
+and then feeds into the adaptive instance normalisation at each convolution layer in the synthesis network.
+the formula of the AdaIN is provide in above picture. the noise will add as a source of variation for each 
+section of the generator model.
+
+![styleGan discriminator structure](./readme_png/styleGAN_discriminator_structure.PNG)
+### Discriminator model
+
+As StyleGAN does not change anything about the discriminator, it still uses the discriminator model 
+in Progressive Growing of GAN (PGGAN). Therefore, the discriminator structure include two convolution layer
+with LeakReLU activation in each section of the network. the aim of discriminator is to distinguish the fake and real images.
+
+## Environment and Dataset Setup
+### Dependencies
+  - Tensorflow 2.10.0
+  - numpy 1.23.3
+  - matplotlib 3.5.2
+  - python 3.10.6
+
+### Dataset preparation
+The Preprocessed OASIS MRI Dataset need to be extracted to the same level as the ```train.py```.
+
+## Training / Predicting
+### Training the model
+
+Run this code to train the model:
+
+```python train.py```
+
+or if you want to choose the number of epochs, run this code:
+
+```python train.py -e EPOCHS```
+
+Note: Replace EPOCHS to the number you choose.
+
+The training step will generate an image and a plot of loss for each epoch. During the training, it will print the progress of the training.
+
+
+### Prediction using the trained model
+After the model training is finished, it will generate a checkpoint folder and do not remove it.
+the prediction will use the latest trained model and will generate an image for the prediction.
+To make a prediction, run this code:
+
+```python predict.py```
+
+## Result
+
+### Training
+
+![Training progress](./readme_png/training_progress.png)
+
+### Visualise the loss for each training epoch
+
+![Loss for each training epoch](./readme_png/Loss_output_epoch.png)
+
+### Visualise the loss for entire training step
+
+![entire loss for each epoch](./readme_png/Loss_Training_output.png)
+
+### Prediction output
+![Prediction using trained model](./readme_png/Prediction.png)
+
+## Reference
+[1] TensorFlow. 2022. Deep Convolutional Generative Adversarial Network  |  TensorFlow Core. [online] Available at: <https://www.tensorflow.org/tutorials/generative/dcgan> [Accessed 11 October 2022].
+
+[2] Karras, T., Laine, S. and Aila, T., 2019. A Style-Based Generator Architecture for Generative Adversarial Networks. [online] arXiv.org. Available at: <https://arxiv.org/abs/1812.04948> [Accessed 9 October 2022].
+
+[3] Karras, T., Aila, T., Laine, S. and Lehtinen, J., 2018. Progressive Growing of GANs for Improved Quality, Stability, and Variation. [online] arXiv.org. Available at: <https://arxiv.org/abs/1710.10196> [Accessed 10 October 2022].

recognition/45183735/dataset.py

@@ -0,0 +1,64 @@
+import tensorflow as tf
+import matplotlib.pyplot as plt
+import os
+
+
+# normalise the data between -1 to 1
+def normalise(data):
+    data = tf.cast(data/255., tf.float32)
+    return data
+
+
+class Dataset:
+
+    def __init__(self, ds_path, batch_size, image_size):
+        # path for the dataset folder
+        self.path = ds_path
+        self.batch_size = batch_size
+        # image must be squared e.g. image_size = 256 -> image is 256x256
+        self.image_size = image_size
+        # dataset for training
+        self.train_ds = self.get_train_ds()
+        # dataset for testing
+        self.test_ds = self.get_test_ds()
+        # dataset for validation
+        self.val_ds = self.get_val_ds()
+
+    # get the training dataset from the path
+    def get_train_ds(self):
+        train_ds = tf.keras.preprocessing.image_dataset_from_directory(
+            directory=self.path + '/keras_png_slices_train',
+            label_mode=None,
+            color_mode='grayscale',
+            batch_size=self.batch_size,
+            image_size=(self.image_size, self.image_size)
+        )
+
+        train_ds = train_ds.map(normalise).prefetch(buffer_size=tf.data.experimental.AUTOTUNE).cache()
+
+        return train_ds
+
+    # get the test dataset from the path
+    def get_test_ds(self):
+        test_ds = tf.keras.preprocessing.image_dataset_from_directory(
+            directory=self.path + '/keras_png_slices_test',
+            label_mode=None,
+            color_mode='grayscale',
+            batch_size=self.batch_size,
+            image_size=(self.image_size, self.image_size)
+        )
+        test_ds = test_ds.map(normalise).prefetch(buffer_size=tf.data.experimental.AUTOTUNE).cache()
+        return test_ds
+
+    # get the validation dataset from the path
+    def get_val_ds(self):
+        val_ds = tf.keras.preprocessing.image_dataset_from_directory(
+            directory=self.path + '/keras_png_slices_validate',
+            label_mode=None,
+            color_mode='grayscale',
+            batch_size=self.batch_size,
+            image_size=(self.image_size, self.image_size)
+        )
+        val_ds = val_ds.map(normalise).prefetch(buffer_size=tf.data.experimental.AUTOTUNE).cache()
+        return val_ds
+

recognition/45183735/modules.py

@@ -0,0 +1,180 @@
+import tensorflow as tf
+
+
+# Mapping Network
+class G_Mapping:
+
+    def __init__(self, latent_size):
+        self.latent_size = latent_size
+        # Mapping network for generator
+        self.model = self.mapping_nw()
+
+    # build the mapping network
+    def mapping_nw(self):
+        fc_input = tf.keras.layers.Input(shape=(self.latent_size,))
+        fc1 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc_input)
+        fc2 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc1)
+        fc3 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc2)
+        fc4 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc3)
+        fc5 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc4)
+        fc6 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc5)
+        fc7 = tf.keras.layers.Dense(self.latent_size, activation="relu")(fc6)
+        return tf.keras.Model(inputs=[fc_input], outputs=[fc7])
+
+# Synthesis network
+class G_Synthesis:
+    def __init__(self, latent_size, g_mapping, input_size):
+        # input image size
+        self.input_size = input_size
+        self.latent_size = latent_size
+        # Latent inputs
+        self.z = self.get_latent_inputs()
+        # Mapping network
+        self.g_mapping = g_mapping
+        # non_liner mapping network to map z -> w
+        self.w = self.get_w()
+        # Noises input
+        self.noises_input = self.get_noises_input()
+        # Noises
+        self.noises = self.get_noises()
+        # Synthesis network
+        self.nw = self.get_synthesis_nw()
+
+    def get_latent_inputs(self):
+        z = []
+        for i in range(7):
+            z.append(tf.keras.layers.Input(shape=(self.latent_size,)))
+        return z
+
+    def get_w(self):
+        w = []
+        for i in range(7):
+            w.append(self.g_mapping.model(self.z[i]))
+        return w
+
+    def get_noises_input(self):
+        noises_input = []
+        for i in range(7):
+            noises_input.append(tf.keras.layers.Input(shape=(4 * 2 ** i, 4 * 2 ** i, 1)))
+        return noises_input
+
+    def get_noises(self):
+        noises = []
+        for i in range(7):
+            noises.append(tf.keras.layers.Dense(32, activation="relu")(self.noises_input[i]))
+        return noises
+
+    # Adaptive instance normalisation
+    def get_AdaIN(self, x, ys, yb):
+        x_mean, x_std = tf.keras.backend.mean(x), tf.keras.backend.std(x)
+        ys = tf.reshape(ys, (-1, 1, 1, tf.shape(ys)[-1]))
+        yb = tf.reshape(yb, (-1, 1, 1, tf.shape(yb)[-1]))
+        return tf.add(tf.multiply(ys, tf.divide(x - x_mean, x_std + 1e-7)), yb)
+
+    def get_synthesis_nw(self):
+        layer = tf.keras.layers.Dense(4 * 4 * 32, activation="relu")(self.z[0])
+        layer = tf.keras.layers.Reshape((4, 4, 32))(layer)
+        noise_b = tf.keras.layers.Dense(32)(self.noises[0])
+        # add noise
+        layer = tf.keras.layers.Add()([layer, noise_b])
+        # add the style in AdaIN
+        layer = self.get_AdaIN(layer, tf.keras.layers.Dense(32)(self.w[0]), tf.keras.layers.Dense(32)(self.w[0]))
+        layer = tf.keras.layers.Conv2D(32, (3, 3), padding="same")(layer)
+        # add noise
+        layer = tf.keras.layers.Add()([layer, noise_b])
+        # add the style in AdaIN
+        layer = self.get_AdaIN(layer, tf.keras.layers.Dense(32)(self.w[0]), tf.keras.layers.Dense(32)(self.w[0]))
+
+        # for 8x8 to 256x256
+        for i in range(6):
+            layer = tf.keras.layers.UpSampling2D()(layer)
+            layer = tf.keras.layers.Conv2D(32, (3, 3), padding="same")(layer)
+            noise_b = tf.keras.layers.Dense(32)(self.noises[i + 1])
+            layer = tf.keras.layers.Add()([layer, noise_b])
+            layer = self.get_AdaIN(layer, tf.keras.layers.Dense(32)(self.w[i + 1]),
+                                   tf.keras.layers.Dense(32)(self.w[i + 1]))
+            layer = tf.keras.layers.Conv2D(32, (3, 3), padding="same")(layer)
+            layer = tf.keras.layers.Add()([layer, noise_b])
+            layer = self.get_AdaIN(layer, tf.keras.layers.Dense(32)(self.w[i + 1]),
+                                   tf.keras.layers.Dense(32)(self.w[i + 1]))
+
+        layer = tf.keras.layers.Dense(1)(layer)
+        layer = tf.keras.layers.Activation("sigmoid")(layer)
+        return layer
+
+
+# generator model
+class G_style:
+
+    def __init__(self, latent_size, input_size, g_synthesis):
+        self.input_size = input_size
+        self.latent_size = latent_size
+        self.g_synthesis = g_synthesis
+        self.model = self.generation_model()
+
+    def generation_model(self):
+        model = tf.keras.Model(inputs=self.g_synthesis.z + self.g_synthesis.noises_input, outputs=[self.g_synthesis.nw])
+        model.summary()
+        return model
+
+
+# as the styleGan does not modify discriminator in any way, so we using the discriminator structure of PGGan
+class Discriminator:
+
+    def __init__(self, input_size):
+        self.input_size = input_size
+        self.d_model = self.generate_discriminator_model()
+
+    def generate_discriminator_model(self):
+        D_model = tf.keras.models.Sequential()
+
+        # 256x256 -> 128x128
+        D_model.add(tf.keras.layers.Conv2D(16, (3, 3), strides=(1, 1), padding='same',
+                                           input_shape=[self.input_size, self.input_size, 1]))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(16, (3, 3), strides=(1, 1), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(32, (3, 3), strides=(2, 2), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Dropout(0.2))
+
+        # 128x128 -> 64x64
+        D_model.add(tf.keras.layers.Conv2D(32, (3, 3), strides=(1, 1), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(64, (3, 3), strides=(2, 2), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Dropout(0.3))
+
+        # 64x64 -> 32x32
+        D_model.add(tf.keras.layers.Conv2D(64, (3, 3), strides=(1, 1), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(128, (3, 3), strides=(2, 2), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Dropout(0.4))
+
+        # 32x32 -> 16x16
+        D_model.add(tf.keras.layers.Conv2D(128, (3, 3), strides=(1, 1), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(256, (3, 3), strides=(2, 2), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Dropout(0.4))
+
+        # 16x16 -> 8x8
+        D_model.add(tf.keras.layers.Conv2D(256, (3, 3), strides=(1, 1), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(512, (3, 3), strides=(2, 2), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Dropout(0.4))
+
+        # 8x8 -> 4x4
+        D_model.add(tf.keras.layers.Conv2D(512, (3, 3), strides=(1, 1), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Conv2D(512, (3, 3), strides=(2, 2), padding='same'))
+        D_model.add(tf.keras.layers.LeakyReLU())
+        D_model.add(tf.keras.layers.Dropout(0.4))
+
+        D_model.add(tf.keras.layers.Flatten())
+        D_model.add(tf.keras.layers.Dense(1, activation="sigmoid"))
+
+        return D_model
+