Protein function prediction with GO - Part 3

[aditya0by0]

• PR for the Issue Protein function prediction with GO #36

Note: The above issue will be implemented in 3 PRs:

• Protein function prediction with GO #39 (Merged)
• Protein function prediction with GO - Part 2 #57 (Merged)
• Protein function prediction with GO - Part 3 #64

Changes to be done in this PR

evaluation: Evaluate using the same metrics as DeepGO for comparing the models

From comment #36 (comment)

• on a new branch: metrics for evaluation (I talked to Martin about the Fmax score: Although it has some methodological issues, we should include it in our evaluation to do a comparison with DeepGO)
• DeepGO-SE (paper): use these results as a baseline, integrate their data into our pipeline (there is a link to the dataset on their github page

[aditya0by0]

I have made the suggested changes for migration. Please check.

Config for DeepGO1:

class_path: chebai.preprocessing.datasets.go_uniprot.DeepGO1MigratedData
init_args:
  go_branch: "MF"
  max_sequence_length: 1002
  reader_kwargs: {n_gram: 3}

Config for DeepGO2:

class_path: chebai.preprocessing.datasets.go_uniprot.DeepGO2MigratedData
init_args:
  go_branch: "MF"
  max_sequence_length: 1000
  reader_kwargs: {n_gram: 3}

[sfluegel05]

I ran the migration script for DeepGO2 and tried training a model with the data. I noticed three issues:

• The max_sequence_length parameter seems to have no effect (I set it to 1,000, but the processed dataset contains feature-entries that have a length of more than 4,000)
• The labels are empty lists in the processed data (although a classes_deep_go2.txt file does get generated)
• The dataset contains "invalid" amino acids (O and U for pyrrolysine and selenocysteine) - as you pointed out, DeepGO does not include them as valid, but as far as I can tell, that is not enforced. Either way, currently, when processing the migrated data, chebai raises an error and stops when reaching either an O or U in the dataset. This is not very useful. We should either raise a warning and skip the protein or include O and U as valid (I would suggest to do both, i.e., adding O and U as safe and raising a warning for other letters that are definitely invalid).

@aditya0by0 Can you have a look at that?

Edit: I used the following commands:
Migration:

python chebai/preprocessing/migration/deep_go/migrate_deep_go_2_data.py migrate --data_dir=data/deepgo2-train-data --go_branch=mf

Run:

python3 -m chebai fit --trainer=configs/training/default_trainer.yml --trainer.min_epochs=10 --trainer.max_epochs=10 --model=configs/model/electra.yml --model.train_metrics=configs/metrics/micro-macro-f1.yml --model.test_metrics=configs/metrics/micro-macro-f1.yml --model.val_metrics=configs/metrics/micro-macro-f1.yml --data=configs/data/deepGO2.yml --trainer.logger=configs/training/csv_logger.yml --model.out_dim=898 --model.criterion=configs/loss/bce.yml --data.init_args.batch_size=10 --data.init_args.num_workers=10 --model.pass_loss_kwargs=false --trainer.logger.init_args.name=DeepGO2_MF --data.init_args.splits_file_path=data/GO_UniProt/GO_MF_1000/processed/splits_deep_go2.csv

[aditya0by0]

• DeepGO does not include them as valid, but as far as I can tell, that is not enforced.

• We should either raise a warning and skip the protein or include O and U as valid (I would suggest to do both, i.e., adding O and U as safe and raising a warning for other letters that are definitely invalid).

I agree that invalid amino acids like "O" and "U" are not explicitly enforced as invalid by DeepGO2, but they are also not explicitly treated as valid in their data pipeline.

To elaborate, DeepGO2 includes the amino acid notation "X" in its set of valid amino acids, where "X" represents any/unknown amino acids (as per [Wikipedia].

In their pipeline, when invalid amino acids such as "U", "O", "B", "Z", "J", or "*" are encountered in the protein sequences, they are effectively mapped to "X". This is evident in their implementation, where the index of "X" is used for any amino acid that doesn't belong to the valid set.

You can see this behavior in their code here:
[DeepGO2 Amino Acid relevant code].

So, while "O" and "U" are not explicitly handled, the use of "X" as a catch-all ensures that any invalid amino acids are safely represented.

If we want to follow the approach mentioned above, we would need to replace every invalid amino acid in the sequence with "X" as part of a pre-processing step before tokenization. This to avoid inconsistencies in the n-gram tokenization process.

Please let me know how we want to proceed with it.

[sfluegel05]

Ok, so if DeepGO2 replaces every not explicitly valid amino acid with X, then we should do the same. I think that is the easiest solution.

[aditya0by0]

I have done the suggested changes (1b8b270 and 85c47a0). Please review.

[aditya0by0]

Commit which added ESM2 embedding in migration and training process : e7b3d80

This commit has changes which also includes the ESM2 embeddings from in deepgo2 migration process.
Please run chebai/preprocessing/migration/deep_go/migrate_deep_go_2_data.py (migration script) before starting training process.

@sfluegel05, Please find the config to train model on DeepGO 2 migrated data with ESM2 embeddings.

class_path: chebai.preprocessing.datasets.deepGO.go_uniprot.DeepGO2MigratedData
init_args:
  go_branch: "MF"
  max_sequence_length: 1000
  use_esm2_embeddings: True

[sfluegel05]

I added the ESM2 config as well as a simple feed forward network.

ESM2 with Electra does not work out of the box, since ESM2 uses real values that may be negative. Electra expects positive values only. Also, I'm not sure how sensible using Electra here would be - Electra expects a sequence, but ESM2 provides an embedding vector that is not directly related to the sequence (tell me if I'm wrong here).

[aditya0by0]

I added the ESM2 config as well as a simple feed forward network.

ESM2 with Electra does not work out of the box, since ESM2 uses real values that may be negative. Electra expects positive values only. Also, I'm not sure how sensible using Electra here would be - Electra expects a sequence, but ESM2 provides an embedding vector that is not directly related to the sequence (tell me if I'm wrong here).

Yes, you're absolutely right about the compatibility issues between ESM2 and Electra. ESM2 embeddings can have negative values as they are activation values from specific layer within ESM2 network.
As, Electra expects positive values and there is no direct token-level alignment in ESM2 embeddings, Electra (specifically its discriminator) might not work well with it.

We can perform ReLU or normalization as pre-processing to ESM2 embeddings to shift them into positive domain, before feeding it to Electra, but I doubt whether it will still work due to previous stated reasons.
Using just Feed Forward Network instead of Electra seems like a more suitable approach.