Multitenancy does not work in CAPOCI using instance principals.

[yimw]

What would you like to be added:
Support Multitenancy in CAPOCI

Why is this needed:
for the current CAPOCI we tested, we created multiple clusters on different Tenancy by using One management cluster:
~  k get clusters -A
NAMESPACE NAME PHASE AGE VERSION
default capioke Provisioned 37h (# This one on Tenancy A compartment A1)
default capoci Provisioned 13h (# This one on Tenancy A compartment A2)
default simplecluster Provisioned 13h (# This one on Tenancy B compartment B1)

And in the CAPOCI controller log, we found such error info:
~  k logs capoci-controller-manager-6574d4c58-nlxp6 -n cluster-api-provider-oci-system
failed to reconcile VCN for OCICluster default/capioke: Error returned by VirtualNetwork Service. Http Status Code: 404. Error Code: NotAuthorizedOrNotFound. Opc request id: 96b0b9de49c13bd31c62771134e90561/66A16B024E825D025C4E6B533054C281/B8ABA034DB613580462A833F052E3214. Message: Authorization failed or requested resource not found.

The failed reason I guess is the credentials we used for creating simplecluster which cannot be authorized to check other clusters.

And in our env, we don't have instance principals support, we could only leverage user principals.

[cheherna]

As per clarification

• In my personal opinion Resource Principals work better for this particular use case, why we don't use them instead of plain instance principals?
• How the error of not been authorized (as before) has to be handled? (Retry, terminate the operation, etc)

[shyamradhakrishnan]

@yimw , for the error you are getting, it seems like you have not provided proper permissions on the network-family resource. But we tested the same and we have found a bug in OCI Service Gateway due to which network creation using CAPOCI is not working. We will update here once that bug is fixed.
@cheherna Resource Principals are not externally visible feature which CAPOCI can use, atleast as per last information I have. When we get an auth error, we go into a exponential backoff and retry mechanism.