Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] Invalidate org membership cache when organization is_active flag is changed #357 #379

Conversation

kaushikaryan04
Copy link
Contributor

@kaushikaryan04 kaushikaryan04 commented May 23, 2024

When the status ( is_active ) of an organization changes it now invalidates the cache of all the organizationsusers.

Fixes #357.

When the status ( is_active ) of an organization changes it now invalidates the cache of all the organizationsusers.
Copy link
Member

@nemesifier nemesifier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kaushikaryan04 for contributing!

The code is not formatted according to our standards and the build will surely fail the QA checks. Please read here how to run QA checks locally and how to reformat the code with black: https://openwisp.io/docs/developer/contributing.html#python-code-conventions.

We need to add an automated test for this which asserts that changing the is_active field of an organization invalidates the cache. Look in the test suite for test code doing similar assertion for the invalidation of the user cache when user details are changed.

@@ -98,11 +98,18 @@ def set_default_settings(self):
def connect_receivers(self):
OrganizationUser = load_model('openwisp_users', 'OrganizationUser')
OrganizationOwner = load_model('openwisp_users', 'OrganizationOwner')
Organization = load_model("openwisp_users" , "Organization")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use single quotes ('), please maintain consistency, applies to all other changes.

return
if getattr(old_obj , "is_active") != getattr(instance , "is_active") :
for user in OrganizationUsers.objects.filter(organization = instance):
cls._invalidate_user_cache(user)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the database is big enough, this task can take enough time to cause the webserver to time out.
We need to move this logic to a background celery task.
We have a number of examples in other modules, eg: https://github.com/openwisp/openwisp-controller/blob/master/openwisp_controller/config/tasks.py#L75-L80
Ideally the celery task calls a class method in the model which takes care of the rest.

logger.exception(f"An error occurred while fetching the organization: {e}")
return
if getattr(old_obj , "is_active") != getattr(instance , "is_active") :
for user in OrganizationUsers.objects.filter(organization = instance):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We must not load the entire set in memory as it can crash big installations, to do that we use the iterator() method of the Django ORM, eg:

Suggested change
for user in OrganizationUsers.objects.filter(organization = instance):
for user in OrganizationUsers.objects.filter(organization = instance).iterator():

except ObjectDoesNotExist:
logger.warning(f"Organization with pk {instance.pk} does not exist.")
return
except Exception as e:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this block can be removed

OrganizationUsers = load_model("openwisp_users" ,"OrganizationUser")
try:
old_obj = Organization.objects.get(pk=instance.pk)
except ObjectDoesNotExist:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
except ObjectDoesNotExist:
except Organization.DoesNotExist:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey sorry for all that inconsistency should have read the guidelines.
I make these changes and make another commit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, ping me in the dev chat when it's ready for review again.

@kaushikaryan04
Copy link
Contributor Author

I added the suggested changes but the some tests are failing as the no of running queries is one more than expected

FAIL: test_organization_slug_post_custom_validation_api (openwisp_users.tests.test_api.test_api.TestUsersApi) [Expecting 6 SQL queries]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/aryankaushik/Desktop/openwisp_users_main/env/lib/python3.9/site-packages/openwisp_utils/tests.py", line 148, in __exit__
    self.test_case.assertEqual(
AssertionError: 7 != 6 : 7 queries executed, 6 expected
Captured queries were:
1. SELECT "openwisp_users_user"."password", "openwisp_users_user"."last_login", "openwisp_users_user"."is_superuser", "openwisp_users_user"."username", "openwisp_users_user"."first_name", "openwisp_users_user"."last_name", "openwisp_users_user"."is_staff", "openwisp_users_user"."is_active", "openwisp_users_user"."date_joined", "openwisp_users_user"."id", "openwisp_users_user"."email", "openwisp_users_user"."bio", "openwisp_users_user"."url", "openwisp_users_user"."company", "openwisp_users_user"."location", "openwisp_users_user"."phone_number", "openwisp_users_user"."birth_date", "openwisp_users_user"."notes", "openwisp_users_user"."language", "openwisp_users_user"."password_updated" FROM "openwisp_users_user" WHERE "openwisp_users_user"."id" = 'dc2f49d54db24f2cb4e09102804fe994' LIMIT 21
2. SELECT "openwisp_users_organization"."name", "openwisp_users_organization"."is_active", "openwisp_users_organization"."created", "openwisp_users_organization"."modified", "openwisp_users_organization"."slug", "openwisp_users_organization"."id", "openwisp_users_organization"."description", "openwisp_users_organization"."email", "openwisp_users_organization"."url" FROM "openwisp_users_organization" WHERE "openwisp_users_organization"."slug" = 'test-org' ORDER BY "openwisp_users_organization"."name" ASC LIMIT 1
3. SELECT 1 AS "a" FROM "openwisp_users_organization" WHERE "openwisp_users_organization"."slug" = 'test-org' LIMIT 1
4. SELECT 1 AS "a" FROM "openwisp_users_organization" WHERE "openwisp_users_organization"."id" = '6854249477df4b6dbd7fb2fd49cd5d94' LIMIT 1
5. SELECT "openwisp_users_organization"."name", "openwisp_users_organization"."is_active", "openwisp_users_organization"."created", "openwisp_users_organization"."modified", "openwisp_users_organization"."slug", "openwisp_users_organization"."id", "openwisp_users_organization"."description", "openwisp_users_organization"."email", "openwisp_users_organization"."url" FROM "openwisp_users_organization" WHERE "openwisp_users_organization"."id" = '16302821e4cc46d4814f1fdc679c4059' LIMIT 21
6. SELECT "openwisp_users_organization"."name", "openwisp_users_organization"."is_active", "openwisp_users_organization"."created", "openwisp_users_organization"."modified", "openwisp_users_organization"."slug", "openwisp_users_organization"."id", "openwisp_users_organization"."description", "openwisp_users_organization"."email", "openwisp_users_organization"."url" FROM "openwisp_users_organization" WHERE (NOT ("openwisp_users_organization"."id" = '16302821e4cc46d4814f1fdc679c4059') AND "openwisp_users_organization"."slug" = 'test-org') ORDER BY "openwisp_users_organization"."name" ASC
7. INSERT INTO "openwisp_users_organization" ("name", "is_active", "created", "modified", "slug", "id", "description", "email", "url") VALUES ('test-org', 1, '2024-05-27 17:41:05.471960', '2024-05-27 17:41:05.472193', 'test-org', '16302821e4cc46d4814f1fdc679c4059', '', '', '')

======================================================================
FAIL: test_remove_organization_owner_api (openwisp_users.tests.test_api.test_api.TestUsersApi) [Expecting 11 SQL queries]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/aryankaushik/Desktop/openwisp_users_main/env/lib/python3.9/site-packages/openwisp_utils/tests.py", line 148, in __exit__
    self.test_case.assertEqual(
AssertionError: 12 != 11 : 12 queries executed, 11 expected
Captured queries were:
1. SELECT "openwisp_users_user"."password", "openwisp_users_user"."last_login", "openwisp_users_user"."is_superuser", "openwisp_users_user"."username", "openwisp_users_user"."first_name", "openwisp_users_user"."last_name", "openwisp_users_user"."is_staff", "openwisp_users_user"."is_active", "openwisp_users_user"."date_joined", "openwisp_users_user"."id", "openwisp_users_user"."email", "openwisp_users_user"."bio", "openwisp_users_user"."url", "openwisp_users_user"."company", "openwisp_users_user"."location", "openwisp_users_user"."phone_number", "openwisp_users_user"."birth_date", "openwisp_users_user"."notes", "openwisp_users_user"."language", "openwisp_users_user"."password_updated" FROM "openwisp_users_user" WHERE "openwisp_users_user"."id" = '7d5f9551f23d457b96c0a5dfc0d32cde' LIMIT 21
2. SELECT "openwisp_users_organization"."name", "openwisp_users_organization"."is_active", "openwisp_users_organization"."created", "openwisp_users_organization"."modified", "openwisp_users_organization"."slug", "openwisp_users_organization"."id", "openwisp_users_organization"."description", "openwisp_users_organization"."email", "openwisp_users_organization"."url" FROM "openwisp_users_organization" WHERE "openwisp_users_organization"."id" = '1c1754ecbcf44722ac045f5141f33348' LIMIT 21
3. SELECT 1 AS "a" FROM "openwisp_users_organizationowner" WHERE "openwisp_users_organizationowner"."organization_id" = '1c1754ecbcf44722ac045f5141f33348' LIMIT 1
4. SELECT 1 AS "a" FROM "openwisp_users_organizationowner" WHERE "openwisp_users_organizationowner"."organization_id" = '1c1754ecbcf44722ac045f5141f33348' LIMIT 1
5. SELECT "openwisp_users_organizationowner"."created", "openwisp_users_organizationowner"."modified", "openwisp_users_organizationowner"."id", "openwisp_users_organizationowner"."organization_user_id", "openwisp_users_organizationowner"."organization_id" FROM "openwisp_users_organizationowner" WHERE "openwisp_users_organizationowner"."organization_id" = '1c1754ecbcf44722ac045f5141f33348' ORDER BY "openwisp_users_organizationowner"."id" ASC LIMIT 1
6. DELETE FROM "openwisp_users_organizationowner" WHERE "openwisp_users_organizationowner"."id" IN ('5ab96a836f684456aeb2883004219aa9')
7. SELECT "openwisp_users_organizationuser"."created", "openwisp_users_organizationuser"."modified", "openwisp_users_organizationuser"."is_admin", "openwisp_users_organizationuser"."id", "openwisp_users_organizationuser"."user_id", "openwisp_users_organizationuser"."organization_id" FROM "openwisp_users_organizationuser" WHERE "openwisp_users_organizationuser"."id" = 'af21273fe11346a69838f9b0ad14c77a' LIMIT 21
8. SELECT "openwisp_users_user"."password", "openwisp_users_user"."last_login", "openwisp_users_user"."is_superuser", "openwisp_users_user"."username", "openwisp_users_user"."first_name", "openwisp_users_user"."last_name", "openwisp_users_user"."is_staff", "openwisp_users_user"."is_active", "openwisp_users_user"."date_joined", "openwisp_users_user"."id", "openwisp_users_user"."email", "openwisp_users_user"."bio", "openwisp_users_user"."url", "openwisp_users_user"."company", "openwisp_users_user"."location", "openwisp_users_user"."phone_number", "openwisp_users_user"."birth_date", "openwisp_users_user"."notes", "openwisp_users_user"."language", "openwisp_users_user"."password_updated" FROM "openwisp_users_user" WHERE "openwisp_users_user"."id" = '243282a1f4c34e098c1956d90fb75b86' LIMIT 21
9. SELECT "openwisp_users_organizationuser"."created", "openwisp_users_organizationuser"."modified", "openwisp_users_organizationuser"."is_admin", "openwisp_users_organizationuser"."id", "openwisp_users_organizationuser"."user_id", "openwisp_users_organizationuser"."organization_id", "openwisp_users_organization"."name", "openwisp_users_organization"."is_active", "openwisp_users_organization"."created", "openwisp_users_organization"."modified", "openwisp_users_organization"."slug", "openwisp_users_organization"."id", "openwisp_users_organization"."description", "openwisp_users_organization"."email", "openwisp_users_organization"."url", "openwisp_users_organizationowner"."created", "openwisp_users_organizationowner"."modified", "openwisp_users_organizationowner"."id", "openwisp_users_organizationowner"."organization_user_id", "openwisp_users_organizationowner"."organization_id" FROM "openwisp_users_organizationuser" INNER JOIN "openwisp_users_organization" ON ("openwisp_users_organizationuser"."organization_id" = "openwisp_users_organization"."id") LEFT OUTER JOIN "openwisp_users_organizationowner" ON ("openwisp_users_organizationuser"."id" = "openwisp_users_organizationowner"."organization_user_id") WHERE ("openwisp_users_organization"."is_active" AND "openwisp_users_organizationuser"."user_id" = '243282a1f4c34e098c1956d90fb75b86') ORDER BY "openwisp_users_organization"."name" ASC, "openwisp_users_organizationuser"."user_id" ASC
10. SELECT "openwisp_users_organization"."name", "openwisp_users_organization"."is_active", "openwisp_users_organization"."created", "openwisp_users_organization"."modified", "openwisp_users_organization"."slug", "openwisp_users_organization"."id", "openwisp_users_organization"."description", "openwisp_users_organization"."email", "openwisp_users_organization"."url" FROM "openwisp_users_organization" WHERE "openwisp_users_organization"."id" = '1c1754ecbcf44722ac045f5141f33348' LIMIT 21
11. UPDATE "openwisp_users_organization" SET "name" = 'org1', "is_active" = 1, "created" = '2024-05-27 17:41:08.827987', "modified" = '2024-05-27 17:41:08.974391', "slug" = 'org1', "description" = '', "email" = '', "url" = '' WHERE "openwisp_users_organization"."id" = '1c1754ecbcf44722ac045f5141f33348'
12. SELECT "openwisp_users_organizationowner"."created", "openwisp_users_organizationowner"."modified", "openwisp_users_organizationowner"."id", "openwisp_users_organizationowner"."organization_user_id", "openwisp_users_organizationowner"."organization_id" FROM "openwisp_users_organizationowner" WHERE "openwisp_users_organizationowner"."organization_id" = '1c1754ecbcf44722ac045f5141f33348' LIMIT 21

And one testcase which is failing is this. I am working on this it is failing because of the line which is invalidating cache in tasks.py which I am working on but any suggestion are appreciated.

======================================================================
FAIL: test_can_change_inline_org_owner (openwisp_users.tests.test_admin.TestUsersAdmin) [owner can edit inline org owner]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/aryankaushik/Desktop/openwisp_users_main/openwisp-users/openwisp_users/tests/test_admin.py", line 1423, in test_can_change_inline_org_owner
    self.assertEqual(org_owners.count(), 1)
AssertionError: 0 != 1

Copy link
Member

@nemesifier nemesifier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for following up @kaushikaryan04, you can increase the query count.
Please follow our commit message guidelines or the QA checks for the CI build will fail (please see the CI build results below).
I will need more time to look into the outstanding issue.

@kaushikaryan04
Copy link
Contributor Author

Okay I will change the query count and will follow the commit message guidelines.
About the Issue I have tried a lot of things. If you find something out let me know.

Copy link
Member

@nemesifier nemesifier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I will change the query count and will follow the commit message guidelines. About the Issue I have tried a lot of things. If you find something out let me know.

Please resolve the issues I pointed out, I will look into to help on any outstanding issue once those are sorted out.

I made a pre_save signal that calls a celery task that would invalidate cache of all users in the organization if the status is changed.
But it is failing a test case which i have mentioned.

Fixes openwisp#357
@kaushikaryan04
Copy link
Contributor Author

I have made a commit with those issues solved

Copy link
Member

@nemesifier nemesifier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that the test which is failing is slightly flawed, it seems to me the organization is disabled when it executes. I am still investigating. But your patch looks good, I left some more comments to keep improving it. I will keep you informed about my investigation on the failing test.

openwisp_users/apps.py Show resolved Hide resolved
@classmethod
def handle_organization_update(cls, instance, **kwargs):
try:
old_instance = instance._meta.model.objects.get(pk=instance.pk)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
old_instance = instance._meta.model.objects.get(pk=instance.pk)
old_instance = Organization.objects.only('is_active').get(pk=instance.pk)

Calling only should be more efficient.

def handle_organization_update(cls, instance, **kwargs):
try:
old_instance = instance._meta.model.objects.get(pk=instance.pk)
except instance._meta.model.DoesNotExist:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
except instance._meta.model.DoesNotExist:
except Organization.DoesNotExist:



@shared_task
def organization_update_task(organization_pk):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's rename this to invalidate_org_membership_cache

@shared_task
def organization_update_task(organization_pk):
"""
Invalidates cache of users when organization become inactive
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Invalidates cache of users when organization become inactive
Invalidates organization membership cache of all users of an
organization when organization.is_active changes
(organization is disabled or enabled again).

"""
Invalidates cache of users when organization become inactive
"""
OrganizationUsers = load_model('openwisp_users', 'OrganizationUser')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can place this on the top of the file, after User.

Refactored the code, changed name of a function that describes it better and made a query more efficient.
@kaushikaryan04
Copy link
Contributor Author

Made the changes suggested by you. Thanks for guiding line by line. I will try to understand more about this project meanwhile.

Copy link
Member

@nemesifier nemesifier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for following up and welcome @kaushikaryan04.

Upon inspection again with fresh eyes today I found a few more details that can be improved/simplified. See below.

"""
OrganizationUsers = load_model('openwisp_users', 'OrganizationUser')
qs = OrganizationUsers.objects.filter(
qs = OrganizationUser.objects.filter(
organization_id=organization_pk
).select_related('user')
User = get_user_model()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is redundant, User is already imported at the top of the file.

pre_save.connect(
self.handle_organization_update,
sender=Organization,
dispatch_uid='handle_organization_is_active_change',
Copy link
Member

@nemesifier nemesifier May 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dispatch_uid='handle_organization_is_active_change',
dispatch_uid='handle_org_is_active_change',

Comment on lines 102 to 104
if not isinstance(user, User):
user = user.user
user._invalidate_user_organizations_dict()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is all this really necessary? Why do you feel the need of checking if user.user is an instance of User? Could it be anything different? Let me know.

I think this could be simplified as following:

Suggested change
if not isinstance(user, User):
user = user.user
user._invalidate_user_organizations_dict()
user._invalidate_user_organizations_dict()

If I am missing something let me know.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the if condition can be removed but I cannot do this

user._invalidate_user_organizations_dict()

This will cause error as organization user does not have this method.
So we can do this

user.user._invalidate_user_organizations_dict()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the variable name user is misleading here, this is an OrganizationUser instance, so let's call it org_user.
Then you can call: org_user.user._invalidate_user_organizations_dict().

openwisp_users/apps.py Show resolved Hide resolved
@@ -130,6 +137,18 @@ def connect_receivers(self):
dispatch_uid='make_first_org_user_org_owner',
)

@classmethod
def handle_organization_update(cls, instance, **kwargs):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's rename this for consistency: handle_org_is_active_change

organization_id=organization_pk
).select_related('user')
User = get_user_model()
for user in qs.iterator():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
for user in qs.iterator():
for org_user in qs.iterator():

Comment on lines 102 to 104
if not isinstance(user, User):
user = user.user
user._invalidate_user_organizations_dict()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the variable name user is misleading here, this is an OrganizationUser instance, so let's call it org_user.
Then you can call: org_user.user._invalidate_user_organizations_dict().

@nemesifier
Copy link
Member

nemesifier commented May 31, 2024

@kaushikaryan04 come in the OpenWISP dev chat to coordinate when you can 🙏

Added a condition that reduced query count in 2 testcases and imporved naming of variables and funtions.
@kaushikaryan04 kaushikaryan04 force-pushed the issues/357-inactive-organizations-users-visible branch from 834602a to 2020fea Compare June 1, 2024 11:26
@nemesifier nemesifier added the bug label Jun 1, 2024
@nemesifier nemesifier changed the title [openwisp_users/apps.py] fixes issue #357 [fix] Invalidate org membership cache when organization is_active flag is changed #357 Jun 5, 2024
Added is_active parameter in post request to
avoid disabling org.
@coveralls
Copy link

Coverage Status

coverage: 97.84% (-0.09%) from 97.929%
when pulling 63544b4 on kaushikaryan04:issues/357-inactive-organizations-users-visible
into e85cf62 on openwisp:master.

Copy link
Member

@nemesifier nemesifier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kaushikaryan04 👍 👏

@nemesifier nemesifier merged commit 57d542b into openwisp:master Jun 5, 2024
20 checks passed
@kaushikaryan04 kaushikaryan04 deleted the issues/357-inactive-organizations-users-visible branch June 12, 2024 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

Successfully merging this pull request may close these issues.

[bug] Objects from inactive organizations are visible to the user
3 participants