Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix concurrent map access by using a synchronized map #348

Merged
merged 1 commit into from
Sep 11, 2024

Conversation

stippi2
Copy link
Contributor

@stippi2 stippi2 commented Sep 3, 2024

Fixes #216

Copy'ing the analysis from the issue here:

  • There is a global Namespace informer. The generic_replicator uses it to register hook functions to be informed on namespace additions and updates.
  • The generic_replicator also uses its own informer to be notified of changes to other resources.
  • Our understanding is that the functions that the Informers call, are called in dedicated Go routines.
  • The same map is iterated from the NamespaceAdded hook as well as written from the ResourceAdded hook called by the other Informer.

We switched to using a sync.Map for ReplicateToList and ReplicateToMatchingList.
We did not change the DependencyMap, since it is only used in the resources informer hooks.

We also created a simple test to see whether there is any noticeable change to the performance:

  • Create a fake.NewClientset() K8s client
  • Create a test secret that is push replicated via a pattern
  • Create 1000 test namespaces
  • Measure the time until the replication is finished.

We did not see any change in the execution time. But of course this test may not be extensive, so any insights are appreciated.

@stippi2 stippi2 marked this pull request as ready for review September 3, 2024 14:48
@stippi2
Copy link
Contributor Author

stippi2 commented Sep 10, 2024

Could someone please have a look? It should be very easy to verify that this PR fixes the concurrent map access in the generic_replicator.

@stippi2
Copy link
Contributor Author

stippi2 commented Sep 11, 2024

@Lucaber Could you perhaps have a look? I saw you recently approved and merged a PR in this repo. Many thanks!

Copy link
Member

@martin-helmich martin-helmich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for my unresponsiveness. 🙏 Thanks for your work in debugging this and also providing a fix. The code looks good to me; will merge as soon as the pipeline passes. 👍

@martin-helmich martin-helmich enabled auto-merge (squash) September 11, 2024 08:25
@martin-helmich martin-helmich merged commit aa2d92c into mittwald:master Sep 11, 2024
3 checks passed
@stippi2 stippi2 deleted the fix-concurrent-map-access branch September 11, 2024 09:19
daniel-ciaglia added a commit to lynqtech/kubernetes-replicator that referenced this pull request Sep 16, 2024
* introduce generic synced map, use in generic_iterator (mittwald#348)
* Bump go to 1.22 (mittwald#343)

Co-authored-by: JorgeQ <[email protected]>
Co-authored-by: Martin Helmich <[email protected]>

* Bump chartVersion and appVersion to 'v2.10.2'

---------

Co-authored-by: Stephan Aßmus <[email protected]>
Co-authored-by: JorgeAndresQuintero <[email protected]>
Co-authored-by: JorgeQ <[email protected]>
Co-authored-by: Martin Helmich <[email protected]>
Co-authored-by: Mittwald Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

fatal error: concurrent map iteration and map write dealing with short-living namespaces
2 participants