Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PerformanceWarning instances raised on population dataframe creation #1174

Closed
matt-graham opened this issue Nov 6, 2023 · 0 comments · Fixed by #1186
Closed

PerformanceWarning instances raised on population dataframe creation #1174

matt-graham opened this issue Nov 6, 2023 · 0 comments · Fixed by #1186
Assignees
Labels
warning Warnings generated by code that should be addressed

Comments

@matt-graham
Copy link
Collaborator

Since the update to pandas v2 we get a series of warnings

PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`

from the lines

props = pd.DataFrame(index=pd.RangeIndex(stop=size, name="person"))
for module in self.sim.modules.values():
for prop_name, prop in module.PROPERTIES.items():
props[prop_name] = prop.create_series(prop_name, size)

I'm not sure, but I think the performance issue mentioned is with respect to the cost of creating a dataframe with lots of columns by inserting one by one rather than adding altogether, rather than having an implication for the subsequent cost of accessing data in that dataframe (which would be much more of an issue in our case), but it would still be worth fixing this to avoid the warnings.

Originally posted by @matt-graham in #1159 (comment)

@matt-graham matt-graham self-assigned this Nov 7, 2023
@matt-graham matt-graham added the warning Warnings generated by code that should be addressed label Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
warning Warnings generated by code that should be addressed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant