Inconsistent output for code quality checks

[stephenbach]

See this for an example: bc0a8f1

make style says that it is reformatting promptsource/promptsource.py but there is no diff afterwards. Rerunning make style produces the same output and make quality fails, saying that it would be reformatted.

[tianjianjiang]

I will investigate it.
(IIRC I encountered a similar situation, but since the make style/quality commands are copied from HuggingFace so I figure that would be a low priority issue.)

[stephenbach]

Thanks! Some more info:

This is the diff it wants

black --diff --line-length 119 --target-version py38 promptsource
--- promptsource/promptsource.py	2021-05-31 15:07:06.548574 +0000
+++ promptsource/promptsource.py	2021-05-31 15:08:40.062640 +0000
@@ -1,11 +1,10 @@
 import datasets
 import requests
 import streamlit as st
 from session import _get_state
-from utils import (_ADDITIONAL_ENGLISH_DATSETS, removeHyphen,
-                   renameDatasetColumn)
+from utils import _ADDITIONAL_ENGLISH_DATSETS, removeHyphen, renameDatasetColumn
 
 from templates import Template, TemplateCollection

When I make that change manually, and then run make style it says 5 files left unchanged BUT it actually changes promptsource.py back to the 2-line import and we're back in the same state.

The only change I've made to your initial config @tianjianjiang is that I added --max-line-length 119 to flake8 because it seemed inconsistent with the black config.

[tianjianjiang]

Thanks! Some more info:

When I make that change manually, and then run make style it says 5 files left unchanged BUT it actually changes promptsource.py back to the 2-line import and we're back in the same state.

I suspect that's an unwelcome side effect from isort, I will look into isort's options.

The only change I've made to your initial config @tianjianjiang is that I added --max-line-length 119 to flake8 because it seemed inconsistent with the black config.

That is simplified from datasets and transformers but admittedly the latter has a set of more sophisticated formatting routines that could be more precise and consistent. I will also look into it.

[VictorSanh]

Yes, if i remember correctly, there is some inconsistency between black and isort import conventions -> PyCQA/isort#1518
Don't have a solution on top of my head, until we find it, let's ignore tests and merge @stephenbach

[arnaudstiegler]

If the problem is from a black/isort compatibility, I believe we can fix that using this:
https://pycqa.github.io/isort/docs/configuration/black_compatibility/

You define a config file for isort and have "black" for the profile

[VictorSanh]

@tianjianjiang after reading @arnaudstiegler's comment, you also need to import the setup.cfg file!
I fixed it in commit c1ec687 and the tests seem to be back to normal

[tianjianjiang]

@VictorSanh @arnaudstiegler Thank you guys and sorry for the late reaction.
BTW, I was thinking at least we can move isort to the top and introduce autoflake for following flake8 recommendations, but just something less urgent I suppose.