can`t reach the F1 80.4

[midnight2104]

hi , SeoSangwoo, when i run your code , the result of f1 is 81.56, but the paper of f1 is 84 , how about your socre finally?

[heslowen]

i run this code and the F1 result is only 68%?... i had not change any parameters

[ZHOUJessie]

F1 result is only 68%,but the paper of f1 is 84 , how about your socre finally?

[xxxxyan]

hi , SeoSangwoo, thank you very much for sharing the code firstly , and when i run the code ,i also encountered the problems mentioned above ,the result of f1 is not same as that in paper , i found that PI was not considered in the code,isn't it?

[Jerryten]

Does anyone get the best F1? Is it convenient to reveal the design of super parameters?

[xxxxyan]

Hello,I am sorry to just find this email and can't reply to you timely. I did not reproduce the code of the article either. I would like to ask you a question about the code, why did use tf.reduce_sum here to reduce dimensions?is it the beat way?I would appreciate it if you could answer my questions. output = tf.reduce_sum(inputs * tf.expand_dims(alphas, -1), 1)

…

------------------ 原始邮件 ------------------ 发件人: "Jerry"<[email protected]>; 发送时间: 2019年6月12日(星期三) 下午3:16 收件人: "SeoSangwoo/Attention-Based-BiLSTM-relation-extraction"<[email protected]>; 抄送: "肖燕"<[email protected]>;"Comment"<[email protected]>; 主题: Re: [SeoSangwoo/Attention-Based-BiLSTM-relation-extraction] can`treach the F1 80.4 (#14) Does anyone get the best F1? Is it convenient to reveal the design of super parameters? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[wang-h]

<<< (9+1)-WAY EVALUATION TAKING DIRECTIONALITY INTO ACCOUNT -- OFFICIAL >>>:

Confusion matrix:
C-E C-W C-C E-D E-O I-A M-C M-T P-P O <-- classified as
+--------------------------------------------------+ -SUM- xDIRx skip ACTUAL
C-E | 295 1 0 0 6 1 0 2 2 17 | 324 4 0 328
C-W | 1 245 3 0 0 6 10 5 4 28 | 302 10 0 312
C-C | 0 3 159 8 1 0 2 1 0 17 | 191 1 0 192
E-D | 0 2 5 265 0 0 0 0 0 20 | 292 0 0 292
E-O | 5 2 1 2 225 0 0 1 1 20 | 257 1 0 258
I-A | 3 7 0 2 1 91 0 1 11 39 | 155 1 0 156
M-C | 0 10 1 1 2 0 195 0 2 18 | 229 4 0 233
M-T | 2 5 0 1 1 0 1 217 0 29 | 256 5 0 261
P-P | 4 3 0 0 6 6 1 5 173 31 | 229 2 0 231
O | 22 38 17 31 24 16 33 27 28 218 | 454 0 0 454
+--------------------------------------------------+
-SUM- 332 316 186 310 266 120 242 259 221 437 2689 28 0 2717

Coverage = 2717/2717 = 100.00%
Accuracy (calculated for the above confusion matrix) = 2083/2717 = 76.67%
Accuracy (considering all skipped examples as Wrong) = 2083/2717 = 76.67%
Accuracy (considering all skipped examples as Other) = 2083/2717 = 76.67%

Results for the individual relations:
Cause-Effect : P = 295/( 332 + 4) = 87.80% R = 295/ 328 = 89.94% F1 = 88.86%
Component-Whole : P = 245/( 316 + 10) = 75.15% R = 245/ 312 = 78.53% F1 = 76.80%
Content-Container : P = 159/( 186 + 1) = 85.03% R = 159/ 192 = 82.81% F1 = 83.91%
Entity-Destination : P = 265/( 310 + 0) = 85.48% R = 265/ 292 = 90.75% F1 = 88.04%
Entity-Origin : P = 225/( 266 + 1) = 84.27% R = 225/ 258 = 87.21% F1 = 85.71%
Instrument-Agency : P = 91/( 120 + 1) = 75.21% R = 91/ 156 = 58.33% F1 = 65.70%
Member-Collection : P = 195/( 242 + 4) = 79.27% R = 195/ 233 = 83.69% F1 = 81.42%
Message-Topic : P = 217/( 259 + 5) = 82.20% R = 217/ 261 = 83.14% F1 = 82.67%
Product-Producer : P = 173/( 221 + 2) = 77.58% R = 173/ 231 = 74.89% F1 = 76.21%
_Other : P = 218/( 437 + 0) = 49.89% R = 218/ 454 = 48.02% F1 = 48.93%

Micro-averaged result (excluding Other):
P = 1865/2280 = 81.80% R = 1865/2263 = 82.41% F1 = 82.10%

MACRO-averaged result (excluding Other):
P = 81.33% R = 81.03% F1 = 81.04%

<<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 81.04% >>> using the default hyper-parameters and small batch size=10, Glove 6d.100.txt

[heslowen]

By using BiLSTM+attention+pytorch ，I got the best marco-F1(exclude relation 'other') 79.2% . I guess that some tricks may lead to a worse result compare to the F1 reported from the paper. hao <[email protected]> 於 2019年8月27日週二 下午4:28寫道：

…

I also cannot reach 80%, my best result using Pytorch is only 71.3% (BiLSTM + ATTN), 69.9% (BiLSTM), is the result reported in the paper correct? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#14?email_source=notifications&email_token=AFKQGUIKHYL47EVZLAEXWKLQGTQS5A5CNFSM4GKSMIL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5G6FAY#issuecomment-525197955>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFKQGULNFNRQZIV5VZZKTS3QGTQS5ANCNFSM4GKSMILQ> .

[zhijing-jin]

For everyone's inference,

I ran the code on Sept 9, 2019, did not change anything, and obtained macro-averaged F1 = 81.37%.

<<< The official score is (9+1)-way evaluation with directionality taken into account: macro-averaged F1 = 81.37% >>>

My environment is

• Python: 3.6
• TensorFlow: 1.14.0

[Just-silent]

通过使用BiLSTM + attention + pytorch，我获得了79.2％的最佳marco-F1（排除“其他”关系）。我猜想，与本文报道的F1相比，一些技巧可能会导致更糟糕的结果。hao [email protected]于2019年8月27日周二下午4:28时间：
…
我也无法达到80％，使用Pytorch的最佳结果仅为71.3％（BiLSTM + ATTN），69.9％（BiLSTM），论文中报告的结果正确吗？—您收到此评论是因为您发表了评论。回复此电子邮件直接，查看它在GitHub < ＃14？email_source =通知＆email_token = AFKQGUIKHYL47EVZLAEXWKLQGTQS5A5CNFSM4GKSMIL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5G6FAY＃issuecomment-525197955>，或静音螺纹< https://github.com/notifications/unsubscribe-auth/AFKQGULNFNRQZIV5VZZKTS3QGTQS5ANCNFSM4GKSMILQ >。

hello,i never used the tensorflow ,can you share your code with me.Rencently,i copy one code from other,which's p can only 62% , i can't find some questions ,so i want study your code by pytorch