more norms

[maxaalexeeva]

• addressing some of the issues in Normalization issues #642
• adding relevant tests
• allowing a negative check in numerical entity tests (e.g., to make sure we did not extract 108.0 in from Sahel 108 in Senegal)

[maxaalexeeva]

@kwalcock would you please give me the power to check what jenkins is up to?

[kwalcock]

LGTM but I wish that meeting the needs of habitus wasn't requiring changes to processors. It should be possible to adapt locally. That's not your problem, though.

[kwalcock]

This isn't necessary, but one way to do it without the buffer is

    for {
      m <- mentions
      converted <- Try(converter(m))
          .recover { case throwable: Throwable =>
            System.err.println(s"WARNING: $converterName conversion failed! Recovering and continuing...")
            throwable.printStackTrace()
            Seq.empty[Mention]
          }
          .get
    } yield converted

[maxaalexeeva]

Makes sense. With processors I basically go the path of fewest changes :)

[kwalcock]

Probably all of processors needs to be reviewed for its use of println, System.out, and System.err. These should probably be logged rather than printed. That's not your problem.

[maxaalexeeva]

I wish that meeting the needs of habitus wasn't requiring changes to processors.

A lot of these changes seem pretty general and could be useful for other projects as well? (other than the WS/DS ones maybe) But yeah, the pipeline of me fixing a few newly discovered issues -> releasing processors might not be the most convenient.

[maxaalexeeva]

Hi @kwalcock, I notice that the version in project is still the old snapshot (8.5.1-SNAPSHOT). Maybe I branched off unpulled master. However, I do not see this difference reflected here under Files changed. Should I merge master into my branch to fix this? I am a little confused.

[kwalcock]

AFAICT this branched from master just before 8.5.1 was released. There are no problems when I merge locally. We've probably just been working on non-intersecting parts of the code. I don't think there is need to merge master into this branch without merge conflicts.

[maxaalexeeva]

@kwalcock thanks for checking!

[maxaalexeeva]

@kwalcock @MihaiSurdeanu fyi, converting this to draft for now. I am going through habitus tests to see what else needs fixed and whether these changes help in habitus, and I keep finding more things to do.

[maxaalexeeva]

@MihaiSurdeanu I think I am done with this for now. Please have a look.

One concern I have is this commit 2b935c6
The intention was to avoid finding in in released as Sahel 108 in Senegal in 1994 as a unit (inch). My previous solution of just disallowing units before NNPs did not work because fertilizers in habitus are frequently tagged as NNP. I also can't add a constraint to the rule based on location entity because based on what I can tell from the code, entities like locations are not available at the stage of numerical info extraction. My solution, though, feels to be a bit overkill. Any suggestions? If not, keep my solution or let it go with this example?

[MihaiSurdeanu]

Hi @maxaalexeeva,
Yes, that commit is problematic. Would it be simpler to disallow units if the POS tag is "IN" or "TO"? Assuming the POS tagger is correct, this should fix the "in Senegal" bug because "in" there should be a preposition.
I think the question is: does the POS tagger label units as IN in legit cases?
Thanks!

[maxaalexeeva]

@MihaiSurdeanu if I remember correctly, in(ch) got tagged as IN as well, but I'll double check tomorrow.

[maxaalexeeva]

@MihaiSurdeanu, same tag:

Lemmas: release as sahel 108 in senegal .
POS tags: VBN IN NNP CD IN NNP .

vs.

Lemmas: that year , we see 108 in of snow .
POS tags: DT NN , PRP VBD CD IN IN NN .

and

Lemmas: snow level be 108 in .
POS tags: NNP NN VBD CD IN .

[MihaiSurdeanu]

I see...
Then I don't have a better solution. Please merge it as is, @maxaalexeeva or @kwalcock.
Thanks!

[maxaalexeeva]

@kwalcock I don't think I have the power to merge, so feel free to merge if you have no further comments!

[kwalcock]

Will do.