Update DW Version #342
Update DW Version #342
Conversation
src/fundus/publishers/de/dw.py
Outdated
| ) | ||
|
|
||
|
|
||
| class DWParser(ParserProxy): | ||
| class V2(BaseParser): | ||
| VALID_UNTIL = datetime.date(2024, 1, 18) | ||
|
|
||
| _paragraph_selector = CSSSelector("div.rich-text > p") |
There was a problem hiding this comment.
DW seems to sometimes have additional tags at the end of the article (https://www.dw.com/de/trump-in-verleumdungsprozess-gegen-e-jean-carroll-zu-83-millionen-dollar-schadenersatz-verurteilt/a-68100499, https://www.dw.com/de/us-regierung-genehmigt-verkauf-von-f-16-kampfjets-an-türkei/a-68100064), unfortunately I didn't see an easy fix at the first glance, since they aren't any special tags and not always there.
There was a problem hiding this comment.
They also sometimes mention an update of the article at the end, this should be able to be handled easily since it's in italics: https://www.dw.com/de/esc-2024-schwedische-k%C3%BCnstler-gegen-teilnahme-israels/a-68011968
There was a problem hiding this comment.
DW seems to sometimes have additional tags at the end of the article
Are you talking about this jj/sti (dpa, afp, rtr)?
I can only think of regular expressions. But I don't know how to get them working in XPath.
Anyways, that would be a regular expression matching the above string.
.*\((rtr, |dpa, |afp, ){0,2}(rtr|dpa|afp)\)$
They also sometimes mention an update of the article at the end, this should be able to be handled easily since it's in italics
I added it to the selector.
There was a problem hiding this comment.
Yes, although it's not restricted to those: here are some that I found: AR/jj (dpa, rtr, ap) kle/jj (kna, dpa, rtr, afp) pg/AR/kle (dpa, afp) pg/AR/haz (afp, dpa) kle/haz (dpa, rtr, afp) not really sure what it's limited to though
There was a problem hiding this comment.
So unless we can use reg exp in XPath there is nothing much to do about it anyways, but updating the parser would be very important because it currently does not work.
Update: i added a comment about the author line.
src/fundus/publishers/de/dw.py
Outdated
| # which seems to be rather hard to omit. Some examples: | ||
| # AR/jj (dpa, rtr, ap), kle/jj (kna, dpa, rtr, afp), pg/AR/kle (dpa, afp), | ||
| # pg/AR/haz (afp, dpa), kle/haz (dpa, rtr, afp) | ||
| _paragraph_selector = XPath("//div[contains(@class, 'rich-text')] /p[not(em) or text()]") |
There was a problem hiding this comment.
This might be an option
| _paragraph_selector = XPath("//div[contains(@class, 'rich-text')] /p[not(em) or text()]") | |
| _paragraph_selector = XPath("//div[contains(@class, 'rich-text')] /p[(not(em) or text()) and not(re:test(text(), '.*\((rtr, |dpa, |afp, |epd, |ap, ){0,3}(ap|rtr|dpa|afp|epd)\)$'))]" | |
| , namespaces={'re': 'http://exslt.org/regular-expressions'}) |
There was a problem hiding this comment.
I altered the regex a bit to make it independent of the actual acronyms.
^([a-z]{2,3}\/|[A-Z]{2,3}\/)*([a-z]{2,3}|[A-Z]{2,3})\s\(([a-z]{2,3}, )*([a-z]{2,3})\)$
Testing this, the regex works but the selector doesn't. Can you confirm that? Did re:test work on your side?
I tried the following
_author_regex = r"^([a-z]{2,3}\/|[A-Z]{2,3}\/)*([a-z]{2,3}|[A-Z]{2,3})\s\(([a-z]{2,3}, )*([a-z]{2,3})\)$"
_paragraph_selector = XPath(
f"//div[contains(@class, 'rich-text')] /p[not(em) or text() and not(re:test(text(), '{_author_regex}'))]",
namespaces={"re": "http://exslt.org/regular-expressions"},
)There was a problem hiding this comment.
That looks good. It works for me when I add parentheses around the or statement:
_author_regex = r"^([a-z]{2,3}\/|[A-Z]{2,3}\/)*([a-z]{2,3}|[A-Z]{2,3})\s\(([a-z]{2,3}, )*([a-z]{2,3})\)$" _paragraph_selector = XPath( f"//div[contains(@class, 'rich-text')] /p[(not(em) or text()) and not(re:test(text(), '{_author_regex}'))]", namespaces={"re": "http://exslt.org/regular-expressions"}, )
src/fundus/publishers/de/dw.py
Outdated
| # pg/AR/haz (afp, dpa), kle/haz (dpa, rtr, afp) | ||
| _paragraph_selector = XPath("//div[contains(@class, 'rich-text')] /p[not(em) or text()]") | ||
| # https://regex101.com/r/uZLwyb/1 | ||
| _author_regex = r"^([a-z]{2,3}\/|[A-Z]{2,3}\/)*([a-z]{2,3}|[A-Z]{2,3})\s\(([a-z]{2,3}, )*([a-z]{2,3})\)$" |
There was a problem hiding this comment.
I found a couple more edge cases: https://regex101.com/r/9r8bUf/1 (nice tool btw)
_author_regex = r"^([A-z]{2,3}\/)*([A-z]{2,3})\s\(([A-z]{2,3}, ?)*([A-z ]{2,9})\)$"
No description provided.