Addition of Braunschweiger Zeitung

[addie9800]

No description provided.

[MaxDall]

Thanks for adding this 👍

[MaxDall]

This seems like a NewsMap to me and not a Sitemap

PS:
There also seems to be a sitemap judging by this link https://www.braunschweiger-zeitung.de/sitemaps/archive/sitemap-2017-08-p00.xml.gz
Looks like a monthly pattern with pagination.

Could you check for what date range you get something back and if there is a second page p01?
And afterwards add this as a sitemap?

[addie9800]

Interesting, how did you find it? I checked the robots.txt file and that only mentioned the NewsMap. Regarding the SiteMap: it seems rather weird. The earliest I can find is: https://www.braunschweiger-zeitung.de/sitemaps/archive/sitemap-2006-11-p00.xml.gz (at least if you consider the dates in the url), but if you access the sitemap, it covers articles from September 2016. I think that is the correct lower bound of available dates though. If using dates after that, it seems to work just fine. There's no page 1, one file seems to contain all articles of that month. In case I would want to add it, how would I go about it? I am having trouble to find an index for the sitemaps

[MaxDall]

I got it by searching for braunschweiger-zeitung.de sitemap in google.

from datetime import datetime
from dateutil.rrule import rrule, MONTHLY

from fundus import Sitemap

sitemaps = [
    Sitemap(
        f"https://www.braunschweiger-zeitung.de/sitemaps/archive/sitemap-{d.year}-{str(d.month).zfill(2)}-p00.xml.gz"
    )
    for d in reversed(list(rrule(MONTHLY, dtstart=datetime(2016, 9, 1), until=datetime.now())))
]

This snippet generates all available sitemaps. You can add it with the + operator to sources.

[MaxDall]

Thanks for adding this 👍

[MaxDall]

There are merge conflicts with master that should be resolved before i can review this.

[addie9800]

Sorry about that, I fixed the conflicts now.