Merge branch 'dev'

.travis.yml

@@ -28,6 +28,7 @@ jobs:
         - docker pull pierky/bird:1.6.8
         - docker pull pierky/bird:2.0.8
         - docker pull pierky/openbgpd:6.8p1
+        - docker pull pierky/openbgpd:6.9p0-patches
         - docker pull pierky/exabgp:4.2.7
         - docker pull nlnetlabs/routinator:v0.8.3
     - env: TOXENV=py36-coverage

CHANGES.rst

@@ -3,6 +3,35 @@ Change log
 
 .. note:: **Upgrade notes**: after upgrading, run the ``arouteserver setup-templates`` command to sync the local templates with those distributed with the new version. More details on the `Upgrading <https://arouteserver.readthedocs.io/en/latest/INSTALLATION.html#upgrading>`__ section of the documentation.
 
+next release
+------------
+
+Starting with this release, the default target version for OpenBGPD will be the latest stable (6.9 in this case).
+
+- New: Add support for OpenBGPD/OpenBSD 6.9 and OpenBGPD Portable 6.9p0, also added to the integration testing suite.
+
+- New (OpenBGPD): add support for RTR sessions starting with version 6.9.
+
+  Please note the following issues with OpenBGPD 6.9 if you want to enable RTR sessions; you might want to apply the available patches:
+
+  - ``Invalid argument`` error and RTR session not coming up (`issue #23 on GitHub <https://github.com/openbgpd-portable/openbgpd-portable/issues/23>`__ and `"bgpd, fix RTR connect" <https://marc.info/?l=openbsd-tech&m=162004696829635&w=2>`__ post on openbsd-tech)
+
+  - non blocking ``connect()`` call for RTR session establishment (`"bgpd behaviour when RTR endpoint is not available" <https://marc.info/?l=openbgpd-users&m=161997334304946&w=2>`__ post on openbgpd-users and `"bgpd, non-blocking rtr connect" <https://marc.info/?l=openbsd-tech&m=162005636502085&w=2>`__ post on openbsd-tech)
+
+- New (OpenBGPD): enable support for path-hiding mitigation.
+
+  Even though OpenBGPD supports path-hiding mitigation starting with version 6.9, the feature is not automatically enabled by the ``configure`` command because of some issues that might impair the stability of the routing ecosystem:
+
+  - withdrawal of 2nd best route with ``rde evaluate all`` (`issue #21 on GitHub <https://github.com/openbgpd-portable/openbgpd-portable/issues/21>`__ and `"bgpd fix for rde evaluate all" <https://marc.info/?l=openbsd-tech&m=162011500326166&w=2>`__ post on openbsd-tech)
+
+  - advertisement of 2nd best routes on reload with ``rde evaluate all`` (`issue #21 on GitHub <https://github.com/openbgpd-portable/openbgpd-portable/issues/21>`__ and `"bgpd better reload behaviour" <https://marc.info/?l=openbsd-tech&m=162021735205669&w=2>`__ post on openbsd-tech)
+
+  Please apply the existing patches before enabling it on a production environment, and acknowledge the error produced by ARouteServer using the ``--ignore-issues path_hiding_69`` CLI option.
+
+- Improvement: the default list of `"transit free" <https://arouteserver.readthedocs.io/en/latest/GENERAL.html#transit-free-networks-transit-free>`__ ASNs has been updated and some networks have been removed.
+
+  See also `GitHub PR73 <https://github.com/pierky/arouteserver/pull/73>`_.
+
 v1.5.1
 ------
 

README.rst

@@ -42,7 +42,7 @@ How it works
 
 #. `Jinja2`_ built-in templates are used to render the final route server's configuration file.
 
-   Currently, **BIRD** (>= 1.6.3 up to 1.6.8), **BIRD v2** (starting from 2.0.7 - support for BIRD v2 is in `early stages <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html>`_) and **OpenBGPD** (OpenBSD 6.1 up to 6.8 and also OpenBGPD Portable 6.5p1 up to 6.8p1) are supported, with almost `feature parity <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html#supported-features>`_ between them.
+   Currently, **BIRD** (>= 1.6.3 up to 1.6.8), **BIRD v2** (starting from 2.0.7 - support for BIRD v2 is in `early stages <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html>`_) and **OpenBGPD** (OpenBSD 6.1 up to 6.9 and also OpenBGPD Portable 6.5p1 up to 6.9p0) are supported, with almost `feature parity <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html#supported-features>`_ between them.
 
 **Validation** and testing of the configurations generated with this tool are performed using the built-in **live tests** framework: `Docker`_ instances are used to simulate several scenarios and to validate the behaviour of the route server after configuring it with ARouteServer. More details on the `Live tests <https://arouteserver.readthedocs.io/en/latest/LIVETESTS.html>`_ section.
 
@@ -163,6 +163,8 @@ Who is using ARouteServer?
 
 - `DO-IX <https://www.do-ix.net/>`__, BIRD.
 
+- `EVIX <https://evix.org/>`__, BIRD.
+
 - `FCIX <https://fcix.net/>`__, BIRD.
 
 - `GAVLIX <https://gavlix.se/>`__.
@@ -185,7 +187,7 @@ Who is using ARouteServer?
 
 - `QCIX <http://www.qcix.net/>`__, BIRD.
 
-- `RO-CIX <https://roix.net//>`__, OpenBGPD.
+- `RO-CIX <https://roix.net/>`__, OpenBGPD.
 
 - `SwissIX <https://www.swissix.ch/>`__, OpenBGPD.
 

config.d/general.yml

@@ -37,7 +37,8 @@ cfg:
   #   accepted is found or all routes for that network are rejected."
   #   (http://bird.network.cz/?get_doc&f=bird-6.html#bgp-secondary)
   #
-  # OpenBGPD: not implemented in ARouteServer. Single RIB only.
+  # OpenBGPD: 'rde evaluate all' is used.
+  #   (https://man.openbsd.org/bgpd.conf#rde)
   #
   # Default: True
   path_hiding: True
@@ -174,9 +175,8 @@ cfg:
       # Comma separated list of ASNs which are considered
       # transit-free. Used only if an 'action' is provided above.
       asns: >
-        174, 209, 286, 701, 1239, 1299, 2828, 2914,
-        3257, 3320, 3356, 3549, 5511, 6453, 6461,
-        6762, 6830, 7018, 12956
+        174, 701, 1299, 2914, 3257, 3320, 3356, 5511, 6453,
+        6461, 6762, 6830, 7018, 12956
 
     never_via_route_servers:
       # Similarly to what happens with the 'transit_free' config
@@ -517,17 +517,19 @@ cfg:
     # Can be one of the following options:
     # - 'rtr': ROAs are loaded from an external RTR source.
     #   rtrllib (https://github.com/rtrlib/bird-rtrlib-cli) can be
-    #   used for BIRD 1.6.x; in BIRD v2 there is built-in support
-    #   for the RTR protocol.
+    #   used for BIRD 1.6.x; in BIRD v2 and OpenBGPD (starting with
+    #   version 6.9) there is built-in support for the RTR protocol.
     #   The name of the table where send the ROAs to is 'RPKI' for
     #   BIRD 1.6.x and 'RPKI4' and 'RPKI6' for BIRD v2.
-    #   In BIRD v2, an external file 'rpki_rtr_config.local' must be
+    #   When the built-in implementation is used for OpenBGPD (> 6.9)
+    #   or BIRD v2, an external file 'rpki_rtr_config.local' must be
     #   found within the same directory where the main configuration
-    #   file is stored (/etc/bird usually) and must contain the BIRD 2
-    #   configuration for 'protocol rpki'. An example on how to
-    #   setup that file can be found in the examples/bird2_rpki_rtr
-    #   directory (please note, in order to use the RTR protocol BIRD
-    #   must be compiled with --enable-libssh).
+    #   file is stored (/etc/bird or /etc/bgpd usually) and must
+    #   contain the configuration of the RTR sessions specific for
+    #   that daemon.
+    #   An example on how to setup that file can be found in the
+    #   examples/rpki_rtr directory (please note, in order to use
+    #   the RTR protocol BIRD must be compiled with --enable-libssh).
     # - 'ripe-rpki-validator-cache': ROAs are loaded from a JSON
     #   file in RIPE NCC RPKI Validator cache format.
     #
@@ -541,9 +543,6 @@ cfg:
     #   instance of a RPKI validator is provided below in the
     #   'ripe_rpki_validator_url' option.
     #
-    # OpenBGPD: only the 'ripe-rpki-validator-cache' source
-    # is currently supported.
-    #
     # Default: ripe-rpki-validator-cache
     source: "ripe-rpki-validator-cache"
 

docs/CONFIG.rst

@@ -244,15 +244,18 @@ ROAs sources
 
 A couple of methods can be used to acquire RPKI data (ROAs):
 
-- (BIRD and OpenBGPD) the builtin method based on `RIPE RPKI Validator format <https://rpki-validator.ripe.net>`__ JSON export file (also generated by other validators like Routinator, rpki-client, OctoRPKI): the URL of a local and trusted instance of a RPKI validator should be provided to ensure that a trusted dataset is used. By default, the URLs of some  public instances are used.
+- the builtin method based on `RIPE RPKI Validator format <https://rpki-validator.ripe.net>`__ JSON export file (also generated by other validators like Routinator, rpki-client, OctoRPKI): the URL of a local and trusted instance of a RPKI validator should be provided to ensure that a trusted dataset is used. By default, the URLs of some  public instances are used.
 
-- (BIRD only) external resources can be used to pull ROAs from using the RTR protocol:
+- RTR protocol (only on BIRD and OpenBGPD >= 6.9):
 
   - BIRD 1.6.x: the `rtrlib <http://rpki.realmv6.org/>`_ suite: `rtrlib <https://github.com/rtrlib>`__ and `bird-rtrlib-cli <https://github.com/rtrlib/bird-rtrlib-cli>`__.
 
   - BIRD v2: the `built-in RTR protocol <https://bird.network.cz/?get_doc&v=20&f=bird-6.html#ss6.13>`_ implementation.
 
-  One or more trusted local validating caches should be used to get and validate ROAs before pushing them to BIRD. An overview is provided on the `rtrlib GitHub wiki <https://github.com/rtrlib/rtrlib/wiki/Background>`__, where also an `usage guide <https://github.com/rtrlib/rtrlib/wiki/Usage-of-the-RTRlib>`__ can be found. For BIRD v2, an example of how to configure the RTR protocol can be found in the ``examples/bird2_rpki_rtr`` directory (`also on GitHub <https://github.com/pierky/arouteserver/tree/master/examples/bird2_rpki_rtr>`_).
+  - OpenBGPD >= 6.9: the `built-in RTR protocol <https://man.openbsd.org/bgpd.conf#rtr>`_ implementation.
+
+  One or more trusted local validating caches should be used to get and validate ROAs before pushing them to BIRD or OpenBGPD. Extensive information on RPKI and how to setup validating caches can be found on `https://rpki.readthedocs.io/ <https://rpki.readthedocs.io/>`__.
+  For BIRD v2 and OpenBGPD, an example of how to configure the RTR protocol and use it with ARouteServer can be found in the ``examples/rpki_rtr`` directory (`also on GitHub <https://github.com/pierky/arouteserver/tree/master/examples/rpki_rtr>`_).
 
 The configuration of ROAs source can be done within the ``rpki_roas`` section of the ``general.yml`` file.
 
@@ -508,8 +511,6 @@ The following list of limitations is based on the currently supported versions o
 
 - OpenBGPD
 
-  - Currently, **path hiding** mitigation is not implemented for OpenBGPD configurations. Only single-RIB configurations are generated.
-
   - **ADD-PATH** is not supported by OpenBGPD.
 
   - For max-prefix filtering, only the ``shutdown`` and the ``restart`` actions are supported by OpenBGPD. Restart is configured with a 15 minutes timer.

docs/EXAMPLES.rst

@@ -163,7 +163,7 @@ https://github.com/pierky/arouteserver/blob/master/examples/auto-config
 
 bird-general.yml.html - See the `textual representation of this configuration <_static/examples_auto-config_bird-general.yml.html>`__.
 
-openbgpd64-general.yml.html - See the `textual representation of this configuration <_static/examples_auto-config_openbgpd64-general.yml.html>`__.
+openbgpd-general.yml.html - See the `textual representation of this configuration <_static/examples_auto-config_openbgpd-general.yml.html>`__.
 
 IX-F Member Export files
 ------------------------
@@ -172,19 +172,22 @@ The files reported within this directory were generated using the ``ixf-member-e
 
 https://github.com/pierky/arouteserver/blob/master/examples/ixf-member-export
 
-BIRD v2 RPKI RTR configuration
-------------------------------
+BIRD v2 and OpenBGPD RPKI RTR configuration
+-------------------------------------------
 
-This is an example of using BIRD v2 with an external source for RPKI ROAs.
+This is an example of how to use BIRD v2 or OpenBGPD with an external source for RPKI ROAs based on the RTR protocol.
 
-BIRD v2 has built-in support for the RTR protocol, that allows to connect the BGP daemon directly to a local cache (a "validator").
+BIRD v2 and OpenBGPD (starting with release 6.9) have built-in support for the RTR protocol, that allows to connect the BGP daemon directly to a local cache (a "validator").
 
-To configure BIRD v2 with ARouteServer in order to fetch ROAs using RTR, the ``rpki_roas.source`` option must be set to ``rtr`` and a local *rpki_rtr_config.local* file must be placed inside the same directory where the main BIRD configuration file is created (*/etc/bird* by default, or a custom one set using the ``--local-files-dir`` command line argument of ARouteServer).
+To configure the daemons with ARouteServer in order to fetch ROAs using RTR, the ``rpki_roas.source`` option must be set to ``rtr`` and a local *rpki_rtr_config.local* file must be placed inside the same directory where the main configuration file is created (*/etc/bird* or */etc/bgpd* by default, or a custom one set using the ``--local-files-dir`` command line argument of ARouteServer).
 
-The *rpki_rtr_config.local* file is expected to contain the snippet of BIRD config needed to setup a *rpki protocol*, accordingly to what is documented in the official BIRD web site: https://bird.network.cz/?get_doc&v=20&f=bird-6.html#ss6.13
+The *rpki_rtr_config.local* file is expected to contain the snippet of BIRD or OpenBGPD config needed to setup one or more RTR sessions:
+- BIRD v2: https://bird.network.cz/?get_doc&v=20&f=bird-6.html#ss6.13
 
-The names of the tables where ROAs will be injected into must be ``RPKI4`` and ``RPKI6``.
+  **Please note:** the names of the tables where ROAs will be injected into must be ``RPKI4`` and ``RPKI6``.
 
-An example configuration is reported in the *rpki_rtr_config.local* file that can be found within this directory.
+- OpenBGPD: https://man.openbsd.org/bgpd.conf#rtr
 
-https://github.com/pierky/arouteserver/blob/master/examples/bird2_rpki_rtr
+Example configurations are reported in the *rpki_rtr_config.local.BIRD* and *rpki_rtr_config.local.OpenBGPD* files that can be found within this directory.
+
+https://github.com/pierky/arouteserver/blob/master/examples/rpki_rtr

docs/FEATURES.rst

@@ -32,7 +32,7 @@ How it works
 
 #. `Jinja2`_ built-in templates are used to render the final route server's configuration file.
 
-   Currently, **BIRD** (>= 1.6.3 up to 1.6.8), **BIRD v2** (starting from 2.0.7 - support for BIRD v2 is in `early stages <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html>`_) and **OpenBGPD** (OpenBSD 6.1 up to 6.8 and also OpenBGPD Portable 6.5p1 up to 6.8p1) are supported, with almost `feature parity <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html#supported-features>`_ between them.
+   Currently, **BIRD** (>= 1.6.3 up to 1.6.8), **BIRD v2** (starting from 2.0.7 - support for BIRD v2 is in `early stages <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html>`_) and **OpenBGPD** (OpenBSD 6.1 up to 6.9 and also OpenBGPD Portable 6.5p1 up to 6.9p0) are supported, with almost `feature parity <https://arouteserver.readthedocs.io/en/latest/SUPPORTED_SPEAKERS.html#supported-features>`_ between them.
 
 **Validation** and testing of the configurations generated with this tool are performed using the built-in **live tests** framework: `Docker`_ instances are used to simulate several scenarios and to validate the behaviour of the route server after configuring it with ARouteServer. More details on the `Live tests <https://arouteserver.readthedocs.io/en/latest/LIVETESTS.html>`_ section.
 

docs/FUTUREWORK.rst

@@ -20,7 +20,6 @@ Mid term
 Long term
 ---------
 
-- New feature: path-hiding mitigation technique on OpenBGPD
 - New feature: routing policies based on RPSL import-via/export-via
 - New feature: other BGP speakers support (GoBGP, ...)
 - New feature: balance clients among *n* different configurations (for multiple processes - see `Scaling BIRD Routeservers <https://ripe73.ripe.net/presentations/115-e-bru-20161026-RIPE73-scaling-bird-routeservers-final.pdf>`_)

docs/GENERAL.rst

@@ -61,7 +61,8 @@ General options: ``cfg``
   (http://bird.network.cz/?get_doc&f=bird-6.html#bgp-secondary)
 
 
-  OpenBGPD: not implemented in ARouteServer. Single RIB only.
+  OpenBGPD: 'rde evaluate all' is used.
+  (https://man.openbsd.org/bgpd.conf#rde)
 
 
   Default: **True**
@@ -331,9 +332,9 @@ in the left-most position.
   .. code:: yaml
 
      asns: >
-          174, 209, 286, 701, 1239, 1299, 2828, 2914,
-          3257, 3320, 3356, 3549, 5511, 6453, 6461,
-          6762, 6830, 7018, 12956
+          174, 701, 1299, 2914, 3257, 3320, 3356, 5511, 6453,
+          6461, 6762, 6830, 7018, 12956
+
 
 
 
@@ -955,17 +956,19 @@ when **filtering.irrdb.use_rpki_roas_as_route_objects** or
 
   - **rtr**: ROAs are loaded from an external RTR source.
     rtrllib (https://github.com/rtrlib/bird-rtrlib-cli) can be
-    used for BIRD 1.6.x; in BIRD v2 there is built-in support
-    for the RTR protocol.
+    used for BIRD 1.6.x; in BIRD v2 and OpenBGPD (starting with
+    version 6.9) there is built-in support for the RTR protocol.
     The name of the table where send the ROAs to is **RPKI** for
     BIRD 1.6.x and **RPKI4** and **RPKI6** for BIRD v2.
-    In BIRD v2, an external file **rpki_rtr_config.local** must be
+    When the built-in implementation is used for OpenBGPD (> 6.9)
+    or BIRD v2, an external file **rpki_rtr_config.local** must be
     found within the same directory where the main configuration
-    file is stored (/etc/bird usually) and must contain the BIRD 2
-    configuration for 'protocol rpki'. An example on how to
-    setup that file can be found in the examples/bird2_rpki_rtr
-    directory (please note, in order to use the RTR protocol BIRD
-    must be compiled with --enable-libssh).
+    file is stored (/etc/bird or /etc/bgpd usually) and must
+    contain the configuration of the RTR sessions specific for
+    that daemon.
+    An example on how to setup that file can be found in the
+    examples/rpki_rtr directory (please note, in order to use
+    the RTR protocol BIRD must be compiled with --enable-libssh).
 
 
   - **ripe-rpki-validator-cache**: ROAs are loaded from a JSON
@@ -988,10 +991,6 @@ when **filtering.irrdb.use_rpki_roas_as_route_objects** or
   **ripe_rpki_validator_url** option.
 
 
-  OpenBGPD: only the **ripe-rpki-validator-cache** source
-  is currently supported.
-
-
   Default: **ripe-rpki-validator-cache**
 
   Example:

docs/LIVETESTS_SCENARIOS.rst

@@ -3,7 +3,6 @@
 .. toctree::
    :maxdepth: 1
 
-   LIVETESTS_SCENARIOS_bird2_rpki_rtr_example
    LIVETESTS_SCENARIOS_communities
    LIVETESTS_SCENARIOS_default
    LIVETESTS_SCENARIOS_global
@@ -12,5 +11,6 @@
    LIVETESTS_SCENARIOS_path_hiding
    LIVETESTS_SCENARIOS_rich_example
    LIVETESTS_SCENARIOS_rpki
+   LIVETESTS_SCENARIOS_rpki_rtr_example
    LIVETESTS_SCENARIOS_tag_as_set
    LIVETESTS_SCENARIOS_tag_reject_policy

docs/LIVETESTS_SCENARIOS_rpki_rtr_example.rst

@@ -0,0 +1 @@
+.. include:: ../tests/live_tests/scenarios/rpki_rtr_example/README.rst

docs/STATUS.txt

@@ -23,6 +23,8 @@ Who is using ARouteServer?
 
 - `DO-IX <https://www.do-ix.net/>`__, BIRD.
 
+- `EVIX <https://evix.org/>`__, BIRD.
+
 - `FCIX <https://fcix.net/>`__, BIRD.
 
 - `GAVLIX <https://gavlix.se/>`__.