diff --git a/2024/day27.md b/2024/day27.md index 01d083523..901ec222f 100644 --- a/2024/day27.md +++ b/2024/day27.md @@ -128,7 +128,7 @@ ansible \ Go on and check if the web servers are running on the respective hosts. -> [!TIP] +> [!HINT] > Ansible is **idempotent** - try running the commands again and see how the output differs. @@ -166,3 +166,176 @@ ansible-playbook \ ``` + +### Lab 2: Event-Driven Ansible + +
+ +Receive Generic Events via Webhook + +#### Receive Generic Events via Webhook + +If you followed the setup instructions for the EDA lab, you should already have a running EDA instance on the `eda-controller.example.com` VM. + +If you navigate to `/etc/edacontroller/rulebook.yml` on the VM, you'll see the following rulebook: + +```yaml +--- +- name: Listen to webhook events + hosts: all + sources: + - ansible.eda.webhook: + host: 0.0.0.0 + port: 5000 + rules: + - name: Debug event output + condition: 1 == 1 + action: + debug: + msg: "{{ event }}" + +- name: Listen to Alertmanager alerts + hosts: all + sources: + - ansible.eda.alertmanager: + host: 0.0.0.0 + port: 9000 + data_alerts_path: alerts + data_host_path: labels.instance + data_path_separator: . + rules: + - name: Restart MySQL server + condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing' + action: + run_module: + name: ansible.builtin.service + module_args: + name: mysql + state: restarted + - name: Debug event output + condition: 1 == 1 + action: + debug: + msg: "{{ event }}" + +``` + +For this part of the lab, the **first rule** is the one we're interested in: It listens to a generic webhook on port `5000` and prints the event's **metadata** to its logs. + +To test this, we can use the `curl` command to send a `POST` request to the webhook `/endpoint` from the VM itself: + +```console +curl \ + -X POST \ + -H "Content-Type: application/json" \ + -d '{"foo": "bar"}' \ + http://localhost:5000/endpoint +``` + +If you now check the logs of the EDA controller, you should see the following output: + +```console +journalctl -fu eda-controller + +Jan 11 16:35:29 eda-controller ansible-rulebook[56882]: {'payload': {'foo': 'bar'}, 'meta': {'endpoint': 'endpoint', +'headers': {'Host': 'localhost:5000', 'User-Agent': 'curl/7.76.1', 'Accept': '*/*', 'Content-Length': '21', +'Content-Type': 'application/x-www-form-urlencoded'}, 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, +'received_at': '2024-01-11T15:35:29.798401Z', 'uuid': '6ebf8dd2-60a2-455a-9383-97b81f535366'}} +``` + +A rule that always evaluates to `true` is not very useful, so let's change the rule to only print the the value of `foo` if the `foo` key is present in the event's payload, and `no foo :(` otherwise: + +```yaml +--- +- name: Listen to webhook events + hosts: all + sources: + - ansible.eda.webhook: + host: 0.0.0.0 + port: 5000 + rules: + - name: Foo + condition: event.payload.foo is defined + action: + debug: + msg: "{{ event.payload.foo }}" + - name: No foo + condition: 1 == 1 + action: + debug: + msg: "no foo :(" +``` + +Send the same `curl` request again and check the logs, you should see a line saying `bar` now. + +Let's also try a `curl` request with a different payload: + +```console +curl \ + -X POST \ + -H "Content-Type: application/json" \ + -d '{"bar": "baz"}' \ + http://localhost:5000/endpoint +``` + +This time, the output should be `no foo :(`. + +
+ +
+ +Restarting Services Automatically with EDA + +#### Restarting Services Automatically with EDA + +The last lab is more of a demo - it shows how you can use EDA to automatically react on events observed by **Prometheus** and **Alertmanager**. + +For this demo, the second **ruleset** in our rulebook is the one we're interested in: + +```yaml +- name: Listen to Alertmanager alerts + hosts: all + sources: + - ansible.eda.alertmanager: + host: 0.0.0.0 + port: 9000 + data_alerts_path: alerts + data_host_path: labels.instance + data_path_separator: . + rules: + - name: Restart MySQL server + condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing' + action: + run_playbook: + playbook: ./playbook.yml + - name: Debug event output + condition: 1 == 1 + action: + debug: + msg: "{{ event }}" +``` + +With this rule, we can restart our MySQL server if it's not running! But how do we get the event to trigger? With **Prometheus** and **Alertmanager**! + +When you ran the setup playbook, it installed **Prometheus** and **Alertmanager** on the `eda-controller.example.com` VM. You can access the **Prometheus** UI at `http://:9090` and the **Alertmanager** UI at `http://:9093`. + +It also installed a **Prometheus exporter** for the **MySQL** database that runs on the server. + +With this setup, we can now shut down our MySQL server and see what happens - make sure to watch the output of the EDA controller's logs: + +```console +systemctl stop mysql +journalctl -fu edacontroller +``` + + +Within 30-90 seconds, you should see EDA running our **playbook** and restarting the MySQL server. You can track that process by watching the Prometheus/Alertmanager UIs for firing alerts. + +Once you see the playbook being executed in the logs, you can check the MySQL state once more: + +```console +systemctl status mysql +``` + +MySQL should be up and running again! +