Merge pull request #356 from jacobdotcosta/improve_ocp_deployment

Improve ocp deployment

.gitignore

@@ -31,7 +31,10 @@ output/**
 
 # Testing purposes
 sandbox/cluster_up/test.sh
+
+# Python
 .snowdrop-venv/
+.*env
 
 #######
 # IDE #

README.adoc

@@ -14,6 +14,7 @@ endif::[]
 
 == Introduction
 
+[.lead]
 This project details the `prerequisites` and `steps` necessary to automate the installation of a Kubernetes (aka k8s) cluster or Openshift 4 top of one of the following cloud provider:
 
 * Red Hat https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/[OpenStack] (RHOS-PSI)
@@ -31,28 +32,66 @@ NOTE: kind is not a cloud provider but a tool able to run a k8s cluster on a con
 All the commands mentioned on this project are to be executed at the root folder of the repository, except if stated otherwise.
 ====
 
-This project uses Ansible. Check the link:ansible/README.adoc[Ansible Document] for the
-installation and usage instrutions.
-
 == Prerequisites
 
+This project uses Ansible. Check the link:ansible/README.adoc[Ansible Document] for the
+installation and usage instructions.
+
 To use the scripts, playbooks, part of this project, some prerequisites are needed. It is not mandatory to install
-all of them and the following chapters will mention which ones are needed.
+all of them and the following chapters will mention which ones are needed. 
 
 * https://kind.sigs.k8s.io/docs/user/quick-start/#installation[kind]
 * https://docs.docker.com/engine/install/[Docker] or https://podman.io/docs/installation[podman]
 * https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-and-upgrading-ansible[Ansible]. Version >= 2.14
 * https://www.python.org/downloads/[Python]. Version >= 3.11
 * https://www.passwordstore.org/[passwordstore]
 * https://github.com/hetznercloud/cli[hcloud] (optional)
+* 
+
+=== Python
+
+Several requirements are provided as Python libraries, including Ansible, 
+ and are identified on the link:requirements.txt[] file.
+
+Using a Python Virtual Environment is recommended and can be created using 
+ the following command:
+
+[source,bash]
+----
+python3 -m venv .snowdrop-venv
+----
 
+After creating the virtual environment start using it with the following command:
+
+[source,bash]
+----
+source .snowdrop-venv/bin/activate
+----
+
+The venv will be in use when the `(.snowdrop-venv)` prefix is shown on the bash prompt.
+
+The python requirements can be installed by executing:
+
+[source,bash]
+----
+pip3 install -r requirements.txt
+----
+
+[NOTE]
+====
+For more information check the link:ansible/README.adoc#python-venv[Python Virtual Env] section on our Ansible README. 
+====
 
-== Locally
+=== Ansible
 
-The word `locally` should be understood as the process to run on your developer laptop the cluster, using also a CI/CD platform
-such as GitHub Actions, etc.
+Several Ansible Galaxy collections are used as part of this project and 
+ are listed in the link:collections/requirements.yml[] file. 
+ To install them execute the following command.
 
-We recommend to use 2 tools to run locally a kubernetes cluster: kind or minikube
+[source,bash]
+----
+ansible-galaxy collection install -r ./collections/requirements.yml --upgrade
+----
 
 === Kind
 
@@ -76,7 +115,7 @@ The provisioning process towards the cloud providers relies on the following ass
 - Password store is installed/configured and needed k/v created
 - Flavor, volume, capacity (cpu/ram/volume) and OS can be mapped with the playbook of the target cloud provider
 - Permissions have been set to allow to provision a VM top of the target cloud provider
-- Ssh key exist and has been imported (or could be created during provisioning process)
+- SSH key exist and has been imported (or could be created during provisioning process)
 
 and will include the following basic steps:
 
@@ -96,7 +135,8 @@ This section details how to provision an Openshift 4 cluster using one of Red Ha
 
 _Tools: password store, ansible_
 
-The link:openstack/README.adoc[OpenStack] page explains how to create an OpenStack cloud vm using
+The link:openstack/README.adoc[OpenStack] page explains the process using 
+ the RHOS cloud provider.
 
 ==== https://resourcehub.redhat.com/[Resource Hub]
 

ansible.cfg

@@ -7,6 +7,9 @@ hash_behaviour = merge
 roles_path = ansible/roles/
 
 callback_whitelist = profile_tasks
-gather_timeout = 5000
+gather_timeout = 30000
 
 log_path=/opt/log/ansible.log
+
+ansible_python_interpreter=/usr/bin/python3
+interpreter_python=auto

ansible/README.adoc

@@ -1,12 +1,9 @@
 = Ansible
 Snowdrop Team (Antonio costa)
-Snowdrop Team (Antonio costa)
 :icons: font
 :revdate: {docdate}
-:revdate: {docdate}
 :toc: left
 :toclevels: 3
-:toclevels: 3
 :description: This document introduces some of the key concepts that you should be aware when you play with Ansible in order to configure the environment to let Ansible to access the different machines.
 ifdef::env-github[]
 :tip-caption: :bulb:
@@ -15,13 +12,6 @@ ifdef::env-github[]
 :caution-caption: :fire:
 :warning-caption: :warning:
 endif::[]
-ifdef::env-github[]
-:tip-caption: :bulb:
-:note-caption: :information_source:
-:important-caption: :heavy_exclamation_mark:
-:caution-caption: :fire:
-:warning-caption: :warning:
-endif::[]
 
 == Conventions
 
@@ -32,6 +22,11 @@ The exception goes to the playbooks that are executed against `localhost`. This
 
 NOTE: Check the Ansible https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix[requirement] page for Python compatibility !
 
+[#ansible-inventory]
+== Ansible Inventory
+
+The Ansible Inventory is managed by 
+
 == Installation guide
 
 In order to play with the playbooks/roles of this project, it is needed to:
@@ -48,14 +43,15 @@ In order to play with the playbooks/roles of this project, it is needed to:
 
 [NOTE]
 ====
-Since passwordstore is integrated with [git](https://git-scm.com/), all changes made locally to a pass repository are automatically committed to the local git repo.
+Since passwordstore is integrated with link:https://git-scm.com/[git], all changes made locally to a pass repository are automatically committed to the local git repo.
 ====
 
-[NOTE]
+[WARNING]
 ====
 Don't forget to `git push` and `git pull` often in order to have your local repository synchronized with other team members as well as publishing to the team your changes. 
 ====
 
+[#python-venv]
 === Python Virtual Environments
 
 This project suggests using a link:https://docs.python.org/3/library/venv.html[python virtual environment] 
@@ -282,7 +278,7 @@ Because a host can already be defined under the store, prior to execute the play
 
 [source,bash]
 ----
-$ pass hetzner
+pass hetzner
 hetzner
 ├── ...
 ├── host-1
@@ -312,7 +308,7 @@ If a host has already been created, it can be imported within the inventory usin
 
 [source,bash]
 ----
-$ ansible-playbook ansible/playbook/passstore_controller_inventory.yml -e vm_name=<VM_NAME> -e pass_provider=hetzner
+ansible-playbook ansible/playbook/passstore_controller_inventory.yml -e vm_name=<VM_NAME> -e pass_provider=hetzner
 ----
 
 where `<VM_NAME>` corresponds to the host key created under `hetzner`.
@@ -334,7 +330,7 @@ This is done using the `passstore_controller_inventory_remove` playbook. More in
 
 [source,bash]
 ----
-$ ansible-playbook ansible/playbook/passstore_controller_inventory_remove.yml -e vm_name=<vm_name> -e pass_provider=<provider>
+ansible-playbook ansible/playbook/passstore_controller_inventory_remove.yml -e vm_name=<vm_name> -e pass_provider=<provider>
 ----
 
 === Create Server

ansible/ansible-inventory.adoc

@@ -1,6 +1,25 @@
 = Ansible Inventory
+:icons: font
+:revdate: {docdate}
 :toc: left
-:description: This document describes the Ansible inventory implementation.
+:toclevels: 3
+:description: Ansible Inventory
+ifdef::env-github[]
+:tip-caption: :bulb:
+:note-caption: :information_source:
+:important-caption: :heavy_exclamation_mark:
+:caution-caption: :fire:
+:warning-caption: :warning:
+endif::[]
+
+== Introduction
+
+[.lead]
+This document described the implementation of the Ansible Inventory on this 
+ project.
+
+It used a mix of a passwordstode database and static files to maintain all 
+ host information and properties.
 
 == Introduction to Ansible Inventory
 
@@ -14,7 +33,7 @@ The two most important files are:
 
 **Remark**: More information on the Ansible Inventory and how to build it is defined within the [Ansible User Guide](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html).
 
-=== `hosts.yml`
+== `hosts.yml` file
 
 This file contains static information for the inventory such as:
 * Group 
@@ -24,7 +43,8 @@ This file contains static information for the inventory such as:
 
 Here is a sample of a *hosts yaml* file designed using YAML. 
 
-```yaml
+[source,yaml]
+----
 all: # keys must be unique, i.e. only one 'hosts' per group
     hosts:
         host1:
@@ -51,15 +71,17 @@ all: # keys must be unique, i.e. only one 'hosts' per group
                 host1:
             vars:
                 group_last_var: value
-```
+----
 
 More information on these documents are available:
-* [yaml – Uses a specific YAML file as an inventory source](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/yaml_inventory.html)
-* [How to build your inventory](https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html)
 
-This project already includes a static inventory, at [../inventory/hosts.yml](../inventory/hosts.yml) file.
+* link:https://docs.ansible.com/ansible/latest/collections/ansible/builtin/yaml_inventory.html[yaml – Uses a specific YAML file as an inventory source]
+* link:https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html[How to build your inventory]
 
-=== Groups
+This project already includes a static inventory, at the
+ link:../inventory/hosts.yml[] file.
+
+== Groups
 
 Ansible hosts can be grouped into...well groups. This allows the execution of playbooks and the definition of variables in a common matter for different hosts.
 
@@ -68,7 +90,8 @@ values assigned to each group.
 
 Host group assignment is made in `passstore` by managing entries in the `provider/host/groups` folder being each entry a group to which the host belongs. 
 
-```text
+[source]
+----
 ├── provider
 |   ├── host_1
 │   │   ├── groups
@@ -79,19 +102,46 @@ Host group assignment is made in `passstore` by managing entries in the `provide
 │   │   ├── groups
 │   │   │   ├── group_2
 │   │   │   ├── group_3
-```
+----
 
 For instance, we wanted to define the ports that a k8s master needs to open. This has been done in the `hosts.yml` file having the following variable assigned to 
 the `masters` group, which is also inside a group structure so other variables are inherited. 
 
-```
+[source,yml]
+----
 firewalld_public_ports:
   - 6443/tcp
   - 10250/tcp
   - 10255/tcp
   - 8472/udp
   - 30000-32767/tcp
-```
+----
 
 For information regarding actually managing host-group assignment check the [`passstore_manage_host_groups` section](#passstore_manage_host_groups). 
 
+== `pass_inventory.py` Inventory Python script
+
+This Python script build the Ansible Inventory from the passwordstore database.
+
+To collect information on a host execute the following command.
+
+[source,bash]
+----
+./ansible/inventory/pass_inventory.py --host <1>
+----
+<1> Name of the host in the Ansible inventory.
+
+Example
+
+[source,bash]
+----
+./ansible/inventory/pass_inventory.py --host ocp-xyz-tmp-bootstrap-server
+----
+
+To list all the inventory simply execute with the --list attribute.
+
+[source,bash]
+----
+./ansible/inventory/pass_inventory.py --list
+----
+

ansible/ansible_collections/snowdrop/godaddy/playbooks/roles

@@ -0,0 +1 @@
+../roles/

ansible/inventory/pass_inventory.py

@@ -77,9 +77,20 @@
                                     passLines = pipe.stdout.readlines()
                                     passEntry = passLines[0].replace('\n', '')
                                     if ('os_user' == passEntryName):
+                                        host_vars.update({passEntryName:passEntry})
                                         host_vars.update({'ansible_user':passEntry})
                                     elif ('ip_address' == passEntryName):
+                                        host_vars.update({passEntryName:passEntry})
+                                        if (not 'floating_ip' in host_vars):
+                                          host_vars.update({'ansible_ssh_host':passEntry})
+                                    elif ('floating_ip' == passEntryName):
+                                        host_vars.update({passEntryName:passEntry})
+                                        # floating_ip overrides any other host variable
                                         host_vars.update({'ansible_ssh_host':passEntry})
+                                        host_vars.update({'floating_ip':passEntry})
+                                    elif ('ansible_ssh_host' == passEntryName):
+                                        if (not 'ansible_ssh_host' in host_vars):
+                                          host_vars.update({'ansible_ssh_host':passEntry})
                                     # elif ('ssh_port' == passEntryName):
                                     #     host_vars.update({'ansible_ssh_port':passEntry})
                                     else: