Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes configuration example #167

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ Voice support is only available on Linux. On Mac or Windows, you can use the fol
docker run --name almond -p 3000:3000 stanfordoval/almond-server:latest-portable
```

### I am Kubernetes!

As an alternative, Almond-Server can run under Kubernetes; see sample configuration file in the [examples](https://github.com/stanford-oval/almond-server/tree/master/examples) directory for information on how to get started.

## Development setup

To develop almond-server, you should clone this repository, then install the dependencies with:
Expand Down
85 changes: 85 additions & 0 deletions examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Kubernetes deployment

This is a very basic configuration file for running almond-server on a Kubernetes cluster.

After applying this manifest, two containers are created by the pod and provides the following:
almond-server as the main deployment
nginx as a reverse proxy sidecar
sets securityContex for a specific user; in this example 'pi' with UID & GID of 1000.
configures a NodePort for Home-Assistant and local access

It is also assumed that PulesAudio runs in user mode

## Getting Started

Find your UID and GID from the command line.
Copy file 'kubernetes-deployment-example.yaml' to your prefered location, edit the following settings if necessary.

Default values
---
deployment
image: stanfordoval/almond-server ... your image here if not amd64
---
env:
TZ: Europe/London ... your timezone
---
SecurityContes: this ensures that the container runs as a specific user
runAsGroup: 1000 ... enter your GID and UID here
runAsUser: 1000 ... find it by typing 'id' from the command line
---
volumes:
hostPath:
path: /run/user/1000/pulse ... this is just $XDG_RUNTIME_DIR/pulse
path: /home/pi/.config/almond-server path to your Almond-Server config directory
---
service
spec:
nodePort: 32000 ... a valid port value of your choice
---
configmap
server
allow 10.1.12.1; ... example only, check nginx logs for connection failures
---

### Run almond-server

kubectl apply -f kubernetes-deployment-example.yaml

Navigate to http://<almond host>:32000 for almond web interface
You should reach Almond's Initial Configuration page where you are invited to set a password.

Home-Assistant config for this example.

```yaml
# Example configuration.yaml entry
almond:
type: local
host: http://<almond host>:32000
```

### Troubleshooting

You may see an error, 'access forbidden by rule', output in nginx logs, as in example below;
which IP 10.1.12.1 is my kubenetes instance and my host 192.168.0.2.

```logs

2021/03/06 10:06:18 [error] 31#31: *1 access forbidden by rule, client: 10.1.12.1, server: _, request: "GET / HTTP/1.1", host: "192.168.0.2:32000"
10.1.12.1 - - [06/Mar/2021:10:06:18 +0000] "GET / HTTP/1.1" 403 125 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15"
2021/03/06 10:06:18 [error] 31#31: *2 access forbidden by rule, client: 10.1.12.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "192.168.0.2:32000", referrer: "http://192.168.0.2:32000/"
10.1.12.1 - - [06/Mar/2021:10:06:18 +0000] "GET /favicon.ico HTTP/1.1" 403 125 "http://192.168.0.2:32000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15"

```

Update the configmap to allow client 10.1.12.1, e.g.

```yaml

allow 127.0.0.1; # localhost
allow 10.1.12.1; # kubernetes
deny all;

```

Restart Nginx container.
You are good to go.
166 changes: 166 additions & 0 deletions examples/kubernetes-deployment-example.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
# This file contains a basic configuration for kubernetes

# DEPLOYMENT
apiVersion: apps/v1
kind: Deployment
metadata:
name: almond-server
# namespace <your namespace>
spec:
selector:
matchLabels:
app: almond-server
role: backend
replicas: 1
template:
metadata:
labels:
app: almond-server
role: backend
spec:
#hostNetwork: true
#dnsPolicy: ClusterFirstWithHostNet
containers:
- name: almond-server
image: stanfordoval/almond-server
env:
- name: TZ
value: Europe/London
- name: THINGENGINE_HOST_BASED_AUTHENTICATION
value: local-ip
Copy link
Contributor

@gcampax gcampax Mar 10, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be proxied-ip and should you set THINGENGINE_HAS_REVERSE_PROXY?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback.
I haven’t tested against the latest master; but I will and investigate your suggestions above.

I am running almond on an arm64 based platform so will need to recompile.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have now build an arm64 image from the latest master branch and tagged it 1.99.0.
However, could you point me in the direction of where, for example; THINGENGINE_HOST_BASED_AUTHENTICATION and THINGENGINE_HAS_REVERSE_PROXY are defined so that I can carry out some further tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are defined/used in config.js

imagePullPolicy: "IfNotPresent"
resources: {}
securityContext:
allowPrivilegeEscalation: true
capabilities: {}
readOnlyRootFilesystem: false
runAsGroup: 1000
runAsUser: 1000
seLinuxOptions: {}
volumeMounts:
- name: run-user-1000
mountPath: /run/pulse
- name: home-user-config-almond-server
mountPath: /var/lib/almond-server
workingDir: /opt/almond
- name: nginx-sidecar
image: nginx:alpine
env:
- name: TZ
value: Europe/London
volumeMounts:
- name: nginx-sidecar-config
mountPath: /etc/nginx/nginx.conf
subPath: nginx.conf
volumes:
- hostPath:
path: /run/user/1000/pulse
type: Directory
name: run-user-1000
- hostPath:
path: /home/pi/.config/almond-server
type: Directory
name: home-user-config-almond-server
- name: nginx-sidecar-config
configMap:
name: nginx-sidecar-conf
status: {}

---

# SERVICE
# Uses nodeport for local access

apiVersion: v1
kind: Service
metadata:
name: almond-service
# namespace <your namespace>
labels:
app: almond-service
spec:
type: NodePort
ports:
- name: almond-web
port: 80
targetPort: 3001
protocol: TCP
nodePort: 32000
selector:
app: almond-server
role: backend
status: {}

---

# CONFIGMAP
# nginx sidecar configmap
# modify the allow section to allow only host you trust; deny all otherwise
# monitor nginx logs for errors: e.g. kubectl logs -n namespace -c nginx-sidecar <pod id>

apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-sidecar-conf
# namespace: <your namespace>
data:
nginx.conf: |-
user nginx;
worker_processes 1;
pid /var/run/nginx.pid;
error_log /dev/stdout info;

events {
worker_connections 1024;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
proxy_read_timeout 1d;
gzip on;
gzip_disable "msie6";

upstream backend {
ip_hash;
server 127.0.0.1:3000;
#server almond-service:3000;
}

map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}

# API & Ingress
server {
listen 3001 default_server;
listen 8099 default_server;
listen [::]:3001 default_server;
listen [::]:8099 default_server;

allow 127.0.0.1; # localhost
deny all;

server_name _;
access_log /dev/stdout combined;

client_max_body_size 4G;
keepalive_timeout 5;

root /dev/null;

location / {
proxy_redirect off;
proxy_pass http://backend;

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host "127.0.0.1:3000";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting this might confuse the logic to compute the right redirect URLs for OAuth. DId you try with any OAuth skill (such as Spotify)?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as previous comment; will investigate and report back.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed setting the header in that way dose confuse the logic because when you try to configure Spotify for example although you are presented with your Spotify account login page, the end result is: ‘ cannot open page because it cannot connet to the server’. More ideas needed.
But isn’t this a restriction on the local almond server or have there been some other changes since 1.8.0?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There has been changes here as we try to make the experience smoother for users of the Home Assistant add-on. I additionally changed how we check for host-based authentication and how the origin check works, in a way that should make this hack less necessary.

proxy_set_header Origin "http://127.0.0.1:3000";
}
}
}