From e7def6633a92fd9aec3ca8afe4728efd5e2ff199 Mon Sep 17 00:00:00 2001
From: Alexios Mavronas <73764054+twicechild@users.noreply.github.com>
Date: Thu, 17 Feb 2022 15:37:28 +0200
Subject: [PATCH] Full README
---
README.md | 510 ++++++++++++++++++++++++++++++++----------------------
1 file changed, 303 insertions(+), 207 deletions(-)
diff --git a/README.md b/README.md
index f230ab4..e5195cb 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,6 @@
-# A Matrix (Synapse) Stack with coturn, bots, bridges and more
-A docker-compose stack with Synapse, Postgres, Element-Web, Turn and more
+# A Matrix (Synapse) Stack with Traefik, coturn, bots, bridges and more
-This is how I was "serving" a small chat server on an Ubuntu virtual machine.
+This is a stack in a single `docker-compose.yaml` file. The guide starts by preconfiguring the various services and finally bringing the stack up.
The stack follows some specific logic concerning the file organization and a couple "bad practices" (exposing ports and folders) that should not be a problem for a non production environment.
@@ -18,9 +17,12 @@ The stack follows some specific logic concerning the file organization and a cou
# Assuptions
+
+
+
## Domain and subdomains
-You should have a locally (at least) resolved domain (During the instructions we will use `domain.ltd`). We also use the following subdomains at various points:
+You should have a locally (at least) resolved domain (During the instructions we will use `ms.local`). We also use the following subdomains at various points:
- matrix.ms.local
- turn.ms.local
- webhooks.ms.local
@@ -28,25 +30,30 @@ You should have a locally (at least) resolved domain (During the instructions we
- maubot.ms.local
+
+
## Certificates
The guide assumes you have a wildcard ceritificate for your domain name (`WILDCARD.ms.local`) in `CERT_PATH` folder.
```
-/mnt/
- certs/
- WILDCARD.domain.ltd.crt
- WILDCARD.domain.ltd.key
+/${CERT_PATH}/
+ WILDCARD.ms.local.crt
+ WILDCARD.ms.local.key
```
+You can genarate a self-signed certificate folowing guide from @cecilemuller:
+https://gist.github.com/cecilemuller/9492b848eb8fe46d462abeb26656c4f8
+
You can ofcource use diffrent certificates for every service.
-_Certificate generation is outside of the scope of this guide, for now._
+
+
+
## Folder hiercacy
The docker-compose.yaml file assumes the following hiecrasy:
```
-/BASE_FOLDER/
- configs/
+${CONF_PATH}/
db/
homeserver/
webchat/
@@ -55,10 +62,10 @@ The docker-compose.yaml file assumes the following hiecrasy:
facebook-bridge/
webhook-service/
maubot/
- data/
+${DATA_PATH}
homeserver_media-store
turn
- certs/
+${CERT_PATH}/
```
- `/configs/` : configuration persistent data
@@ -66,13 +73,12 @@ The docker-compose.yaml file assumes the following hiecrasy:
- `/data/` : other kind of persistent data (like synapse media store etc.)
-## Docker volumes and networks
-
-- Create a docker volume for postgres: `sudo docker volume create db-data`
-- The three required networks (`db`,`bots` and `ms`) will be created automatically. If the names overlap with anything already running, you should edit `docker-compose.yaml`
+
-# Initialization
+
+
+# Initialization and preconfigurations
## Expsose ENV
@@ -82,10 +88,36 @@ export DOMAIN=ms.local
export CONF_PATH=/mnt/configs
```
-Some of the services need to initialize some config files before you can finally start them. Below are the steps and a reasoning behind them:
+Some of the services need to initialize some config files before you can finally start them.
+
+
+
+## Proxy
+1. Create file `traefik-ssl.toml` in `${CONF_PATH}/proxy/` and paste the following:
+ ```
+ [tls]
+ [tls.stores]
+ [tls.stores.default]
+ [tls.stores.default.defaultCertificate]
+ certFile = "/certs/WILDCARD.ms.local.crt"
+ keyFile = "/certs/WILDCARD.ms.local.key"
+ ```
+ Change the file name of the certificate if you have to.
+
+
+
+## Prostgres
+
+- Create a docker volume: `sudo docker volume create db-data`
+- Change the values in `db.env` to your liking. You should at least change `POSTGRES_PASSWORD=`
+
+
## Synapse
-Use the following command to generate a `homeserver.yaml` file in `${CONF_PATH}/homeserver/`. __IMPORTANT: the subdomain (`matrix.${DOMAIN}`) CANNOT be changed later. Make sure you have decided correctly.__
+Generate a `homeserver.yaml` file in `${CONF_PATH}/homeserver/`. You can find a sample config at `sample_configs/homeserver/homeserver.yaml`
+
+
+ __IMPORTANT: the subdomain (`matrix.${DOMAIN}`) CANNOT be changed later. Make sure you have decided correctly.__
```
sudo docker run -it --rm \
@@ -94,14 +126,13 @@ sudo docker run -it --rm \
-e SYNAPSE_REPORT_STATS=yes \
matrixdotorg/synapse:latest generate
```
-After that you can edit the file however you want. Some important fields are:
-
-- `server_name` will be autofilled
+Edit/Uncomment some important fields:
+- `server_name` will be autofilled
```
server_name: "matrix.ms.local"
```
-- We add an https listener for secure connections, bind it to all addresses and enable federation.
+- Add an https listener for secure connections, bind it to all addresses and enable federation.
```
listeners:
- port: 8448
@@ -134,7 +165,7 @@ database:
name: psycopg2
args:
user: synapse
- password: 12345
+ password:
database: synapse_db
host: db
cp_min: 5
@@ -152,213 +183,224 @@ tls_certificate_path: "/certs/WILDCARD.ms.local.crt"
tls_private_key_path: "/certs/WILDCARD.ms.local.key"
```
-- Finally, enable registrations
+- Enable registrations
```
enable_registration: true
```
-
+- Enable user directory search. (This will help us find the bot accounts later)
+```
+user_directory:
+ enabled: true
+ search_all_users: true
+ prefer_local_users: true
+```
- Save the file (_We will edit more while configuring Turn, Bridges and Bots_)
+
+
+
+
+
## Bridges and Bots
+
+
### Telegram Brige
_Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram
-1. Run:
-```
-sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest
-```
-This will generate a `config.yaml` that you should edit.
+1. Run the command to generate a `config.yaml`:
+ ```
+ sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest
+ ```
+
-2. You need to set at least the following:
-- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
-```
-homeserver:
- # The address that this appservice can use to connect to the homeserver.
- address: https://homeserver:8448
- # The domain of the homeserver (for MXIDs, etc).
- domain: matrix.ms.local
- # Whether or not to verify the SSL certificate of the homeserver.
- # Only applies if address starts with https://
- verify_ssl: false
+2. Edit the file (reference `sample_configs/telegram-bridge/config.yaml`):
+ - Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
+ ```
+ homeserver:
+ # The address that this appservice can use to connect to the homeserver.
+ address: https://homeserver:8448
+ # The domain of the homeserver (for MXIDs, etc).
+ domain: matrix.ms.local
+ # Whether or not to verify the SSL certificate of the homeserver.
+ # Only applies if address starts with https://
+ verify_ssl: false
-appservice:
- # The address that the homeserver can use to connect to this appservice.
- address: http://telegram-bridge:29317
- database: sqlite:////data/telegram-bridge.db
-```
-- Bridge permissions
+ appservice:
+ # The address that the homeserver can use to connect to this appservice.
+ address: http://telegram-bridge:29317
+ database: sqlite:////data/telegram-bridge.db
+ ```
+ - Bridge permissions
-We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following:
-```
-permissions:
- "*": relaybot
- "matrix.ms.local": admin
-```
-- Telegram API key
-```
-telegram:
- # Get your own API keys at https://my.telegram.org/apps
- api_id: 12345
- api_hash: tjyd5yge35lbodk1xwzw2jstp90k55qz
+ We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following:
+ ```
+ permissions:
+ "*": relaybot
+ "matrix.ms.local": admin
+ ```
+ - Telegram API key
+ ```
+ telegram:
+ # Get your own API keys at https://my.telegram.org/apps
+ api_id: 12345
+ api_hash: tjyd5yge35lbodk1xwzw2jstp90k55qz
-```
+ ```
3. Run the docker command again to generate a 'registration.yaml'
-```
-sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest
-```
- The `registration.yaml` file is mounted on the `homeserver` cotainer.
+ ```
+ sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest
+ ```
+ The `registration.yaml` file is mounted on the `homeserver` cotainer.
+
+
### Facebook Bridge (Almost identical to Telegram bridge)
_Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook
-Run:
-```
- docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
-```
-This will generate a `config.yaml` that you should edit. You need to set at least the following:
-- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
+1. Run the command to generate a `config.yaml`:
+ ```
+ sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
+ ```
+
-```
-homeserver:
- # The address that this appservice can use to connect to the homeserver.
- address: https://homeserver:8448
- # The domain of the homeserver (for MXIDs, etc).
- domain: matrix.ms.local
- # Whether or not to verify the SSL certificate of the homeserver.
- # Only applies if address starts with https://
- verify_ssl: false
+2. Edit the file (reference `sample_configs/facebookm-bridge/config.yaml`):
+ - Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
+ ```
+ homeserver:
+ # The address that this appservice can use to connect to the homeserver.
+ address: https://homeserver:8448
+ # The domain of the homeserver (for MXIDs, etc).
+ domain: matrix.ms.local
+ # Whether or not to verify the SSL certificate of the homeserver.
+ # Only applies if address starts with https://
+ verify_ssl: false
-appservice:
- # The address that the homeserver can use to connect to this appservice.
- address: http://facebook-bridge:29317
- database: sqlite:////data/facebook-bridge.db
-```
-- Bridge permissions
+ appservice:
+ # The address that the homeserver can use to connect to this appservice.
+ address: http://facebook-bridge:29317
+ database: sqlite:////data/facebook-bridge.db
+ ```
+ - Bridge permissions
-We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following:
-```
-permissions:
+ We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following:
+ ```
+ permissions:
"*": "relay"
"matrix.ms.local": "admin"
-```
+ ```
+3. Run the docker command again to generate a 'registration.yaml'
+ ```
+ sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
+ ```
+ The `registration.yaml` file is mounted on the `homeserver` cotainer.
+
+
+
-Run the docker command again to generate a 'registration.yaml'
-```
-sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
-```
-
-
- The `registration.yaml` file is mounted on the `homeserver` cotainer.
-
### Webhook App Service
Source: https://github.com/turt2live/matrix-appservice-webhooks#docker
1. Create an `appservice-registration-webhooks.yaml` file in `${CONF_PATH}/webhooks` and copy the following (make sure you generate `hs_token` and `as_token`):
-```
-id: webhooks
-hs_token: A_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS
-as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS
-namespaces:
- users:
- - exclusive: true
- regex: '@_webhook.*'
-url: 'http://webhook-service:9000'
-sender_localpart: webhooks
-rate_limited: false
-```
+ ```
+ id: webhooks
+ hs_token: A_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS
+ as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS
+ namespaces:
+ users:
+ - exclusive: true
+ regex: '@_webhook.*'
+ url: 'http://webhook-service:9000'
+ sender_localpart: webhooks
+ rate_limited: false
+ ```
2. Create an `config.yaml` file in `${CONF_PATH}/webhooks` and copy/edit the following:
-```
-# Configuration specific to the application service. All fields (unless otherwise marked) are required.
-homeserver:
- # The domain for the client-server API calls.
- url: "http://homeserver:8008"
+ ```
+ homeserver:
+ url: "http://homeserver:8008"
+ domain: "matrix.ms.local"
- # The domain part for user IDs on this home server. Usually, but not always, this is the same as the
- # home server's URL.
- domain: "matrix.ms.local"
+ webhookBot:
+ localpart: "webhooks"
+ appearance:
+ displayName: "Webhook Bridge"
+ avatarUrl: "http://i.imgur.com/IDOBtEJ.png"
-# Configuration specific to the bridge. All fields (unless otherwise marked) are required.
-webhookBot:
- # The localpart to use for the bot. May require re-registering the application service.
- localpart: "webhooks"
+ provisioning:
+ secret: 'CHANGE_ME'
- # Appearance options for the Matrix bot
- appearance:
- displayName: "Webhook Bridge"
- avatarUrl: "http://i.imgur.com/IDOBtEJ.png" # webhook icon
+ web:
+ hookUrlBase: 'https://webhooks.ms.local'
-# Provisioning API options
-provisioning:
- # Your secret for the API. Required for all provisioning API requests.
- secret: 'CHANGE_ME'
-
-# Configuration related to the web portion of the bridge. Handles the inbound webhooks
-web:
- hookUrlBase: 'https://webhooks.domain.ltd'
-
-logging:
- file: logs/webhook.log
- console: true
- consoleLevel: debug
- fileLevel: verbose
- writeFiles: true
- rotate:
- size: 52428800 # bytes, default is 50mb
- count: 5
-
-```
+ logging:
+ file: logs/webhook.log
+ console: true
+ consoleLevel: debug
+ fileLevel: verbose
+ writeFiles: true
+ rotate:
+ size: 52428800 # bytes, default is 50mb
+ count: 5
+ ```
3. Create a `database.json` file in `${CONF_PATH}/webhooks` and copy the following:
-```
+ ```
+ {
+ "defaultEnv": {
+ "ENV": "NODE_ENV"
+ },
+ "development": {
+ "driver": "sqlite3",
+ "filename": "/data/development.db"
+ },
+ "production": {
+ "driver": "sqlite3",
+ "filename": "/data/production.db"
+ }
+ }
+ ```
-{
- "defaultEnv": {
- "ENV": "NODE_ENV"
- },
- "development": {
- "driver": "sqlite3",
- "filename": "/data/development.db"
- },
- "production": {
- "driver": "sqlite3",
- "filename": "/data/production.db"
- }
-}
+4. Run the command to check for errors:
+ ```
+ sudo docker run --rm -v ${CONF_PATH}/webhooks:/data turt2live/matrix-appservice-webhooks
+ ```
+ _If you get an `[ERROR] ConnectionError: request failed: getaddrinfo ENOTFOUND homeserver homeserver:8008`, this is normal since we don't have a working homeserver yet._
-```
-
-4. Run:
-```
-sudo docker run --rm -v ${CONF_PATH}/webhooks:/data turt2live/matrix-appservice-webhooks
-```
-Check the logs for any errors. If you get an `[ERROR] ConnectionError: request failed: getaddrinfo ENOTFOUND homeserver homeserver:8008`, this is normal since we don't have a working homeserver yet.
+
### Maubot Manager
_Source_: https://docs.mau.fi/maubot/usage/setup/docker.html
-1. Run:
-```
-sudo docker run --rm -v ${CONF_PATH}/maubot:/data:z dock.mau.dev/maubot/maubot:latest
-```
+1. Run the command to generate a `config.yaml`:
+ ```
+ sudo docker run --rm -v ${CONF_PATH}/maubot:/data:z dock.mau.dev/maubot/maubot:latest
+ ```
+
-This will generate a `config.yaml` file.
+2. Update the file to add your homeserver:
+ ```
+ homeservers:
+ matrix.ms.local
+ url: https://homeserver:8448
+ secret:
+ ```
-2. Update the file to your liking. You should at least add your homeserver:
-```
-homeservers:
- matrix.ms.local
- url: https://homeserver:8448
- secret:
+3. Create an admin user
+ ```
+ admins:
+ root: ''
+ admin: '12345' #use a password you like
+ ```
+4. Save the file
-```
-3. Save the file
+
+
### Registering the new services to the home server:
@@ -371,6 +413,8 @@ app_service_config_files:
```
(in the docker-compose file we have mounted each file in the `homeserver` container)
+
+
## Turn server (for audio and video calls)
Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the sample file from: https://github.com/coturn/coturn/blob/master/docker/coturn/turnserver.conf
@@ -378,42 +422,94 @@ Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the
Edit the following in the file:
- Specify and external ip
-```
-external-ip=
-external-ip=
-```
+ ```
+ external-ip=
+ external-ip=
+ ```
- Specify a port range
-```
-min-port=64000
-max-port=65535
-```
-This range worked perfectly for me but you should define your own depending on your network setup
+ ```
+ min-port=64000
+ max-port=65535
+ ```
+ This range worked perfectly for me but you should define your own depending on your network setup
- Certificates:
-```
-cert=/certs/WILDCARD.ms.local.crt
-pkey=/certs/WILDCARD.ms.local.key
-```
+ ```
+ cert=/certs/WILDCARD.ms.local.crt
+ pkey=/certs/WILDCARD.ms.local.key
+ ```
- Define a realm
-```
-realm=turn.domain.ltd
-```
+ ```
+ realm=turn.ms.local
+ ```
- Uncomment `use-auth-secret`. Generate a alphanumeric and fill `static-auth-secret=`.
-- In `homeserver.yaml` in `##TURN##` secrion paste the same alpanumeric at `turn_shared_secret: "ALPHANUMERIC"` and add the following `turn_uris`
-```
-turn_uris:
- - "turn:turn.domain.ltd?transport=udp"
- - "turn:turn.domain.ltd?transport=tcp"
- - "turns:turn.domain.ltd:5349?transport=udp"
- - "turns:turn.domain.ltd:5349?transport=tcp"
-```
+- In `homeserver.yaml` in `##TURN##` section paste the same alpanumeric at `turn_shared_secret: "ALPHANUMERIC"` and add the following `turn_uris`
+ ```
+ turn_uris:
+ - "turn:turn.ms.local?transport=udp"
+ - "turn:turn.ms.local?transport=tcp"
+ - "turns:turn.ms.local:5349?transport=udp"
+ - "turns:turn.ms.local:5349?transport=tcp"
+ ```
+
+
# Bringing up the Chat Server
-If everything is correctly initialized we can bring up the stack with `sudo docker-compose up`
+If everything is correctly initialized we can bring up the stack with `sudo docker-compose up`.
+After a while we should be able to visit the web element UI at `https://webchat.${DOMAIN}`, and register a new user.
-After a while we should be able to visit the web element UI at `http://:10000`, and register a new user.
+
+
+
+
+# Final Notes
+
+- This is by __no means__ a production ready setup. Some of the things that should be changed are:
+ - Diffrent certificates for every service (plus for the bots)
+ - Postgres for the bridges databases
+ - No `--serverstransport.insecureskipverify=true` in traefik commands
+ - Use `secrets` for sensitive information
+- Turn server runs in host network mode. This is needed to better communicate with UDP. This means that your devices and containers (especially `homeserver`) should be able to resolve `turn.ms.local` to you docker host IP
+- There are some more things to setup for the homeserver, bots and bridges. Please refer to their respective documentations.
+
+
+# Disclaimer
+
+It goes without saying that I'm not responsible for anything that might go wrong. __BUT__ I will be more than happy to help in any situation. If you have any suggestions on how this guide can be better (I'm sure there are a lot), please feel free to contact me!
+
+# Sources and links
+
+- Synapse
+ - Github: @matrix-org | https://github.com/matrix-org/synapse
+ - Documentation: https://matrix-org.github.io/synapse/latest/
+ - Docker image: https://hub.docker.com/r/matrixdotorg/synapse/
+- Postgres
+ - Github: @postgres | https://github.com/postgres/postgres
+ - Documentation: https://www.postgresql.org/docs/current/
+ - Docker image: https://hub.docker.com/_/postgres/
+- Coturn
+ - Github: @coturn | https://github.com/coturn/coturn
+ - Documantation: https://github.com/coturn/coturn/wiki/
+ - Docker image: https://hub.docker.com/r/instrumentisto/coturn/
+- Element.io Web
+ - Github: @vector-im
+ - Docker image: https://hub.docker.com/r/vectorim/element-web/
+- Synapse Admin
+ - Github: @Awesome-Technologies | https://github.com/Awesome-Technologies/synapse-admin
+ - Docker image: https://hub.docker.com/r/awesometechnologies/synapse-admin
+- Telegram Bridge
+ - Github: @mautrix | https://github.com/mautrix/telegram
+ - Documentation: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram
+- Facebook Bridge
+ - Github: @mautrix | https://github.com/mautrix/facebook
+ - Documentation: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook
+- Maubot Manager
+ - Github: @maubot | https://github.com/maubot/maubot
+ - Documentation: https://docs.mau.fi/maubot/usage/setup/docker.html
+- Webhook Appservice
+ - Github: @turt2live | https://github.com/turt2live/matrix-appservice-webhooks