diff --git a/README.md b/README.md index f230ab4..e5195cb 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ -# A Matrix (Synapse) Stack with coturn, bots, bridges and more -A docker-compose stack with Synapse, Postgres, Element-Web, Turn and more +# A Matrix (Synapse) Stack with Traefik, coturn, bots, bridges and more -This is how I was "serving" a small chat server on an Ubuntu virtual machine. +This is a stack in a single `docker-compose.yaml` file. The guide starts by preconfiguring the various services and finally bringing the stack up. The stack follows some specific logic concerning the file organization and a couple "bad practices" (exposing ports and folders) that should not be a problem for a non production environment. @@ -18,9 +17,12 @@ The stack follows some specific logic concerning the file organization and a cou # Assuptions + +
+ ## Domain and subdomains -You should have a locally (at least) resolved domain (During the instructions we will use `domain.ltd`). We also use the following subdomains at various points: +You should have a locally (at least) resolved domain (During the instructions we will use `ms.local`). We also use the following subdomains at various points: - matrix.ms.local - turn.ms.local - webhooks.ms.local @@ -28,25 +30,30 @@ You should have a locally (at least) resolved domain (During the instructions we - maubot.ms.local +
+ ## Certificates The guide assumes you have a wildcard ceritificate for your domain name (`WILDCARD.ms.local`) in `CERT_PATH` folder. ``` -/mnt/ - certs/ - WILDCARD.domain.ltd.crt - WILDCARD.domain.ltd.key +/${CERT_PATH}/ + WILDCARD.ms.local.crt + WILDCARD.ms.local.key ``` +You can genarate a self-signed certificate folowing guide from @cecilemuller: +https://gist.github.com/cecilemuller/9492b848eb8fe46d462abeb26656c4f8 + You can ofcource use diffrent certificates for every service. -_Certificate generation is outside of the scope of this guide, for now._ + +
+ ## Folder hiercacy The docker-compose.yaml file assumes the following hiecrasy: ``` -/BASE_FOLDER/ - configs/ +${CONF_PATH}/ db/ homeserver/ webchat/ @@ -55,10 +62,10 @@ The docker-compose.yaml file assumes the following hiecrasy: facebook-bridge/ webhook-service/ maubot/ - data/ +${DATA_PATH} homeserver_media-store turn - certs/ +${CERT_PATH}/ ``` - `/configs/` : configuration persistent data @@ -66,13 +73,12 @@ The docker-compose.yaml file assumes the following hiecrasy: - `/data/` : other kind of persistent data (like synapse media store etc.) -## Docker volumes and networks - -- Create a docker volume for postgres: `sudo docker volume create db-data` -- The three required networks (`db`,`bots` and `ms`) will be created automatically. If the names overlap with anything already running, you should edit `docker-compose.yaml` +
-# Initialization +
+ +# Initialization and preconfigurations ## Expsose ENV @@ -82,10 +88,36 @@ export DOMAIN=ms.local export CONF_PATH=/mnt/configs ``` -Some of the services need to initialize some config files before you can finally start them. Below are the steps and a reasoning behind them: +Some of the services need to initialize some config files before you can finally start them. + +
+ +## Proxy +1. Create file `traefik-ssl.toml` in `${CONF_PATH}/proxy/` and paste the following: + ``` + [tls] + [tls.stores] + [tls.stores.default] + [tls.stores.default.defaultCertificate] + certFile = "/certs/WILDCARD.ms.local.crt" + keyFile = "/certs/WILDCARD.ms.local.key" + ``` + Change the file name of the certificate if you have to. + +
+ +## Prostgres + +- Create a docker volume: `sudo docker volume create db-data` +- Change the values in `db.env` to your liking. You should at least change `POSTGRES_PASSWORD=` + +
## Synapse -Use the following command to generate a `homeserver.yaml` file in `${CONF_PATH}/homeserver/`. __IMPORTANT: the subdomain (`matrix.${DOMAIN}`) CANNOT be changed later. Make sure you have decided correctly.__ +Generate a `homeserver.yaml` file in `${CONF_PATH}/homeserver/`. You can find a sample config at `sample_configs/homeserver/homeserver.yaml` + + + __IMPORTANT: the subdomain (`matrix.${DOMAIN}`) CANNOT be changed later. Make sure you have decided correctly.__ ``` sudo docker run -it --rm \ @@ -94,14 +126,13 @@ sudo docker run -it --rm \ -e SYNAPSE_REPORT_STATS=yes \ matrixdotorg/synapse:latest generate ``` -After that you can edit the file however you want. Some important fields are: - -- `server_name` will be autofilled +Edit/Uncomment some important fields: +- `server_name` will be autofilled ``` server_name: "matrix.ms.local" ``` -- We add an https listener for secure connections, bind it to all addresses and enable federation. +- Add an https listener for secure connections, bind it to all addresses and enable federation. ``` listeners: - port: 8448 @@ -134,7 +165,7 @@ database: name: psycopg2 args: user: synapse - password: 12345 + password: database: synapse_db host: db cp_min: 5 @@ -152,213 +183,224 @@ tls_certificate_path: "/certs/WILDCARD.ms.local.crt" tls_private_key_path: "/certs/WILDCARD.ms.local.key" ``` -- Finally, enable registrations +- Enable registrations ``` enable_registration: true ``` - +- Enable user directory search. (This will help us find the bot accounts later) +``` +user_directory: + enabled: true + search_all_users: true + prefer_local_users: true +``` - Save the file (_We will edit more while configuring Turn, Bridges and Bots_) +
+ +
+ + ## Bridges and Bots +
+ ### Telegram Brige _Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram -1. Run: -``` -sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest -``` -This will generate a `config.yaml` that you should edit. +1. Run the command to generate a `config.yaml`: + ``` + sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest + ``` + -2. You need to set at least the following: -- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_) -``` -homeserver: - # The address that this appservice can use to connect to the homeserver. - address: https://homeserver:8448 - # The domain of the homeserver (for MXIDs, etc). - domain: matrix.ms.local - # Whether or not to verify the SSL certificate of the homeserver. - # Only applies if address starts with https:// - verify_ssl: false +2. Edit the file (reference `sample_configs/telegram-bridge/config.yaml`): + - Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_) + ``` + homeserver: + # The address that this appservice can use to connect to the homeserver. + address: https://homeserver:8448 + # The domain of the homeserver (for MXIDs, etc). + domain: matrix.ms.local + # Whether or not to verify the SSL certificate of the homeserver. + # Only applies if address starts with https:// + verify_ssl: false -appservice: - # The address that the homeserver can use to connect to this appservice. - address: http://telegram-bridge:29317 - database: sqlite:////data/telegram-bridge.db -``` -- Bridge permissions + appservice: + # The address that the homeserver can use to connect to this appservice. + address: http://telegram-bridge:29317 + database: sqlite:////data/telegram-bridge.db + ``` + - Bridge permissions -We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following: -``` -permissions: - "*": relaybot - "matrix.ms.local": admin -``` -- Telegram API key -``` -telegram: - # Get your own API keys at https://my.telegram.org/apps - api_id: 12345 - api_hash: tjyd5yge35lbodk1xwzw2jstp90k55qz + We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following: + ``` + permissions: + "*": relaybot + "matrix.ms.local": admin + ``` + - Telegram API key + ``` + telegram: + # Get your own API keys at https://my.telegram.org/apps + api_id: 12345 + api_hash: tjyd5yge35lbodk1xwzw2jstp90k55qz -``` + ``` 3. Run the docker command again to generate a 'registration.yaml' -``` -sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest -``` - The `registration.yaml` file is mounted on the `homeserver` cotainer. + ``` + sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest + ``` + The `registration.yaml` file is mounted on the `homeserver` cotainer. + +
### Facebook Bridge (Almost identical to Telegram bridge) _Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook -Run: -``` - docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest -``` -This will generate a `config.yaml` that you should edit. You need to set at least the following: -- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_) +1. Run the command to generate a `config.yaml`: + ``` + sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest + ``` + -``` -homeserver: - # The address that this appservice can use to connect to the homeserver. - address: https://homeserver:8448 - # The domain of the homeserver (for MXIDs, etc). - domain: matrix.ms.local - # Whether or not to verify the SSL certificate of the homeserver. - # Only applies if address starts with https:// - verify_ssl: false +2. Edit the file (reference `sample_configs/facebookm-bridge/config.yaml`): + - Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_) + ``` + homeserver: + # The address that this appservice can use to connect to the homeserver. + address: https://homeserver:8448 + # The domain of the homeserver (for MXIDs, etc). + domain: matrix.ms.local + # Whether or not to verify the SSL certificate of the homeserver. + # Only applies if address starts with https:// + verify_ssl: false -appservice: - # The address that the homeserver can use to connect to this appservice. - address: http://facebook-bridge:29317 - database: sqlite:////data/facebook-bridge.db -``` -- Bridge permissions + appservice: + # The address that the homeserver can use to connect to this appservice. + address: http://facebook-bridge:29317 + database: sqlite:////data/facebook-bridge.db + ``` + - Bridge permissions -We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following: -``` -permissions: + We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following: + ``` + permissions: "*": "relay" "matrix.ms.local": "admin" -``` + ``` +3. Run the docker command again to generate a 'registration.yaml' + ``` + sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest + ``` + The `registration.yaml` file is mounted on the `homeserver` cotainer. + + +
-Run the docker command again to generate a 'registration.yaml' -``` -sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest -``` - - - The `registration.yaml` file is mounted on the `homeserver` cotainer. - ### Webhook App Service Source: https://github.com/turt2live/matrix-appservice-webhooks#docker 1. Create an `appservice-registration-webhooks.yaml` file in `${CONF_PATH}/webhooks` and copy the following (make sure you generate `hs_token` and `as_token`): -``` -id: webhooks -hs_token: A_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS -as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS -namespaces: - users: - - exclusive: true - regex: '@_webhook.*' -url: 'http://webhook-service:9000' -sender_localpart: webhooks -rate_limited: false -``` + ``` + id: webhooks + hs_token: A_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS + as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS + namespaces: + users: + - exclusive: true + regex: '@_webhook.*' + url: 'http://webhook-service:9000' + sender_localpart: webhooks + rate_limited: false + ``` 2. Create an `config.yaml` file in `${CONF_PATH}/webhooks` and copy/edit the following: -``` -# Configuration specific to the application service. All fields (unless otherwise marked) are required. -homeserver: - # The domain for the client-server API calls. - url: "http://homeserver:8008" + ``` + homeserver: + url: "http://homeserver:8008" + domain: "matrix.ms.local" - # The domain part for user IDs on this home server. Usually, but not always, this is the same as the - # home server's URL. - domain: "matrix.ms.local" + webhookBot: + localpart: "webhooks" + appearance: + displayName: "Webhook Bridge" + avatarUrl: "http://i.imgur.com/IDOBtEJ.png" -# Configuration specific to the bridge. All fields (unless otherwise marked) are required. -webhookBot: - # The localpart to use for the bot. May require re-registering the application service. - localpart: "webhooks" + provisioning: + secret: 'CHANGE_ME' - # Appearance options for the Matrix bot - appearance: - displayName: "Webhook Bridge" - avatarUrl: "http://i.imgur.com/IDOBtEJ.png" # webhook icon + web: + hookUrlBase: 'https://webhooks.ms.local' -# Provisioning API options -provisioning: - # Your secret for the API. Required for all provisioning API requests. - secret: 'CHANGE_ME' - -# Configuration related to the web portion of the bridge. Handles the inbound webhooks -web: - hookUrlBase: 'https://webhooks.domain.ltd' - -logging: - file: logs/webhook.log - console: true - consoleLevel: debug - fileLevel: verbose - writeFiles: true - rotate: - size: 52428800 # bytes, default is 50mb - count: 5 - -``` + logging: + file: logs/webhook.log + console: true + consoleLevel: debug + fileLevel: verbose + writeFiles: true + rotate: + size: 52428800 # bytes, default is 50mb + count: 5 + ``` 3. Create a `database.json` file in `${CONF_PATH}/webhooks` and copy the following: -``` + ``` + { + "defaultEnv": { + "ENV": "NODE_ENV" + }, + "development": { + "driver": "sqlite3", + "filename": "/data/development.db" + }, + "production": { + "driver": "sqlite3", + "filename": "/data/production.db" + } + } + ``` -{ - "defaultEnv": { - "ENV": "NODE_ENV" - }, - "development": { - "driver": "sqlite3", - "filename": "/data/development.db" - }, - "production": { - "driver": "sqlite3", - "filename": "/data/production.db" - } -} +4. Run the command to check for errors: + ``` + sudo docker run --rm -v ${CONF_PATH}/webhooks:/data turt2live/matrix-appservice-webhooks + ``` + _If you get an `[ERROR] ConnectionError: request failed: getaddrinfo ENOTFOUND homeserver homeserver:8008`, this is normal since we don't have a working homeserver yet._ -``` - -4. Run: -``` -sudo docker run --rm -v ${CONF_PATH}/webhooks:/data turt2live/matrix-appservice-webhooks -``` -Check the logs for any errors. If you get an `[ERROR] ConnectionError: request failed: getaddrinfo ENOTFOUND homeserver homeserver:8008`, this is normal since we don't have a working homeserver yet. +
### Maubot Manager _Source_: https://docs.mau.fi/maubot/usage/setup/docker.html -1. Run: -``` -sudo docker run --rm -v ${CONF_PATH}/maubot:/data:z dock.mau.dev/maubot/maubot:latest -``` +1. Run the command to generate a `config.yaml`: + ``` + sudo docker run --rm -v ${CONF_PATH}/maubot:/data:z dock.mau.dev/maubot/maubot:latest + ``` + -This will generate a `config.yaml` file. +2. Update the file to add your homeserver: + ``` + homeservers: + matrix.ms.local + url: https://homeserver:8448 + secret: + ``` -2. Update the file to your liking. You should at least add your homeserver: -``` -homeservers: - matrix.ms.local - url: https://homeserver:8448 - secret: +3. Create an admin user + ``` + admins: + root: '' + admin: '12345' #use a password you like + ``` +4. Save the file -``` -3. Save the file + +
### Registering the new services to the home server: @@ -371,6 +413,8 @@ app_service_config_files: ``` (in the docker-compose file we have mounted each file in the `homeserver` container) +
+ ## Turn server (for audio and video calls) Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the sample file from: https://github.com/coturn/coturn/blob/master/docker/coturn/turnserver.conf @@ -378,42 +422,94 @@ Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the Edit the following in the file: - Specify and external ip -``` -external-ip= -external-ip= -``` + ``` + external-ip= + external-ip= + ``` - Specify a port range -``` -min-port=64000 -max-port=65535 -``` -This range worked perfectly for me but you should define your own depending on your network setup + ``` + min-port=64000 + max-port=65535 + ``` + This range worked perfectly for me but you should define your own depending on your network setup - Certificates: -``` -cert=/certs/WILDCARD.ms.local.crt -pkey=/certs/WILDCARD.ms.local.key -``` + ``` + cert=/certs/WILDCARD.ms.local.crt + pkey=/certs/WILDCARD.ms.local.key + ``` - Define a realm -``` -realm=turn.domain.ltd -``` + ``` + realm=turn.ms.local + ``` - Uncomment `use-auth-secret`. Generate a alphanumeric and fill `static-auth-secret=`. -- In `homeserver.yaml` in `##TURN##` secrion paste the same alpanumeric at `turn_shared_secret: "ALPHANUMERIC"` and add the following `turn_uris` -``` -turn_uris: - - "turn:turn.domain.ltd?transport=udp" - - "turn:turn.domain.ltd?transport=tcp" - - "turns:turn.domain.ltd:5349?transport=udp" - - "turns:turn.domain.ltd:5349?transport=tcp" -``` +- In `homeserver.yaml` in `##TURN##` section paste the same alpanumeric at `turn_shared_secret: "ALPHANUMERIC"` and add the following `turn_uris` + ``` + turn_uris: + - "turn:turn.ms.local?transport=udp" + - "turn:turn.ms.local?transport=tcp" + - "turns:turn.ms.local:5349?transport=udp" + - "turns:turn.ms.local:5349?transport=tcp" + ``` +
+
# Bringing up the Chat Server -If everything is correctly initialized we can bring up the stack with `sudo docker-compose up` +If everything is correctly initialized we can bring up the stack with `sudo docker-compose up`.
+After a while we should be able to visit the web element UI at `https://webchat.${DOMAIN}`, and register a new user. -After a while we should be able to visit the web element UI at `http://:10000`, and register a new user. +
+ +
+ +# Final Notes + +- This is by __no means__ a production ready setup. Some of the things that should be changed are: + - Diffrent certificates for every service (plus for the bots) + - Postgres for the bridges databases + - No `--serverstransport.insecureskipverify=true` in traefik commands + - Use `secrets` for sensitive information +- Turn server runs in host network mode. This is needed to better communicate with UDP. This means that your devices and containers (especially `homeserver`) should be able to resolve `turn.ms.local` to you docker host IP +- There are some more things to setup for the homeserver, bots and bridges. Please refer to their respective documentations. + + +# Disclaimer + +It goes without saying that I'm not responsible for anything that might go wrong. __BUT__ I will be more than happy to help in any situation. If you have any suggestions on how this guide can be better (I'm sure there are a lot), please feel free to contact me! + +# Sources and links + +- Synapse + - Github: @matrix-org | https://github.com/matrix-org/synapse + - Documentation: https://matrix-org.github.io/synapse/latest/ + - Docker image: https://hub.docker.com/r/matrixdotorg/synapse/ +- Postgres + - Github: @postgres | https://github.com/postgres/postgres + - Documentation: https://www.postgresql.org/docs/current/ + - Docker image: https://hub.docker.com/_/postgres/ +- Coturn + - Github: @coturn | https://github.com/coturn/coturn + - Documantation: https://github.com/coturn/coturn/wiki/ + - Docker image: https://hub.docker.com/r/instrumentisto/coturn/ +- Element.io Web + - Github: @vector-im + - Docker image: https://hub.docker.com/r/vectorim/element-web/ +- Synapse Admin + - Github: @Awesome-Technologies | https://github.com/Awesome-Technologies/synapse-admin + - Docker image: https://hub.docker.com/r/awesometechnologies/synapse-admin +- Telegram Bridge + - Github: @mautrix | https://github.com/mautrix/telegram + - Documentation: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram +- Facebook Bridge + - Github: @mautrix | https://github.com/mautrix/facebook + - Documentation: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook +- Maubot Manager + - Github: @maubot | https://github.com/maubot/maubot + - Documentation: https://docs.mau.fi/maubot/usage/setup/docker.html +- Webhook Appservice + - Github: @turt2live | https://github.com/turt2live/matrix-appservice-webhooks