netshade/matrix-synapse-stack
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Full README

Browse Source
This commit is contained in:
Alexios Mavronas 2022-02-17 15:37:28 +02:00 committed by GitHub
parent 433aa146da
commit e7def6633a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 303 additions and 207 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

394
README.md
View File

@ -1,7 +1,6 @@
# A Matrix (Synapse) Stack with coturn, bots, bridges and more # A Matrix (Synapse) Stack with Traefik, coturn, bots, bridges and more
A docker-compose stack with Synapse, Postgres, Element-Web, Turn and more
This is how I was "serving" a small chat server on an Ubuntu virtual machine. This is a stack in a single `docker-compose.yaml` file. The guide starts by preconfiguring the various services and finally bringing the stack up.
The stack follows some specific logic concerning the file organization and a couple "bad practices" (exposing ports and folders) that should not be a problem for a non production environment. The stack follows some specific logic concerning the file organization and a couple "bad practices" (exposing ports and folders) that should not be a problem for a non production environment.
@ -18,9 +17,12 @@ The stack follows some specific logic concerning the file organization and a cou
# Assuptions # Assuptions
<br/>
## Domain and subdomains ## Domain and subdomains
You should have a locally (at least) resolved domain (During the instructions we will use `domain.ltd`). We also use the following subdomains at various points: You should have a locally (at least) resolved domain (During the instructions we will use `ms.local`). We also use the following subdomains at various points:
- matrix.ms.local - matrix.ms.local
- turn.ms.local - turn.ms.local
- webhooks.ms.local - webhooks.ms.local
@ -28,25 +30,30 @@ You should have a locally (at least) resolved domain (During the instructions we
- maubot.ms.local - maubot.ms.local
<br/>
## Certificates ## Certificates
The guide assumes you have a wildcard ceritificate for your domain name (`WILDCARD.ms.local`) in `CERT_PATH` folder. The guide assumes you have a wildcard ceritificate for your domain name (`WILDCARD.ms.local`) in `CERT_PATH` folder.
``` ```
/mnt/ /${CERT_PATH}/
certs/ WILDCARD.ms.local.crt
WILDCARD.domain.ltd.crt WILDCARD.ms.local.key
WILDCARD.domain.ltd.key
``` ```
You can genarate a self-signed certificate folowing guide from @cecilemuller:
https://gist.github.com/cecilemuller/9492b848eb8fe46d462abeb26656c4f8
You can ofcource use diffrent certificates for every service. You can ofcource use diffrent certificates for every service.
_Certificate generation is outside of the scope of this guide, for now._
<br/>
## Folder hiercacy ## Folder hiercacy
The docker-compose.yaml file assumes the following hiecrasy: The docker-compose.yaml file assumes the following hiecrasy:
``` ```
/BASE_FOLDER/ ${CONF_PATH}/
configs/
db/ db/
homeserver/ homeserver/
webchat/ webchat/
@ -55,10 +62,10 @@ The docker-compose.yaml file assumes the following hiecrasy:
facebook-bridge/ facebook-bridge/
webhook-service/ webhook-service/
maubot/ maubot/
data/ ${DATA_PATH}
homeserver_media-store homeserver_media-store
turn turn
certs/ ${CERT_PATH}/
``` ```
- `/configs/` : configuration persistent data - `/configs/` : configuration persistent data
@ -66,13 +73,12 @@ The docker-compose.yaml file assumes the following hiecrasy:
- `/data/` : other kind of persistent data (like synapse media store etc.) - `/data/` : other kind of persistent data (like synapse media store etc.)
## Docker volumes and networks <br/>
- Create a docker volume for postgres: `sudo docker volume create db-data`
- The three required networks (`db`,`bots` and `ms`) will be created automatically. If the names overlap with anything already running, you should edit `docker-compose.yaml`
# Initialization <br/>
# Initialization and preconfigurations
## Expsose ENV ## Expsose ENV
@ -82,10 +88,36 @@ export DOMAIN=ms.local
export CONF_PATH=/mnt/configs export CONF_PATH=/mnt/configs
``` ```
Some of the services need to initialize some config files before you can finally start them. Below are the steps and a reasoning behind them: Some of the services need to initialize some config files before you can finally start them.
<br/>
## Proxy
1. Create file `traefik-ssl.toml` in `${CONF_PATH}/proxy/` and paste the following:
```
[tls]
[tls.stores]
[tls.stores.default]
[tls.stores.default.defaultCertificate]
certFile = "/certs/WILDCARD.ms.local.crt"
keyFile = "/certs/WILDCARD.ms.local.key"
```
Change the file name of the certificate if you have to.
<br/>
## Prostgres
- Create a docker volume: `sudo docker volume create db-data`
- Change the values in `db.env` to your liking. You should at least change `POSTGRES_PASSWORD=`
<br/>
## Synapse ## Synapse
Use the following command to generate a `homeserver.yaml` file in `${CONF_PATH}/homeserver/`. __IMPORTANT: the subdomain (`matrix.${DOMAIN}`) CANNOT be changed later. Make sure you have decided correctly.__ Generate a `homeserver.yaml` file in `${CONF_PATH}/homeserver/`. You can find a sample config at `sample_configs/homeserver/homeserver.yaml`
__IMPORTANT: the subdomain (`matrix.${DOMAIN}`) CANNOT be changed later. Make sure you have decided correctly.__
``` ```
sudo docker run -it --rm \ sudo docker run -it --rm \
@ -94,14 +126,13 @@ sudo docker run -it --rm \
-e SYNAPSE_REPORT_STATS=yes \ -e SYNAPSE_REPORT_STATS=yes \
matrixdotorg/synapse:latest generate matrixdotorg/synapse:latest generate
``` ```
After that you can edit the file however you want. Some important fields are: Edit/Uncomment some important fields:
- `server_name` will be autofilled - `server_name` will be autofilled
``` ```
server_name: "matrix.ms.local" server_name: "matrix.ms.local"
``` ```
- We add an https listener for secure connections, bind it to all addresses and enable federation. - Add an https listener for secure connections, bind it to all addresses and enable federation.
``` ```
listeners: listeners:
- port: 8448 - port: 8448
@ -134,7 +165,7 @@ database:
name: psycopg2 name: psycopg2
args: args:
user: synapse user: synapse
password: 12345 password: <SAME AS db.env>
database: synapse_db database: synapse_db
host: db host: db
cp_min: 5 cp_min: 5
@ -152,28 +183,41 @@ tls_certificate_path: "/certs/WILDCARD.ms.local.crt"
tls_private_key_path: "/certs/WILDCARD.ms.local.key" tls_private_key_path: "/certs/WILDCARD.ms.local.key"
``` ```
- Finally, enable registrations - Enable registrations
``` ```
enable_registration: true enable_registration: true
``` ```
- Enable user directory search. (This will help us find the bot accounts later)
```
user_directory:
enabled: true
search_all_users: true
prefer_local_users: true
```
- Save the file (_We will edit more while configuring Turn, Bridges and Bots_) - Save the file (_We will edit more while configuring Turn, Bridges and Bots_)
<br/>
<br/>
## Bridges and Bots ## Bridges and Bots
<br/>
### Telegram Brige ### Telegram Brige
_Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram _Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram
1. Run: 1. Run the command to generate a `config.yaml`:
``` ```
sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest
``` ```
This will generate a `config.yaml` that you should edit.
2. You need to set at least the following:
- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_) 2. Edit the file (reference `sample_configs/telegram-bridge/config.yaml`):
``` - Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
homeserver: ```
homeserver:
# The address that this appservice can use to connect to the homeserver. # The address that this appservice can use to connect to the homeserver.
address: https://homeserver:8448 address: https://homeserver:8448
# The domain of the homeserver (for MXIDs, etc). # The domain of the homeserver (for MXIDs, etc).
@ -182,47 +226,50 @@ homeserver:
# Only applies if address starts with https:// # Only applies if address starts with https://
verify_ssl: false verify_ssl: false
appservice: appservice:
# The address that the homeserver can use to connect to this appservice. # The address that the homeserver can use to connect to this appservice.
address: http://telegram-bridge:29317 address: http://telegram-bridge:29317
database: sqlite:////data/telegram-bridge.db database: sqlite:////data/telegram-bridge.db
``` ```
- Bridge permissions - Bridge permissions
We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following: We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following:
``` ```
permissions: permissions:
"*": relaybot "*": relaybot
"matrix.ms.local": admin "matrix.ms.local": admin
``` ```
- Telegram API key - Telegram API key
``` ```
telegram: telegram:
# Get your own API keys at https://my.telegram.org/apps # Get your own API keys at https://my.telegram.org/apps
api_id: 12345 api_id: 12345
api_hash: tjyd5yge35lbodk1xwzw2jstp90k55qz api_hash: tjyd5yge35lbodk1xwzw2jstp90k55qz
``` ```
3. Run the docker command again to generate a 'registration.yaml' 3. Run the docker command again to generate a 'registration.yaml'
``` ```
sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest sudo docker run --rm -v ${CONF_PATH}/telegram-bridge:/data:z dock.mau.dev/mautrix/telegram:latest
``` ```
The `registration.yaml` file is mounted on the `homeserver` cotainer. The `registration.yaml` file is mounted on the `homeserver` cotainer.
<br/>
### Facebook Bridge (Almost identical to Telegram bridge) ### Facebook Bridge (Almost identical to Telegram bridge)
_Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook _Source_: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook
Run: 1. Run the command to generate a `config.yaml`:
``` ```
docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
``` ```
This will generate a `config.yaml` that you should edit. You need to set at least the following:
- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
```
homeserver: 2. Edit the file (reference `sample_configs/facebookm-bridge/config.yaml`):
- Main connection configurations (_Since this is a dev/testing server we will use HTTPS but we won't verify any certificates between the bridge and the homeserver. Same goes for othe bridges and services_)
```
homeserver:
# The address that this appservice can use to connect to the homeserver. # The address that this appservice can use to connect to the homeserver.
address: https://homeserver:8448 address: https://homeserver:8448
# The domain of the homeserver (for MXIDs, etc). # The domain of the homeserver (for MXIDs, etc).
@ -231,79 +278,67 @@ homeserver:
# Only applies if address starts with https:// # Only applies if address starts with https://
verify_ssl: false verify_ssl: false
appservice: appservice:
# The address that the homeserver can use to connect to this appservice. # The address that the homeserver can use to connect to this appservice.
address: http://facebook-bridge:29317 address: http://facebook-bridge:29317
database: sqlite:////data/facebook-bridge.db database: sqlite:////data/facebook-bridge.db
``` ```
- Bridge permissions - Bridge permissions
We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following: We should also give permission to some users to use the bridge. Since we don't even have a homeserver yet we will give admin permissions to all users that share the domain `matrix.ms.local` . Edit the following:
``` ```
permissions: permissions:
"*": "relay" "*": "relay"
"matrix.ms.local": "admin" "matrix.ms.local": "admin"
``` ```
Run the docker command again to generate a 'registration.yaml'
```
sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
```
3. Run the docker command again to generate a 'registration.yaml'
```
sudo docker run --rm -v ${CONF_PATH}/facebook-bridge:/data:z dock.mau.dev/mautrix/facebook:latest
```
The `registration.yaml` file is mounted on the `homeserver` cotainer. The `registration.yaml` file is mounted on the `homeserver` cotainer.
<br/>
### Webhook App Service ### Webhook App Service
Source: https://github.com/turt2live/matrix-appservice-webhooks#docker Source: https://github.com/turt2live/matrix-appservice-webhooks#docker
1. Create an `appservice-registration-webhooks.yaml` file in `${CONF_PATH}/webhooks` and copy the following (make sure you generate `hs_token` and `as_token`): 1. Create an `appservice-registration-webhooks.yaml` file in `${CONF_PATH}/webhooks` and copy the following (make sure you generate `hs_token` and `as_token`):
``` ```
id: webhooks id: webhooks
hs_token: A_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS hs_token: A_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS
as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING # CHANGE THIS
namespaces: namespaces:
users: users:
- exclusive: true - exclusive: true
regex: '@_webhook.*' regex: '@_webhook.*'
url: 'http://webhook-service:9000' url: 'http://webhook-service:9000'
sender_localpart: webhooks sender_localpart: webhooks
rate_limited: false rate_limited: false
``` ```
2. Create an `config.yaml` file in `${CONF_PATH}/webhooks` and copy/edit the following: 2. Create an `config.yaml` file in `${CONF_PATH}/webhooks` and copy/edit the following:
``` ```
# Configuration specific to the application service. All fields (unless otherwise marked) are required. homeserver:
homeserver:
# The domain for the client-server API calls.
url: "http://homeserver:8008" url: "http://homeserver:8008"
# The domain part for user IDs on this home server. Usually, but not always, this is the same as the
# home server's URL.
domain: "matrix.ms.local" domain: "matrix.ms.local"
# Configuration specific to the bridge. All fields (unless otherwise marked) are required. webhookBot:
webhookBot:
# The localpart to use for the bot. May require re-registering the application service.
localpart: "webhooks" localpart: "webhooks"
# Appearance options for the Matrix bot
appearance: appearance:
displayName: "Webhook Bridge" displayName: "Webhook Bridge"
avatarUrl: "http://i.imgur.com/IDOBtEJ.png" # webhook icon avatarUrl: "http://i.imgur.com/IDOBtEJ.png"
# Provisioning API options provisioning:
provisioning:
# Your secret for the API. Required for all provisioning API requests.
secret: 'CHANGE_ME' secret: 'CHANGE_ME'
# Configuration related to the web portion of the bridge. Handles the inbound webhooks web:
web: hookUrlBase: 'https://webhooks.ms.local'
hookUrlBase: 'https://webhooks.domain.ltd'
logging: logging:
file: logs/webhook.log file: logs/webhook.log
console: true console: true
consoleLevel: debug consoleLevel: debug
@ -312,13 +347,11 @@ logging:
rotate: rotate:
size: 52428800 # bytes, default is 50mb size: 52428800 # bytes, default is 50mb
count: 5 count: 5
```
```
3. Create a `database.json` file in `${CONF_PATH}/webhooks` and copy the following: 3. Create a `database.json` file in `${CONF_PATH}/webhooks` and copy the following:
``` ```
{
{
"defaultEnv": { "defaultEnv": {
"ENV": "NODE_ENV" "ENV": "NODE_ENV"
}, },
@ -330,35 +363,44 @@ logging:
"driver": "sqlite3", "driver": "sqlite3",
"filename": "/data/production.db" "filename": "/data/production.db"
} }
} }
```
``` 4. Run the command to check for errors:
```
sudo docker run --rm -v ${CONF_PATH}/webhooks:/data turt2live/matrix-appservice-webhooks
```
_If you get an `[ERROR] ConnectionError: request failed: getaddrinfo ENOTFOUND homeserver homeserver:8008`, this is normal since we don't have a working homeserver yet._
4. Run: <br/>
```
sudo docker run --rm -v ${CONF_PATH}/webhooks:/data turt2live/matrix-appservice-webhooks
```
Check the logs for any errors. If you get an `[ERROR] ConnectionError: request failed: getaddrinfo ENOTFOUND homeserver homeserver:8008`, this is normal since we don't have a working homeserver yet.
### Maubot Manager ### Maubot Manager
_Source_: https://docs.mau.fi/maubot/usage/setup/docker.html _Source_: https://docs.mau.fi/maubot/usage/setup/docker.html
1. Run: 1. Run the command to generate a `config.yaml`:
``` ```
sudo docker run --rm -v ${CONF_PATH}/maubot:/data:z dock.mau.dev/maubot/maubot:latest sudo docker run --rm -v ${CONF_PATH}/maubot:/data:z dock.mau.dev/maubot/maubot:latest
``` ```
This will generate a `config.yaml` file.
2. Update the file to your liking. You should at least add your homeserver: 2. Update the file to add your homeserver:
``` ```
homeservers: homeservers:
matrix.ms.local matrix.ms.local
url: https://homeserver:8448 url: https://homeserver:8448
secret: <THE registration_shared_secret FROM homeserver.yaml> secret: <THE registration_shared_secret FROM homeserver.yaml>
```
``` 3. Create an admin user
3. Save the file ```
admins:
root: ''
admin: '12345' #use a password you like
```
4. Save the file
<br/>
### Registering the new services to the home server: ### Registering the new services to the home server:
@ -371,6 +413,8 @@ app_service_config_files:
``` ```
(in the docker-compose file we have mounted each file in the `homeserver` container) (in the docker-compose file we have mounted each file in the `homeserver` container)
<br/>
## Turn server (for audio and video calls) ## Turn server (for audio and video calls)
Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the sample file from: https://github.com/coturn/coturn/blob/master/docker/coturn/turnserver.conf Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the sample file from: https://github.com/coturn/coturn/blob/master/docker/coturn/turnserver.conf
@ -378,42 +422,94 @@ Create a new file `turnserver.conf` in `${CONF_PATH}/turn/`. Copy and paste the
Edit the following in the file: Edit the following in the file:
- Specify and external ip - Specify and external ip
``` ```
external-ip=<YOUR PUBLIC IP, IF YOU PLAN TO USE IT FROM THE INTERNET> external-ip=<YOUR PUBLIC IP, IF YOU PLAN TO USE IT FROM THE INTERNET>
external-ip=<YOUR DOCKER HOST IP> external-ip=<YOUR DOCKER HOST IP>
``` ```
- Specify a port range - Specify a port range
``` ```
min-port=64000 min-port=64000
max-port=65535 max-port=65535
``` ```
This range worked perfectly for me but you should define your own depending on your network setup This range worked perfectly for me but you should define your own depending on your network setup
- Certificates: - Certificates:
``` ```
cert=/certs/WILDCARD.ms.local.crt cert=/certs/WILDCARD.ms.local.crt
pkey=/certs/WILDCARD.ms.local.key pkey=/certs/WILDCARD.ms.local.key
``` ```
- Define a realm - Define a realm
``` ```
realm=turn.domain.ltd realm=turn.ms.local
``` ```
- Uncomment `use-auth-secret`. Generate a alphanumeric and fill `static-auth-secret=`. - Uncomment `use-auth-secret`. Generate a alphanumeric and fill `static-auth-secret=`.
- In `homeserver.yaml` in `##TURN##` secrion paste the same alpanumeric at `turn_shared_secret: "ALPHANUMERIC"` and add the following `turn_uris` - In `homeserver.yaml` in `##TURN##` section paste the same alpanumeric at `turn_shared_secret: "ALPHANUMERIC"` and add the following `turn_uris`
``` ```
turn_uris: turn_uris:
- "turn:turn.domain.ltd?transport=udp" - "turn:turn.ms.local?transport=udp"
- "turn:turn.domain.ltd?transport=tcp" - "turn:turn.ms.local?transport=tcp"
- "turns:turn.domain.ltd:5349?transport=udp" - "turns:turn.ms.local:5349?transport=udp"
- "turns:turn.domain.ltd:5349?transport=tcp" - "turns:turn.ms.local:5349?transport=tcp"
``` ```
<br/>
<br/>
# Bringing up the Chat Server # Bringing up the Chat Server
If everything is correctly initialized we can bring up the stack with `sudo docker-compose up` If everything is correctly initialized we can bring up the stack with `sudo docker-compose up`. <br/>
After a while we should be able to visit the web element UI at `https://webchat.${DOMAIN}`, and register a new user.
After a while we should be able to visit the web element UI at `http://<DOCKER-HOST-IP>:10000`, and register a new user. <br/>
<br/>
# Final Notes
- This is by __no means__ a production ready setup. Some of the things that should be changed are:
- Diffrent certificates for every service (plus for the bots)
- Postgres for the bridges databases
- No `--serverstransport.insecureskipverify=true` in traefik commands
- Use `secrets` for sensitive information
- Turn server runs in host network mode. This is needed to better communicate with UDP. This means that your devices and containers (especially `homeserver`) should be able to resolve `turn.ms.local` to you docker host IP
- There are some more things to setup for the homeserver, bots and bridges. Please refer to their respective documentations.
# Disclaimer
It goes without saying that I'm not responsible for anything that might go wrong. __BUT__ I will be more than happy to help in any situation. If you have any suggestions on how this guide can be better (I'm sure there are a lot), please feel free to contact me!
# Sources and links
- Synapse
- Github: @matrix-org | https://github.com/matrix-org/synapse
- Documentation: https://matrix-org.github.io/synapse/latest/
- Docker image: https://hub.docker.com/r/matrixdotorg/synapse/
- Postgres
- Github: @postgres | https://github.com/postgres/postgres
- Documentation: https://www.postgresql.org/docs/current/
- Docker image: https://hub.docker.com/_/postgres/
- Coturn
- Github: @coturn | https://github.com/coturn/coturn
- Documantation: https://github.com/coturn/coturn/wiki/
- Docker image: https://hub.docker.com/r/instrumentisto/coturn/
- Element.io Web
- Github: @vector-im
- Docker image: https://hub.docker.com/r/vectorim/element-web/
- Synapse Admin
- Github: @Awesome-Technologies | https://github.com/Awesome-Technologies/synapse-admin
- Docker image: https://hub.docker.com/r/awesometechnologies/synapse-admin
- Telegram Bridge
- Github: @mautrix | https://github.com/mautrix/telegram
- Documentation: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=telegram
- Facebook Bridge
- Github: @mautrix | https://github.com/mautrix/facebook
- Documentation: https://docs.mau.fi/bridges/python/setup/docker.html?bridge=facebook
- Maubot Manager
- Github: @maubot | https://github.com/maubot/maubot
- Documentation: https://docs.mau.fi/maubot/usage/setup/docker.html
- Webhook Appservice
- Github: @turt2live | https://github.com/turt2live/matrix-appservice-webhooks