59 Commits

Author SHA1 Message Date
Ingo Oppermann
71b613371a
Add debug flag to cluster to disable FFmpeg check 2023-07-10 11:07:24 +02:00
Ingo Oppermann
c5d52dbeb3
Add gops agent
The gops agent can be enabled by providing an address to
CORE_DEBUG_AGENT_ADDRESS where it will listen to requests
from the gops utility.
2023-07-03 21:27:44 +02:00
Ingo Oppermann
adcbd98467
Add CORE_TLS_SECRET configuration
This secret will be used to encrypt automatically obtained secrets at
rest, i.e. in a storage. They will be decrypted on demand. If the
secret is wrong, stored certificates can't be decrypted. For changing
the secret, the stored certificated must be deleted first in order
to obtain new ones that will be encrypted with the new secret.
2023-07-03 16:02:39 +02:00
Ingo Oppermann
a6d454b03f
Add CORE_TLS_STAGING
Set this value to true in order to use the Let's Encrypt staging CA,
otherwise the production CA will be used, which is the default.
2023-06-28 11:55:56 +02:00
Ingo Oppermann
72a0566bc6
Add CORE_CLUSTER_STARTUP_TIMEOUT_SEC configuration value 2023-06-28 11:41:24 +02:00
Ingo Oppermann
dd128ac99b
Fix gathering of hostnames, add /v3/cluster/db/kv endpoint 2023-06-27 14:52:32 +02:00
Ingo Oppermann
9fc0bb0a64
Fix env variable for session log path pattern 2023-06-20 15:26:08 +02:00
Ingo Oppermann
d3eed2a417
Allow to log each finished session to filesystem
By providing CORE_SESSIONS_SESSION_LOG_PATH_PATTERN (e.g. "/log/%Y-%m-%d.log") all
finished sessions will be logged to a file according to the provided strftime-pattern. The
actual value is calculated from when the session closed. CORE_SESSIONS_PERSIST must
be set. Default: not set.

Set CORE_SESSIONS_SESSION_LOG_BUFFER_SEC to the number of seconds the log should be
buffered in memory before persisted to disk. Default 15 seconds.
2023-06-20 15:19:25 +02:00
Ingo Oppermann
a728cc7839
Fix clusterNode/proxyNode 2023-06-14 22:05:59 +02:00
Ingo Oppermann
ccf4bee56d
WIP: cluster node revamp, add version and config comparison 2023-06-13 10:25:08 +02:00
Ingo Oppermann
7e7d1caca7
Add EmergencyLeaderTimeout parameter, ignore throttling nodes, implement NodeRecoverTimeout, introduce processNodeMap in cluster DB 2023-06-07 22:08:07 +02:00
Ingo Oppermann
bd75a5ad0f
Add SyncInterval and NodeRecoverTimeout to cluster config 2023-06-07 10:18:23 +02:00
Ingo Oppermann
3ac7ead20d
Add the resource limits to the metrics 2023-06-06 15:20:59 +02:00
Ingo Oppermann
a79cfa0c77
Merge branch 'clusteriam' into vod 2023-06-01 15:16:54 +02:00
Ingo Oppermann
710d5c595f
Merge branch 'iam' into clusteriam 2023-05-25 11:05:26 +02:00
Ingo Oppermann
3c89cbb831
Create identities for basic auth access to mount points 2023-05-24 14:29:14 +02:00
Ingo Oppermann
d201921a33
Allow to provide complete cluster configuration
Replace CORE_CLUSTER_JOIN_ADDRESS with CORE_CLUSTER_PEERS. This is
a comma separated list of cluster members with their IDs of the form
ID@host:port

On startup the node tries to connect to all the peers. In case of
sudden deaths of a node this will allow to find back into the
cluster. The list in CLUSTER_PEERS is a starting point of known
peers. Other node that are not in that list can still join the
cluster.

File and stream proxy has been moved to the Proxy type.
2023-05-03 16:13:05 +02:00
Ingo Oppermann
74f98e8c7e
Define system CPU resources in percent 0-100, acquiring resources check with available system resources 2023-04-28 10:27:33 +02:00
Ingo Oppermann
7527f7833b
Add resource manager 2023-04-27 17:02:20 +02:00
Ingo Oppermann
9b17ab2b29
Add global limits in config, fix not using process limits 2023-04-25 21:49:55 +02:00
Ingo Oppermann
75c11eb475
WIP: add recover option 2023-04-24 10:54:53 +02:00
Ingo Oppermann
27f19f9188
WIP: internal API 2023-04-17 14:01:20 +02:00
Ingo Oppermann
7643959bf8
WIP: raft leadership 2023-04-13 21:44:24 +02:00
Ingo Oppermann
a1682b7aa4
Fix parsing S3 storage definition from environment variable 2023-04-13 12:19:20 +02:00
Ingo Oppermann
8326160b28
Add option to set GOMAXPROCS automatically
Use the debug.auto_max_procs config value or the CORE_DEBUG_AUTO_MAX_PROCS
environment variable. Set it to "true" to enable this feature.
2023-04-11 17:12:01 +02:00
Ingo Oppermann
507b4f4b9a
Merge branch 'dev' into vod 2023-04-11 08:29:39 +02:00
Ingo Oppermann
98c561554d
Fix S3 storage parsing from environment variable 2023-04-05 14:43:45 +02:00
Ingo Oppermann
baf1c3391a
Deprecate ENV names that do not correspond to JSON name 2023-04-03 21:21:02 +02:00
Ingo Oppermann
4ce8a0eaa3
Add config value for how many minimal process report should be kept in the history
A minimal history is a history entry without log and prelude.

The corresponding config entry is ffmpeg.log.max_minimal_history. This value is
added on top of the ffmpeg.log.max_history value. I.e. the latest max_history
entries contain the log and prelude, and the remaining entries don't have the
log and prelude. In total there are max_minimal_history+max_history history
entries.

If you want no history, set both values to 0.
If you want only full history, set max_minimal_history to 0.
If you want only minimal history, set max_history to 0.
2023-03-16 12:25:06 +01:00
Ingo Oppermann
175cfc2324
Fix using prefix syntax for filesystem operations 2023-03-03 16:18:46 +01:00
Ingo Oppermann
1c04961fc1
Fix tests 2023-02-21 12:57:33 +01:00
Ingo Oppermann
2a3288ffd0
Use abstract filesystem for stores 2023-02-01 16:09:20 +01:00
Ingo Oppermann
e3d25b1bdd
Add check for duplicate filesystem names 2023-01-31 15:53:48 +01:00
Ingo Oppermann
f519acfd71
Add S3 storage support 2023-01-31 14:45:58 +01:00
Ingo Oppermann
ad3538d224
Add tests 2023-01-24 13:40:36 +01:00
Ingo Oppermann
59b7978470
Add tests 2023-01-24 11:22:31 +01:00
Ingo Oppermann
e374f83377
Fix config timestamps
created_at represents the time when the configuration has been persisted to disk.
loaded_at represents the time when the configuration has actually been used.

If created_at is larger than loaded_at, then the Core needs a reload in order
to apply the latest configuration.

if created_at is lower than laoded_at, then the Core applied the latest
configuration.

The value of updated_at is irrelevant and shouldn't be used.
2023-01-19 16:13:53 +01:00
Ingo Oppermann
ea79b87236
Add format annotation for integer types for swagger documentation 2023-01-11 21:05:40 +01:00
Ingo Oppermann
1bbb7a9c1f
Use config locations for import and ffmigrage 2023-01-03 11:45:10 +01:00
Ingo Oppermann
17c9f6ef13
Test different standard location for config file
If no path is given in the environment variable CORE_CONFIGFILE, different
standard locations will be probed:
- os.UserConfigDir() + /datarhei-core/config.js
- os.UserHomeDir() + /.config/datarhei-core/config.js
- ./config/config.js
If the config.js doesn't exist in any of these locations, it will be
assumed at ./config/config.js
2023-01-03 07:55:55 +01:00
Ingo Oppermann
378a3cd9cf
Allow to set a soft memory limit for the binary itself
The setting debug.memory_limit_mbytes should not be used in conjuction
with debug.force_gc because the memory limit influences the garbage
collector.
2023-01-02 11:58:54 +01:00
Ingo Oppermann
992b04d180
Allow alternative syntax for auth0 tenants as environment variable 2023-01-02 11:39:58 +01:00
Ingo Oppermann
391681447e
Fix MustDir config type to create directory 2023-01-02 10:54:29 +01:00
Ingo Oppermann
9ad19fbdd6
Fix reading partial config
If the config on the disk doesn't have all fields, then the missing
fields are now populated with their defaults.
2022-11-08 14:44:47 +01:00
Ingo Oppermann
4d4e70571e
Fix proper version handling for uploading a new config 2022-10-10 16:19:45 +02:00
Ingo Oppermann
33bd7bd384
Set default email address 2022-09-30 12:25:01 +02:00
Ingo Oppermann
fe2e9d375c
Use LE porduction CA, allow to configure an email address 2022-09-30 12:12:36 +02:00
Ingo Oppermann
5cb0592854
Exclude .m3u8 and .mpd files from disk cache by default 2022-08-26 11:35:56 +03:00
Ingo Oppermann
6802830c62
Don't use deprecated functions from io/ioutil 2022-08-18 10:27:33 +03:00
Ingo Oppermann
273ca0abbc
Add cache block list for extensions not to cache 2022-08-02 19:10:28 +02:00