80 Commits

Author SHA1 Message Date
Ingo Oppermann
a79cfa0c77
Merge branch 'clusteriam' into vod 2023-06-01 15:16:54 +02:00
Ingo Oppermann
f21ca2055e
Use IAM for placeholders, simplify proxy nodes 2023-05-31 21:49:25 +02:00
Ingo Oppermann
89cc42c4b9
Add cluster IAM, update dependencies 2023-05-30 21:24:12 +02:00
Ingo Oppermann
e9034aa171
Create identity and access packages for IAM 2023-05-25 16:16:29 +02:00
Ingo Oppermann
710d5c595f
Merge branch 'iam' into clusteriam 2023-05-25 11:05:26 +02:00
Ingo Oppermann
10df8d51b1
Simplify IAM user config, only create users and policies if not previously run 2023-05-24 22:28:24 +02:00
Ingo Oppermann
3c89cbb831
Create identities for basic auth access to mount points 2023-05-24 14:29:14 +02:00
Ingo Oppermann
ccac2ffd5d
Various updates
- rebrand group to domain
- move IAM to the API (rest and graph) for enforcing "process:" rules
- add abstraction layer for restream store in order to decouple internal format from format on disk
- move playout handler into restreamHandler
- remove user from restream interface
- add TaskID type that includes the process id and its domain
2023-05-23 15:47:06 +02:00
Ingo Oppermann
cd58cf3df7
Fix panics if cluster is not enabled 2023-05-05 11:56:39 +02:00
Ingo Oppermann
7f59c188cf
Remove node storage, use raft configuration instead; re-establish file and stream proxying 2023-05-04 19:49:53 +02:00
Ingo Oppermann
d201921a33
Allow to provide complete cluster configuration
Replace CORE_CLUSTER_JOIN_ADDRESS with CORE_CLUSTER_PEERS. This is
a comma separated list of cluster members with their IDs of the form
ID@host:port

On startup the node tries to connect to all the peers. In case of
sudden deaths of a node this will allow to find back into the
cluster. The list in CLUSTER_PEERS is a starting point of known
peers. Other node that are not in that list can still join the
cluster.

File and stream proxy has been moved to the Proxy type.
2023-05-03 16:13:05 +02:00
Ingo Oppermann
4eb0258ba4
Add automatic bootstrap and recovery 2023-05-02 15:49:46 +02:00
Ingo Oppermann
7527f7833b
Add resource manager 2023-04-27 17:02:20 +02:00
Ingo Oppermann
3a1825cf61
Expose resource usage in report history 2023-04-25 15:55:32 +02:00
Ingo Oppermann
386bdc0b6e
WIP: add emergency leader, api circuit breaker 2023-04-21 17:19:21 +02:00
Ingo Oppermann
27f19f9188
WIP: internal API 2023-04-17 14:01:20 +02:00
Ingo Oppermann
7643959bf8
WIP: raft leadership 2023-04-13 21:44:24 +02:00
Ingo Oppermann
e6b64dbe97
WIP: raft 2023-04-13 09:42:19 +02:00
Ingo Oppermann
ffdf6d3323
Merge branch 'dev' into cluster 2023-04-12 15:27:38 +02:00
Ingo Oppermann
8326160b28
Add option to set GOMAXPROCS automatically
Use the debug.auto_max_procs config value or the CORE_DEBUG_AUTO_MAX_PROCS
environment variable. Set it to "true" to enable this feature.
2023-04-11 17:12:01 +02:00
Ingo Oppermann
0bd68d6dc3
Fix debug.force_gc behaviour 2023-04-11 11:08:11 +02:00
Ingo Oppermann
507b4f4b9a
Merge branch 'dev' into vod 2023-04-11 08:29:39 +02:00
Ingo Oppermann
84817f137a
Merge branch 'dev' into iam 2023-04-03 15:44:16 +02:00
Ingo Oppermann
99c571d623
Add event filter support 2023-03-23 14:50:59 +01:00
Ingo Oppermann
b84fdddd81
Add /api/v3/events endpoint 2023-03-23 11:11:47 +01:00
Ingo Oppermann
674c444d17
Merge branch 'dev' into vod 2023-03-17 18:41:16 +01:00
Ingo Oppermann
3149572a64
Fix freeing up S3 mounts 2023-03-17 18:40:20 +01:00
Ingo Oppermann
7e7aadc6cb
Allow to bulk delete files based on ListOptions 2023-03-17 15:56:15 +01:00
Ingo Oppermann
4ce8a0eaa3
Add config value for how many minimal process report should be kept in the history
A minimal history is a history entry without log and prelude.

The corresponding config entry is ffmpeg.log.max_minimal_history. This value is
added on top of the ffmpeg.log.max_history value. I.e. the latest max_history
entries contain the log and prelude, and the remaining entries don't have the
log and prelude. In total there are max_minimal_history+max_history history
entries.

If you want no history, set both values to 0.
If you want only full history, set max_minimal_history to 0.
If you want only minimal history, set max_history to 0.
2023-03-16 12:25:06 +01:00
Ingo Oppermann
6e93c1d5a1
Get rid of $localhost pseudo user 2023-03-09 21:10:04 +01:00
Ingo Oppermann
a0ef3ab5ee
WIP: add /api/v3/iam/group/* endpoints 2023-03-09 19:03:58 +01:00
Ingo Oppermann
d101a76e9e
Change anon user to localhost user only if DisableLocalhost is set 2023-03-08 15:21:55 +01:00
Ingo Oppermann
8755117e92
Add /api/v3/iam/user endpoints 2023-03-07 16:31:58 +01:00
Ingo Oppermann
be718eac0a
Add support for date placeholder in process config
Because it doesn't make sense to replace the date placeholder at
process creation, it has to be replaced at every start of the process.

On process creation only the static placeholders (such as process ID)
are replaced. Dynamic placeholders (so far only "date") are not
replaced. On process start, a callback has been introduced that gives
the chance to change the command line. This is the point where
the restreamer replaces the date placeholders.
2023-02-28 17:46:08 +01:00
Ingo Oppermann
8215c20ae6
Fix basic auth, disable localhost, replace template func 2023-02-17 17:27:39 +01:00
Ingo Oppermann
ad4f392bd0
user only for api: 2023-02-17 10:30:38 +01:00
Ingo Oppermann
1e74f11f5e
WIP: policy update 2023-02-16 21:47:56 +01:00
Ingo Oppermann
acc4d7ec5a
WIP: verify policies for processes, rtmp, and srt. rewriter for local urls 2023-02-15 21:43:37 +01:00
Ingo Oppermann
74d0a96e1c
Merge branch 'dev' into iam 2023-02-14 19:08:37 +01:00
Ingo Oppermann
05a176370a
Fix missing filesystem metadata and middlewares 2023-02-14 16:16:35 +01:00
Ingo Oppermann
eac49ad11a
Define default policies to mimic current behaviour 2023-02-10 15:14:30 +01:00
Ingo Oppermann
312f65d110
WIP: add verfiers, fix bugs, implement middleware, support rtmp 2023-02-09 21:33:45 +01:00
Ingo Oppermann
11e55fc2c7
WIP: add casbin to access manager, allow to persist identities 2023-02-06 17:07:20 +01:00
Ingo Oppermann
8f1ff2d1a2
WIP: designing interfaces, detecting identity, enforcing policies 2023-02-03 17:43:06 +01:00
Ingo Oppermann
2a3288ffd0
Use abstract filesystem for stores 2023-02-01 16:09:20 +01:00
Ingo Oppermann
49b16f44a8
Add templates for s3 filesystems 2023-01-31 15:54:40 +01:00
Ingo Oppermann
f519acfd71
Add S3 storage support 2023-01-31 14:45:58 +01:00
Ingo Oppermann
0147651de6
Extend placeholders
1. Allow variables in placeholders for parameter values, e.g.
   {rtmp,name=$processid}. The variable starts with a $ letter.
   The recognized variables are provided with the Replace func.

2. The template func recieves the process config and the name of
   the section where this placeholder is located, i.e. "global",
   "input", or "output".
2023-01-20 13:38:33 +01:00
Ingo Oppermann
e374f83377
Fix config timestamps
created_at represents the time when the configuration has been persisted to disk.
loaded_at represents the time when the configuration has actually been used.

If created_at is larger than loaded_at, then the Core needs a reload in order
to apply the latest configuration.

if created_at is lower than laoded_at, then the Core applied the latest
configuration.

The value of updated_at is irrelevant and shouldn't be used.
2023-01-19 16:13:53 +01:00
Ingo Oppermann
17c9f6ef13
Test different standard location for config file
If no path is given in the environment variable CORE_CONFIGFILE, different
standard locations will be probed:
- os.UserConfigDir() + /datarhei-core/config.js
- os.UserHomeDir() + /.config/datarhei-core/config.js
- ./config/config.js
If the config.js doesn't exist in any of these locations, it will be
assumed at ./config/config.js
2023-01-03 07:55:55 +01:00