330 Commits

Author SHA1 Message Date
Ingo Oppermann
cd58cf3df7
Fix panics if cluster is not enabled 2023-05-05 11:56:39 +02:00
Ingo Oppermann
7f59c188cf
Remove node storage, use raft configuration instead; re-establish file and stream proxying 2023-05-04 19:49:53 +02:00
Ingo Oppermann
d201921a33
Allow to provide complete cluster configuration
Replace CORE_CLUSTER_JOIN_ADDRESS with CORE_CLUSTER_PEERS. This is
a comma separated list of cluster members with their IDs of the form
ID@host:port

On startup the node tries to connect to all the peers. In case of
sudden deaths of a node this will allow to find back into the
cluster. The list in CLUSTER_PEERS is a starting point of known
peers. Other node that are not in that list can still join the
cluster.

File and stream proxy has been moved to the Proxy type.
2023-05-03 16:13:05 +02:00
Ingo Oppermann
c0a5325f05
Merge branch 'dev' into cluster 2023-05-03 11:17:54 +02:00
Ingo Oppermann
0921384568
Fix double import of path package 2023-04-29 07:48:29 +02:00
Ingo Oppermann
1d7fd960d5
Merge branch 'dev' into vod 2023-04-28 18:10:24 +02:00
Ingo Oppermann
74110dae54
Fix possible infinite loop with HLS session rewriter 2023-04-28 17:38:36 +02:00
Ingo Oppermann
b723f4e5fd
Propagate number of logical CPUs in resource usage 2023-04-27 10:58:41 +02:00
Ingo Oppermann
b70382c40b
Expose resource usage in process state 2023-04-25 16:20:59 +02:00
Ingo Oppermann
3a1825cf61
Expose resource usage in report history 2023-04-25 15:55:32 +02:00
Ingo Oppermann
c1bbb4065c
Merge branch 'dev' into vod 2023-04-24 12:16:52 +02:00
Ingo Oppermann
317d6eb4d9
Add updated_at field in process infos 2023-04-24 12:05:01 +02:00
Ingo Oppermann
386bdc0b6e
WIP: add emergency leader, api circuit breaker 2023-04-21 17:19:21 +02:00
Ingo Oppermann
78686c81f4
Fix keeping matches in the report history 2023-04-17 15:03:25 +02:00
Ingo Oppermann
b6a9fa7965
Merge branch 'dev' into cluster 2023-04-13 16:25:53 +02:00
Ingo Oppermann
5d9db55032
Merge branch 'dev' into vod 2023-04-13 15:23:09 +02:00
Ingo Oppermann
d807becc8a
Add support for input framerate data from jsonstats patch 2023-04-13 15:22:33 +02:00
Ingo Oppermann
ffdf6d3323
Merge branch 'dev' into cluster 2023-04-12 15:27:38 +02:00
Ingo Oppermann
507b4f4b9a
Merge branch 'dev' into vod 2023-04-11 08:29:39 +02:00
Ingo Oppermann
51cbb8e74e
Fix assembling return value for filtered report history 2023-04-05 13:58:41 +02:00
Ingo Oppermann
7372b3210d
Fix test 2023-04-05 12:21:19 +02:00
Ingo Oppermann
ec4607aec8
Fix process report selection 2023-04-05 12:01:21 +02:00
Ingo Oppermann
508d82d579
Fix high memory consumption when copying files to S3 2023-04-05 10:30:17 +02:00
Ingo Oppermann
7e9e6fce8d
Add number of keyframes and extradata size to process progress data 2023-04-04 20:44:57 +02:00
Ingo Oppermann
84817f137a
Merge branch 'dev' into iam 2023-04-03 15:44:16 +02:00
Ingo Oppermann
0dd4a8fb60
Add TestFilesystemsPurgeCache 2023-04-03 14:02:41 +02:00
Ingo Oppermann
30be82c9fb
Merge branch 'dev' into vod 2023-04-03 10:29:01 +02:00
Ingo Oppermann
6eefa5ca2b
Fix purging default file from HTTP cache 2023-04-03 10:27:04 +02:00
Ingo Oppermann
75332ade0a
Merge branch 'dev' into vod 2023-04-03 10:25:06 +02:00
Ingo Oppermann
029bf208e7
Add bandwidth limit to filesystem operations, rename fields 2023-03-30 09:51:29 +02:00
Ingo Oppermann
0a3117bbd0
Add log_patterns to process config
log_patterns allow to filter the FFmpeg log messages based on regular
expressions. Each entry of log_patterns is interpreted as regular
expression and matched against every non-progress log line emitted from
FFmpeg. All matching lines are returned in the matches array of the
report.
2023-03-27 15:50:25 +02:00
Ingo Oppermann
814975dfee
Fix duplicate ID in swagger annotations 2023-03-27 14:35:50 +02:00
Ingo Oppermann
7188a7dcdd
Potentially fix panic on event subscriber disconnect 2023-03-23 21:07:21 +01:00
Ingo Oppermann
094e666926
Fix swagger documentation for /api/v3/events 2023-03-23 19:46:24 +01:00
Ingo Oppermann
99c571d623
Add event filter support 2023-03-23 14:50:59 +01:00
Ingo Oppermann
48678fb4c6
Fix purging default file from HTTP cache 2023-03-23 11:17:13 +01:00
Ingo Oppermann
b84fdddd81
Add /api/v3/events endpoint 2023-03-23 11:11:47 +01:00
Ingo Oppermann
b21aba5f9d
Remove /process/:id/report/:at endpoint, extend /process/:id/report endpoint 2023-03-22 12:31:41 +01:00
Ingo Oppermann
c92f2fd1df
Fix JSON name of process timeout 2023-03-21 14:54:33 +01:00
Ingo Oppermann
b63b23527b
Add scheduler and timeout to processes
The scheduler allows to define when a process should run. It can be either
a timestamp in RFC3339 format or a crontab expression. If a scheduler is
given, reconnect and the reconnect delay will only apply to processes that
exited as failed.

The timeout allows to define when a process should be gracefully stopped.
It is measured from the actual start of that process including all reconnects
due to failures. If the process finished regularly, the timeout will be
reset.
2023-03-21 14:51:43 +01:00
Ingo Oppermann
a692f88b69
Add DELETE /api/v3/fs/:storage endpoint
Works similar to the GET endpoint. It has the same query parameter (without
"sort" and "order"). A glob pattern has to be provided. To select all
files, use the glob pattern "/**".
2023-03-17 17:05:56 +01:00
Ingo Oppermann
ecfbbe3857
Allow to list files by ranges of size and/or lastmod
The listing options are implemented by the query parameters size_min,
size_max, lastmod_start, and lastmod_end, or by the new ListOptions
type. size_min and size_max expect a number of bytes, lastmod_start
and lastmod_end expect a unix timestamp. All values are inclusive.
2023-03-17 15:15:20 +01:00
Ingo Oppermann
2e8f8fb1a7
Fix swagger tags 2023-03-17 13:58:40 +01:00
Ingo Oppermann
ffaa3d9324
Merge branch 'dev' into vod 2023-03-17 13:56:07 +01:00
Ingo Oppermann
562b7aed92
Use better naming for storage endpoint documentation 2023-03-17 13:55:19 +01:00
Ingo Oppermann
4ce8a0eaa3
Add config value for how many minimal process report should be kept in the history
A minimal history is a history entry without log and prelude.

The corresponding config entry is ffmpeg.log.max_minimal_history. This value is
added on top of the ffmpeg.log.max_history value. I.e. the latest max_history
entries contain the log and prelude, and the remaining entries don't have the
log and prelude. In total there are max_minimal_history+max_history history
entries.

If you want no history, set both values to 0.
If you want only full history, set max_minimal_history to 0.
If you want only minimal history, set max_history to 0.
2023-03-16 12:25:06 +01:00
Ingo Oppermann
5b2b2243bb
Fix report history tests 2023-03-14 15:24:05 +01:00
Ingo Oppermann
1d298038dd
Add date of when a process exited in report history 2023-03-14 15:10:06 +01:00
Ingo Oppermann
6e93c1d5a1
Get rid of $localhost pseudo user 2023-03-09 21:10:04 +01:00
Ingo Oppermann
a0ef3ab5ee
WIP: add /api/v3/iam/group/* endpoints 2023-03-09 19:03:58 +01:00