Reliability and Observability Enhancements
Enhanced dashboards with Time to First Token metrics and improved routing resilience
Cailos improves system observability and reliability by introducing Time to First Token and Tokens Per Second metrics to team analytics and health dashboards, with sorting and skipped check reporting. Endpoint reliability rankings now penalise failures by 30% and use TTT rather than total latency for more accurate assessment. Additionally, new request routing race executor tests endpoints concurrently and seamlessly hands over tasks if primary stalls, improving request resilience.
-
added
Team analytics dashboards displayed Time to First Token and Tokens Per Second metrics alongside latency and errors.
-
added
Health dashboard included Time to First Token data, allowing sorting by this metric and reporting skipped checks.
-
changed
Endpoint reliability rankings penalised failures by 30% and used Time to First Token instead of total latency.
-
added
Introduced a request routing race executor that tested endpoints concurrently and handed over tasks if the primary stalled.