Generation (/api/v1/generation) cannot be reached by our automated monitoring.

Chat (/api/v1/chat/completions) cannot be reached by our automated monitoring.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Chat (/api/v1/chat/completions) is operational again.

Chat (/api/v1/chat/completions) cannot be reached by our automated monitoring.

Generation (/api/v1/generation) is operational again.

Generation (/api/v1/generation) cannot be reached by our automated monitoring.

Chat (/api/v1/chat/completions) is operational again.

Chat (/api/v1/chat/completions) cannot be reached by our automated monitoring.

This incident has been resolved.

The issue has been identified and recovery has begun. Logs will continue to be applied from the existing backlog. The chatroom may return 408 errors, but the inference APIs will continue to succeed.

Generations API is also impacted.

We are currently investigating an issue with delayed logging of API requests. This extends to delays in updating budgets and billing events.