Monitoring

Assistant-Engine provides extensive monitoring and traceability features. You can track and analyze all AI interaction history, including messages, instructions, context, data source usage, and detailed API call (timing) information. The platform ensures full visibility into the execution of tasks and assistants, allowing you to debug and optimize performance efficiently. Additionally, Assistant-Engine helps monitor and manage LLM usage, offering insights into potential cost drivers and enabling more effective resource allocation.

Monitoring Metrics

You can monitor various metrics and information such as:

Timings: Track how long each step of a run takes to execute. See run steps and run duration as well as Token per second counts and statuses (incomplete, completed)

Tool Calls: Monitor the tools called in every single run and their request and response details.

LLM Configuration: For every message sent you can view historic information about Instructions, Context, Tools, User Message, Chat History & LLM Response used as well as the total Raw Data sent to and received from the LLM.

Prompt Tokens and Completion Tokens: Token count used in each request and response to/from the LLM

Assistant Execution Monitoring

Go to the Assistants Overview.

Select an Assistant and click on the History tab to view its execution history.

You will see a list of all the times the Assistant was executed, along with details such as

• Date and Time of conversation
• Conversation id
• User
• Status (complete, incomplete,error, initializing)
• Message

Click any table entry to get further information on its specific conversation-id including:

• Entire conversation(-id) history
• Context and User Information
• Conversation status: active, inactive
• Total Prompt Tokens: Aggregated outgoing token count to the LLM for the selected conversation
• Total Completion Tokens: Aggregated incoming token count from the LLM for the selected conversation
• Total Token Count: Total Prompt Tokens+ Total Completion Tokens
• Last Run Status: Status of the last run in the conversation - complete, incomplete,error, initializing
• Instruction Context or error message

Select a message to get additional information on the specific run for that message including:

• Run Configuration: Details about Model, Token limits and Truncation Strategy
• Run Results: Detailed run information about Status, Duration and Token counts
• LLM Data: All information processed for the specific run
• Instruction: Instruction valid at the time of run
• Context: Context as defined at the time of run
• Tools: Tools assigned to and used by Assistant at the time of run
• Chat: Chat History and User Message used in run as well as Response
• Raw: Entirety of LLM input data based on the aforementioned details AND Raw LLM output data.

Click Steps tab to get detailled information broken down into the steps taken

• Step Type: Message Creation, Tool Calls
• Processing Status: complete, incomplete,error, initializing
• Step Duration: in seconds
• Tokens per second: LLM output/execution speed
• Token counts: Prompt Tokens, Completion Tokens and Total Tokens for this step of the run
• Tool Calls: Request and Response information for the individual tool call

Click Timing tab to get visual insights into detailed runtimes for each action.

Task Execution Monitoring

Go to the Overall Tasks section within your project (Alternatively: Navigate to the underlying assistant and open the corresponding tasks from there)

Click on the "Run details" tab to view further information. To see execution details for a specific tasks click on the Logs Icon of the desired task

You will see a list of all the times the Task was executed, along with initial details about

• Date and Time of conversation
• Status (complete, incomplete,error, initializing)
• Steps: Number of steps executed for task
• Tools: Number of Tool Calls during task execution
• Duration: Total time of task execution (empty if not complete)

Click any table entry to get further information on its specific conversation-id including:

• Entire conversation(-id) history
• Input and output message
• Total Prompt Tokens: Aggregated outgoing token count to the LLM for the selected task
• Total Completion Tokens: Aggregated incoming token count from the LLM for the selected task
• Total Token Count: Total Prompt Tokens+ Total Completion Tokens
• Last Run Status: Status of the last run in the conversation - complete, incomplete,error, initializing
• Instruction Context or error message

Click the input or output message to get additional information on the specific run for the task including:

• Run Configuration: Details about Model, Token limits and Truncation Strategy
• Run Results: Detailed run information about Status, Duration and Token counts
• LLM Data: All information processed for the specific run
• Instruction: Instruction valid at the time of run
• Context: Context as defined at the time of run
• Tools: Tools assigned to and used by Task at the time of run
• Chat: User Message used in run as well as Response
• Raw: Entirety of LLM input data based on the aforementioned details AND Raw LLM output data.

Click Steps tab to get detailled information broken down by steps taken during task execution

• Step Type: Message Creation, Tool Calls
• Processing Status: complete, incomplete,error, initializing
• Step Duration: in seconds
• Tokens per second: LLM output/execution speed
• Token counts: Prompt Tokens, Completion Tokens and Total Tokens for this step of the run
• Tool Calls: Request and Response information for the individual tool call

Click Timing tab to get visual insights into detailed runtimes for each action.

Tasks Monitoring Run Timing

Data Source Execution Monitoring

• Go to the Data Source Management section within your project.
• Select a Data Source and click on the Tool Calls tab to view its execution history.
• You will see a list of all the times the Tool was called, along with details such as
  • Assistant (and Task) that triggered the tool call
  • Conversation id
  • Response status
  • Request function
  • Date and Time of call
  • Tools Monitoring Overview

Click any table entry to get further information on the specific tool execution

The API-Request tab provides additional information