Analysis of Unresolved Conversations

When the Request Analyzer service auto-categorizes the bot user Requests, it creates the following visualizations and analytics:

1. Top Flow Gaps - This visualization identifies the flows that were served in requests, which were subsequently tagged as either "Flow Gap Terminal Node" or "Flow Gap Non-Terminal Node."

2. Top KB Gaps - This visualization identifies the knowledge documents that were served in requests, which were subsequently tagged as “KB Gap”.

3. Top Recommended Actions - This visualization identifies top recommended/ suggested actions that a user can take to address the top gaps affecting the highest number of requests. Each of these recommended actions will correspond to a possible impact that the user can make upon addressing the gaps.

The Auto-Categorization algorithm identifies and tags all requests where the application (bot) served a correct RAG response / Knowledge Base Article fulfillment, but still received negative feedback from the end user.

Possible impact = Number of requests with a given gap (auto-tag) / Total number of unresolved requests

The goal of Unresolved Analytics is to:

• Get a holistic view of Unresolved Requests:

  • Type of Unresolved Requests

  • Track Top Unresolved Intents

  • Track Not Understood and Not Helpful Rate - Not Understood Requests are requests that have no intent identified by intent identifying engine of Aisera’s bot. However, a correct fulfillment could have been served from a different engine. On the other hand, Not Helpful Requests are requests where a fulfillment was served but the user clicked thumbs down

• Find appropriate Intents, KBs and Flows for Unresolved Requests

Unresolved Analytics has four major components:

• Not Understood Request Clusters

• Not Helpful Request Intents

• Top Unresolved KPIs

• Auto-Categorization

How does the Aisera platform handle Clusters that are Not Understood?

Step 1: View Top Cluster that are Not Understood

Aisera automatically clusters all “Not Understood” conversation requests by looking at existing entities from ontology. Clustering job is executed periodically every day automatically. Each phrase cluster can contain multiple unresolved conversation requests.

Step 2: Select Cluster

Select a cluster that has a large bucket of unresolved conversations. Each phrase cluster contains information on cluster name, number of Requests and date/time stamp when it was last updated.

Step 3: Take on of the following Actions:

Once a cluster has been identified and scanned, admin needs to click on the action menu. The two possible options are:

1. Add to an existing intent

2. Create a new Intent

Step 4: Select Relevant Intent

Select all or most applicable phrase clusters. Click on Action 🡪 Add to existing intent and then select the relevant intent that phrase applies to.

Step 5: Preview Intent Modifications

1. Review all phrases to add to existing intent

2. Add/Remove/make modifications to phrases inline

3. Review existing phrases of intent as well to get context and click on

4. This action also removes these phrases from existing cluster in real-time

Step 6: Intent Review

1. Go to Intent Review Page

2. View the newly annotated intent with phrases

3. View the Review Status of the annotation as “Needs review”, “Approved – Validated” and “Rejected”

4. Admin has option to either approve or reject the proposed changes

Note: The Annotator role can propose changes to Intents but cannot approve or reject changes.

Step 7: Run Validation job

1. Once the intent changes have been approved, proceed to run the “Validate” job

2. This is equivalent to taking all newly added phrases and running it through AI lens

Step 8: View Validation Success

1. Once validation is successful, you can go back to AI lens and try to do a spot check

2. Spot check should include if the newly added phrases are associated with expected intent

How does the Aisera platform handle Requests that are Not Helpful?

Step 1: View Top Requests that are Not Helpful

1. Aisera automatically clusters all “Not Helpful” conversation requests by looking at existing entities from ontology

2. Clustering job is executed periodically every day automatically

3. Each phrase cluster can contain multiple unresolved conversation requests

Step 2: Select a cluster or multiple requests

1. Identify and Select a cluster that has a large bucket of Not Helpful conversations

2. Each phrase cluster contains information on cluster name, number of Requests and date/time stamp when it was last updated

Step 3: Analyze Missing Information in Fulfillment Attached

• For Workflow >> Optimize the Flow

• For Knowledge >> Enrich the KB