PrimeNG + NestJS + STT API = Speech Analytics

[Sanatel Consulting]

We developed speech analytics for a contact center. Speech recognition is handled via the Yandex Speechkit service, while the analysis of the resulting text is performed within our own solution. During development, we encountered some interesting aspects that I will try to describe.

1. Audio player in the transcript card.

The transcript card looks like all other transcript cards in analytics: information about the audio recording and a player slider at the top, and the dialogue replicas at the bottom.



When clicking on a replica, the player should jump to the corresponding fragment of the audio. This turned out to be simple: we read the audio element from the document, and this element already has the necessary methods. We added a start_time field to replicas, measured in seconds from the beginning of the audio recording. When clicking on a replica, we call the method of the audio element and pass the required seconds:

this.audio = document.querySelector('audio');

playAudioOnClickOnItem(time: number) {
this.audio.currentTime = time; // В секундах
this.audio.play();
}

2. Highlighting words in replicas and tags.

In replicas, we need to underline or highlight with color the words found in dictionaries. Also, in replicas, there is a column where dictionary tags are placed. The question arose: where should we prepare this HTML markup of replicas? On the one hand, it seems like a frontend task. On the other hand, since the backend analytics already parses the replicas and counts the words, why not also prepare the tags there? Moreover, if we generate the HTML markup on the frontend, the backend would still need to send data about where and how many words were found for each replica. In the end, we settled on preparing it on the backend. For each replica, the backend sends a ready-made HTML string with the necessary styles, and the frontend simply displays them.



3. Editing words in a fragment.

Fragments are needed to compose a conversation script to control adherence to scripts during dialogue. A fragment consists of a set of words. The frontend receives from the backend a fragment entity, and inside it is an array of words included in this fragment. On the frontend, we created a form in which the user can change the set of words, correct words, etc. When the “Save” button is pressed, the fragment with the new word array is sent back to the backend.



4. Selecting columns in the records registry.

Usually, the registry of records does not contain many columns, so the set of columns is static. But in the case of audio transcripts, the entity can have a lot of fields, ranging from “Incoming” or “Outgoing” call to department, city, call score, number of words from one side, and so on. Therefore, in the call registry in analytics, there should be a choice of columns that the user considers necessary. The only static (mandatory) column is the clickable column with the call ID, which links to the call card.



The selected set of columns in the registry is saved in cookies in the browser, and when the user re-enters the section, the selected columns are restored.

5. JWT and user account.

There are many articles and videos on the Internet about authorization and roles in NestJs, but the topic is often not fully covered. Typically, the author explains how to create a JWT token, declares strategies and guards, and shows that authorization works — and stops there. But that’s only about 5–10% of the topic of authorization and user roles. First, the backend needs to understand which user is making the request, and then work with that.

To achieve this, we declare a simple decorator. In this example, the user account is equal to their email. The decorator selects the fields available in the payload of the JWT token. If our JWT token contains:

async login(user: any) {

const payload = {
email: user.email,
username: user.username,
role: user.role
};

return {
access_token: this.jwtService.sign(payload),
};
}

Then in the decorator, we can extract any field from the payload, for example, the email:

export const User = createParamDecorator(
(data: any, ctx: ExecutionContext): string => {
const request = ctx.switchToHttp().getRequest();

if(
request.user !== undefined
&&
request.user.email !== undefined
)
{
return request.user.email;
}
else {
return 'unknown_user'
}
});

Then in the controller, we receive the user data and can do something with it, for example, check access rights to read records:

import { User } from '../user/user.decorator';

@Get()
@ApiOperation({ summary: 'Get coversation list' })
@ApiOkResponse({ description: 'Coversation list', type: SpeechEntity, isArray: true })
findAll(@User() userMail: string): Promise<SpeechEntity[]> {
return this.speechService.findAll( userMail );
}

6. Roles and access rights — the most controversial topic.

Several questions arise:

Where in the NestJs project architecture is it correct to check access rights? In the controller or already in the service?

It seems logical to check access rights in the controller and pass conditions to the service for search. For example, a user has rights to read only the records where they are responsible.

Then:

@Get()
@ApiOperation({ summary: 'Get contact list' })
@ApiOkResponse({ description: 'Contact list', type: ContactEntity, isArray: true })
findAll( @User() userMail: string ): Promise<ContactEntity[]> {
const userAccessWhereOptions = this.userService.getUserAccessWhereOptions( userMail, controllerName );
return this.contactService.findAll( userAccessWhereOptions);
}

But inside the findAll service, there may be its own conditions for search, for example, category = new, which will need to be attached to the conditions of responsibility. And there might also be cases where “own” records are those where the user is either responsible or the author. Then the findAll service would receive two OR conditions, and we would have to add the category = new condition to both of them.

Additionally, there’s the issue of deleting records, which in reality is not deletion but setting deleted = yes. A regular user sees “own” records under the condition deleted = no, while an admin sees all records.

So far, we’ve settled on this: the access rights service returns the condition for find, already considering access rights, and also with the condition for deleted records.

where = [
{
'owner': userEmail, // Responsible for record
'deleted': 'no'
}, // OR
{
'father': userEmail, // Creator of record
'deleted': 'no'
}
];

And inside findAll, there’s a function that iterates through all conditions and adds its own conditions to them:

async findAll( userAccessWhereOptions: {}[] ): Promise<ContactEntity[]> {
const options: FindManyOptions = {
where: UtilWhereAssign(
userAccessWhereOptions,
{
'category': 'new'
}
),
order: { updated: 'DESC' },
};
const result = await this.contactRepository.find( options );
return result;
}

From UtilWhereAssign, for a “regular user,” a new search condition will be returned:

where = [
{
'owner': userEmail,
'deleted': 'no',
'category': 'new'
}, // ИЛИ
{
'father': userEmail,
'deleted': 'no',
'category': 'new'
}
];

7. On the frontend, we consider user rights.

But we also need to inform the user on the frontend that they don’t have rights for the requested operation. The backend has a method that returns the user’s role. Accordingly, if a user doesn’t have rights to add records, the frontend should disable the “Add record” or “Delete record” button, etc.



8. Change history.

Since we need to keep a log of who made what changes to entities, we create a historyService, and in all create, update, and remove methods in various services, we pass to the historyService data about the ID of the modified record and the user.

To avoid implementing change history creation for each entity, we created a universal method calcChangeAndCreate in historyService, to which we pass the old entity instance, the new entity instance, and the username. Example of update for a contact:

async update(recordId: string, updateDto: ContactUpdateDTO, userMail: string): Promise<ContactEntity> {
const options = {
where: [{ 'id': recordId }]
}
const oldResult = await this.contactRepository.findOne( options );
await this.contactRepository.update( { "id": recordId }, updateDto);
const newResult = await this.contactRepository.findOne( options );

await this.historyService.calcChangeAndCreate( 'contact', newResult, oldResult, userMail, recordId );

return newResult;
}

Inside the calcChangeAndCreate method, we compare the fields of the old and new objects, and if there are changes, we create a record in the change history:

calcChange( newEntity: object, oldEntity: object ): HistoryRecordChange {
let result = {} as HistoryRecordChange;
result.fieldChangeList = [];
result.haveChange = false;
result.user = 'no_user';
result.table = 'no_table';
result.record = 'no_record';

for( let key of Object.keys( newEntity) ) {
if( key != 'created' && key != 'updated' ) {

let newValue: string;
if(
typeof newEntity[key] == 'object'
&&
newEntity[key] != null
) {
newValue = newEntity[key].toISOString() + ' (UTC)';
}
else {
newValue = newEntity[key];
}

let oldValue: string;
if(
typeof oldEntity[key] == 'object'
&&
oldEntity[key] != null
) {
oldValue = oldEntity[key].toISOString() + ' (UTC)';
}
else {
oldValue = oldEntity[key];
}

if( newValue != oldValue ) { // Field changed
result.haveChange = true;
let fieldChange = {} as HistoryFieldChange;
fieldChange.field = key;
fieldChange.oldValue = oldValue;
fieldChange.newValue = newValue;
result.fieldChangeList.push( fieldChange );
}
}

return result;
}

The same applies to creating and deleting entities. There are also child entities, for example, a phone number in the contact’s phone list. Changes to child entities are recorded in the history of the parent entity.

Accordingly, we add the section “Change History”.



Continue reading...

 

Join 𝕋𝕄𝕋 on Telegram
Channel PREVIEW: