Sign language is a communicative system based on gestures (hand and facial gestures, lip articulation and emotions) rather than sound form. It is a natural visual- spatial language that is used for communication of the deaf and those who are hard of hearing.

In sign languages, information is encoded by hand movements, body, facial expressions and perceived visually. This determines the ability of sign language to express the emotions and feelings of a deaf person through a variety of language means of emotional vocabulary through the expression of movement and emotional facial expressions on the face of the sign language speaker.

Given the peculiarities of sign language reproduction, which require a frank and more expressive emotional and articulatory process on the face and gestures, the analysis of emotion and articulation in sign language reproduction is a very important factor in understanding the semantics of information transmitted with its subsequent elemental synthesis (Krak, Lozynska, Pasichnyk, Ternov & Shkilnyuk, 2016).

The scope of modern automatic gesture recognition systems comprise the recognition and modelling of dactyl language, emotional-articulatory visual component of the speech process, technologies for gestures reproduction and translation from verbal language into sign language. The abovementioned activities are of great interest and performed in collaboration with IT specialists, whereas linguistics rather focus on translation and interpretation of fingertelling into the verbal language.

There are two tasks that can be distinguished in the translation of sign language into the language of words and sounds: sign recognition and summarization and generation of coherent text based on key words and expressions obtained from word- to-word translation of a sign phrase. Taking into account the polysemy of gestures it is necessary to carefully monitor the correspondence of the generated text to the current subject area. On the other hand, the translation of natural language into sign language poses some problems as well.

With an increase in the level of education of the deaf audience the number of gestures used increases and the structure of utterances approaches the norms of the language, i.e. each word (or a combination of words) has its own gesture, and the sequence of gestures corresponds to the sequence of words in a phrase. So when working with the sign language, the interpreter should first translate the sentence into “the language of the deaf” creating the required sequence of gestures before actually showing gestures (Voskresenskij, 2002).

In educational, scientific and technological contexts, the urgent task is to develop software that helps in the study and translation of Ukrainian sign language. The problem of developing multimedia systems for recognizing elements of dactyl language as one of the means of communication between people with hearing impairments is quite relevant for domestic science and the world community. Today, many leading research institutions are working on the development of sign language recognition systems which are divided into the following groups:

• gesture recognition systems using gloves with different types of sensors;
• gesture recognition systems using 3D cameras and stereo cameras;
• gesture recognition systems using video cameras (Krak et al., 2016).

Such systems work by tracking the movement of the hand, recognizing the sequence of gestures shown and displaying them in a text form. Such a task is rather difficult for several reasons:

• there are gestures in the manual alphabet that are difficult to distinguish even for a person;
• the speed of fingerspelling may vary during different displays of the same gestures;
• the size of the palm differs significantly from person to person;
  • factors such as different skin colors and changes in illumination should be taken into account;
  • gesture recognition must be done in real time;
  • regardless of the rules, each person has his own peculiarities of gestures;
  • as a rule, dactyl communication is performed smoothly and jointly which complicates the process of segmentation of the gestures (Nagapetian & Hachumov, 2013);
  • a high percentage of errors in translating ambiguous words and homonyms;
  • incomplete description of the sign language grammatical system (Grif, Korolkova & Prihodko, 2021).

The sign language of the deaf is a rather complex language. It uses grammar and gesture rules just like natural language. The manual alphabet is only a part of the gestures used in the language of the deaf and dumb. Despite this, the automatic hand­alphabet recognition system paves the way for more natural human-machine interfaces, removes the communication limitations that deaf people face in their daily lives.

References

1. Grif, M., Korolkova, O., & Prihodko, A. (2021). Raspoznavanie jzestovoy rechi s uchetom kombinatornyh izmeneniy jzestov [Sign language recognition with combinatorial gesture changes]. Retrieved from: https://cutt.ly/rRHL7h5
2. Krak, Yu., Lozynska, O., Pasichnyk, V., Ternov, A., & Shkilnyuk, D. (2016). Matematychni metody ta prykladni informatsiyni tehnolohii modeliuvannya, perekladu ta navchannia dlia ukrainskoi jzestovoi movy [Mathematical methods and applied informational technologies of modelling, translation and education for Ukrainian sign language]. Visnyk Natsionalnogo universytetu “Lvivska politehnika” (pp. 210-227). Retrieved from: http://science.lpnu.ua/sites/default/files/iournal-paper/2018/jun/12982/21210-227.pdf
3. Nagapetian, V., & Hachumov, V. (2013). Avtomaticheskoe preobrazovanie jzestov russkoi ruchnoi azbuki v tekstovyi vid [Automatic conversion of Russian
4. manual alphabet gestures in text form], Iskusstvennyi intelekt i priniatie reshenii. (pp. 59-66). Retrieved from: http://www.isa.ru/aidt/images/documents/2013-03/59 66.pdf
5. Voskresenskij, L. (2002). Nepriznannyi yazyk (Yazyk jzestov gluhih i kompiuternaia lingvistika) [Unacknowledged language (sing language and computing linguistics)]. Proceedings of International conference on computational linguistics and intellectual technologies “Dialogue 2002 ”. Retrieved from: http://www.dialog- 21.ru/digest/2002/articles/voskresenskyi/