Multimodality as a Subject in Foreign Linguistic Studies

   Relevance. Modern life is characterized by using information technologies in all spheres, which affects both communication process and approaches to its studying. However, as can be concluded from our experience, despite technologically advanced communication forms, the process is often ineffective, because students lack the skills to extract information from multimodal texts. Consequently, the problem of developing the skills for effective communication in the digital multimodal environment which depend rather on the skills of analytical and synthetic information processing than on the computer literacy skills, remains unresolved.
   The purpose of this work is to comprehensively analyze foreign studies on multimodality issues to reveal research and development trends.
   The objectives set by the authors involve collection, analysis, classification, and systematization of the foreign studies data on multimodality; clarification of the concepts of ‘multimodal text’ and ‘multimodal analysis’; identification of the multimodal analysis implementation techniques when reading/viewing multimodal scientific texts in order to increase communication effectiveness.
   Methodology. To fulfil the objectives set we applied such general scientific methods as acquisition and processing of information, methods of classification and systematization.
   Results. The results obtained showed that the issues related to multimodality are considered from different positions: multimodality as a phenomenon and as an approach to its studying; a multimodal text is a text which use several modes of communication simultaneously; the concept of ‘literacy’ is expanded to ‘multiliteracy’, interpreted as an ability to ‘read the world’. A multimodal analysis is defined as the analysis, search, and extraction of semiotic patterns in multimedia databases in order to explore the relationship between semiosis, text, and context, those patterns that reveal individual, social, cultural and global patterns of meaning creation. Practice-oriented studies propose various models for multimodal analysis implementation: J. Álvarez’s fourcomponent model, F. Serafini’s model, and a Model for Cultivating Multimodal Literacy by K. Danielsson and K. Selander. A number of authors analyze the work of students/children when reading / visualizing digital multimodal texts, but they investigate perception rather than methods of information processing.
   Conclusion. Thus, in the presence of an extensive multimodal communication theoretical and practical developments, the analyzed works do not provide algorithms for performing multimodal analysis. In this regard, the problem raised by the authors remains topical, and the research requires further development to improve learning process effectiveness when using digital multimodal scientific texts.

1. Annenkova A. V. Verbal'nye i vizual'nye kody social'noj reklamy: opyt eksperimental'nogo issledovaniya [Verbal and visual codes of social advertising: the experience of experimental research]. Uchenye zapiski: elektronnyj nauchnyj zhurnal Kurskogo gosudarstvennogo universiteta = Scientific Notes: electronic scientific journal of Kursk State University, 2014, no. 2 (30), pp. 117–122. Available at: https://cyberleninka.ru/article/n/verbalnye-i-vizualnye-kody-sotsialnoy-reklamy-opyt-eksperimentalnogo-issledovaniya

2. Annenkova A. V. Modelirovanie processa formirovaniya cennostnogo otnosheniya k budushchej professii kak otrazhenie realij sovremennogo obshchestva (na primere budushchego uchitelya inostrannogo yazyka) [Modeling of the process of forming a value attitude to the future profession as a reflection of the realities of modern society (on the example of a future foreign language teacher)]. Izvestiya Yugo-Zapadnogo gosudarstvennogo universiteta. Seriya: Lingvistika i pedagogika = Proceedings of the Southwest State University. Series: Linguistics and Pedagogics, 2021, vol. 11, no. 1, pp. 94–112.

3. Levin A. I., Levina L. V. Kompetentnostnyj podhod v podgotovke upravlencheskih kadrov kak faktor preodoleniya funkcional'noj negramotnosti [Competence-based approach in the training of managerial personnel as a factor in overcoming functional illiteracy]. Izvestiya Yugo-Zapadnogo gosudarstvennogo universiteta. Seriya: Lingvistika i pedagogika = Proceedings of the Southwest State University. Series: Linguistics and Pedagogics, 2021, vol. 11, no. 1, pp. 124–134.

4. Zanina O. N., Mahova V. V. [Formation of foreign language speech competence among law students]. Koncept i kul'tura. Sb. nauch. st. IV Mezhdunar. nauch. konf. [Concept and culture. Collection of scientific articles of the IV International Scientific Conference]. Kemerovo, 2010, pp. 591-595 (In Russ.). Developing L2 communicative competence in law students training

5. Zanina O.N. Rechevye oshibki kak material issledovaniya [Speech errors as a research material]. Izvestiya Yugo-Zapadnogo gosudarstvennogo universiteta. Seriya: Lingvistika i pedagogika = Proceedings of the Southwest State University. Series: Linguistics and Pedagogics, 2012, no. 2, pp. 69-72.

6. Zanina O. N. [Some aspects of teaching translation in the field of professional communication: the formation of supports of the internal language]. Obuchenie inostrannomu yazyku: sovremennost' i perspektivy. Sb. nauch. st. region. nauch.-metod. konf. [Teaching a foreign language: modernity and prospects: collection of scientific articles of the region. scientific method. conf.] Kursk, 2019, pp. 127-132 (In Russ.).

7. Zanina O. N. [Some features of teaching adults a foreign language (from the experience of teaching English in groups of the teaching staff of the Southwestern State University)]. Yazyk dlya special'nyh celej: sistema, funkcii, sreda. Sb. nauchnyh statej VII Mezhdunar. nauch.-prakt. konf. [Language for special purposes: system, functions, environment. Collection of scientific articles of the VII International Scientific and Practical Conference]. Kursk, 2018, pp. 69-74 (In Russ.). Some Issues of Teaching Adults a Foreign Language (in Academic Teaching Staff Groups)

8. Mahova V. V., Kevin M. B. [Specificity of modern English-language scientific and technical texts]. Yazyk dlya special'nyh celej: sistema, funkcii, sreda. Sb. nauchnyh statej VIII Mezhdunar. nauch.-prakt. konf. [Language for special purposes: system, functions, environment. Collection of scientific articles of the VIII International Scientific and Practical Conference]. Kursk, 2020, pp. 291-299 (In Russ.). The Features of Modern Scientific and Technical Texts

9. Mahova V. V. [Modern approaches to teaching textual activity]. Obuchenie inostrannomu yazyku: sovremennost' i perspektivy : Sbornik nauchnyh statej regional'noj nauchno-metodicheskoj konferencii [Teaching a foreign language: modernity and prospects. Collection of scientific articles of the regional scientific and methodological conference]. Kursk, 2019, pp. 255-268 (In Russ.). Modern Approaches to Textual Activities Teaching

10. Mahova V. V. Vliyanie kognitivnyh skhem na aktualizaciyu fonovyh znanij i adekvatnoe izvlechenie informacii iz inoyazychnogo nauchnogo teksta [The influence of cognitive schemes on the actualization of background knowledge and adequate extraction of information from a foreign scientific text]. Kognitivnye issledovaniya yazyka = Cognitive Studies of Language, 2016, no. 27, pp. 524-532. Influence of Cognitive Schemes on Background Knowledge Actualization and Relevant Extraction of Information from Foreign Language Scientific Text.

11. Mahova V. V. Smyslovoj analiz nauchno-tekhnicheskih tekstov kak sposob formirovaniya inoyazychnoj kompetentnosti v sfere professional'noj kommunikacii (na primere obucheniya studentov neyazykovyh vuzov) [Semantic analysis of scientific and technical texts as a way of forming foreign-language competence in the field of professional communication (on the example of teaching students of non-linguistic universities)]. Kursk, 2015. 207 p. Concept analysis of scientific and technical texts as a way of forming foreign language competence in the field of professional communication (on the example of teaching students of non-linguistic universities).

12. Mahova V. V. [From the experience of using interactive methods of teaching a foreign language (On the example of studying the topic " My University”]. Yazyk dlya special'nyh celej: sistema, funkcii, sreda. Sbornik nauchnyh statej VII Mezhdunarodnoj nauchno-prakticheskoj konferencii [Language for special purposes: system, functions, environment. Collection of scientific articles of the VII International Scientific and Practical Conference]. Kursk, 2018, pp. 193-200 (In Russ.). The Experience of Implementing Interactive Methods of Foreign Language Mastering (Mastering the Topic “My University”)

13. Mahova V. V., Zanina O. N. O rezul'tatah vnedreniya metodiki formirovaniya professional'noj inoyazychnoj kompetentnosti u studentov-yuristov [On the results of the implementation of the methodology for the formation of professional foreign language competence among law students]. Voprosy prikladnoj lingvistiki = Questions of Applied Linguistics, 2010, is. 3.

14. Niederhauser D. Educational hypertext research. In J. M. Spector, M. D., Merrill, J. van Merrienboer, & M. P. Driscoll (Eds.), Handbook of research on educational communications and technology. 3^rd ed., New York: Lawrence Erlbaum Associates Publishers, 2008, pp. 199–210

15. Shapiro A., Niederhauser D. Learning from hypertext: Research issues and findings. In D. Jonassen (Ed.), Handbook of research for educational commu-nications and technology 2^nd ed., Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers, 2004, pp. 605–620.

16. Spiro R. J., Feltovitch P., Jacobson M., & Coulson R. Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in illstructured domains. In T. Duffy and D. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation Hillsdale, NJ: Lawrence Erlbaum, 1992, pp. 57–75. Available at: https://pdfs.semanticscholar.org/0d37/bee40117f53104a1522eb301f4b285db3c18.pdf (accessed 05. 04. 21).

17. O’Halloran K. L., A. Smith Bradley. Multimodal Text Analysis. Available at: http://multimodal-analysis-lab.org/_docs/encyclopedia/01-Multimodal_Text_Analysis-O%27Halloran_and_Smith.pdf (accessed 10. 10. 20).

18. Kitson L., Reconceptualising Understandings of Texts, Readers and Contexts: One English Teacher’s Response to Using Multimodal Texts and Interactive Whiteboards English in Australia, 2011, vol. 46, no 3. Available at: https://core.ac.uk/download/pdf/143854887.pdf (accessed 10. 10. 20).

19. Kress G. R., Leeuwen T. Reading images: the grammar of visual design / Gunther Kress and Theo van Leeuwen. 2^nd ed. Routledge. Taylor & Francis Group: London and New York, 2006. 321 p.

20. O'Halloran K. L. (in press). Multimodal Analysis and Digital Technology. In A. Baldry and E. Montagna (Eds.), Interdisciplinary Perspectives on Multimodality: Theory and Practice. Campobasso: Palladino. Available at: http://multimodal-analysis-lab.org/_docs/Multimodal%20Analysis%20and%20Digital%20Technology.pdf (accessed 15. 04. 21).

21. Cazden C., Cope B., Fairclough N., Gee J. et al. A pedagogy of multiliteracies: Designing social futures Harvard Educational Review; Spring 1996; pp. 60-92. Available at: http://newarcproject.pbworks.com/f/Pedagogy+of+Multiliteracies_New+London+Group.pdf (accessed 21. 11. 20).

22. Multiliteracies: Literacy Learning and the Design of Social Futures. Edited by Kalantzis, M., Cope, B., London: Routledge, 2000. 288 p.

23. Cope B., Kalantzis M. ‘Multiliteracies’: New Literacies. New Learning. 2001. Available at: https://newlearningonline.com/files/2009/03/M-litsPaper13Apr08.pdf (accessed 21. 11. 20).

24. Fairclough N., Critical discourse analysis: The critical study of language. London: Long-man, 1995, pp. 265. Available at: https://www.felsemiotica.com/descargas/Fairclough-Norman-Critical-Discourse-Analysis.-The-Critical-Study-of-Language.pdf (accessed 21. 11. 20).

25. Anstey M., Bull G. Helping teachers to explore multimodal texts. Curriculum & Leader-ship Journal: an electronic journal for leaders in education, 2010, vol. 8 (16). Available at: http://www.curriculum.edu.au/leader/helping_teachers_to_explore_multimodal_texts,31522.html?issueID=12141 (accessed 29. 11. 20).

26. Kress G. Social'naya Semiotika i vyzovy mul'timodal'nosti. Politicheskaya nauka, 2016, no. 3, pp. 77-100. Available at: https://elibrary.ru/item.asp?id=27429205 (accessed 29. 11. 20).

27. Álvarez J. Meaning Making and Communication in the Multi-modal Age: Ideas for Language Teachers. Colomb. Appl. Linguist, 2016, 18(1), pp. 98-115. doi: 10.14483/calj.v18n1.8403 (accessed 29. 11. 20).

28. Serafini F. Expanding the four resources model: reading visual and multi-modal texts. Pedagogies. An International Journal, 2012, vol. 7, no. 2, pp. 150-164. doi: 10.1080/1554480X.2012.656347 (accessed 29. 11. 20).

29. Freebody P., Luke A. Literacies programs: Debates and demands in cultural context. Prospect: Australian Journal of TESOL, 1990, 5 (7), pp. 7–16. Available at: https://eprints.qut.edu.au/49099/ (accessed 15. 04. 21).

30. Tandiana S. T., Abdullah F., Saputra Yu. Learning Multimodality through Genre-Based Multimodal Texts Analysis: Listening to Students’ Voices. Vision: Journal for Language and Foreign Language Learning, 2020, vol. 9, no. 2. Pp. 101-114. doi: 10.21580/vjv9i25406. Available at: https://www.researchgate.net/publication/344772851_Learning_Multimodality_through_Genre-Based_Multimodal_Texts_Analysis_Listening_to_Students%27_Voices (accessed 12. 03. 2020).

31. Chenkai C., Sefton T. The Creation of Multimodal Texts of Adult English Eanguage Learners in a Canadian university – A Case Study. ERL Journal (Re-)Shaping One’s Identity with Language, 2020, vol. 2 (4), DOI: 10.36534/erlj. 2020.02.11. Pp. 100-114 Available at: https://www.researchgate.net/publication/350316554_The_creation_of_multimodal_texts_of_adult_English_language_learners_in_a_Canadian_university_-_A_case_study (accessed 12. 03. 2020).

32. Danielsson K., Selander S. Reading Multimodal Texts for Learning – a Model for Cultivating Multimodal Literacy. Designs for Learning, 2016, no. 8 (1), pp. 25–36. doi: 10.16993/dfl.72 (accessed 12. 03. 2020).

33. Philippe S., Souchet A. D., Lameras P., et..al. Multimodal teaching, learning and training in virtual reality: a review and case study. Virtual Reality & Intelligent Hardware, 2020, vol. 2 (5), pp. 421-442. doi: 10.1016/j.vrih.2020.07.008 (accessed 20. 03. 2020).

34. Abdu R., Helden, G., Alberto, R., Bakker, A. Multimodal dialogue in small-group mathematics learning. Learning, Culture and Social Interaction, 2021, vol. 29. doi: 10.1016/j.lcsi.2021.100491 (accessed 09. 04. 21).

35. Strømman E. Crossover literacies: A study of seventh graders’ multimodal representations in texts about Pokémon Go. Computers and Composition, 2021, vol. 59. doi: 10.1016/j.compcom.2021.102629 (accessed 09. 04. 21).

36. Kuo F. Yua P.-T., Hsiao W.-H. Develop and evaluate the effects of multimodal presentation system on elementary student learning effectiveness: within classroom English learning activity. Procedia - Social and Behavioral Sciences, 2015, vol. 176, pp. 227-235. Available at: https://www.sciencedirect.com/science/article/pii/S1877042815005029 (accessed 09. 04. 21).

37. Education for All Global Monitoring Report 2006. Chapter 6 Understandings of literacy. Available at: http://www.unesco.org/education/GMR2006/full/chapt6_eng.pdf (accessed 09. 04. 21).