Article Type: Research Article

 

Article Citation: Patrick Anderson Matias de Araújo, and Eder Ahmad Charaf Eddine. (2020). THE REVIEWS OF USERS OF THE DUOLINGO APPLICATION: USABILITY AND OBJECTIVITY IN THE LEARNING PROCESS. International Journal of Research -GRANTHAALAYAH, 8(9), 5-15. https://doi.org/10.29121/granthaalayah.v8.i9.2020.1326

 

Received Date: 24 August 2020

 

Accepted Date: 21 September 2020

 

Keywords:

Teaching of Foreign Language

Mobile Applications

Human Computer Interaction Duolingo
ABSTRACT

Since 2011, the Duolingo platform helps those who seek to learn a second language. Present in all popular mobile operating systems, it offers, in the Brazilian interface, the teaching of Spanish, French, and English language, being English the first to be made available in the platform. Given the platform’s mission of personalized teaching, making learning fun, and universally accessible, the research aims to analyze the opinions of users on the Google Play Store. The objective was to understand the usability process, ease of access, and the user-learning relationship of the application and to achieve it, it was used the assumptions of Human-Computer Interaction, that investigates the form of communication between people and computational systems, as well as forms of accessibility and facilitation of this communication. It will be used the quantitative methodology of the comments available on the digital store in the entire month of January 2020. The results show that 85.13% of the users rated the tool with the highest score, the scores range from 1 to 5, and that usability is a positive component that facilitates learning. It is noteworthy, though, that the tool can be used as an aid in foreign language teaching processes.

1.      INTRODUCTION

 

The National Common Curricular Base is a document that defines curricular and pedagogical guidelines for Brazilian children, elementary and high school education. Approved in 2018, this document changes the perspective for educational policies and places the student as the protagonist. One aspect that draws attention in the set of rules is what concerns digital inclusion and technologies, that is, everything that mankind make to let their lives more pleasant and simple, such as the human speech, the alphabet, writing, numbers, mathematics, musical notation, art, etc., all of this is technology, that is, Information and Communication Technology. [1].

Currently, the greatest technological representation is the computer, be it the microcomputer, laptop, smartphone, or any other device. Computers are so present in our society and their importance is unquestionable. The way we work, play, study and communicate are evolving. [2]

The need to merge the new technologies with the existing ones in the classroom is a challenge to improve learning, considering that the technology can be used both inside and outside it. In both cases, as a complement and aid to the teaching and learning process, if well used, provided that the teachers are acquaintance and have the autonomy to decide when and with what strategies they will use them. The internet can exponentially increase the horizons of the classroom as well as being a means of distraction for the student if there is no adequacy of the purposes of using it. At the beginning of the 21st century, making the class interesting and iterative is also a challenge when considering that the entering high school student must probably have been born after the turn of the millennium, that is, immersed several devices with a broadband connection and with a lot of information.

This “information” should also be the subject of discussion and debate both inside and outside the school to educate and instruct how to proceed in the face of so much information, in addition to classifying and distinguishing genuine information from that considered false and harmful. There is also a need for an approach on how to proceed on social networks and instant messaging applications, in view of the efficiency of these channels in spreading and propagating hate speech, non-relevant information, and fake news. This type of content can confuse and mislead less educated people on how to proceed in this digital environment.

In order to start researching how the technological tools that can assist in the teaching and learning processes, the research proposes, at first, to research what the users of the Duolingo tool think about the platform. There are several online forums that group user comments about applications. Considering that the most used operating system until the survey was conducted is Android, which has an application store called the Google Play Store, for this reason, it was decided that the evaluations of this tool would be used, which are available both in the application for Android devices and in the website.

The research proposes to analyze the comments made by Portuguese speakers on Duolingo, a technological tool that helps in the process of teaching and learning a foreign language. For the analysis corpus, comments from the Google Play Store platform were selected in the period from January 1 from 2020 to January 31, 2020. Were found 10,943 occurrences. The comments are named by the platform as review and are limited to 500 characters and have notes, the user can choose from one to five stars. With this period defined, the research proposes to verify and analyze the users’ evaluation and use the evaluations.

The Duolingo app, is a language teaching platform created by Luis von Ahn and Severin Hacker, in the United States, to assist in foreign language learning. For Portuguese speakers, he teaches German, Italian, English, French, Esperanto, and Spanish. In the Google Play Store^[1] application store, the tool has more than 100 million^[2] downloads and is ranked first in the education category of the same store. In 2016, the Duolingo platform had around 120 million registered users and, at the beginning of 2020, that number exceeds 200 million.

 

2.      RESEARCH IN HUMAN-COMPUTER INTERACTION

 

Human-Computer Interaction is the area of study that investigates the form of communication between people and computer systems as well as ways of accessing and facilitating this communication. This area of study is an intersection between several sciences such as psychology, linguistics, the arts, and computing.

According to Chancellor et al. (2019), the recent research areas at HCI are concerned with examining areas considered poignant such as mental health, crises, transitions, harassment, and graphic content; the latter two being the cause of emotional and physical vulnerabilities.

As an example of research at HCI, [4], they introduced the concept of MIS in the context of social networks. MIS means Mental Illness Severity and is usually associated with cognitive functions, emotional instability, and limitations in performing mundane activities. Through the analysis of posts, tags, and users of Instagram (Social Network owned by Facebook), the research quantified MIS levels in the social network and reports that studies in HCI and CSCW (Computer Supported Cooperative Work) seek appropriate interventions for the population with mental illnesses, besides, to focus on identifying traces of mental illnesses, such as depression, but not as broadly as in MIS applied in any community prone to mental illnesses. The authors developed an approach that predicts an individual’s future risk.

 

Another article that addresses the specialty of mental health, examined the monitoring of patients with schizophrenia through the CrossCheck tool, which passively collects data on patient’s smartphone use in order to detect relapse [5]. The study concludes that there are statistically significant correlations between automatically monitored behavioral characteristics related to sleep, conversations, mobility, use of smartphones, and self-reported mental health indicators in schizophrenia.

Pater & Mynatt's (2017) research seeks to define digital self-mutilation. By digital self-mutilation, the researchers defined by the digital activity and communication that orientates, ratify, or encourages desired non-suicidal injury or harm of an individual and its physical wellbeing (Pater & Mynatt, 2017, p. 3). The researchers understand that HCI and CSCW research focuses on prosocial studies with positive or neutral results in encouraging health. Finally, they conclude that not studying this subject, the tendency is for HCI and CSCW research to continue with the idyllic look at health and outline an agenda for understanding the mutilation correlation.

Research on the HCI interface and teaching and learning has been gaining ground in recent years. The use of computational and virtual technologies in the educational process is not new, however, this use in foreign language teaching and, consequently, research in these areas, do not go beyond formal means of teaching [7]. Can also be said that human-computer interaction can affect the area of education, since, according to Chancellor et al. (2019, p. 3), “... HCI more broadly recognizes the importance of working through sensitive contexts …”.

The authors Ware et al. (2017) evaluated an English training and education program to determine its suitability for the elderly population, considering that research shows that learning a new language can be beneficial for the senile public. The researchers assessed that the intervention is feasible for the population in question, through applied questionnaires and the analysis of qualitative data.

Hadjerrouit's (2017) research, looked for the characteristics of the concepts of affordance and constraints in the use of the SimReal+ tool in the training of mathematics teachers. Affordance is an HCI term and can be summed up as the purpose of using a certain thing. Following that same line, El Janati et al. (2018), point out that the SMART learning tool is extremely useful in the teaching and learning process. Papadimitriou & Gyftodimos (2017), on the other hand, investigated the characteristics of learners through current adaptive educational systems. The article from the authors’ Strecker et al. (2018) seeks to understand the potential of technologies for higher education, so the authors interviewed four speakers with experience in teaching with technologies to inquire about their observations. These papers indicate that the educational system is constantly changing and that they come associated with new learning methods and techniques with the use of computer technologies. For this, the user interface must be improved to make it accessible to different users.

In this perspective, Lind et al. (2019) report that the teaching of technologies is compulsory in Swedish schools and that this means that there is a need for students to understand the impacts of the use of technologies on society.

In another type of approach, Juraschek et al. (2018) seek to introduce Mixed Reality applications in Learning Factories. The authors report that Mixed Reality is in increasing use in the development of human-machine interfaces since they can increase the effects of training and teach with technical systems.

Bartolomé et al. (2018) analyzed, through a literature review of the period between 1960 and 2015, the educational perspective that is present in the environments and experiences proposed in educational technology that is addressed in the literature of personalization in education. The text points out that programs for learning are developed with human-computer interaction as an important aspect. The authors point out that the greatest efforts of educational technologies focus on higher education and an explicit pedagogical perspective is lacking in the analyzed experiences.

Specifically, about Duolingo, a paper analyzes the tool in conjunction with the WhatsApp instant messaging application in order to observe its contributions as an aid to classroom French lessons. To this end, the authors Souza & Mourão (2017) reported on a pedagogical practice within the monitoring project at Federal University of Piaui. It was noted that the services can pedagogically enhance the learning of French.

Two other articles that address Duolingo seek to investigate the experiences of those who have used the application. In the article by the authors Marques-Schäfer & Orlando (2018), it is reported that the application is the most downloaded by language students, because it is free and because of its attractive interface. Through the analysis of data from a questionnaire applied to 114 learners of the platform, problems and dissatisfactions are found, such as a discrepancy between the discourse and the application’s practice, as well as a discrepancy between the user’s learning concept and their habits. The article by Şendurur et al. (2017), in addition to investigating user experiences, on the other hand, analyzes the reason for the upward preference of mobile learning apps, however, the 18 students interviewed believe that it is impossible to learn a new language only with apps, but they find the tool convenient.

Like the previous one, the article by the authors Guaqueta & Castro-Garces (2018) uses the tool in conjunction with the Kahoot application to improve vocabulary construction for English learners during a semester at a rural area high school. It is noted that the context of rural areas is a matter of controversy, due to the quality of the internet connection. To assess the effectiveness, tests were performed before the tool was started to being used, and at the end of the tool’s use.

Unlike the article by Marques-Schäfer & Orlando (2018), it was noted that there is evidence that the use of technology in conjunction with face-to-face classes can become an ally to improve vocabulary construction. The correlation between the performance of the course students’ performance and their grades obtained in the application was the focus of the study conducted by Brick & Cervi-Wilson (2019), on the other hand. The quantitative data obtained by the researchers revealed a weak correlation between the high marks obtained in the app and the grades obtained in the course evaluation.

Loewen et al. (2019) point to the lack of studies regarding the effectiveness of studying in second language learning tools, although, emphasize the authors, that a commissioned research found favorable learning results. The authors’ research seeks to investigate the experience and results of nine Turkish language learners during a semester using the Duolingo tool. The investigation showed improvement in the learning parameters of the apprentices, in addition to a positive/moderate correlation between the time spent in the application and learning gains. As in the research by Şendurur et al. (2017), the apprentices found the tool convenient, flexible, and positive gamification aspects. However, frustrations were also expressed with the instructional materials.

Finally, Lotherington (2018) evaluates the four most downloaded apps in the online app stores: Duolingo, Babbel, Busuu, and Memrise. The research was followed by empirical studies on the pedagogical and epistemological approaches used in these applications. A case study indicated that the applications depended on outdated pedagogical and structural concepts. It was also noted that the issue of mobility in pedagogical approaches to applications was used more for marketing purposes than for innovative and instructive design.

 

3.      COMMENTS AND THEIR IMPORTANCE IN THE ANALYSIS OF HUMAN-COMPUTER INTERACTION

 

User comments and opinions are an important part of human-computer interaction research. According to Barbosa and Silva (2010), the “[...] the sooner users are involved in the design process, the sooner it will be possible to learn about their needs and, thus, positively influence the synthesis of the solution, as well as identify and correct possible problems ” [22].

The Google Play Store is an online store that sells digital content and is currently categorized into applications (apps), electronic games (games), music, movies, books, and magazines. This content is available free of charge or for a fee.

Paid content can be sold/bought or rented. With regard to applications and games, the fact that the application is made available free of charge depends on the policy adopted by the developer, an application can be made available for free and generate monetization for its developer through ads displayed to the user of the application. The Duolingo app qualifies in this case, since in order not to have ads displayed constantly it is necessary to pay rent.

All content available on the Google Play Store has a forum for evaluating that content, the structure is the same regardless of the content evaluated. The notes are staggered from one to five. The forum consists of three areas divided into statistical data, creating a new assessment of the content and assessments of those who criticized the content.

 

In the area dedicated to statistical data, the information displayed is:

·         the average of the marks given by those who evaluated;

·         the total number of assessments;

·         how these notes are distributed.

 

In the area to create a new content evaluation, the user who is willing to criticize will make his observation. For this, the user must be logged in to the platform, where the name of the user who commented and the date of the comment will be automatically extracted. This area has a field for filling of the:

·         numerical evaluation of the content, ranging from one to five;

·         comment that will be posted.

 

Finally, the area where the published evaluations are displayed has the various evaluations carried out on an object and can be organized in a way to order the evaluations according to the most recent, according to the criteria of the relevance of the store or according to the grade. These reviews can be voted by other users as useful and have a field with:

·         the name of the commenter;

·         the evaluation score;

·         the comment;

·         the date of publication;

·         how many people rated the review as useful.

 

4.      MATERIALS AND METHODS

 

The Google Play Store presents a WWW (World Wide Web) page and to analyze the comments of Duolingo users, it was extracted from this page by creating an algorithm. These comments displayed on the Google Play Store are available both on the website of the store and on the Android platform application, with the same comments and it was decided to extract the website as it is the most viable way.

With the period of dates that are relevant to this study, to extract the comments already defined, the next step was to sort the comments on the site by the most recent, this resulted in a list of comments sorted from the most recent to the oldest. Comments are of interest until the earliest date in the defined date range as a criterion for extracting comments. When all comments were displayed until the lowest date defined as criteria, it was necessary to save the HTML page to a directory on the computer.

An algorithm was developed in JAVA programming language in order to facilitate the analysis and extraction process of the analysis components. For this, the algorithm extracts the comments from the saved HTML file and provides a file in the “.txt” format as output. This file can have its result varied depending on what is chosen and passed as a parameter to the program. You can choose to have the program provide an output file with all the reviews of the HTML file, or you can delimit a period of dates, with a start date and an end date, the last being a more recent date than the first. For any chosen period, the file is organized as a spreadsheet, organized by columns as follows:

·         General index - Description: Index of the comment considering all the notes. If there are n comments, the index starts with 1 and ends with n. Value type: Integer. Value type in JAVA language: int.

·         Index by grade - Description: Index of the comment filtered considering the grade specified. If there are n comments with a score of x the index will start at 1 and end at n. Value type: Integer. Value type in JAVA language: int.

·         Grade - Description: Grade that was given by the appraiser (Ranges from one to five), on a scale where grade one represents the worst grade and grade five the highest grade. Value type: Integer. Value type in JAVA language: int.

·         Useful - Description: Number of times the evaluation was considered useful by other evaluators. Value type: Integer. Value type in JAVA language: int.

·         Date - Description: Date on which the assessment was published. Value type: Integer. Value type in JAVA language: int.

·         Comment - Description: The content of the comment in more detail. Value type: Text. Value type in JAVA language: String.

 

5.      RESULTS AND DISCUSSIONS

 

A total of 10,943 comments were returned from the period between January 1, 2020, and January 31, 2020. Of that total, 2.24% rated the tool with a grade of 1, 0.78 voted for grade two, 2.33 % rated it in three, in grade four it corresponded to 9.52% of the assessments and 85.13% of the comments fit in grade five.

 

Table 1: Distribution of grades

 

It can be noted that the extreme majority of comments have a score of 5, the highest score available for evaluation. Therefore, grade 1 is the lowest. The Google Play Store makes a constant average of all grades and the Duolingo app scored 4.7 on June 12, 2020.

We will have the highest score, 5, which expresses a positive rating of the application, and the lowest score, 1, as a negative rating^[3]. Considering all those who rate the tool as 4 and 5, we have approximately 95%. Comments 1 and 2 totaled 3% of the amount, representing a minority, as well as comments 3, with the remaining 2%, as shown in the graph in Figure 1.

 

Figure 1: Distribution of grades

 

The rating can also be given for the comment, there is an option for other users to rate the comment as “Useful”, Table 2 shows the distribution of comments rated as useful in each note. Thus, of the total of 10,943 comments, 416 (3.77%) have at least one rating as useful. Comments rated as useful are distributed with 10% of those comments grouped in note 1, which corresponds to 40 comments. In note 2, there are 13 comments in this category, which represents 3% of the amount of comments. In note 3, on the other hand, we have 10% of the comments rated as useful, resulting in 40 comments. Corresponding to 15% of the comments in this category, note 4 has 64 comments. And finally, note 5 has 259 comments that rate it as useful (62%).

 

Table 2: Distribution of the evaluation as “useful”

 

It can be seen that notes 4 and 5 groups most of the evaluations that attribute the quality of useful and added together result in 77% of the comments in this category, according to the graph on Figure 2:

 

 

Figure 2: Distribution of useful comments by grades

 

Table 3 shows the calculation of the number of comments that were rated as useful per grade divided by the total of all comments - which includes “useful” ratings and that reviews that were not rated as “useful” - by grades.

 

Table 3: Number of useful assessments distributed in grades divided by number of assessments distributed in grades

 

Analyzing the total universe of comments, was noticed that 3.8% are considered “useful” and it can be noted that the proportion of comments rated as useful is higher, as the grades are lower, the highest proportion being in grade one. Table 4 shows the number of comments with useful rating divided by the total of comments.

 

Table 4: Number of useful comments per total number of grades

 

When comparing Table 4 with Table 5, was noticed that grade five is where most of the comments are found to be useful, representing 2.37 % of the total comments, thus, considering the total of comments, the comments with the highest marks have a greater representation in the total of 10,943 comments.

 

Organizing these 416 comments by rating and displaying the most rated comment as useful the following organization was synthetized:

1)     Grade 1

·         Quantity: 40 comments rated as useful out of a total of 245 comments.

·         Percentage: 16.33 % of comments are considered as useful out of a total of comments.

·         Most rated comment in the useful category

§  Quantity of rated as useful: 363

§  Date: January 16, 2020.

§  Comment in Portuguese: “muita militância e politização de extrema esquerda e nada de ensino e efetividade, não tem porque colocar um casal de lésbicas no exercício da lua de mel, não tem porque colocar propaganda ao veganismo.” (IG 6131)

§  Comment translated to English: “a lot of militancy and extreme left politicization and no teaching and effectiveness, there is no reason to put a lesbian couple on the honeymoon exercise, there is no reason to put propaganda on veganism.” (IG 6131)

2)     Grade 2

·         Quantity: 13 comments rated as useful out of a total of 85 comments.

·         Percentage: 15.29 % of comments are considered as useful out of a total of comments.

·         Most rated comment in the useful category

§  Quantity of rated as useful: 113

§  Date: January 31, 2020.

§  Comment in Portuguese: “Gosto muito do app, ele ensina muito bem porém deixou à desejar nas pronúncias, falta mais prática na fala onde podemos continuar à visualiza/escutar as frases da língua desejada e pronúncia corrente. Ou seja, gostaria que houvesse mais atividades de pronúncia, como conversação, feedback etc. Vocês ajudam bastante a gente, porém se poder dar uma reforçada estou agradecida. Obrigada!!!” (IG 458)

§  Comment translated to English: “I like the app a lot, it teaches very well but it left a lot to be desired in pronunciations, there is a lack of more practice in the speech where we can continue to visualize/listen to the phrases of the desired language and current pronunciation. That is, I would like to have more pronunciation activities, such as conversation, feedback, etc. You help us a lot, but if you can reinforce it, I’m grateful. Thanks!!!" (IG 458)

3)     Grade 3

·         Quantity: 40 comments rated as useful out of a total of 255 comments.

·         Percentage: 15.69 % of comments are considered as useful out of a total of comments.

·         Most rated comment in the useful category

§  Quantity of rated as useful: 357

§  Date: January 30, 2020.

§  Comment in Portuguese: “O app é muito bom, muito intuitivo e fácil de usar, mas tem um problema grave para o aprendizado: ele é extremamente repetitivo, a ponto de quase fazer o usuário desistir. Estou aprendendo francês e mesmo já tendo concluído 90 tarefas, o app ainda usa as mesmas palavras. Sempre usa gato, cachorro, menino, maçã, vermelho... Podia usar outros exemplos para que o usuário pudesse adquirir mais vocabulário e melhorar a didática a que o app se propõe.” (IG 1)

§  Comment translated to English: “The app is very good, very intuitive and easy to use, but it has a serious problem for learning: it is extremely repetitive, to the point of almost making the user give up. I'm learning French and even though I’ve completed 90 tasks, the app still uses the same words. Always uses cat, dog, boy, apple, red... I could use other examples so that the user could acquire more vocabulary and improve the didactics that the app proposes.” (IG 1)

4)     Grade 4

·         Quantity: 63 comments rated as useful out of a total of 1,042 comments.

·         Perce: 6.14 % of comments are considered as useful out of a total of comments.

·         Most rated comment in the useful category

§  Quantity of rated as useful: 211

§  Date: January 26, 2020.

§  Comment in Portuguese: “Eu amava essse aplicativo porque eu uso ele à 5 anos e nunca tive nenhum problema com ele, até agora. Com essa nova atualização, o aplicativo fica te impulsionando a comprar o Plus. Enfim, a ideia do aplicativo é perfeita e antes dessa nova atualização, ele era o meu aplicativo preferido. Gostaria que pudesse continuar sem os corações (vidas) , restringe a educação do aplicativo.” (IG 2763)

§  Comment translated to English: “I used to love this app because I have been using it for 5 years and have never had any problems with it until now. With this new update, the app is driving you to buy Plus. Anyway, the idea of the app is perfect and before this new update, it was my favorite app. I wish I could continue without the hearts (lives), restricts the education of the application.” (IG 2763)

5)     Grade 5

·         Quantity: 259 comments rated as useful out of a total of 9,316 comments

·         Percentage: 2.78 % of comments are considered as useful out of a total of comments.

·         Most rated comment in the useful category

§  Quantity of rated as useful: 1041

§  Date: January 23, 2020.

§  Comment in Portuguese: “O Duolingo está evoluindo bastante, estou aprendendo muito, muito obrigado aos desenvolvedores por manterem um aplicativo de tanta qualidade.” (IG 3905)

§  Comment translated to English: “Duolingo is evolving a lot, I am learning a lot, thanks to the developers for maintaining an application of such quality.” (IG 3905)

 

6.      CONCLUSIONS AND RECOMMENDATIONS

 

The relationship between technology and education is not recent in the history of mankind, but the forms that the new Information and Communication Technologies are presenting is something that has a direct impact on the subject and society, to the point of having a Network Society[23].

The impact of these technologies on education would not be different, as we have seen the teaching and learning process can also be optimized by new forms of knowledge transmission. The results of this research show that usability is a major factor and that 85.13% of users rated the tool with a score of 5, which is very positive, as it is the highest score available on the Google Play Store platform.

 

Considering usability, the following comment summarizes it:

The best app out there! Easy to use, great for learning a language from the basics. I always recommend it to my friends, but they end up uninstalling it because they now have five lives, so my friends say it’s not worth it. Such a good app, but now for my friends it is not serving much, because the lives end. It is very expensive for an app. It is useful, but for those who cannot pay it is not serving. Unfortunately. (IG 9436, our translation)

 

The platform shows itself as easy to use and that achieves its goals. The comment also points to the non-free application, which would make it difficult to use in Brazilian public schools.

In this way, the tool can be thought of as support and auxiliary for the foreign language teacher. It has data on usability, ease of access, interactive and achieves non-native tongue learning objectives.

The data also point out the need for further qualitative content analysis that will be carried out by the researchers in the second stage of this research.

 

SOURCES OF FUNDING

 

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

 

CONFLICT OF INTEREST

 

The author have declared that no competing interests exist.

 

ACKNOWLEDGMENT

 

We would like to express an especial thanks of gratitude to Luciano de Jesus Golçalves, his careful reading, grammatical corrections, and support were paramount to the development of this paper. To Ananda Liz Matias de Araújo and Luciana Matias de Barros, for their assistance and support on this research.

 

REFERENCES

 

     [1]        Secretaria Municipal de Educação, Orientações curriculares: proposições de expectativas de aprendizagem - Tecnologias de Informação e Comunicação. São Paulo: SME / DOT, 2010.

     [2]        P. C. Masiero, Ética em Computação. São Paulo: EDUSP, 2008.

     [3]        S. Chancellor, N. Andalibi, L. Blackwell, D. Nemer, and W. Moncur, “Sensitive Research, Practice and Design in HCI,” in Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, May 2019, pp. 1–8, doi: 10.1145/3290607.3299003.

     [4]        S. Chancellor, Z. Lin, E. L. Goodman, S. Zerwas, and M. De Choudhury, “Quantifying and Predicting Mental Illness Severity in Online Pro-Eating Disorder Communities,” in Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW ’16, 2016, pp. 1169–1182, doi: 10.1145/2818048.2819973.

     [5]        R. Wang et al., “CrossCheck: Toward passive sensing and detection of mental health changes in people with schizophrenia,” in Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Sep. 2016, pp. 886–897, doi: 10.1145/2971648.2971740.

     [6]        J. Pater and E. Mynatt, “Defining Digital Self-Harm,” in Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Feb. 2017, pp. 1501–1513, doi: 10.1145/2998181.2998224.

     [7]        E. Şendurur, E. Efendioğlu, N. Y. Çalışkan, N. Boldbaatar, E. Kandin, and S. Namazli, THE M-LEARNING EXPERIENCE OF LANGUAGE LEARNERS IN INFORMAL SETTINGS. International Association for the Development of the Information Society. e-mail: secretariat@iadis.org; Web site: http://www.iadisportal.org, 2017.

     [8]        C. Ware et al., “Maintaining Cognitive Functioning in Healthy Seniors with a Technology-Based Foreign Language Program: A Pilot Feasibility Study,” Front. Aging Neurosci., vol. 9, Mar. 2017, doi: 10.3389/fnagi.2017.00042.

     [9]        S. Hadjerrouit, “Assessing the Affordances of SimReal+ and their Applicability to Support the Learning of Mathematics in Teacher Education,” Issues Informing Sci. Inf. Technol., vol. 14, pp. 121–138, 2017, doi: 10.28945/3692.

   [10]     S. El Janati, A. Maach, and D. El Ghanami, “SMART Education Framework for Adaptation Content Presentation,” Procedia Comput. Sci., vol. 127, pp. 436–443, 2018, doi: 10.1016/j.procs.2018.01.141.

   [11]     A. Papadimitriou and G. Gyftodimos, “The Role of Learner Characteristics in the Adaptive Educational Hypermedia Systems: The Case of the MATHEMA,” Int. J. Mod. Educ. Comput. Sci., vol. 9, no. 10, pp. 55–68, Oct. 2017, doi: 10.5815/ijmecs.2017.10.07.

   [12]     S. Strecker, D. Kundisch, F. Lehner, J. M. Leimeister, and P. Schubert, “Higher Education and the Opportunities and Challenges of Educational Technology,” Bus. Inf. Syst. Eng., vol. 60, no. 2, pp. 181–189, Apr. 2018, doi: 10.1007/s12599-018-0522-8.

   [13]     J. Lind, S. Pelger, and A. Jakobsson, “Students’ ideas about technological systems interacting with human needs,” Int. J. Technol. Des. Educ., vol. 29, no. 2, pp. 263–282, Mar. 2019, doi: 10.1007/s10798-018-9449-0.

   [14]     M. Juraschek, L. Büth, G. Posselt, and C. Herrmann, “Mixed Reality in Learning Factories,” Procedia Manuf., vol. 23, pp. 153–158, 2018, doi: 10.1016/j.promfg.2018.04.009.

   [15]     A. Bartolomé, L. Castañeda, and J. Adell, “Personalisation in educational technology: the absence of underlying pedagogies,” Int. J. Educ. Technol. High. Educ., vol. 15, no. 1, p. 14, Dec. 2018, doi: 10.1186/s41239-018-0095-0.

   [16]     T. B. Souza and M. I. Z. N. da S. Mourão, “Ensinar Francês por dispositivos móveis: uma experiência com Duolingo e whatsApp,” Texto Livre, vol. 10, no. 2, pp. 206–219, 2017, doi: 10.17851/1983-3652.10.2.206-219.

   [17]     G. Marques-Schäfer and A. A. D. S. Orlando, “Concepções de aprendizagem de línguas e o Duolingo: uma análise crítica sobre sua proposta e experiências de aprendizes / Languages learning conceptions and Duolingo: a critical analysis on its proposals and learners experiences,” Texto Livre Ling. e Tecnol., vol. 11, no. 3, p. 228, Dec. 2018, doi: 10.17851/1983-3652.11.3.228-251.

   [18]     C. A. Guaqueta and A. Y. Castro-Garces, “The Use of Language Learning Apps as a Didactic Tool for EFL Vocabulary Building,” English Lang. Teach., vol. 11, no. 2, p. 61, Jan. 2018, doi: 10.5539/elt. v11n2p61.

   [19]     B. Brick and T. Cervi-Wilson, “Enhancing learners’ professional competence via Duolingo classroom,” in Professional competencies in language learning and teaching, Research-publishing.net, 2019, pp. 19–29.

   [20]     S. Loewen et al., “Mobile-assisted language learning: A Duolingo case study,” ReCALL, vol. 31, no. 3, pp. 293–311, Sep. 2019, doi: 10.1017/S0958344019000065.

   [21]     H. Lotherington, HOW MOBILE ARE TOP-RATED MOBILE LANGUAGE LEARNING APPS? International Association for the Development of the Information Society. e-mail: secretariat@iadis.org; Web site: http://www.iadisportal.org, 2018.

   [22]     S. D. J. Barbosa and B. S. da Silva, Interação humano-computador. Rio de Janeiro: Elsevier, 2010.

   [23]     M. Castells, Sociedade em rede. São Paulo: Paz e Terra, 1999.

   [24]     APP-Sindicato, “Aplicativo Aula Paraná recebe nota 1,5 e é reprovado por estudantes,” App-sindicato, 2020. https://appsindicato.org.br/aplicativo-aula-parana-recebe-nota-15-e-e-reprovado-por-estudantes/ (accessed Jun. 12, 2020).