Article Type: Research Article Article Citation: Tiffany Sutter, and Mingyuan
Zhang. (2020). EXAMINING THE RELATIONSHIP BETWEEN STUDENTS’ CREATION OF
SPECIFIC DIGITAL PRODUCTS AND THEIR TECHNOLOGY ASSESSMENT SCORES. International
Journal of Research -GRANTHAALAYAH, 8(9), 153-175. https://doi.org/10.29121/granthaalayah.v8.i9.2020.1342 Received Date: 27 August 2020 Accepted Date: 30 September 2020 Keywords: Technology Digital Media Technology & Engineering
Literacy (Tel) NAEP Student Computer Use This study presented a secondary analysis of the National Assessment of Educational Progress (NAEP) dataset. The paper examined the impact of eighth grade students’ specific digital product creation on their Technology & Engineering Literacy ICT scores. In order to gain a better understanding of the impact of computer use technology achievement of eighth-grade students, this study used a quantitative descriptive research design to analyze secondary data extracted from the 2018 NAEP data set. The findings include (1) using computers to create, edit, or organize digital media both for school work and activities beyond school increased overall ICT scores. (2) The average score of students who used computers to create presentations in school increased while those who did this activity after school time saw a decrease in scores. (3) Students who reported that they create spreadsheets mostly saw a decrease in their average ICT score no matter the frequency or purpose for the activity. These findings may indicate that there is an ideal frequency for digital product creation in school, but that these types of activities may not be indicative of real-world use which is how students are assessed.
1. INTRODUCTIONWith the advent of the term digital native by Prensky
(2001), many are led to believe that students today are actively using
technology in a meaningful way simply because of the era into which they were
born. While generation is certainly a factor in a user’s comfort with
technology, it is not the only factor that decides if someone should be
considered a digital native nor should age automatically mean proficiency in
technology (Helsper & Eynon,
2010). According to Kirschner and De Bruyckere
(2017), understanding that digital natives are not necessarily sitting in our
classrooms will help teachers “avoid the pitfall of assuming that their
students possess talents and abilities that they do not actually have. The
skills and competences attributed to this generation of students…need to be
properly taught and acquired before they can be applied” (p. 137). It is
through purposeful instruction and inclusion of technology in curricula that
students can increase their abilities. In 2001, the Enhancing Education
Through Technology Act required districts to not only demonstrate technological
integration into curriculum but to prove that all students had met a specific
level of mastery by the completion of eighth grade (Barron et al., 2003; No
Child Left Behind Act of 2001, Title II, Part D, § 2402, 2002). This
incentive-based program and its rewards and consequences directed some of the
technology curricula of K12 schools for numerous years. However, once the
federal aspects of No Child Left Behind were eliminated and the states were
given authority regarding curriculum and governing standards, many states
reverted back to a time when they did not assess students’ overall
technological abilities since it was not required (Davis, 2015; Layton, 2015). In order to gauge students’ abilities regarding technology,
the National Assessment of Educational Progress (NAEP, 2014) designed and
administered a national assessment to eighth-graders involving technology and
engineering challenges. In 2014, only 43% of tested students performed at or
above the proficient level (NAEP, 2014). While the number of proficient
students grew significantly in 2018, to 46%, there is still a clear majority of
students who are not meeting basic levels of abilities (NAEP, 2018). In order
to discern characteristics that may positively impact scores, this research is
concerned with whether a relationship exists between students’ achievement in
the Technology and Engineering Literacy (TEL) assessment and how often and in
what ways they use a computer for school work and activities unrelated to
school. In particular, computer use will constitute creating and organizing
digital media and creating presentations and spreadsheets. Despite a lack of basic
proficiency, our digital native students are utilizing technology both in and
out of school more than previous generations. Teachers are integrating the
creation of digital media, presentations, and spreadsheets into their classroom
instruction which not only gives students hands-on experience but provides a
variety of ways to utilize technology in a constructive way to meet standards.
In fact, Helsper and Eynon
(2010) found that experience with technology helps to create a competent
digital user as does the breadth of that use. Therefore, it is these factors
that should be looked at when evaluating users’ abilities with technology. This
study seeks to bridge the gap between the scores and some of the factors which
may positively contribute to overall proficiency. The data of computer use in
and out of school and its relationship to TEL scores may be a helpful reference
for teachers to understand what kind of computer-created products to include in
their lessons. This information may also serve as a catalyst for encouraging
students to create their own products beyond the classroom. The purpose of this study is to explore the relationship
between eighth-grade students’ 2018 Technology and Engineering Literacy (TEL)
scores and specific computer-created products for school work and outside of
school activities. In order to explore this relationship, this study focused on
these research questions: Does using computers to create
media, presentations, and spreadsheets for school work have an impact on
eighth-grade TEL scores? Does using computers to create
media, presentations, and spreadsheets outside of school for activities
unrelated to their school work have an impact on eighth-grade TEL scores? Our theoretical framework for this
research adopts a scientific inquiry-based approach. The framework was
described in great details in The Impact of Conversations on Fourth Grade
Reading Performance - What NAEP Data Explorer Tells? (Bond & Zhang, 2017).
In summary, the research methods combined the inquiry process with scientific
knowledge, reasoning, and critical thinking. We started with an extensive
exploration of the dataset, and that led to the designing of the research
questions. The research questions further guided us to mine the data with great
in-depth. 2. LITERATURE REVIEWDigital literacy is no longer a way to differentiate oneself
from others who do not have that skill; it is imperative that all graduating
students reach at least a basic, proficient level in their abilities with
technology. In fact, the ability of students to be digitally literate is no
longer optional but has become a necessary survival skill for all regardless of
curricular requirements (Eshet-Alkalai, 2004; Eshet-Alkalai, 2012). Even though technology will
constantly change, and current tools will shift and evolve, the concepts of
digital citizenship and literacies will not only sustain but expand (National
Research Council, 1999). Through direct instruction, independent discovery, and
hands-on experiences, students need opportunities to learn about and with
technologies (National Research Council, 1999, p. 55). Often, students can be
introduced to digital tools in the classroom which may prove useful for them
beyond the walls of the school as well. 2.1. TECHNOLOGY USE IN SCHOOL
Since the choice to use technology or not within a lesson is
a personal one, it is important to understand that teachers’ beliefs and
perceptions impact the overall design of their classrooms. The decisions of
what types of activities occur in a classroom each day are largely dependent on
teachers’ pedagogical choices and knowledge of their content. Additionally, the
correlation between teachers’ beliefs about technology and how and when they
choose to integrate it into their lessons has been established by several
researchers (Albion & Ertmer, 2002; Badia et al., 2014; Chen, 2008; Ertmer,
2005; Taimalu & Luik,
2019). Taking it one step further, Ritzhaupt et al.
(2012) noted that how teachers choose to integrate technology into the
classroom directly impacted the frequency with which students used technology
for learning. More than a decade ago, findings were already showing an
encouraging trend where teachers had students use more advanced, creation
software rather than basic answer-input technologies (Niederhauser
et al., 2007). For example, students can create digital presentations,
spreadsheets, and concept maps and 76% of the activities Niederhauser
et al. (2007) witnessed included these types of activities (pp. 495-496).
Furthermore, in the case study by Ezquerra et al.
(2014), the researchers found that by working with digital media in class,
students not only reported an increase in their overall abilities, but also an
increase in their potential to replicate the activity on their own. More
recently, Schmid and Petko (2019) found that a clear
majority of students used computers in their classes regularly. Combined, these
data show that teachers are choosing to increase technological experiences for
students in school, and they are utilizing higher-level computer use and
creation rather than using the technology solely as a productivity tool or for
instructional purposes. 2.2. TECHNOLOGY USE OUT OF SCHOOL
Today’s students not only use technology in school
environments, but most have better access to technology outside of their formal
learning as well. Wang et al. (2014) found that a vast majority of students
have regular use of smartphones, laptops, and desktops at home and tend to use
technology more at home than in school. Research has found, however, that students
are more likely to be creators of content in school whereas outside of school
they may be more of a consumer; the difference in technology use may be
attributed to students’ personal entertainment purposes during their free time
as well as requirements from their instructors to produce rather than share
others’ work (Lu, Hao, & Jing, 2016). In school, students were much more
likely to engage in the use and creation of spreadsheets and presentations than
outside of school where other media engagement surpassed the in-school use of
the same tools (Wang et al., 2014). According to DeBell
and Chapman (2006), a majority of students use home computers to play games and
be entertained and 47% use the home computer to complete school work with word
processors being the most common tool. Therefore, while students have more home
access to technology and computers than ever before, they do not appear to be
using it productively most of the time. 2.3. STUDENTS’ TECHNOLOGY
ABILITIES
With the prevalence of technology today,
students regularly use digital and social media tools both within and outside
of the classroom. However, Erstad (2015) noted that
this “can easily be misleading and give the impression that all young people
today are super-users and highly competent in their use of different media” (p.
85). In fact, often, the opposite is true as teachers report that students do
not always know how to use new and emerging technologies without devoting
school instructional time to navigating the tool alone (Wang et al., 2014).
Furthermore, the types of assessment necessary to judge students’ abilities
cannot be done with typical standardized testing scenarios (Hohlfeld
et al., 2010). This means that most of the data about students’ abilities are
tied to self-reported surveys and observations done by teachers and
researchers. A survey by Kaminski et al.,
(2003) found that students had relatively low software proficiencies despite
having technology access and only about half of the respondents, or less,
reported having some basic skills in hardware and information gathering.
Additionally, they found that students preferred to increase their
understanding through classroom instruction designed at improving their skills
rather than seeking out to advance on their own (Kaminski et al., 2003). This
was in contrast to the survey findings of McEuen
(2001) where a majority of students stated that they troubleshoot problems and
learn new technologies on their own. The McEuen
(2001) survey also found that a majority of students self-reported average to
expert capabilities in word processing, media creation, spreadsheeting, and
using the internet. Overall, students who use
integrated technology in school will be better tech users than those who are
not exposed to digital production and creation mandated by teachers. Moreover,
students who have technologies available at home and choose to use them beyond
consumerism will continue their learning in authentic ways. Just as classroom
use of technology can impact home computer use, students who increase their
abilities beyond the classroom may try to incorporate that knowledge into their
school work. Whether students are interacting with digital tools formally or
informally, there should be an overall impact on their general capabilities when
it comes to technology. 3.
METHODOLOGY
The National Center for Educational Statistics (NCES) designed the National Assessment of Educational Progress (NAEP), also known as The Nation’s Report Card, in 1969 to measure what U.S. students knew and could do within various subject areas. NAEP is a congressionally mandated project administered by the NCES within the U.S. Department of Education and the Institute of Education Sciences (NCES, 2018). The present study will examine a subcomponent of the eighth-grade Technology & Engineering Literacy (TEL) scores of students at the national level. Of the three content classifications within the TEL score, the data used in this study is from the Information and Communication Technology (ICT) segment. This score ranges from 0-300 and was chosen because it “measures students' knowledge of the software and systems used for accessing, creating, and communicating information, and for facilitating creative expression” (The Nation’s Report Card, 2018a, para. 5). This definition was most closely related to the variables needed for the research questions and aligned with the categorization of the same variables in research completed by Park et al. (2017). 3.1. PARTICIPANTS AND SCHOOL
SELECTION
Using a probability sample design, participants for the assessments are selected randomly to ensure accurate, proportional representation of the greater population. According to a NAEP sampling infographic, NAEP uses a primary sampling unit (PSU) to identify single counties or groups of contiguous counties which are a representative sample of the nation at large (NCES, 2019). The national sample tests, such as the TEL, include public and non-public schools. Then, schools are chosen within PSUs to ensure that there is an equal representation of location type, race or ethnicity, and student achievement. Finally, students are then selected randomly from the appropriate grade level and students are assigned to a test. NAEP provides accommodations for special education students, students with disabilities, and English-language learners (NCES, 2019). Given every four years, the most recent TEL assessment was administered between January and March of 2018 to approximately 15,400 students from 600 schools (The Nation’s Report Card, 2018c). 3.2. NAEP SAMPLING AND DATA
COLLECTION
The TEL assessment is given completely via computer and does not break down collected data by state. According to The Nation’s Report Card (2018a), the complete TEL assessment contains 15 scenario-based tasks and 77 standalone questions; however, students do not answer each question. Students test for a total of 60 minutes which is broken into two 30-minute sessions that vary based on the participant; the assessment is accompanied by a student questionnaire where participants have an opportunity to respond about their education and experiences both in and out of school (The Nation’s Report Card, 2018a). The present study uses the average ICT scaled scores for the 2018 TEL assessment and there were six variables selected for analysis. Results were restricted to national public schools which exclude scores from the Bureau of Indian Education schools and Department of Defense Education Activity schools but include charter schools. 3.3. DATA ANALYSISThe NAEP website includes a Data Explorer which allows users to access state and national results from all the assessments given. By inputting selected criteria, researchers can access data tables and charts and run basic statistical analyses. The Data Explorer was used to choose the appropriate TEL sub-score for eighth-graders and the desired variables before running statistical tests. The six variables chosen for this study were all student-reported. Three of the variables asked students to consider the instruction received within the classroom while the other three focused on their time outside of school. The specific variables are presented in Table 1. Table 1: Student-Reported
NAEP Variables Chosen for Analysis Against TEL Scores
The possible answers to all six of these student questions
were a 5-point Likert-type scale indicating frequency from “never/almost never”
to “almost every day.” After selecting the proper sub-score and these
variables, statistical significance tests were run to determine differences
between groups. 4.
RESULTS
AND DISCUSSION
The NAEP Data Explorer provided summary means and standard deviations of TEL scores in relation to each category answer for the variables. The average ICT scaled score for all eighth-grade participants was 151 (SD=37) in 2018. Presented below are the mean scores for each variable related to the research questions as well as significance testing results. When significance was found, Cohen’s d was calculated by hand using an effect size calculator from the University of Colorado (https://www.uccs.edu/lbecker/). 4.1. RESEARCH QUESTION #1 -
COMPUTER USE IN CLASS
During the student questionnaire,
students were asked about their computer use in the classroom and the ways in which
they interact with specific technologies during instruction and assessment. To
address the first research question, three variables were chosen for how
students work during classroom activities. Table 2: Students’ ICT Scores by Frequency of Specific Computer Use for School Work
The average ICT scaled score, as well as the standard deviations, by question and response are shown in Table 2. Each variable is then broken down into its significance in the sections that follow. 4.2. CREATE, EDIT, OR ORGANIZE
DIGITAL MEDIA
The highest overall scores for students using computers to
create, edit, or organize digital media for school work were from the students
who used technology for this purpose once or twice a week or almost every day.
The overall lowest score earned in the ICT portion was from the students who
never or almost never used a computer for this purpose. The results of multiple
t-tests for the first variable, taken from Data Explorer, are shown in Table
3. Table 3: Difference in ICT Scale Scores Between Variables for Use Computer to Create, Edit, or Organize Digital Media for School Work
Students reporting using a
computer to create, edit, or organize digital media for school work a few times
a year (M=153, SD=36), 1-2 times a month (M=152, SD=37), once or twice a week
(M=156, SD=37), and almost every day (M=155, SD=38) had significantly higher
average scale scores than those in the never or almost never group (p<0.01).
The mean of the once or twice a week group was also significantly higher
(M=156, SD=37) than the mean of the 1-2 times a month group (M=152, SD=37). Table 4: Effect sizes of Significant Mean Score Differences when Using a Computer to Create, Edit, or Organize Digital Media for School Work
To report the effect sizes of the significant variable
responses, Cohen’s d was calculated and is presented in Table 4.
According to Cohen (1992), an effect size of 0.2 is considered small, a 0.5 is
medium, and a 0.8 is large. For this variable, the effect sizes ranged from
0.11 to 0.24 indicating relatively small significance. 4.3. CREATE PRESENTATIONSIn looking at how often students create presentations on the
computer for their school work, the highest average scores for the ICT portion
was earned by students who did these activities 1-2 times a month or once or
twice a week. The lowest overall scaled score was earned by those students who
never or almost never use a computer in this way. The results of multiple
t-tests for the second variable, taken from NAEP, are shown in Table 5. Table 5: Difference in ICT Scale Scores Between Variables for Use Computer to Create Presentations for School Work
Students reporting using a
computer to create presentations for school work a few times a year (M=152,
SD=36), 1-2 times a month (M=157, SD=36), once or twice a week (M=154, SD=37),
and almost every day (M=146, SD=38) had significantly higher average scale
scores than those in the never or almost never group (p<0.001). The mean of
the 1-2 times a month group was also significantly higher (M=157, SD=36) than
the mean score of 152 (SD=36) of the few times a year group (p<0.001).
Significantly lower scaled scores were found between the group who responded
almost every day (M=146, SD=38) and those in the few times a year (M=152,
SD=36), 1-2 times a month (M=157, SD=36), and once or twice a week (M=154,
SD=37) groups (p<0.001). Finally, the 3-point scaled score difference
between the group who created presentations 1-2 times a month (M=157, SD=36)
and those who did it once or twice a week (M=154, SD=37) was also a significant
difference (p<0.05). Table 6: Effect sizes
of Significant Mean Score Differences when Using a Computer to Create
Presentations for School Work
To report the effect sizes of the variable responses,
Cohen’s d was again calculated and is presented in Table 6. The range
for the effect size from this variable was from 0.14 to 0.70. Five of the
effect size comparisons were on the small end while the largest significance
was found between the students who use a computer to create presentations for
school 1-2 times a month and those who never do it (0.702). A medium effect
size was found between once or twice a week and never or almost never (0.613)
and a few times a year and never or almost never (0.567). 4.4. CREATE SPREADSHEETSStudents who used a computer to create spreadsheets for
school a few times a year yielded the highest scaled score on the ICT portion
of the TEL while those who did it almost every day had the lowest scores. The
results of multiple t-tests for the third variable, taken from the NAEP Data
Explorer, are shown in Table 7. Table 7: Difference in ICT Scale Scores Between Variables for Use Computer to Create Spreadsheets for School Work
Students reporting using a
computer to create spreadsheets for school work a few times a year (M=158,
SD=36) had significantly higher average scale scores than those in the never or
almost never group (p<0.05). The mean of the 1-2 times a month group was
significantly lower (M=148, SD=36) than the mean score of the never or almost
never (M=155, SD=37) and the few times a year (M=158, SD=36) groups
(p<0.001). Significantly lower scaled scores were found between the group
who responded once or twice a week (M=145, SD=36) and those in the never or
almost never (M=155, SD=37), few times a year (M=158, SD=36), and 1-2 times a
month (M=148, SD=36) groups (p<0.05). Finally, students who use computers in
class for spreadsheet creation almost every day (M=139, SD=38) had
significantly lower scaled scores than those in the never or almost never
(M=155, SD=37), a few times a year (M=158, SD=36), 1-2 times a month (M=148,
SD=36), and once or twice a week (M=145, SD=36) groups (p<0.01). Table 8: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Spreadsheets for School Work
To report the effect sizes of the variable responses,
Cohen’s d was calculated and is presented in Table 8. The overall range
on these calculations was from 0.08 to 0.51 with the majority of the effect
sizes at the small level. A medium effect size was found between the responses
of a few times a year and almost every day (0.513) noting that the lower
creation frequency yielded the higher score. 4.5. RESEARCH QUESTION #2 - COMPUTER USE BEYOND SCHOOLDuring the student questionnaire, students were asked about
their computer use outside of the classroom and the ways in which they interact
with specific technologies for activities unrelated to their school work. To
address the second research question, three variables were chosen for how
students use their personal computer time beyond school. Table 9: Students’ ICT Scores by Frequency of Specific Computer Use for Activities Beyond School
The average ICT scaled score, as well as the standard
deviations, by question and response are shown in Table 9. Each variable is
then broken down into its significance in the sections that follow. 4.6. CREATE, EDIT, OR ORGANIZE DIGITAL MEDIAWhen looking at product creation on the computer for
non-school activities, the highest scores on the ICT portion came from students
who created, edited, or organized digital media almost every day. The lowest
scaled score came from the group of students who did this activity a few times
a year. The results of multiple t-tests for the first variable, taken from the
NAEP Data Explorer, are shown in Table 10. Table 10: Difference in Scale Scores Between Variables for Use Computer to Create, Edit, or Organize Digital Media for Activities Beyond School
Students reporting using a
computer to create, edit, or organize digital media for activities beyond
school almost every day (M=157, SD=38) had a significantly higher average scale
score than those in the never or almost never (M=152, SD=36), a few times a
year (M=150, SD=37), and 1-2 times a month (M=151, SD=38) groups (p<0.01). Table 11: Effect sizes of Significant Mean Score Differences when Using a Computer to Create, Edit, or Organize Digital Media for Activities Beyond School
To report the effect sizes of the
variable responses, Cohen’s d was calculated and is presented in Table
11. They range from 0.14 to 0.19, indicating a small significance. 4.7. CREATE PRESENTATIONSWhen evaluating the variable which asked about students’ use of computers to make presentations unrelated to school, the lowest ICT scores came from those students who did this almost every day while the highest scores were students who never or almost never did this activity. The results of multiple t-tests for the second variable, taken from the NAEP Data Explorer, are shown in Table 12. Table 12: Difference in ICT Scale Scores Between Variables for Use Computer to Create Presentations for Activities Beyond School
Students reporting using a
computer to create presentations for activities beyond school almost every day
(M=136, SD=38) had a significantly lower average scale score than those in the
never or almost never (M=160, SD=35), a few times a year (M=153, SD=38), and
1-2 times a month (M=144, SD=36) groups (p<0.001). The means of the once or
twice a week group (M=141, SD=38) and the 1-2 times a month group (M=144,
SD=36) were both significantly lower than the means of the few times a year
(M=153, SD=38) and the never or almost never (M=160, SD=35) groups
(p<0.001). Finally, the 153 mean scaled score
(SD=38) for the students who created presentations outside of school work a few
times a year was significantly lower than those who never or almost never
(M=160, SD=35) did this activity (p<0.001). Table 13: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Presentations for Activities Beyond School
To report the effect sizes of the variable responses,
Cohen’s d was calculated and is presented in Table 13. For this
variable, mostly small effect sizes were found, ranging from 0.19 to 0.45;
however, a medium effect was found for responses of never or almost never and
once or twice a week (0.520) as well as never or almost never and almost every
day (0.657) with the never or almost never group having the highest score over
the other two groups of students. 4.8. CREATE SPREADSHEETSThe final variable looked at how often students use computers beyond school to create spreadsheets. The highest scaled score was earned by students who never or almost never did this activity while those who did it almost every day had the lowest scores on the ICT portion of the assessment. The results of multiple t-tests for the third variable, taken from the NAEP Data Explorer, are shown in Table 14. Table 14: Difference in ICT Scale Scores Between Variables for Use Computer to Create Spreadsheets for Activities Beyond School
Students reporting using a
computer to create spreadsheets for activities beyond school almost every day
(M=134, SD=39) had a significantly lower average scale score than those in the
never or almost never (M=160, SD=34), a few times a year (M=149, SD=37), and
1-2 times a month (M=139, SD=37) groups (p<0.05). The means of the once or
twice a week group (M=136, SD=36) and the 1-2 times a month group (M=139,
SD=37) were both significantly lower than the means of the few times a year
(M=149, SD=37) and the never or almost never (M=160, SD=34) groups
(p<0.001). Finally, the 149 mean scaled score
(SD=37) for the students who created spreadsheets outside of school work a few
times a year was significantly lower than those who never or almost never
(M=160, SD=34) did this activity (p<0.001). Table 15: Effect sizes of Significant Mean Score Differences when Using a Computer to Create Spreadsheets for Activities Beyond School
To report the effect sizes of the variable responses,
Cohen’s d was calculated and is presented in Table 15. The overall range
of effect sizes was from 0.13 to 0.71. While many of the significance were on
the small side, three medium effect sizes were found between never or almost
never and 1-2 times a month (0.591), never or almost never and once or twice a
week (0.685), and never or almost never and almost every day (0.711). The
strength in these frequency changes is negative as scores decreased with more
use rather than increased. 5. DISCUSSIONThis study explored students’ scores from the TEL assessment on the NAEP website as they related to specific uses of digital technologies both in and out of the classroom. O’Dwyer et al. (2005) noted the importance of using a subscale test score when researching standardized assessments in order to more closely align the data to specific instrument variables; therefore only the ICT scaled score was used as it included “...computers and software learning tools...and other technologies for accessing, creating, and communicating information and for facilitating creative expression” (Firman et al., 2015). The selected variables measured the frequency of students’ ICT use of digital media, presentations, and spreadsheets for school work and personal projects. 5.1. INTEGRATION OF TECHNOLOGY INTO THE CLASSROOMOne challenge to this research is that students are not in control of their use of technology in the classroom. Involvement with and use of digital technologies such as media, presentations, and spreadsheets requires teachers to integrate them into their pedagogy which is a deeply personal instructional choice affected by many factors (Ertmer, 2005; Taimalu & Luik, 2019). Teachers’ knowledge and implementation of specific technologies can hinge on many circumstances including their personal experience and comfort level, their teacher training program, and if any professional development has continued (Li et al., 2019). This study found that students who responded that they used computers in school to create, edit, or organize digital media at any frequency greater than never achieved better scores on the TEL assessment. Additionally, a use for this purpose once or twice a week yielded the highest score and a significant (albeit small) gain over those who only did it 1-2 times a month. This is similar to the findings of Wang et al. (2018) who noted a positive association between in-class creation of digital media and higher ICT assessment scores. Furthermore, this research found that students who responded that they used computers in school to create presentations at any frequency greater than never achieved better scores on the TEL assessment. The work by Lei & Zhao (2007) found that creating presentations was one of the most popular technology uses in school. The best frequency, according to the NAEP data, to create presentations for school work was 1-2 times a month as it had a medium to large effect size over other frequencies and students in this category demonstrated the highest ICT score on the TEL assessment. This means that teachers do not need to integrate this type of activity regularly to realize its benefit. This contradicts the work by Park et al. (2017) who found almost no relationship between students’ TEL scores and their ICT use for school-related work and Carter et al. (2017) who noted a negative association between school computer use and content assessment scores. One paradox was the use of spreadsheets which generally found that the more often a student used this tool, the lower their ICT scores. This is despite the fact that spreadsheet knowledge may be considered an important vocational tool and a necessary component of ICT instruction and knowledge (Tort et al., 2008). The highest score, however, was earned by students who did this type of activity a few times a year, but there was only a small effect size over those who never created spreadsheets. Without knowing specific numbers of students in each category, it is hard to know just how many students created spreadsheets in the classroom at all meaning that a small number of participants could have dictated the scaled score. Regardless, these findings mimic those of Wang et al. (2018). This type of activity may not really be integrated into teaching and learning opportunities or may have very specific uses in math, science, or STEM courses. Moreover, students and teachers may not fully understand the capabilities and options offered with the use of a spreadsheet which may impact the frequency and depth of their use for classwork (Tort et al., 2008). Overall, the increased use of computers at school does not necessarily translate into higher abilities with computers as the frequency of ICT use itself is not enough to develop the higher-level quality addressed on technology assessments (Hatlevik et al., 2018; Wastiau et al., 2013). Teachers cannot control the technology students utilize at home and must integrate important ICT components into the classroom in order to best impact achievement (Hatlevik et al., 2015). This study showed that occasional use of these digital tools and media may impact students’ technology assessment scores but there is other research that has not found the same correlations (Beland & Murphy, 2016; Biagi & Loi, 2013; Claro et al., 2012; Spiezia, 2011). Therefore, as with most in-school pedagogical choices, moderation in a breadth of areas may be the key to helping students slowly and steadily improve their technological abilities. 5.2. UTILIZING TECHNOLOGY BEYOND SCHOOLWhen asking students to self-report their use of technology for projects unrelated to school, surveys rely on them to be forthcoming. Researchers have documented that perceptions do not always reflect reality, especially for sensitive topics, and may be biased or dishonest (Northrup, 1996; Wilson & Zietz, 2004). It is not likely that students taking the TEL assessment seek to be misleading with their answers to the questionnaire; however, they may not consider some of what they do independently as creation of digital media, presentations, or spreadsheets if it does not appear identical to in-class uses. It is with this in mind that some interesting correlations were found between students’ ICT scores and their use frequency of these types of products. Students who responded that they used home computers to create, edit, or organize digital media once or twice a week or almost every day saw higher scaled scores than those who never did these activities beyond school. Significantly higher scores, albeit a small effect size, for those who did it almost every day are similar to findings by Wang et al. (2018) regarding ICT experiences out of school. This also correlates to research which found that students who use a home computer in addition to having regular access in school had greater confidence in their ICT competencies which can account for higher overall scores (Wastiau et al., 2013). In looking at the scores for students who create presentations or spreadsheets beyond school, the data showed that those who never do these two activities scored significantly higher than those who completed these tasks at any frequency. In fact, the participants who did these two tasks almost every day yielded the smallest ICT score of all the frequencies. Wang et al. (2018) also found a negative association between ICT score and more frequent creation of presentations and spreadsheets out of school, and this is also in alignment with the findings of Zhang et al. (2016) when looking at a previous administration of the TEL assessment. These types of tasks are less challenging and may not adequately match scenarios offered by the TEL assessment for real-life problems. Further, the repetition of basic tasks such as creation of presentations and spreadsheets may distract students by taking time away from other TEL-related learning which could better impact scores (Zhang et al., 2016). Wang et al. (2018) found that many of the out of school variables negatively impacted TEL scores. It is possible that the items on the TEL assessment which contribute to the students’ ICT score are not strongly aligned with the types of activities mentioned in the questionnaire especially considering the ways in which students engage with technology at home. Wastiau et al. (2013) found students to be more competent with social media if they had online access at home. However, Uzun & Kilis (2019) found a significant relationship between heavy media usage at home and lower levels of academic performance. This association may explain some of the lower ICT scores on the TEL assessment as students do not practice media, presentation, and spreadsheet product creation at home, yet they spent an immense amount of social time in the digital world. According to a study conducted by the Pew Research Center, at least 92% of students are accessing the Internet daily with approximately 24% who responded that they are online almost constantly (Lenhart, 2015). Students are regularly accessing social media, video games, and text-based communication; in other words, they are using the Internet for entertainment and communication (Lenhart, 2015; Wang et al., 2011). This work confirms that it is not a meaningful occurrence for students to be creating, editing, or organizing digital media, presentations, or spreadsheets in their personal time. However, negative relationships between students’ use of home technology and their standardized achievement scores have been documented in areas of and beyond technology (O’Dwyer et al., 2005). Therefore, it is important to recognize how students’ behaviors outside of the classroom will impact their achievement on standardized assessments such as the TEL in order to combat it with proper classroom instruction involving technology. 6. CONCLUSIONS AND RECOMMENDATIONSStudents today are generally not proficient in technology, as evidenced by the TEL scores overall and the low scores exhibited in the ICT subset; some students are “independent and critical users of ICT,” but there are many who are not (Fraillon et al., 2014, p. 24). O’Dwyer et al. (2005) found a positive relationship between technology use in school and at home and student achievement on assessments. However, Zhang et al. (2016) posited that students do not need daily access to these technologies and activities but can gain proficiency through other instructional methods. This is further confirmed by Carter et al. (2017) who noted higher content assessment scores from students without regular use of technology. The main consideration, therefore, is that technology must be an appropriate pedagogical choice to enhance content and not distract from it or be taught in isolation as it is the quality and not the quantity that impacts scores (Lei & Zhao, 2007). With this in mind, there are three principles to pull from the findings. 6.1. IDEAL FREQUENCIES FOR IN-CLASS PRODUCT CREATIONThis study showed that it is best to have students create, edit, or organize digital media once or twice a week to help improve their ICT scores. Additionally, having them create presentations once or twice a month can increase standardized assessment achievement. It is these two classroom activities which increased scores on the ICT portion of the TEL a minimum of 5 points but up to an astounding 26 points. The implication for teachers is to find ways to meaningfully incorporate the creation of digital media and presentations into all curricular areas so students can use their higher-order processing abilities to see a better connection to real-life applications of these digital tools. After all, students who are highly engaged in technology-based lessons and passionate about the learning at hand are more likely to learn faster and retain it better (Bandekar et al., 2018). 6.2. OUT OF SCHOOL TECHNOLOGY IMPACTS SCORESThe results of this research indicated that what students do outside of school mostly has a negative impact on their standardized assessment ICT scores. Therefore, students need to be aware of how their media usage can impact their academic achievement and the risks of being too involved with technology (Sert et al., 2019; Uzun & Kilis, 2019). Since the only way to positively impact scores is to create, edit, or organize digital media, students should be encouraged to do this at home although research shows they do this type of behavior already (Lenhart, 2015). This can help students to see transference between the topics and tools learned in school and how those types of products can have authentic, real-world applications and uses. 6.3. INCORPORATION OF REAL-LIFE APPLICATIONS FOR TECHNOLOGYThere are marked differences in how technology is used in class, out of class, and what is asked on an assessment. Several of the technology assessments available both nationally and internationally are boiled down to multiple choice questions or may ask for constructed responses about highly abstract concepts, and often find that a vast majority of students are not proficient (Fraillon et al., 2014; Fraillon et al., 2013; Krupczak & Disney, 2013; The Nation’s Report Card, 2018b). These types of assessments may not match how instruction is usually delivered in class and how students utilize technology outside of school which can account for the lower proficiency scores overall and within the ICT subset. Therefore, when considering technology integration, it is important to go beyond simple uses of software programs and encourage scenario- and simulation-based tasks which better encourage problem-solving and allow for better transference of skills to an assessment. Overall, the results of this study imply that teachers need
to be utilizing technology in the classroom in order to see an increase in
students’ technology scores. Since student scores can improve when they create,
edit, or organize digital media within the classroom, then teachers need to be
provided proper professional development to know what this looks like in their
specific content area. Specifically, teachers need to know how to include more
authentic uses of technology for their subject area rather than having students
artificially working with technology. With this information, administrators can
make sure their staff are knowledgeable about some uses of authentic learning
and assessment as well as problem-based learning. Additionally, presentation
creation is one way to see scores rise, so teachers should include this type of
digital activity in their classes although it does not need to be done daily.
In this way, teachers can better equip students with proper technical knowledge
for the 21st century. 6.4. LIMITATIONSThis research had several limitations. The first, already mentioned, is that students provided responses on questionnaires after completing the TEL assessment and may provide skewed answers that do not match reality (Northrup, 1996). The ambiguity of the questions asked, and thus used to analyze results, may also contribute to this problem and may not best match how students accurately use technology both in and out of school. A second limitation is the proficiency bands created by NAEP for use with their assessments to determine student achievement; many of the scores presented here show a lack of student abilities with regard to ICT when the cut scores set by NAEP may simply be too high (U.S. Department of Education, Office of Planning, Evaluation and Policy Development, 2009). This could mean that evaluating scores provided by NAEP is artificial and does not provide adequate information for stakeholders to draw meaningful conclusions. Finally, the NAEP website allows for some limited statistical analyses with the data but does not allow researchers to download original datasets or know N values (NAEP, 2014). This limited scope of use restricts findings to simple correlations and does not account for any regressions or predictions to make impactful changes. 6.5. RECOMMENDATIONS FOR FUTURE STUDYSince a student’s home use of computers cannot be regulated or monitored, it is necessary for administrators and educators to focus on what is being done in classrooms regarding technology use and instruction. Future research with ICT scores from the TEL assessment should look at some of the other variables related to in-class uses of technology such as the frequency of students designing computer programs or using digital tools to test hypotheses. These other variables may provide greater insight into what specifically helps to improve student scores. Additionally, some of the areas which found higher effect sizes may direct future research. For example, looking at the frequency of students creating presentations for the optimal success in addition to looking at what constitutes presentation-making with technology tools available. Ideally, researchers should strive to find classroom technology uses that significantly contribute to higher scores overall as the nation continues to look for ways to increase the technology abilities of our students. SOURCES OF FUNDINGThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. CONFLICT OF INTERESTThe author have declared that no competing interests exist. ACKNOWLEDGMENTNone. REFERENCES
[1]
Albion,
P. R., & Ertmer, P. A. (2002). Beyond the
foundations: The role of vision and belief in teachers' preparation for
integration of technology. TechTrends, 46(5), 34-38.
[2]
Badia, A., Meneses, J., Sigalés, C., & Fàbregues, S. (2014). Factors affecting school teachers’
perceptions of the instructional benefits of digital technology. Procedia -
Social and Behavioral Sciences, 141(C), 357-362.
[3]
Bandekar, B., Yitbarek, E. M., Lopez, K., Tran, C., & Momoh, E.
(2018). Social Media: Effect, Affect, Teens, and Possible Addiction. Collin
College Undergraduate Interdisciplinary Student Research Conference. 2.
https://digitalcommons.collin.edu/ccuisrc/2018/thursday/2
[4]
Barron,
A. E., Kemker, K., Harmes,
C., & Kalaydjian, K. (2003). Large-scale research
study on technology in K–12 schools. Journal of Research on Technology in
Education, 35(4), 489-507. https://doi.org/10.1080/15391523.2003.10782398
[5]
Beland,
L. P., & Murphy, R. (2016). Ill communication: Technology, distraction
& student performance. Labour Economics, 41,
61-76.
[6]
Biagi, F., & Loi, M. (2013). Measuring ICT use and learning outcomes:
Evidence from recent econometric studies. European Journal of Education, 48(1),
28-42.
[7]
Bond,
J., & Zhang, M. (2017). The Impact of Conversations on Fourth Grade Reading
Performance - What NAEP Data Explorer Tells?. European
Journal of Educational Research, 6(4), 407-417. doi:10.12973/eu-jer.6.4.407
[8]
Carter,
S. P., Greenberg, K., & Walker, M. S. (2017). The impact of computer usage
on academic performance: Evidence from a randomized trial at the United States
Military Academy. Economics of Education Review, 56, 118-132.
[9]
Chen, C.
(2008). Why do teachers not practice what they believe regarding technology
integration? Journal of Educational Research, 102(1), 65-75.
[10] Claro, M., Preiss, D. D., San Martín, E., Jara, I., Hinostroza, J. E.,
Valenzuela, S., Cortes, F., & Nussbaum, M. (2012). Assessment of 21st
century ICT skills in Chile: Test design and results from high school level
students. Computers & Education, 59(3), 1042-1053.
[11] Cohen, J. (1992). A power primer.
Psychological Bulletin, 112(1), 155-159.
[12] Davis, J. H. (2015, December 10). President
Obama signs into law a rewrite of No Child Left Behind. The New York Times.
Retrieved January 28, 2019, from https://www.nytimes.com/ 2015/12/11/us/politics/president-obama-signs-into-law-a-rewrite-of-no-child-left-behind.html.
[13] DeBell, M., & Chapman, C. (2006). Computer and
Internet Use by Students in 2003. Statistical Analysis Report. NCES 2006-065.
National Center for Education Statistics.
[14] Erstad, O. (2015). Educating the digital
generation: Exploring media literacy for the 21st century. Nordic Journal of
Digital Literacy, 10, 85-102.
[15] Ertmer, P. (2005). Teacher pedagogical beliefs: The
final frontier in our quest for technology integration? Educational Technology
Research and Development, 53(4), 25-39.
[16] Eshet-Alkalai, Y. (2004). Digital literacy: A conceptual
framework for survival skills in the digital era. Journal of Educational
Multimedia and Hypermedia, 13(1), 93-106.
[17] Eshet-Alkalai, Y. (2012). Thinking in the digital era: A
revised model for digital literacy. Issues in Informing Science and Information
Technology, 9, 267–276.
[18] Ezquerra, A., Manso, J.,
Burgos, M., & Hallabrin, C. (2014). Creation of
audiovisual presentations as a tool to develop key competencies in
secondary-school students: A case study in science class. International Journal
of Education and Development Using Information and Communication Technology,
10(4), 155-170.
[19] Firman, H., Rustaman, N.
Y., & Suwarma, I. R. (2015, November). The
development of technology and engineering literacy through STEM-based
education. In 2015 International Conference on Innovation in Engineering and
Vocational Education. Atlantis Press.
[20] Fraillon, J., Ainley, J.,
Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a
digital age: The IEA international computer and information literacy study
international report. Springer.
[21] Fraillon, J., Schulz, W., & Ainley,
J. (2013). International computer and information literacy study: Assessment
framework. Springer.
[22] Hatlevik, O. E., Guðmundsdóttir,
G. B., & Loi, M. (2015). Digital diversity among
upper secondary students: A multilevel analysis of the relationship between
cultural capital, self-efficacy, strategic use of information and digital
competence. Computers & Education, 81, 345-353.
[23] Hatlevik, O. E., Throndsen,
I., Loi, M., & Guðmundsdóttir,
G. B. (2018). Students’ ICT self-efficacy and computer and information
literacy: Determinants and relationships. Computers & Education, 118,
107-119.
[24] Helsper, E., & Eynon,
R. (2010). Digital natives: Where is the evidence? British Educational Research
Journal, 36(3), 503-520.
[25] Hohlfeld, T., Ritzhaupt,
A., & Barron, A. (2010). Development and validation of the student tool for
technology literacy (ST2L). Journal of Research on Technology in Education,
42(4), 361-389.
[26] Kaminski, K., Seel,
P., & Cullen, K. (2003). Technology literate students. Educause Quarterly,
3, 34-40.
[27] Kirschner, P., & De Bruyckere,
P. (2017). The myths of the digital native and the multitasker. Teaching and
Teacher Education, 67(C), 135-142.
[28] Krupczak, J., & Disney, K. A. (2013).
Technological literacy: assessment and measurement of learning gains. In Makalah disajikan dalam at the 120th American Society for Engineering
Education Annual Conference & Exposition.
[29] Layton, L. (2015, December 10). Obama signs
new K-12 education law that ends No Child Left Behind. The Washington Post.
Retrieved January 28, 2019, from https://
www.washingtonpost.com/local/education/obama-signs-new-k-12-education-law-that-ends-no-child-left-behind/2015/12/10/c9e58d7c-9f51-11e5-a3c5-c77f2cc5a43c_story.html?noredirect=on&utm_term=.9c2e0eb231c0.
[30] Lei, J., & Zhao, Y. (2007). Technology
uses and student achievement: A longitudinal study. Computers & Education,
49(2), 284-296.
[31] Lenhart, A. (2015, April 9). Teens, social
media & technology overview 2015. Pew Research Center.
https://www.pewresearch.org/internet/2015/04/09/teens-social-media-technology-2015/
[32] Li, Y., Garza, V., Keicher,
A., & Popov, V. (2019). Predicting high school teacher use of technology:
Pedagogical beliefs, technological beliefs and attitudes, and teacher training.
Technology, Knowledge and Learning, 24(3), 501-518.
[33] Lu, J., Hao, Q., & Jing, M. (2016).
Consuming, sharing, and creating content: How young students use new social
media in and outside school. Computers in Human Behavior, 64(C), 55-64.
[34] McEuen, S. (2001). How fluent with information
technology (FIT) are our students? Educause Quarterly, 24(4), 8-17.
[35] The Nation’s Report Card. (2018a). About the TEL
assessment: Assessment framework and design. Retrieved February 18, 2020, from
https://www.nationsreportcard.gov/tel/about/ assessment-framework-design/
[36] The Nation’s Report Card. (2018b). About the
TEL assessment: Sample scenario-based tasks and discrete questions. Retrieved
March 20, 2020, from https://www.nationsreportcard.gov/tel/tasks/
[37] The Nation’s Report Card. (2018c). About the
TEL assessment: Samples, inclusion, and participation. Retrieved February 18,
2020, from https://www.nationsreportcard.gov/tel/about/
samples-inclusion-participation/
[38] National Assessment of Educational Progress.
(2014). 2014 Technology & Engineering Literacy (TEL). The Nation’s Report
Card. https://www.nationsreportcard.gov/tel_2014/#
[39] National Assessment of Educational Progress.
(2018). NAEP Technology & Engineering Literacy (TEL) Report Card. The
Nation’s Report Card. https:// www.nationsreportcard.gov/tel/
[40] National Center for Education Statistics.
(2019). Focus on NAEP: Sampling. The Nation’s Report Card. Retrieved February
18, 2020, from https:// www.nationsreportcard.gov/focus_on_naep/files/
sampling_infographic.pdf
[41] National Center for Educational Statistics.
(2018, May 24). About NAEP. Retrieved February 18, 2020, from
https://nces.ed.gov/nationsreportcard/about/
[42] National Research Council. (1999). Being
fluent with information technology. Washington, DC: The National Academies
Press. https://doi.org/10.17226/6482
[43] Niederhauser, D., Lindstrom, D., & Strobel, J.
(2007). Evidence of the NETS*S in K-12 classrooms: Implications for teacher
education. Journal of Technology and Teacher Education, 15(4), 483-512.
[44] No Child Left Behind Act of 2001, PL 107-110,
107th Cong., § 2401-2404, 115 Stat. 1425 (2002)
[45] Northrup, D. A. (Fall 1996). The problem of
the self-report in survey research. Institute for Social Research, 11(3).
Retrieved from http://www.math.yorku.ca/ISR/self.htm
[46] O’Dwyer, L. M., Russell, M., Bebell,
D., & Tucker-Seeley, K. R. (2005). Examining the relationship between home
and school computer use and students’ English/language arts test scores.
Journal of Technology, Learning, and Assessment, 3(3).
[47] Park, B. J., Broer,
M., & Bohrnstedt, G. W. (2017, April 28). The
relationship between students' contextual factors related to technology and
Technology and Engineering Literacy performance. Paper presented at the 2017
annual meeting of the American Educational Research Association. Retrieved
March 2, 2020, from the AERA Online Paper Repository.
[48] Prensky, M. (2001). Digital natives, digital immigrants part 1. On the Horizon, 9(5), 1-6. https://
doi.org/10.1108/10748120110424816
[49] Ritzhaupt, A., Dawson, K., & Cavanaugh, C. (2012).
An investigation of factors influencing student use of technology in K-12 classrooms
Using Path Analysis. Journal of Educational Computing Research, 46(3), 229-254.
[50] Schmid, R., & Petko,
D. (2019). Does the use of educational technology in personalized learning
environments correlate with self-reported digital skills and beliefs of secondary-school
students? Computers & Education, 136, 75-86.
[51] Sert, H., Taskin Yilmaz, F., Karakoc
Kumsar, A., & Aygin, D.
(2019). Effect of technology addiction on academic success and fatigue among
Turkish university students. Fatigue: Biomedicine, Health & Behavior, 7(1),
41-51.
[52] Spiezia, V. (2011). Does computer use increase
educational achievements? Student-level evidence from PISA. OECD Journal:
Economic Studies, 2010(1), 1-22.
[53] Taimalu, M., & Luik,
P. (2019). The impact of beliefs and knowledge on the integration of technology
among teacher educators: A path analysis. Teaching and Teacher Education, 79,
101-110.
[54] Tort, F., Blondel, F. M., & Bruillard, É. (2008, July). Spreadsheet knowledge and
skills of French secondary school students. In International Conference on
Informatics in Secondary Schools-Evolution and Perspectives (pp. 305-316).
Springer, Berlin, Heidelberg.
[55] U.S. Department of Education, Office of
Planning, Evaluation and Policy Development. (2009). Evaluation of the National
Assessment of Educational Progress, final report.
https://www2.ed.gov/about/offices/list/opepd/ppss/reports.html
[56] Uzun, A. M., & Kilis, S. (2019). Does
persistent involvement in media and technology lead to lower academic
performance? Evaluating media and technology use in relation to multitasking,
self-regulation and academic performance. Computers in Human Behavior, 90,
196-203.
[57] Wang, H., Zhou, X., Lu, C., Wu, J., Deng, X.,
& Hong, L. (2011). Problematic Internet use in high school students in
Guangdong Province, China. PloS one, 6(5).
[58] Wang, K., Wang, X., & Li, J. (2018, April
16) Relationship between students' experiences and Technology and Engineering
Literacy performance. Paper presented at the 2018 annual meeting of the
American Educational Research Association. Retrieved March 2, 2020, from the
AERA Online Paper Repository.
[59] Wang, S., Hsu, H., Campbell, T., Coster, D., & Longhurst, C. (2014). An investigation of
middle school science teachers and students use of technology inside and
outside of classrooms: Considering whether digital natives are more technology
savvy than their teachers. Educational Technology Research and Development,
62(6), 637-662.
[60] Wastiau, P., Blamire, R.,
Kearney, C., Quittre, V., Van de Gaer,
E., & Monseur, C. (2013). The Use of ICT in
Education: a survey of schools in Europe. European Journal of Education, 48(1),
11-27.
[61] Wilson, M. L., & Zietz,
J. (2004). Systematic bias in student self-reported data. Journal for Economic
Educators, 4(4), 13-19.
[62] Zhang, T., Xie, Q.,
Park, B. J., Kim, Y. Y., Broer, M., & Bohrnstedt, G. (2016). Computer familiarity and its
relationship to performance in three NAEP digital-based assessments.
Washington, DC: American Institutes for Research.
This work is licensed under a: Creative Commons Attribution 4.0 International License © Granthaalayah 2014-2020. All Rights Reserved. |