Ensuring the Quality of Digital Content for Learning
Recommendations for K12 Education
State Educational Technology Directors Association (SETDA) Policy Brief,
prepared in partnership with Foresight Law + Policy
M
arch 2015
Context and Overview
In the fall of 2012, SETDA released a groundbreaking report, Out of Print: Reimagining the K-12
Textbook in a Digital Age, which provided initial advice to states and districts about managing
the shift from traditional print-based instructional materials to digital content. While the
advantages to increasing the use of digital content by students and teachers are significant, the
policies and practices associated with navigating a successful digital transition are still emerging
and evolving. Recognizing this challenge, SETDA recently published two issue briefs, building
on Out of Print’s recommendations, to assist states in identifying and implementing additional
policies and practices for keeping the digital content transition on track and on target. The two
initial briefs focused on using digital content to enhance accessibility for students with unique
needs, and clarifying the ownership of teacher-created digital content.
This paper complements SETDA’s prior digital transition policy briefs by examining strategies
for ensuring digital content quality, including exploration of the specific quality-control challenges
and opportunities associated with open educational resources (OER). Specifically, this paper
describes: 1) digital content’s unique characteristics; 2) traditional state/district instructional
materials quality-review practices; and 3) recommendations to inform/strengthen state
strategies for ensuring digital resource quality.
Digital Content’s Unique Characteristics and the Future of Content Quality
Control
Recent advancements in technology have been breathtaking, as reflected by the plethora of
powerful and innovative digital devices and tools in our homes, workplaces, and communities.
Recognizing technology’s potential to improve educational outcomes, even as technology use in
schools has lagged behind other sectors, is key to innovative learning. Educators understand
that the use of digital content and devices to enhance educational opportunities benefit K-12
students. Federal, state, and school district leaders are beginning to refine their policies and
practices with the goal of providing faster and more widely distributed digital technology to K-12
students to improve teaching and learning. This technical educational transformation includes a
still nascent, but growing, use of digital content and open educational resources (OER), which
hold immense promise for providing a significantly greater number of educators and students
with the innovative learning tools needed to meet a state’s rigorous college- and career-ready
standards. OER, in particular, provide a unique and cost- effective pathway for equipping all
students with high-quality, up-to-date, and customizable content and tools.
What are Open Educational Resources (OER)?
OER are “teaching and learning materials licensed in such a way that they are free and may be
used, reused, remixed, and otherwise customized to meet specific needs.” In other words, OER
are teaching, learning, and resource materials, tools, and media that are in the public domain or
are available under an open license so that they may be used and repurposed freely by
educators, students, and self-learners.
OER items can range from a single lesson or instructional support material to a complete unit or
full-course materials; they include e-textbooks, videos, rubrics, assessments, and any other
tools that support teaching and learning.
*See SETDA, Out of Print: Reimagining the K-12 Textbook in a Digital Age (2012), available at
http://setda.org/priorities/digital-content/out-of-print/
Digital materials provide many teaching and learning benefits to educators and students. They
can be updated more quickly than traditional print materials, may be adapted to address
students’ learning differences and styles (with an appropriate license), and can offer interactive
functions that pique student interests. These advantages are particularly true of digital OER,
which offer a lawful pathway for tailoring and adapting content to meet a student’s unique
learning needs.
Digital content provides significant instructional and practical advantages; however, many
districts and schools navigating to digital environments often face implementation challenges
and important policy questions. Perhaps most significantly, many states and districts are unsure
how to best ensure the quality and accuracy of digital materials, without sacrificing the powerful
flexibility and differentiation they offer to teachers and students. This uncertainty often creates a
significant hurdle for digital implementation, including slow decision-making, because school,
district, and state leaders do not want to inadvertently compromise the quality of content in their
classrooms.
Therefore, ensuring digital content quality—including building confidence in the materials among
stakeholders—is an important critical step in more swiftly giving students access to the best
possible resources for supporting and advancing their learning. As digital content, including
OER, rapidly becomes more common, educators and other stakeholders must develop policies
on how to select, judge, use, and refine materials over time. It is incumbent on state and district
leaders to adopt and implement new policy frameworks for ensuring that only high-quality digital
learning materials will be used in schools, classrooms, and all other places where students
learn. Achieving this goal will require a thoughtful policy design process that incorporates
relevant elements of existing quality assurance policies and practices (i.e., those best practices
that apply in both the paper and digital worlds) and merges them with new approaches aligned
to the digital environment.
With this goal in mind, state and local efforts to ensure digital content quality should
begin by asking:
• How can new quality assurance approaches be designed to reflect digital content’s unique
characteristics and advantages, without losing existing best practices?
• Who will be responsible for overseeing new approaches to ensure quality and for building
capacity to deliver it?
The policy recommendations provided later in this paper further explore these two areas. Prior
to turning to these specific recommendations, however, we believe it is instructive to reflect on
the characteristics of existing (print focused) quality assurance systems. Some elements of
quality assurance systems for print align well with the digital transition and should be continued,
while others should be abandoned or significantly modified.
Traditional Systems for Ensuring the Quality of Instructional Materials and Why
Digital Demands a Different Approach
This section provides a high-level overview of typical state and district models for evaluating
traditional print materials, and then examines why ensuring the quality of digital materials, and
promoting their development and use, requires a modified approach.
Full Course Content vs. Supplemental Materials
In general, existing processes ensure that instructional materials are of quality. They aim to
provide educators with confidence that the content they use aligns to state standards, is
accurate, and promotes student success. Instructional materials, whether in traditional or digital
form, include: 1) full-course content (such as textbooks); and 2) supplemental materials that
complement full-course resources.
Under existing quality-review systems for print materials, the rigor of review differs between
these two classes of materials. Formal, rigorous state-level reviews typically focus on core—full-
course materials for certain academic subjects because the material represents the core
learning resource relied on by educators and students, and because acquisition of the material
usually involves a very large state or district investment. At the other end of the spectrum, single
lessons, units, and other supplemental materials are not typically subject to a formal review
process (in part because they are usually provided for free with core content). Although
supplemental materials are not typically rigorously reviewed, many quality considerations
currently apply to both full-course content and supplemental materials (e.g., all materials should
be free of error, aligned to state standards, and be free from bias).
The digital transition is beginning to blur the distinction between these types of materials. For
example, the expected lower costs associated with updating digital full-course materials could
diminish some of the fiscal risks associated with exchanging a rigorous front loaded content
review process for a balanced system that is more reliant on classroom evaluations. That
change, however, does not diminish the importance of ensuring that both types of content are
high quality when they are acquired and, especially in the case of OER, as they are adapted
and tailored to meet specific student and classroom needs.
State vs. District Role in Adoption and Quality Assurance
State approaches to assessing full-course print resources typically differ. Some state laws
provide no state review function and instill in local districts the full control of their instructional
materials adoption.
1
Other states vet and approve instructional materials at the state level and
employ either mandatory or advisory approaches for district adoption. For example, in Alabama,
a mandatory adoption state, districts must select textbooks from the list of state-approved
materials.
2
Florida employs a hybrid approach, allowing districts to spend up to 50% of their
instructional materials allocation on non- state-adopted material.
3
A growing number of states
use advisory approaches, in which they vet and approve materials but ultimately defer to local
decision-making, given different contexts and needs. Indiana, for example, shifted from
promulgating a mandatory list of textbooks from which districts had to choose, to an advisory
approach. Indiana continues to offer districts the resources and results of a full state review,
with guidance on the alignment of certain materials to state standards, but permits districts the
autonomy to select a textbook not on the state list.
4
State-level, institutionalized content assurance systems can enable greater continuity for
student educational experiences across the state and also provide cost savings to districts,
which can rely on state-led reviews. Policies and best practices for ensuring digital content
quality, however, need not be defined exclusively at the state level, but instead may be
appropriately integrated into whatever model (state, district, or hybrid) a particular jurisdiction
elects to implement.
Key Elements of Traditional Systems for Ensuring Content Quality
Traditional approaches for assuring the quality of print textbooks and other full-course
instructional materials typically involve significant work over many months, using a variety of
established standards and measures, with formal systems of peer review and expert
assessment. Such systems often include the following overarching steps (in order):
• Establish Adoption Cycle: The State Board of Education, or designated entity, creates an
adoption cycle for subjects in the state’s core curriculum to ensure that they are reviewed
periodically (usually over a defined period of years).
• State Proclamation/Call for New Materials: The State Board of Education, or designated
entity, publishes a request for standards-aligned, accurate materials in a given subject and
grade level (with given specifications, including for example, accessibility requirements).
• Bidders Conference: Interested parties convene for a question and answer session about the
proclamation’s focus and requirements.
• Initial Materials Development and Submission: Publishers create materials consistent with the
proclamation’s requirements and submit them to the state agency.
• Expert Panel Review: The state agency identifies expert educators from across the state in
the relevant subject area at issue in the proclamation to serve on a review committee. They
analyze the material for alignment to state standards and identify factual errors. After an initial
period of independent review, experts are convened to make consensus decisions about
whether the content meets the state’s needs or falls short in certain areas, and to identify further
questions for the publishers.
• Publishers’ Response and Committee Recommendation: Publishers respond to the review
committee’s questions and requirements. If the committee accepts the publisher’s response, it
will recommend that the State Board of Education or relevant authority put the material on the
state approved content list.
• Public Comment and State Board of Education Action: The State Board considers the
committee’s recommendation, including inviting public input, and then adopts or rejects the
content.
This approach, and the quality assurance process timeline, varies somewhat state-by-state, but
this model’s core elements appear in most jurisdictions. The Florida Department of Education,
Office of Instructional Materials (FDE), for example, leads the review of instructional materials
for individual subjects and grade levels to ensure that materials comply with state curricular
requirements.
5
FDE invites standards aligned content from vendors, which is then reviewed
online by two national or state level experts. In the event of a tie, a third expert will review the
content. Superintendents are then asked to supply a district expert to rate two or three of the
materials on the instructional usability of the resources. The review is then open to the public for
their comments. Materials are ultimately adopted or rejected by the Commissioner of Education.
Florida rotates subject areas over the course of a five-year period; once a review and assurance
cycle is complete, traditional print materials typically are not modified until the next cycle, though
there is a substitution policy in place where a publisher may request substitution of materials
after the first six months of the adoption. Florida’s adoption priorities (for illustration) follow in the
table below.
Florida Textbook Adoption Priorities
1
The following are some, but not all, of the criteria included within each of Florida’s three
priorities for the evaluation of instructional materials.
C
ontent Presentation Learning
Alignment with curricular
requirements
Comprehensiveness (teacher
resources
Motivational strategies
(challenge, relevant)
Level of complexity Alignment of components Teaching a few “big ideas”
Accuracy Organization of materials Explicit instruction (clarity of
direction and explanations)
Currentness Readability Guidance and support
Authenticity (life connections,
interdisciplinary treatment)
Pacing of content Active student participation
Multicultural representation Ease of materials use Targeted instructional and
assessment strategies
What Attributes Make Digital Content Distinct from Print Materials?
In contrast to traditional textbooks and other print materials, digital materials may be modified
and refined much more easily, enabling adjustments to address local needs and to ensure
relevance and accuracy over time, with no need to consider the costs of a new round of
printing.
6
These attributes lower the quality assurance process’s stakes because problems may
be addressed quickly (e.g., without waiting for a multi-year print cycle) and at lower cost (e.g.,
digital materials only need to be republished electronically, not physically reprinted). In addition,
digital materials create opportunities for meaningful evaluation of the content’s performance in
the classroom (e.g., through feedback loops from educators), followed by timely improvement,
adjustment, and review before students encounter the new materials. Finally, digital OER’s
unique intellectual property characteristics require somewhat different quality assurance
strategies, including a greater focus on building educators’ and school leaders’ capacity to
skillfully adapt, develop, and judge the quality of digital resources. These intellectual property
characteristics permit content modification and refinement to be undertaken by multiple users,
such as teachers and students, during the periods between regular approvals by state or district
officials.
Given these attributes, digital materials built via collaboration, co-creation, and continuous
refinement necessitate a different approach to quality assessment compared to the kind of
multiple-month, once-every-five-year cycle often employed for print textbooks. Systems for
ensuring the quality of digital materials, including OER, must be more fluid and flexible,
reflecting the continual nature of the materials’ development, refinement, and improvement. And
because digital content can be edited (if licensed appropriately) and revised incrementally over
time at low (or no) cost, states can adjust time- intensive traditional systems without
compromising quality. This adjustment could include exchanging front-end heavy traditional
quality assurance models with more balanced systems that rely more heavily on ongoing
evaluation of the content’s performance in the classroom, supplemented by robust systems for
building the capacity of educators, school leaders, and students to recognize and support
quality assurance.
At the same time, the quality standards employed to assess instructional materials should not
differ based on whether a resource is print or digital. Indeed, digital materials can be assessed
using many of the same quality indicators and metrics that states and districts have used for
print content.
For example, Florida’s state-level reviews of instructional materials focus on three priorities:
content, presentation, and learning. As the table above illustrates, the criteria employed by
Florida for each of these three priorities would likely be relevant for the assessment of any
instructional resource or tool, including those licensed as OER. Given this context, states might
choose to continue using some existing print quality-assurance tools (e.g., rubrics) to evaluate
digital materials.
States and districts might also look to evaluative instruments developed specifically for the
review of OER, but which apply to all types of digital and print resources. For example, Achieve
developed eight OER rubrics that evaluate OER quality aspects, including rubrics focused on
alignment to standards, utility of materials designed to support teaching, and assurance of
accessibility.
7
Rubrics to assess OER also have been developed at Temoa, an open
educational resources portal associated with Mexico’s Tecnologico de Monteree.
8
Other quality
assurance instruments increasingly may be found on online collaborative websites—with the
rubrics themselves being OER tools. In addition, the State Instructional Materials Review
Association (SIMRA) compiles a number of rubric tools, including some developed for use with
OER.
9
The following exhibit outlines major focuses of certain available rubrics for assessing
OER.
Quality Assessment Criteria of Sample OER Rubrics
Achieve Temoa
Degree of explanation to standards Content quality
Quality of explanation of the subject matter Motivation
Utility of materials designed to support
teaching
Presentation design
Quality of assessment Usability
Quality of technological interactivity Accessibility
Quality of instructional and practice exercises Educational value
Assurance of accessibility Overall rating
These rubrics alone, however, are not sufficient to ensure digital content’s quality. The following
sections describe additional steps states should consider when updating their content
assurances processes for digital content.
Policy Recommendations: Ensuring the Quality of Digital Materials
Every state needs a modern—more innovative—framework for evaluating digital instructional
materials. These new frameworks should acknowledge digital content’s unique characteristics.
Traditional approaches to ensuring content quality, with the evaluation of pristine, untouched
print materials examined before long-term use by educators and students, and major
investments in printing, must be re-examined. A more valuable process might involve less
onerous initial state or district screening of digital materials complemented by ongoing
opportunities to receive feedback from the classroom and the field on how the materials work
(or do not work), including assessing student learning outcomes connected to the content’s use.
Such a process should also include a focus on building educator, school leader, and student
capacity to recognize and ensure quality, particularly with regard to the growing use of digital
OER. With this high-level vision in mind, this paper makes five core recommendations for state
and districts to consider when updating their processes for ensuring the quality of digital
resources.
1. Establish a clear vision statement to promote and guide digital content development,
review, and use, grounded in an unwavering commitment to quality, accuracy, and
accessibility, including alignment to college- and career-ready standards. Communicate
that vision to all stakeholders.
2. Designate experienced state and district leaders to lead quality assurance policy
development and oversight, and also empower practitioners, such as curriculum experts,
professional learning specialists, content experts, technology leaders and other
stakeholders involved with supporting implementation strategies for ensuring digital
materials quality. This step must include investing in stakeholders’ capacity—at all
levels—to execute the state or district’s digital content quality assurance strategy,
including establishing classroom evaluation and performance templates/protocols,
complemented by feedback loops designed to lead to timely content improvements and
updates.
3. Provide guidance describing the characteristics of well-balanced quality assurance
systems for digital content, including OER, such as developing and acknowledging tools
and uniform state or local indicators and standards, and advancing an inclusive, not
exclusive, approach to quality review.
4. Support educator preparation and professional learning opportunities focused on
building the educator’s capacity to assess digital material’s quality, including assessing
the initial and ongoing quality of OER, which should evolve and improve over time.
5. Ensure sufficient financial resources to establish and sustain an effective system of
quality assessment of digital materials, including OER.
Policy Recommendations
As digital content nears the tipping point into mainstream adoption and use,
10
states and
districts must plan for and invest in updated and improved systems for the quality assessment of
instructional materials and that support the development and use of OER. To support these
efforts, this section further explores the five recommendations outlined above.
1. Establish a clear vision statement to promote and guide digital content development,
review, and use, grounded in an unwavering commitment to quality, accuracy, and
accessibility, including alignment to college- and career-ready standards. Communicate
that vision to all stakeholders.
State and district leaders should adopt a public strategy for the development, review, and use of
quality digital materials in classrooms and should communicate that vision to educators, school
leaders, students, content-creators, and other stakeholders, emphasizing digital content’s role in
promoting student success. To support implementation of this vision, and build confidence in
digital materials, states and districts must effectively communicate with the public and all
stakeholders about their commitment to ensuring digital content quality, including being
transparent about the strategies that will be used to ensure that only quality digital materials are
used in the classroom.
2. Designate experienced state and district leaders to lead quality assurance policy
development and oversight, and also empower practitioners, such as curriculum experts,
professional learning specialists, content experts, technology leaders and other
stakeholders with supporting implementation strategies for ensuring digital materials
quality. This step must include investing in stakeholders’ capacity—at all levels—to
execute the state or district’s digital content quality assurance strategy, including
establishing classroom evaluation and performance templates/protocols, complemented
by feedback loops designed to lead to timely content improvements and updates.
Quality assessment of digital instructional materials, including OER, should be formalized at the
state or district level and then be embedded in the education enterprise as a central exercise of
educators and school leaders. This step includes establishing a robust process that enables the
assessment of digital materials through different approaches and at various levels—with options
for self-assessment, peer (and student) review, quality branding, and benchmarking or rating
systems.
States and districts should invest in and support institutional mechanisms for the quality
assessment of digital resources. A formal approach can serve as a template for local and
individual efforts. For example, the process led by Washington’s Office of the Superintendent of
Public Instruction (OSPI), described below, includes a quality assurance template for school
districts. States should formalize quality assurance efforts at the state and/or local level and
then model the types of activities and considerations that educators, school leaders, and
students can employ when analyzing the digital content’s effectiveness.
A Case Study: Washington State
In April 2012, the Washington state legislature passed a bill that charged the state education
agency, the Office of the Superintendent of Public Instruction (OSPI), with creating a library of
OER aligned to the Common Core State Standards (CCSS). OSPI also was required to raise
school district awareness of OER.
Pursuant to this directive, OSPI developed and implemented a review process to gauge the
alignment of openly licensed courseware to the CCSS, focusing during two review cycles on
Algebra, Integrated Math, Geometry, and grades 9 to 12 English Language Arts. Engaging
educators and others via webinars and workshops, and selecting and training subject matter
experts to assess OER, OSPI used three review instruments to assess the quality of the
materials— with two instruments relevant for the quality review of all instructional materials,
including traditional materials, and one focused specifically on the assessment of OER.
Specifically, OSPI assessed materials using: 1) the Instructional Material Evaluation Tool
(IMET) created by Student Achievement Partners and based on the publisher’s criteria
developed by the Council of Chief State School Officers, the National Governors Association,
and the lead writers of the CCSS to assess alignment of OER with the CCSS, 2) the EQUIP
rubric developed by Achieve to measure CCSS alignment, and 3) the Achieve rubrics for OER,
discussed in this policy brief.
Following this quality assurance work, OSPI created an online library of OER reviews. For each
resource included in the library, OSPI provides an overview of the review, ratings for each
standard of the three assessment instruments used, and comments from each of the reviewers.
Resources given low ratings are included in the library as well.
OSPI’s process now serves as a model for Washington State school districts considering the
adoption of full-course OER, and the results of its reviews are available to schools and districts
in an online resource library.
For more information, see OSPI, Open Educational Resources, available at
http://digitallearning.k12.wa.us/oer/.
The knowledge level of individuals designated to support formal quality assurance efforts at the
state or district level must also be a major consideration. States and districts should consider,
for example, establishing specific mandatory digital content knowledge or experience, along
with other prescribed expectations for the individuals selected. For example, Washington’s
OSPI selected and trained twenty reviewers per review cycle with subject matter expertise (10
in mathematics and 10 in English, language arts) and familiarity with the Common Core State
Standards to lead the examination of the alignment of available OER courseware to the
Common Core State Standards. Other approaches may necessitate different participant types,
such as content developers, purchasers, users, and researchers. Some states and districts
include students on the teams conducting quality assessments of instructional materials, and
the assessment of digital materials may require information technology specialists. States and
districts should determine the goals and objectives of their digital quality assurance work at the
outset (see recommendation 1) and then select relevant and representative stakeholders who
can lead and execute the quality assurance process.
3. Provide rich guidance describing the characteristics of sound digital content quality
assurance systems, such as developing and acknowledging tools and uniform state or
local indicators and standards, and advancing an inclusive, not exclusive, approach to
quality review.
States and districts should develop and provide educators with guidance about how to ensure
the quality of digital instructional materials, including OER. As described above, many of the
standards and indicators traditionally used by states and districts to assess instructional
materials can and should be included in a quality assessment of digital resources. Additionally,
states and districts can ensure that educators have access to quality assurance materials
designed with OER in mind, which could involve newly developed resources or the identification
of existing tools, like Achieve’s OER rubrics. The interplay of sound traditional approaches and
new quality review tools should be highlighted. The former are time-tested and broadly
applicable and may be well known among educators, while newer digital content quality
assurance strategies can be tailored to acknowledge the iterative nature of digital development
and revision.
For example, broad standards and specific quality measures to consider include the
following:
• Content quality: accuracy; alignment to standards; contemporary; comprehensiveness,
organization, and pacing; and representation of diverse cultural, gender, and linguistic
backgrounds
• Pedagogical quality: cognitive considerations—types of tasks, challenge, and learning
outcomes; accessibility for varying student needs; and motivational strategies
• Resources quality: separate materials to support teacher planning and presenting subject
matter, assessments, and technological interactivity
• Technological quality: accessibility, usability, interoperability across systems and platforms;
and appropriate open licensing
When developing guidance for the field on how to assess digital quality, states and districts
should avoid overly prohibitive approaches with respect to digital content, in which any content
that does not meet a specific mark is barred from the classroom. Digital content can be updated
and altered for different teaching styles and to reflect unique student needs and interests at any
time. For example, a quality assurance review of OER might describe the material’s intended
purpose and identify its shortcomings in such a way that a teacher may recognize the tool as
relevant, address its problems through necessary improvements, and then use the resource,
tailored to student needs. Given this dynamic, state and district policymakers should avoid an
exclusionary approach to digital content when possible.
4. Support educator preparation and professional learning opportunities focused on
skills for the quality assessment of digital materials, including OER.
Teachers increasingly use technology and digital content in their classrooms but report
challenges in identifying and vetting the materials that they discover.
11
States, districts, and
educator preparation programs should provide educators with professional learning
opportunities on how to select and use (and, in the case of OER, create and modify) digital
instructional materials and integrate them into their classrooms.
12
A key focus of such courses
and workshops should be on the skills and strategies needed to assess the quality of digital
materials against state standards and local needs and expectations. Such approaches could
entail use of the quality assessment instruments discussed above, with exercises that guide
educators through the process of examining digital resources against the established quality
standards.
These types of professional learning opportunities could be led and facilitated by highly
experienced teachers and other educators who are skilled in the use of digital materials,
particularly to the degree they are experienced at creating, using, and modifying OER. The Utah
State Office of Education (USOE), for example, engaged teachers in the process of aggregating
OER and aligning materials with state standards to produce completely open textbooks for Utah
science courses.
13
States and districts should build on these types of efforts as they create
relevant seminars, workshops, and the like—and should look to master educators such as the
ones engaged by the USOE to help develop the content of these professional learning
opportunities.
Professional learning activities for (and by) educators not only are useful for formal quality
reviews supported by the state or district in which materials are officially vetted. They also will
empower teachers and other educators to vet available digital materials independently, as the
need arises, in order to identify resources that are valuable for their particular students.
5. Ensure financial resources to establish and sustain an effective system of quality
assessment of digital materials, including OER.
Finally, states and districts must support the use of high-quality digital learning materials,
including OER, in classrooms with necessary investments of funding and other resources. This
requires that states and districts address educator and student access to digital resources and
invest in professional development to enhance educator acumen regarding the development,
use, and assessment of digital materials. As part of this investment, policymakers might
consider incentives that encourage OER development, use, and refinement—for example,
publicly acknowledging exemplars for OER quality assessment, which should contribute to
overall content quality.
States and districts also should ensure dedicated resources for evaluating the impact of digital
learning materials on teacher practice and student learning over time. Technological advances,
including digital data systems, enable states and districts to more easily collect information from
the field and track student progress. For the quality assurance of digital materials, then, states
and districts should consider how they cost-effectively obtain feedback from educators on the
effectiveness of various digital resources, assess a resource’s impact on students’ academic
progress, and implement timely content updates for redistribution to the field.
In light of these shifts, additional support for digital content specialists at the state and local
educational agency level should be considered. States should also examine whether current
policies regarding instructional materials, such as the statutes and regulations that govern
purchasing, create financial or other barriers to the use of digital content.
14
Conclusion
Digital content, including open educational resources, holds tremendous promise for students
and the schools that serve them. Digital tools and resources are increasingly available, and
states and districts must think critically about how to instill in educators the confidence to use,
evaluate, and propose updates to them. Building a strong, multi-level quality assurance system
(recognizing digital content’s unique attributes) is necessary to ensure digital content’s greater
incorporation into the classroom and use by teachers and students. Quality assurance may
occur both formally, building on traditional state review processes that have served as templates
for local reviews, and informally, with individual educators reviewing and selecting the
instructional materials that best fit their needs.
States and districts can ensure that modern frameworks exist to vet and assess the materials
that are used in classrooms by establishing and communicating a clear vision for digital
materials, including OER; tasking relevant stakeholders and practitioners with modeling quality
assessment work; providing guidance and resources on quality assessment; empowering
educators to undertake quality assessment work by providing them with relevant professional
learning opportunities; and supporting sustainable quality assessment with necessary
investments. Every day, educators look critically at the resources they use to promote student
success. With quality assessment part of the everyday fabric of school culture, the efforts
described in this policy brief should not be perceived as an additional burden on the education
enterprise. Rather, they can assist schools and educators with the process of identifying and
creating high-quality digital learning materials that enrich the learning experience for students
and improve student outcomes.
This work is licensed under a Creative Commons Attribution 4.0 International License.
(Footnotes)
1 Fla. Dep’t of Educ., Priorities for Evaluating Instructional Materials: Research Update
(Florida Dep’t of Education,2008), available at
http://www.cimes.fsu.edu/index.cfm?fuseaction=publications.view&publicationID=23&sor
tBy=3 (last accessed Aug. 18, 2014). Although the report is several years old, its
publication represents a significant, multi-year investment by Florida to identify majority
priorities for evaluation of instructional materials, based on nine years of research.\
Below is a list of fourteen sources cited in the previous material.
1. See e.g. Ariz. rev. Stat. Ann. § 15-721.
2. AlA. Code § 16-36-62.
3. FlA. StAt. § 1006.40(3)(b)). See also Fl. Dep’t of Educ., 2013 Policies and Procedures for
the Florida Instructional Materials Adoption, available at
http://www.leg.state.fl.us/STATUTES/index.cfm?App_mode=Display_Statute&Search_String
=&URL=1000-1099/1006/Sections/1006.40.html (last accessed Aug. 22, 2014)
4. ind. Code § 20-2612-1; see also Ind. Dep’t of Educ., Frequently Asked Questions Regarding
New State Textbook Adoption Procedures, Textbook Rental Fees, and 1:1 Device Initiatives
(June 21, 2012), available at http://doe.in.gov/sites/default/files/curriculum/ textbook-faq-6-
21-2012.pdf (last accessed Aug. 22, 2014); Memorandum from Ind. Dep’t of Educ. to
Superintendents, Principals, Textbook Adoption Coordinators (Feb. 8, 2012), available at
http://doe.in.gov/sites/default/files/curriculum/textbook-memo-021312-1. pdf (last accessed
Aug. 22, 2014).
5. FlA. StAt. §§ 1006.31-1006.36. See also Fla. Dep’t of Educ., Priorities for Evaluating
Instructional Materials: Research Update
http://www.cimes.fsu.edu/index.cfm?fuseaction=features.view&featureID=14 (Florida Dep’t
of Education, 2008), available at (last accessed Aug.18, 2014).
6. Of course, even with respect to static instructional materials like textbooks, teachers and
other educators long have enhanced these resources in real time with personalized
modifications and supplements, though these adaptations may not always comply with
traditional copyright frameworks.
7. Achieve, Open Educational Resources, available at http://achieve.org/oer-rubrics (last
accessed Aug. 22, 2014).
8. Temoa, Rubrics to Evaluate Open Educational Resources (OER) (Feb. 15, 2011),
available at http://www.temoa.info/sites/default/files/ OER_Rubrics.pdf (last accessed Aug.
24, 2014).
9. State Instructional Materials Review Association, Documents, available at
http://simra.us/wp/references/documents/ (last accessed Sept. 8, 2014).
10. Nomura Group, Open Education Picking Up Speed (March 14, 2014) (based on global
markets research, recommending reduction in investment in legacy education content
providers, finding “OER is on the edge of the tipping point into mainstream adoption and
further ahead than we thought”).
11. Parthenon Group, US Teacher Survey (n=2,076).
12. See The Accessibility of Learning Content for All Students, Including Students with
Disabilities, Must Be Addressed in the Shiftto Digital Instructional Materials (SETDA, June
2014), available at http://setda.org/wp-content/uploads/2014/03/SETDA_PolicyBrief_
Accessibility_FNL.5.29.pdf (last accessed, Aug. 24, 2014); Clarifying Ownership of Teacher-
Created Digital Content Empowers Educators to Personalize Education, Address Individual
Student Needs (SETDA, May 2014), available at (last accessed Aug. 24, 2014)
13. Utah Education Network, Open Educational Resources, available at http://uen.org/oer/ (last
accessed Aug. 25, 2014). The participation of teachers reflects Utah’s historic confidence in
educator participation in the review of instructional materials; the state’s law establish- ing
the State Instructional Materials Commission requires that the body include a secondary
school principal, elementary school principal, secondary school teacher, and elementary
school teacher. Utah Code Ann. § 53A-14-101.
14. See, e.g., Clarifying Ownership of Teacher-Created Digital Content Empowers Educators to
Personalize Education, Address Individual Student Needs (SETDA, May 2014), supra note
14, which discusses importance of updating certain policies and addressing issue of
ownership of OER
End of source list.
