testing

When we are presented with new information, we try and connect it to information we already hold. This is automatic. Sometimes the information fits in easily; other times the fit is more difficult — perhaps because some of our old information is wrong, or perhaps because we lack some of the knowledge we need to fit them together.

When we're confronted by contradictory information, our first reaction is usually surprise. But if the surprise continues, with the contradictions perhaps increasing, or at any rate becoming no closer to being resolved, then our emotional reaction turns to confusion.

Confusion is very common in the learning process, despite most educators thinking that effective teaching is all about minimizing, if not eliminating, confusion.

But recent research has suggested that confusion is not necessarily a bad thing. Indeed, in some circumstances, it may be desirable.

I see this as an example of the broader notion of ‘desirable difficulty’, which is the subject of my current post. But let’s look first at this recent study on confusion for learning.

In the study, students engaged in ‘trialogues’ involving themselves and two animated agents. The trialogues discussed possible flaws in a scientific study, and the animated agents took the roles of a tutor and a student peer. To get the student thinking about what makes a good scientific study, the agents disagreed with each other on certain points, and the student had to decide who was right. On some occasions, the agents made incorrect or contradictory statements about the study.

In the first experiment, involving 64 students, there were four opportunities for contradictions during the discussion of each research study. Because the overall levels of student confusion were quite low, a second experiment, involving 76 students, used a delayed manipulation, where the animated agents initially agreed with each other but eventually started to express divergent views. In this condition, students were sometimes then given a text to read to help them resolve their confusion. It was thought that, given their confusion, students would read the text with particular attention, and so improve their learning.

In both experiments, on those trials which genuinely confused the students, those students who were initially confused by the contradiction between the two agents did significantly better on the test at the end.

A side-note: self-reports of confusion were not very sensitive, and students’ responses to forced-choice questions following the contradictions were more sensitive at inferring confusion. This is a reminder that students are not necessarily good judges of their own confusion!

The idea behind all this is that, when there’s a mismatch between new information and prior knowledge, we have to explore the contradictions more deeply — make an effort to explain the contradictions. Such deeper processing should result in more durable and accessible memory codes.

Such a mismatch can occur in many, quite diverse contexts — not simply in the study situation. For example, unexpected feedback, anomalous events, obstacles to goals, or interruptions of familiar action sequences, all create some sort of mismatch between incoming information and prior knowledge.

However, all instances of confusion aren’t necessarily useful for learning and memory. They need to be relevant to the activity, and of course the individual needs to have the means to resolve the confusion.

As I said, I see a relationship between this idea of the right level and type of confusion enhancing learning, and the idea of desirable difficulty. I’ve talked before about the ‘desirable difficulty’ effect (see, for example, Using 'hard to read' fonts may help you remember more). Both of these ideas, of course, connect to a much older and more fundamental idea: that of levels of processing. The idea that we can process information at varying levels, and that deeper levels of processing improve memory and learning, dates back to a paper written in 1972 by Craik and Lockhart (although it has been developed and modified over the years), and underpins (usually implicitly) much educational thinking.

But it’s not so much this fundamental notion that deeper processing helps memory and learning, and certain desirable difficulties encourage deeper processing, that interests me as much as idea of getting the level right.

Too much confusion is usually counter-productive; too much difficulty the same.

Getting the difficulty level right is something I have talked about in connection with flow. On the face of it, confusion would seem to be counterproductive for achieving flow, and yet ... it rather depends on the level of confusion, don't you think? If the student has clear paths to follow to resolve the confusion, the information flow doesn't need to stop.

This idea also, perhaps, has connections to effective practice principles — specifically, what I call the ‘Just-in-time rule’. This is the principle that the optimal spacing for your retrieval practice depends on you retrieving the information just before you would have forgotten it. (That’s not as occult as it sounds! But I’m not here to discuss that today.)

It seems to me that another way of thinking about this is that you want to find that moment when retrieval of that information is at the ‘right’ level of difficulty — neither too easy, nor too hard.

Successful teaching is about shaping the information flow so that the student experiences it — moment by moment — at the right level of difficulty. This is, of course, impossible in a factory-model classroom, but the mechanics of tailoring the information flow to the individual are now made possible by technology.

But technology isn't the answer on its own. To achieve optimal results, it helps if the individual student is aware that the success of their learning depends on (or will at least be more effective — for some will be successful regardless of the inadequacy of the instruction) managing the information flow. Which means they need to provide honest feedback, they need to be able to monitor their learning and recognize when they have ‘got’ something and when they haven’t, and they need to understand that if one approach to a subject isn’t working for them, then they need to try a different one.

Perhaps this provides a different perspective for some of you. I'd love to hear of any thoughts or experiences teachers and students have had that bear on these issues.

This post is the fourth and last part in a four-part series on how education delivery is changing, and the set of literacies required in today’s world. Part 1 looked at textbooks; Part 2 at direct instruction/lecturing; Part 3 at computer learning.. This post looks at learning models and types of literacy.

 

Literacy. What does it mean?

Literacy is about being able to access information locked up in a code; it's also about being able to use that code. To be literate is to be able to read and write.

There's also another aspect of literacy that goes beyond mere decoding. This is about reading with understanding, with critical awareness.

Argument around the dangers of modern technology tends, in the way of arguments, to simplistically characterize the players: Internet = short, shallow; Social media = frivolous, distracting; Games = frivolous; Textbooks, Lectures = serious, deep, instructive.

But of course this is ridiculous even if we restrict ourselves to the learning context. Even social media have their uses. Even games can teach. And even textbooks and lectures can be shallow, or uninstructive, or inaccurate. (Indeed, way back in my first year of university I experienced a calculus lecturer who, I believe, reduced my understanding of calculus!)

The internet is, as we all recognize, a two-sided tool (but every tool is). Many people worry about the misinformation, the shallowness of much of the information, the superficiality of surfing, the way people might get stuck in a little corner that reinforces their vulnerabilities or prejudices, and so on.

We can say the same about infographics (data visualization, visual communication, call it what you will). It’s fostered as a way of helping us deal with the complexity and quantity of information (and I’m a big fan of it), but some people have criticized it for its potential for misinformation. Of course, text (wherever found) is far from pure in this respect!

But we don’t deal with misinformation by banning it (well, some of us don’t); we deal with it by providing the tools and the education so that people can recognize when something is being dangerously misleading or just plain wrong.

So, one of the important aspects of literacy (once you get beyond the decoding level) is being able to evaluate the information.

Why do we talk about digital literacy? Do we really need a new term (or terms)?

It comes down to skills. Because that is what literacy is: it's a skill (with all that that implies). And the new literacies do, undeniably, require new skills.

As far as the decoding aspect is concerned, well, text is still text. And textbooks have always included illustrations, so you could say that that is not new either. But that would be a mistake. The problem with visualizations is that it is not obvious that there's a skill to reading them — they're not as transparent as most believe (hence the misinformation claim). Humans have always used pictures to communicate; it is only recently that these have become sufficiently sophisticated to warrant the term 'language'.

So one of the modern literacies must be visual language, which like verbal language (and math and music), comes in different flavors. We wouldn’t use the same strategies to interpret and analyze a novel as we would a chemistry text, or a poem. We need to develop the same understanding of the taxonomy of visual language.

So I think we should include visual literacy in our new literacy set.

But of course, the new information delivery systems have requirements that go beyond content. Being able to use the code goes beyond reading text and pictures. It involves being able to navigate the delivery system. With a book, you just have to turn the pages. But with hyperspace, learning spaces, video-books, and so on, 'reading' is more complicated.

This is the important thing, the qualitative shift: the shift from linearity. Having a space, be it the whole of the internet or a confined learning space, in which you can go in many directions, in which there is no one path, may be empowering and richly layered, but it is not a place you can throw anyone into without training. Not if they are going to truly benefit from it. Like the need for visual literacy, this is another under-recognized need.

The complexity of these spaces and their navigation has, however, led to a number of useful distinctions being made — between digital literacy and computer literacy, information literacy, and media literacy (among others). Basically, these point to the need to distinguish between an ability to use technology (know the language of software — What’s a window? What’s the difference between a browser and a search engine? Do you hashtag your tweets? Do you use folders?) from the ability to find, filter, and evaluate information, and from the ability to actively participate in the information flow across media (Do you change your verbal style appropriately when you move from a tweet to a YouTube script to a written report to a comment on someone’s blog? Do you use different modes of analysis and evaluation when viewing different media?)

Given that we want students to become adept at all of these, how should we teach them?

In an interview, Will Richardson, a teacher whose experiences with interactive Web tools in the classroom led him to write Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms talks about the need for teachers to have a visible presence on the Web, to participate in learning networks, and about how this openness is a huge culture shift for the closed shop of teachers. About network literacy as a key skill: “students should be able to create, navigate, and grow their own personal learning networks in safe, effective, and ethical ways. It’s really about the ability to engage with people around the world in these online networks, to take advantage of learning opportunities that are not restricted to a particular place and time, and to be conversant with the techniques and methodologies involved in doing this.” About how kids may be more technologically savvy, but they need help sorting out which information, and which people, to trust.

My favorite bit: he talks about Rethinking Education in the Age of Technology: The Digital Revolution and Schooling in America (Amazon affiliate link), which apparently discusses how historically we used to have an apprenticeship model of education, which moved to the factory model, where it’s all about training everyone the same way, and now we’re moving back to a more individualized, self-directed and flexible lifelong-learning model. Put in those terms, it seems clear that we can’t just keep tweaking; the changes are more fundamental than that.

He also asks why no one is consciously teaching kids how to read and write in linked environments — which relates back to my point about learning to traverse non-linear spaces.

But the onus shouldn't (and can't) be all on the teacher. They need a structure that supports them.

But as with learning networks and digital tools (Facebook, Twitter, blogs, RSS, Scribd, Flickr, Tumblr,  Mashable, ...), the structures too keep changing under their feet. Blackboard, Moodle, Udemy, Instructure (to pick out some old and some new).

It's perhaps easier when the structure is purely online. (In the U.S., the Keeping Pace with K-12 online learning 2010 report tells us that state virtual schools or state-led online learning initiatives now exist in 39 states, and 27 states plus Washington DC have at least one full-time online school operating statewide.) But mostly online learning occurs side-by-side with face-to-face learning. (The report estimates that about 50% of all districts are operating or planning online and blended learning programs.)

A report profiling 40 schools that have blended-learning programs has found six basic models of blending learning:

• Face-to-face-driver: face-to-face teachers deliver most of the curricula. The physical teacher deploys online learning on a case-by-case basis to supplement or remediate, often in the back of the classroom or in a technology lab.
• Rotation: within a given course, students rotate on a fixed schedule between learning online in a one-to-one, self-paced environment and sitting in a classroom with a traditional face-to-face teacher. The face-to-face teacher usually oversees the online work
• Flex: uses an online platform to deliver most of the curricula. Teachers provide on-site support on a flexible and adaptive as-needed basis through in-person tutoring sessions and small group sessions.
• Online-lab: relies on an online platform to deliver the entire course but in a lab environment. Usually these programs provide online teachers. Paraprofessionals supervise, but offer little content expertise.
• Self-blend: encompassing any time students choose to take one or more courses online to supplement their traditional school’s catalog. The online learning is always remote.

As we can see (and as was also discussed in the Keeping Pace report), online learning is not about making the teacher redundant! No surprise when you consider that a major aspect of online learning (and its attraction for many students) is that it personalizes learning.

This is also echoed at university level. A spokesman for the Pearson Foundation, discussing a survey of over 1,200 college students, said: "There seems to be this belief among students that tablets are going to fundamentally change the way they learn and the way they access what they are learning. Students see these devices as a way to personalize learning." Students don't see tablets as means of accessing digital textbooks as much as a means to access e-mail, manage assignments and schedules, and read non-textbook materials such as study aids, reports, and articles. (You might also like to read about one university's experience introducing iPad's into the classroom)).

In the same way, a study involving students in China and Hong Kong found that Facebook was being used to let them connect with faculty and other students, provide comments to peers/share knowledge, share feelings with peers, join Groups established for subjects, share course schedules and project management calendars, and (via educational applications) organize learning activities.

So one aspect of online learning is management and collaboration.

Of course online learning is not only about personalizing learning. It's also about broadening access to quality educational resources. The open course movement is perhaps more advanced at university level (exemplified by MIT's OpenCourseWare (updated: now Open Education Global), Yale's Open Courses, the Open University's Learning Space), but in Iowa, schools will soon have access to wide variety of digital materials from a central repository using Pearson Education's Equella. Certainly the internet is rife with educational materials aimed at K-12, but there are great benefits from the more formal structure of such a repository.

But this wonderful cornucopia is also the biggest problem. So many resources. And so many structures, programs, digital tools. It takes a lot of time and effort to master each one, and who wants to put in that effort unless they’re sure it’s really important and going to last?

There's no good answer to that, I'm afraid. We are living in a time of transition, and this is the price of that.

But we can try and develop our own 'rules of engagement'. Something to filter out the deluge of new tools and new systems and new resources.

When doing so, we need to consider the two principal, and different, issues involved in this revolution in information delivery systems, which should be kept quite distinct when thinking about them (however muddled together they will be in application). One concerns their use in learning — do textbooks need 'bells and whistles' to be more effective means of learning? what is the best way to frame this information so that students (at their grade level) can understand and remember it? This is the how question. For this we need to work out the different strategies that each delivery system needs to be an effective learning tool, and the different contexts in which each one is effective.

The other issue concerns the world for which the education system is supposedly training students. How is information delivered today? How do people work with information? This is the what question; the issue of content — though not in the 'core knowledge' sense.

Although, part of the issue does concern this question of core content. Because the fact is, however we may pine for the days when we all knew the same things, read the same books, could recite the same poems (no, we never really had those days; we just had smaller groups), there is too much information in the world for that to be possible. And society needs the diversity of many people knowing different things, because there's too much for us all to know the same thing. So what we need from our education system — and I know it's a truism but there you go, doesn't make it less true — is for our students to learn how to learn. Which means they need to know the best strategies for learning from the various information delivery systems they're going to be trying to learn from.

And there's something else, that stems from this point that there's too much for us all to know the same thing. We have this emphasis on doing well as an individual — individuals graduate, become famous, get Nobel Prizes, get remembered in the history books. But science and scholarship, and politics and community development, have always benefited from the stimulation of different minds. The complexity of the world today means that we need that more than ever. The complexity of science today means that most discoveries are the results of a team rather than a single person. Even in mathematics, the archetypal home of the solitary genius.

For example, the Polymath Project began with one mathematical genius who decided to take one of the complex mathematical problems he had struggled to solve to his blog. He threw it out there. And readers threw ideas back. Since then, several papers have been published in journals under the collective name DHJ Polymath.

An example of the open science movement (see the Open Knowledge Foundation and the Open Science Summit), raising the question — is the ‘traditional’ way of doing science really the best way? Let’s bear in mind that the ‘traditional’ way is not in fact all that traditional. It’s a product of its times (and rather recent times at that). We shouldn’t confuse the process of scientific thinking with the institutionalization of science. Proponents of Open Science argue that the advent of the internet can break right through the inertia of the institutions, can allow collaboration and the processing of huge data-sets in ways that are far quicker and more efficient.

This is the world we need to educate for. Educate ourselves and our children. And the heart of it is collaboration. Which is one of the reasons we shouldn't be keeping social media out of the classroom. We just have to use it in the right way.

I began this series with Denmark allowing internet access during exams. So let's finish by returning to this issue.

As with the wider question of education, we need to ask ourselves what testing is for. First of all there's the point that, like note-taking, testing has an obvious purpose and a less obvious one. The obvious one is that it provides a measurement of how well a student knows something (we’ll get to the squirrelly ‘knows’ in a minute); the less obvious is that testing helps students learn. (For note-taking, the obvious purpose is that it provides a record; the less obvious is the same as for testing: it helps you learn.) Many tests may be (or perhaps should be) primarily for learning.

Final exams, on the other hand, are usually solely about assessment. But then we must ask, assessment of what? What do we mean by 'know'? There are topics within subjects which are 'core' — crucial details and understandings without which the subject cannot be understood — cell division in biology; atomic structure in chemistry. But there are many other details that you don't need to have in your head — but you do need to 'know' them enough so that you can find them readily, and fit them into their place readily.

Anyone who can write well and develop an argument in depth on a specialist topic in a three-hour exam period from the internet deserves to pass (I'm assuming, of course, that there are adequate guards against plagiarism!). As with course-work, access to the internet simply raises the standard.

 

These posts have all been rather a grab-bag. This is such a wide topic, with so many issues and everything is such a state of flux. To write coherently on this would require a book. Here I have simply tried to raise some issues, and point to a random diversity of articles and tools that might be of interest. Do add any others (issues, articles, tools) in the comments.