Thinking Outside the Box: Higher-Order Thinking Skills

"For, you see, so many out-of-the-way things had happened lately, that Alice had begun to think that very few things indeed were really impossible."

- Lewis Carrol, Alice in Wonderland

---

I have been thinking about higher-order thinking skills a lot lately.  From trying to make my online courses more engaging to talking to faculty about flipped learning to taking an Adobe course where we explore ways to increase creativity in students, higher-order thinking has been, well, on my mind.

So, what are higher-order thinking skills? I'm glad you asked...

We have all heard of Bloom's taxonomy, a way of classifying learning objectives that has been around for decades. It consisted of six categories named for the type of learning outcome they described: knowledge, comprehension, application, analysis, synthesis, and evaluation.

The taxonomy was modified more recently to use verbs as category names (since learning objectives use verbs to describe observable, measurable behavior) and to modify two categories. Instead of describing types of learning outcomes, the new descriptors defined how learners used and interacted with knowledge: remember, understand, apply, analyze, evaluate, and create.

There are lots of lists out there with verbs you can use to write objectives at each level, as well as directions for how to write learning objectives based on Bloom's taxonomy, but that's not what I want to talk about here.

Bloom's taxonomy is normally described as a hierarchy, with one level leading to the next. The higher-order thinking skills (HOTS) are the ones on the top of the hierarchy, with lower-order thinking skills (LOTS) at the bottom.

It is usually presented in a pyramid diagram like the one to the right. I tend not to like that one, for two reasons.  First, making "Remember" the largest area makes it seem like it is the most important, or that you should have more objectives of that level than others. Second, it puts the levels in a hierarchy, implying you need to master objectives at one level before you can move on to the next.

(Not to mention my OCD gets itchy when you can't fit the labels for the top two levels inside the pyramid...)

Some people present the pyramid flipped, with the point at the bottom. This emphasizes the importance of the upper levels, but it still maintains the hierarchical nature of all the levels.  In my opinion, all of the layers are not hierarchical, so this scheme still is not ideal.

In this scheme, the higher-order thinking skills are Analyze, Evaluate, and Create, while the lower-order ones are Remember, Understand, and Apply.  The lower-order skills are somewhat hierarchical; generally, in order to apply knowledge you need to understand it and to understand it you need to remember it.  However, trying to apply knowledge often leads to greater understanding, and the more you use knowledge, the likelier you are to remember it, so that hierarchy is not perfect.

(And then there's that whole OCD with the labels outside the pyramid boundaries...)

The idea of a hierarchy falls apart at the higher-order levels.  Analyzing involves breaking something down to its component parts, identifying or forming relationships among those parts, and separating facts from inferences or opinions. Evaluation is the process of forming judgments or placing value on ideas or products. Finally, creating is the process of making something new, forming new and original relationships between elements to produce something where the whole is greater than the sum of the parts.  Each one of those higher-order skills are separate, even if they are related. As you create something new, you need to evaluate its effectiveness.  Part of that creation process involves analyzing the relationship among the elements.

I drew this image to describe my thoughts on Bloom's taxonomy. I like this approach because it shows the relative importance of higher-order learning objectives along with the interconnectedness of the lower-order objectives. It represents the lower-order objectives support of the higher-order ones, but does not imply a hierarchy within those orders.

By the way, I used higher-order thinking to draw this picture. I created it, of course, but I also had to analyze what I considered to be the relationships among the categories, and evaluate how well my diagram demonstrated those relationships.  I also analyzed what the traditional diagrams showed and evaluated them to determine what I thought they didn't emphasize.  All of that is built on remembering and understanding Bloom's taxonomy in the first place, and about 35 years of applying it to writing learning objectives.

Why is this important? A number of research studies show the increased importance of creativity as a leadership quality in business. IBM surveyed 1500 CEOs from 60 countries, and creativity emerged as the most crucial factor for success. Adobe surveyed students and teachers globally about the importance of creativity in education.  Both felt creativity will be essential in the future workforce; 94% of teachers believe that GenZ students will have careers that do not yet exist.

Problem-solving and critical thinking are often cited as some of the top skills employers want.  These skills require all of the higher-order thinking skills, analyzing a problem's complexity, identifying relationships among data, creating possible solutions, and evaluating their effectiveness. However, the majority of employers say students lack the problem solving skills needed to succeed in the workplace.

Clearly, higher-order thinking skills are important Flipping the classroom, authentic assessment, problem- and project- based learning are all ways we can encourage the development of higher-order thinking in students. Unfortunately, the traditional method of direct lecture instruction and multiple choice assessment, does not encourage this way of thinking.  It may have been the way most of our college courses were taught, but if we were honest with ourselves, we might realize those courses could have been taught more effectively.

Over the next several weeks, I will talk about specific ways to encourage higher-order thinking skills in the traditional and online classroom.  From things I have tried, like moving to all essay exams and gamifying  deadlines to things I have helped others try like flipping the classroom and learning by doing, all of these techniques require faculty to do certain things.

1. Embrace failure. Be willing to fail and to allow your students to fail. Not everything you do will work, and students will not always be successful.  That doesn't mean you should give up any more than they should. If we never tried anything new for the fear of failing, we never would learn how to walk.
2. Rethink assessment. The ability to assess one's own learning is a higher-order thinking skill, so students need to be taught it.  However, we tend to view assessment as a means to a grade.  Giving students low-stakes ways to assess their learning is key, but it is a balancing act. You need to give them enough incentive to complete the assessment (like a grade) but still make sure that failure doesn't discourage them.
3. Reward the good. We tend to punish bad behavior instead of rewarding good behavior. Instead of assessing a late penalty or not accepting late work, try offering incentives to motivate students to do work on time.  If there is work you want students to complete before class, give them additional time or attempts; students who choose not to do the work on time don't get a penalty for turning work in late (only if it does not cause you more work).
4. Encourage creativity. Why do we insist students write answers to exams? While good written communication skills are important, employers are becoming more interested in verbal and visual communication skills as well.  By giving students options in how to communicate what they have learned, you are making the learning more relevant to them.
5. Be uncomfortable. This image popped up in my twitter feed last year. I think this may be something college faculty, in particular, find challenging, being put in the position of not being the expert in something. In order to learn or try something new, you need to acknowledge that the way you have always done it may not have been the most effective. That makes people uncomfortable. But that's not a bad thing...

So keep thinking about higher-order thinking skills, and use those higher-order thinking skills to try new ways to encourage them in your students.

Mixing It Up with Interactive Online Lectures

“Alice had begun to think that very few things indeed were really impossible."

- Lewis Carrol, Alice in Wonderland

---

If there was one tool I could convince people to try it would be Office Mix.  Whether you are flipping a classroom, teaching online, or making up a snow day, Mix allows you to "mix" multiple sources of information into a PowerPoint deck, narrate your presentation, and include questions students need to answer - and you can incorporate it into Blackboard Learn as a graded assignment.

I used Mix to create learning activities that incorporate videos from NASA and ESA that I remixed using Adobe Premiere and Audition, OER or public domain images and graphics, and illustrations and animations I created myself using Adobe Illustrator and Animator.  I included multiple choice, multiple answer, or true/false questions for students to answer as they worked through the activity. I did not use a traditional textbook; I provided access to a free, online text from OpenStax for students to use as a reference for anything about  which they needed more information.



Feedback from students was quite positive.  Roughly 2/3 of the way through the course, I asked for feedback.  Comments related to the learning activities include:

• "I actually liked this course. It took a different approach than the run of the mill, read a book, take a test. Doing that I do not retain any information."
• "I really enjoy the Learning activities. This is where I feel that I learn the most. I believe that after this semester concludes, I will remember a lot of what I learned from this course."
• "I really enjoy the learning activity lectures at the beginning of the chapter. They are personal, interactive, and informative."
• "This is one of the first online classes that I've taken and I honestly wish they were all like this class. I think the learning activities are really helpful and make it feel like I'm actually in a classroom learning that lesson for the week."
• "I also like the learning activities and they give a classroom feel...Unlike classrooms, I enjoy being able to rewind and listen again, if necessary."
• "So far, I will say this has been the most interactive course I've taken to date.  I like the videos that are made weekly to cover the important topics and bullet points, and that it's actually the teachers voice."
• "The questions spread throughout the presentation help to give me an idea if I understand what I’ve heard so far, and it’s particularly nice that instead of being penalized for getting them wrong, I can instead go back through the section and try again."

Tips for a Good Mix

I wrote earlier in the Teach Me Tuesday blog about how to avoid death by PowerPoint, and all of those tips apply to creating a good Mix.

Write a Script

I strongly recommend writing a narration in the notes area of your PowerPoint slide before you begin your narration.  You can print the notes view to a PDF to share with your students, for those who may prefer to refer back to written material rather than audio.  It also helps you to keep your narration focused and short. The first semester I used Mix, I didn't narrate with a script.  The second semester, I redid the narrations - using the same slide content - and following a script cut the length of the Mix down by about 30% on average.

Keep It Short

Normally, I say keep instructional videos under ten minutes, if not shorter.  There is some research showing that engagement in video actually falls off after about 2 minutes, although that seems to be longer for instructional video. However, the ability to incorporate questions for assessment into your Mix extends that optimal time.  You can combine segments of 2-6 minutes of narration and video with questions in between to extend that engagement time.

After using Mix for a year and looking at the analytics including average time spent by students viewing a Mix, I suggest that a completed Mix should be no more than about a half hour in length, including all of questions for students to answer, with the optimal length around 15-20 minutes I use three learning activities Mixes each week for the Fall and Spring semesters, and 3-5 per week for the Full Summer semester.  Since students also complete a lab, an social activity, and an interactive homework assignment, that's the equivalent of the amount of time students would spend in class for a 3-credit course.

Be Creative

Don't be afraid to be creative.  No Mix is set in stone, so if something doesn't work the way you think it should have, change it for the next semester. Play around with graphics and animation on your PowerPoint slide to make them visually interesting and to support your content.  Look for OER (Open Education Resources) you can incorporate. Do Google image searches and look for graphics you can use with or without attribution. Think about what YOU like to see in videos on YouTube or Facebook, and try to recreate those looks.

And give me a call if you need some help.  I spent a year learning Adobe products, and while I am by no means an expert, I am getting pretty good at editing videos and drawing illustrations, and I am really good at making the most of PowerPoint graphics and animations.  After making and revising over 40 Mixes over the past year, if I haven't done it already, if it can be done, I can probably help you figure out how to make it work.

Start Slow but Persevere

While Mix is easy to learn, it is time consuming. I wouldn't recommend that anyone do what I did, which was to convert an entire course in one semester, while running the course at the same time. 20:1 is an optimistic development ratio - a 30 minute Mix can take 10 hours to create (another reason to keep them short), so instead of doing all of your instruction in Mixes the first time out, try supplementing what you already do with one Mix per week or topic that covers those concepts students struggle with most. Then add in some more that following semester, and before you know it, you will have a completely interactive course.

To get started with Office Mix, read my post in the Teach Me Tuesday Blog.

On Creativity...or...My Year of Learning Dangerously

"Why, sometimes I've believed as many as six impossible things before breakfast."

- Lewis Carroll, Alice's Adventures in Wonderland

---

As I mentioned previously, for a little over a year, I have been taking free online courses from Adobe to learn how to use the Adobe Creative Cloud suite of products.  In those courses, we talk a lot about creativity, what it is, how to assess it, how to encourage it in students.  After 12 courses, comprising almost 300 hours of professional development time, I have come to one conclusion.

Creativity is hard work.

When I talk to colleagues about something I have done for my online courses, I often get a reaction something like this:

"You are so creative! I could never do that."

This assumes creativity is some innate talent that cannot be controlled.  It tends to ignore the effort that goes into innovation. Coming up with a new idea is easy. Making that idea work takes a lot of learning, planning, and trial and error. If you view creativity as something you don't possess, you have no reason to try. You have to believe you have the ability to do it before you try something new.

I already told the story of how I used social media to learn about Pluto and how that experience led me to wanting to ditch the textbook in my online courses. I took a number of MOOCs, and realized I need to make my online courses more visual and interactive. Unfortunately, although I had all sorts of ideas of what I wanted to do, I lacked the technical skills to get them done.

Thus started my Year of Learning Dangerously.

Somehow I stumbled across Adobe's Education Exchange, And there I found heaven in the form of Adobe Generation Professional courses. These are free, online courses, designed specifically to learn how to use Adobe products in education. As an LCCC employee, I could license the entire suite for less than $10 a year..an incredible price! As I said, over the past year, I have completed almost 300 hours of formal coursework.  I spent dozens more learning products not (yet) covered in those courses.  That's about an hour every day for a year. That has nothing to do with creativity and everything to do with commitment.

Taking those courses and learning how to create my own visual learning materials was just part of the equation.  I used Office Mix, an add-on for PowerPoint to create image-based interactive learning activities, that combine the materials I created using what I learned in the #AdobeGenPro courses along with questions that students had to answer while they watched the lecture.

That particular learning activity combines graphics I created using Illustrator:

With an animation I did in Animate:

And a video I "remixed" using public domain material from NASA, recording the narration in Audition and editing the video in Premiere Pro:

In addition to those resources, in a mix I can also include links to web pages, do direct screen capture videos or images, or include simple audio files.

(I like using this example, because it shows the three tools I use most often {Illustrator, Premiere, and Audition} along with one I use infrequently {Animate}, in a learning activity I created from start to finish in one day.)

I also use Spark video - an application I learned in the very first course I took at the start of my Year of Learning Dangerously - to create weekly introductory videos:

The response from students has been very positive. When I asked for feedback, one student said even though I wasn't in the videos, hearing my voice made it seem like he was in a classroom.  Another said she appreciated being able to move back and forth through the lecture to review something she missed. They have their glitches, but since I warn students in advance about some of the problems they might face and how to fix them, they are OK with that.

The advantage to me is I feel like I am in control of my course again, not the publisher, that I am actually teaching, not just grading.  When the Juno probe reached Jupiter, I could update that content in my course right away; I didn't need to wait for a publisher's edition update cycle. When NASA held a press conference about finding several Earth-size planets around another star, I could update the virtual labs on stars and on  exoplanets to include questions about that press conference. I can make the course relevant, up-to-date, and interesting.

That just takes hard work, not "creativity." My astronomy course has 42 learning activities in it, each about 15-30 minutes long.  Each one of those takes about 8 hours to produce, depending on how much I am creating from scratch versus repurposing OER (more on that in another blog). And while I was completing those 300 hours of #AGP courses, I created about two learning activities every week. That's not creative.  That's just insane.

But, I made the commitment to do it, set a schedule, and stuck to it...and my course is the better for it, and my students learn more. Now, instead of telling students that to comment on another students blog, they need to:

1. Click on the In the News Blog link on the Course Menu.
2. Select the Blog on which you want to comment.
3. Locate you name to the right.
4. Click the down arrow under your name to bring up a list of other students.
5. Click on the name of a student to read his or her blog.
6. Click the Comment button.
7. Enter you comment in the Comment box.
8. Click the Add button to submit your comment.
9. Click the arrow under the other student's name on the right to bring up another blog on which to comment

I can show them how to do it:

I realize not everyone wants to learn how to create an avatar in Character Animator or a title sequence in After Effects.  That's the "Oooh, now that's cool!" kind of stuff I like spending time figuring out how to do.

But I don't know any instructor who isn't capable of learning how to use Illustrator or Photoshop to create or modify OER resources for use in their course or to create a screencast and use Premiere to edit it and add simple titles or to make a simple introductory video with Adobe Voice or narrate an interactive lecture with Office Mix.  You just have to be willing to learn something new and put in a little hard work.

Thomas Edison famously said, "Genius is one percent inspiration, ninety nine percent perspiration." I'm committing to trying to provide the inspiration through this blog.  Once a week I will post a new blog, either on an instructional design topic here, an instructional technology topic on Teach Me Tuesday, or a how-to tip on Blackboard Learn Tips and Tricks...if I make the time and I have the inspriation, I'll post more than one.  Let's call it a Year of Blogging Dangerously.

Will you commit to a little perspiration and follow this blog for a year? You might find something to inspire the creativity in you!

My #AdobeGenPro Journey

"and oh ! ever so many lessons to learn ! "

- Lewis Carroll, Alice's Adventures in Wonderland

---

This popped up in my Facebook feed this morning:

One year ago today I submitted the first of what would end up being many assignments for lessons I learned from taking Adobe Generation Professional courses.

At that time, I couldn't draw a straight line with a ruler, the only layers I knew about were in triple chocolate cake, and every time I opened Photoshop I would run screaming for the hills.

Now, I can make a straight line walk across the screen, I can manipulate layers to add dimensionality to illustrations, and I can make a never ending glass of beer (which, I admit, is still what I want after using Photoshop). Oh, yea, and the pen tool is my bitch.

The AdobeGenPro courses are very well designed.  They don't try to teach you everything about the software.  They set up a progression of assignments that let you build on skills, so by the end of the course, you can create a useful product. The instructional videos are great; the ones done by Joseph Labrecque for the Animation course were some of best I have ever seen. But most importantly, they teach you how to learn what else you need to know about the software on your own.

I won't say they are without their hiccups.  There are occasions when I am in a live class thinking WTF are you talking about.  My favorite was when the lecturer was talking about the direct selection tool vs. the selection tool.  I said in chat I had no idea what he was talking about, and a fellow student told me it was the white arrow vs. the black arrow.  As my frustration rose, I said I didn't know what SHE was talking about, since I had never opened the software before.  She patiently explained to me where the black arrow and white arrow were, and what they did.  And now I can use them to draw my own HR diagrams to illustrate the Main Sequence turn off point for star clusters.

Why did I take those courses? Because I wanted to redesign my online astronomy course to be more interactive, more visual, and more effective, to bring it into the 21st century.  I wanted to teach my online students, I didn't want them to teach themselves.  I knew I needed new skills to do that, so I started my MOOC journey which eventually led me to the Adobe Education Exchange site and the Adobe Generation Professional courses.

Now, when I describe the frost line and why different materials in the protoplanetary disk coalesce at different distances from the protosun, I can create an diagram in Illustrator to go along with it.

When I try to explain conservation of angular momentum in that protoplanetary disk and how it leads to all of the planets orbiting the Sun in the same direction, I can create a video in Animate to demonstrate that - and I can record the soundtrack to go along with it in Audition.

Using Premiere Pro, I can even edit videos and visualizations created by NASA that are in the public domain. I have spliced videos together, changes timings, and recorded narrations (again, in Audition) so they meet the instructional needs of my students.

And, of course, I use my old friend, the first Adobe product I learned, Spark Video, to create introductory videos for units in my online courses.

There is still a lot I want to do, primarily using Animate to do things like show the proton-proton chain and CNO cycle nuclear fusion reactions or the path of stars on the HR diagram as they evolve, but I haven't made the time to do those yet.  But, that's the beauty of teaching - you are never done.  This semester I'm improving on the work I did last semester, and next semester I will do things better than I did this semester.

The only thing holding me back is making the time, because there are only so many hours in the day. That's one reason I haven't blogged (at least on this blog) for over a year, even though there are things I want to blog about.  I have had the same sweater on the needles for over a year now.  And if only I could teach my cats how to do housework...but hey, how often do you really need to dust or vacuum...

I know many of you who read this far are saying, "Well, sure, you can do it, Karen, you know how to use technology." I know that, because many of you have said that to my face when I show you some of the things I created.

Frankly, it's a little insulting. I'm just not that special. I don't have a magic "knows everything about every software product ever made" gene.  What I do have is a willingness to try something new and the determination not to give up at the first sign of trouble. If I did, I would have given up halfway through that first course, when I ran into problems with layers in Adobe Mix and Fix.

What I am is willing to put in a lot of time and effort to make this change. Anyone can take the same courses I took and create the tools your students need to learn, if you are willing to put in the same time and effort, because I won't sugarcoat it.  Creativity is hard work.  I just think the pay off is worth it. There is also something very satisfying when you can update your skills on your AdobeEdEx profile.

Because now, a year later, I have the skills and experience to  be able to, with some level of confidence, open up After Effects for the first time and create a simple opening title sequence for videos I am working on for our Distance Education web site.

And I learned all of that without ever opening a textbook...but that's a story for another post.

So please, look into the AdobeGenPro courses.  The next one up is digital video starting on May 1 and running through the middle of June.  Nothing bad happens to you if you don't finish the course (*cough*UX to UI*cough*)(*cough*twice*cough*).  But it you are willing to take a chance and put in some hard work - the same thing you ask your students to do - something good might happen.

Flipping Out: Everything Old Is New Again

"'But then,' thought Alice, 'shall I NEVER get any older than I am now? That'll be a
comfort, one way—never to be an old woman—but then—always to have lessons to learn!
Oh, I shouldn't like THAT!'"

- Lewis Carroll, Alice's Adventures in Wonderland

---

Clearly Alice wasn't a life-long learner.

A faculty member approached me a while ago about linking materials from a third-party provider to Blackboard because she wanted to flip her classroom.  Easy enough to do, but I suggested we talk about some of the challenges of flipping so she could make the most out of the experience.  Based on what I saw of the online materials, they were a bunch of videos already available on YouTube (and of varying quality) along with some electronic exercises. And an e-book.  Somewhere in there, there were links to an e-book.

I wanted to talk to her about how she would guide her students use of the videos.  How would she ensure that her students learned what they needed to learn for the discussions and activities she was using in class? Would she provide them with some guiding questions? Objectives that should be met? Would she provide electronic quizzes so they could test their knowledge and then review what they didn't understand? I tried to explain that to her, but I'm not sure my point sunk in.  She was so excited about flipping her classroom, that I don't think she grasped the preparation needed to make the best use of those technology resources.

Flipping a classroom in its purest form is moving the initial levels of Bloom's taxonomy out of the classroom to make more time for learning at the higher levels in the classroom.  The traditional approach is to focus on the understand, remember, and apply levels in the classroom through lecture and discussion, while addressing the higher order levels outside the classroom through homework, longer term projects, research papers, for example. The idea is it is relatively easy for students to assess their understanding of those initial levels, and to review and relearn what they don't understand.  However, it is harder for them to assess their understanding at the higher levels - but you, as a teacher, are often not available to aid them in that assessment when they are trying to learn those skills.

Side note: As I was discussing the flipped paradigm with someone, he started lecturing me about Bloom's taxonomy.  I had to restrain myself from pointing out that I first learned about Bloom's taxonomy before he was born and from showing off by asking how this relates to the psychomotor and affective domains of the taxonomy. Kids these days.  Think everything is new just because they just learned about it... ;-)

Flipping a classroom is more than just making lecture videos available to students - although that is what comes the mind of most people when you talk about classroom flipping.  As I mentioned in a prior blog, I am a big fan of Gagne's instructional events. I think they are as important in a flipped environment as the traditional one, if not more so. Whether you are in the classroom or online, you still need to tell them what they're going to learn, relate it to what they know, guide them through the learning process, see if they learned it and tell them where they went wrong if they didn't, and then make sure they can apply what they learned outside of the classroom.  Gagne may have first introduced that idea 50 years ago, but it's still valid.

That's the point I tried to make to the above mentioned faculty member.  At the same time, I become involved in a grant where the grant committee decided to employ the flipped classroom to provide technology-enhanced instruction to a certain set of curricula.  Now, I have my issues with that approach, particularly since it was chosen before deciding which course to flip and it was the only technology enhancement to be used, but that wasn't my decision. However, the way we flipped the course was.

The committee seemed to be focused on lecture capture.  Unfortunately, we don't really have the resources to support that college-wide, and part of the purpose of the grant was to transform the institution - that meant anything we did in the curricula covered by the grant needed to be easily applied to others. I was determined, therefore, to use readily available materials already available on the web, and not create a lot of our own.

Here's what I had to work with. The students would access the flipped materials through Blackboard Learn.  The course was an engineering graphics course that used AutoCAD; the textbook included a DVD with tutorials on it.  AutoDESK (the makers of AutoCAD) provide a lot on resources; there are a number of videos on YouTube created by other AutoCAD users and teachers. There really wasn't any point in creating any videos of our own, since the book provided some and others were already available online.

Here's how I applied (most of) Gagne's instructional events.

Inform learners of objectives
Stimulate recall of prior learning

I created folders in Blackboard, one for each major drawing project the student had to complete.  In each folder, I started with a list of "objectives." These are not traditional performance objectives I would write for outcomes assessment; these are just designed to give students an idea of what they should know before coming to class.  For example, here is the list for orthographic projection:

To complete this project, you need to know the following:

• What is orthographic projection?
• What are the different types of lines used in orthographic projection and what do they mean?
• What are the different views used?

In AutoCAD, you should be able to do the following:

• create Layers to control properties like line type, pen width, and color.
• use Blocks to create a title block on your drawing.
• switch between model space and paper space.

Later modules include references back to prior modules; for example, the module on tolerancing references back to "objectives" from the module on dimensioning.

Present the content
Provide “learning guidance”

Then I provided them a "work guide," a PDF document they would print out and use to take notes as they read the book, watched the videos, worked through the tutorials.  This would be turned in at the beginning of class as "proof" that they did the online work.  Here is the guide for orthographic projection.

The final two folders were for AutoDESK resources and for YouTube videos (or other online videos).  It was important for them to get used to using the AutoDESK resources to learn, because AutoCAD updates every year, and they will need to be able to teach themselves when new versions are released.  I also think it's important for them to be able to see examples of effective tutorial videos.  One thing we might do in the future is provide a place in Blackboard (a discussion board or a wiki...maybe a blog) where they can share resources they found useful.  That will let us update the course in the future, and help them learn about to find good resources on their own.

Elicit performance (practice)
Provide feedback

Finally, they had a mastery quiz.  The quiz tested at the remembering and understanding levels, with maybe a couple of application questions.  It consists of all objective questions, so students get immediate feedback.  They can take it as many times as they want with no time limit, but they are advice to repeat it until they reached mastery of 80%.

As of the time I'm writing this, they have completed the first unit.  88% of the active students in the course completed the mastery quiz, and all of them attained an 80% or higher.  Many of them repeated the quiz until the received a perfect score. Note that they don't get a grade for doing any of that work; their only incentive is to be prepared to work on their projects in class.

Assess performance
Enhance retention and transfer to the job

This is what takes place in the classroom.  Instead of doing their AutoCAD projects at home, they now do them in the classroom where the teacher is available to assess how they are doing and to correct any mistakes as they try to make AutoCAD do what they want it to do.  He no longer needs to spend time explaining how layers work, for example, and can instead just tell them to go to their dimensions layer. That gives them more time in the classroom actuallty USING AutoCAD instead of talking about using AutoCAD, which, of course, leads to more effective learning.

So far, so good.  For the first two modules, 94% of enrolled students completed the mastery quiz, with 94% of them achieving mastery on the first module and 88% on the second.  Of the ones who did not achieve mastery (set at 80%), all but one was within 5% of mastery.  We don't have long-term data yet on how the flipped materials affect performance on higher order learning activities, but it's off to a good start.

The key to our (albeit limited, so far) sucess was not the use of videos.  It was the use of guided, active learning activities combined with videos and student self-assessment with immediate feedback.  Just throwing some YouTubes together does not a flipped classroom make.

Cell phones in the Classroom: Tool or Toy

"You are not attending!" said the Mouse to Alice severely. "What are you thinking of?"

-Lewis Carroll, Alice's Adventures in Wonderland

---

I have this bad habit of sitting in my car in the parking lot after I get to work to enjoy a little fresh air and sunshine before I go into my windowless office for several hours.  I use the time to check email, Facebook, Twitter, etc. on my smartphone. The other day, I came across this article on Inside Higher Ed's Facebook feed.  This was about Beloit College's annual Freshman Mindset List for freshmen entering college this year for the class of 2019. I found this comment about the proliferation of cell phones very timely.

I've been thinking a lot about student engagement lately, and how we can foster an interest in lifelong learning in our students.  If you read my blog post on learning about Pluto from Twitter, you will know that I learn a lot from my phone.  Whether it's tweets or Facebook posts or Instagram photos, there is so much fascinating information right at my fingertips.

I know a lot of people view cell phones as distractions.  I view them as a learning gold mine! If I want to know something, I can look it up right away on my smartphone.  When I was in college, if you wanted to look something up, it meant going to this big building called a library, flipping through cards in the aptly named card catalog or paging through the Reader's Guide to Periodical Literature, and then venturing in the dimly lit (and odd smelling) stacks and hoping the book or journal you wanted was there - and intact, since someone could have just ripped out the article you wanted.

Just to show you the apple doesn't fall far from the tree, I gave my old iPad to my mother, who lives in a retirement community.  She takes it with her to meals, carrying it in the basket of her walker.  Whenever there is an argument among the old widow ladies, she pulls out her iPad, looks up the question, and settles the argument.

That got me thinking about how we can encourage students to use their phones as tools rather than toys.  In researching this topic, I came across an interesting article about what affects encoding into long term memory more, engagement or attention.  Not unsurprisingly, it's engagement.  Students can be paying attention to everything you say in a classroom, but if they don't have some level of interest in the topic, if they aren't curious about it, if they aren't motivated in some way to interact with the material, whatever they attend to will just go right out of short term memory and never be encoded.

Which brings me to the idea of using cell phones to foster student engagement.  Now, of course, when a student is looking at their crotch and smiling, chances are they aren't googling Maslow's hierarchy of needs.  But what if instead of making them hide their cell phones away, we encourage them to use them - just in a way that's more beneficial?

According to this infographic (if anyone can find the original source of this data, please leave it in the comments), 77% of college student smartphone users check their phone first thing in the morning.  92% use it during idle times at work or at school.  Two-thirds use it to read news articles, while four-fifths uses it for school related tasks.

And half of them use it when they are in the bathroom.

That sounds like a great opportunity use their cell phones to communicate with them and to encourage them to communicate with each other outside of the classroom (instead of catching them when you can, you can catch them IN the can...). You don't have to give out your personal phone number.  There is a free app and a website called Remind that allows you to contact students via text messages, either through the Remind.com web site or using the app on your phone. You set up a class on Remind (one Remind class of each section you teach), and give the class code to students.  They sign up by texting the class code to specific number. You can set up text messages on the web site in advance to be sent out on the date you select, you can text the entire class either from the web site or the app, or you can text an individual student.  They can text you back, and you can respond to them - again, all without giving out your personal cell phone number.

You could use the scheduling feature to send out reminders for assignment due dates or when tests are due.  You could use the individual feature to text a student who hasn't been to class for a couple of days or who is falling behind in their work. We all know how hard it is to get students to read their student email...but I bet they check their phone when that text message chimes.

But what about in class?  How do we get them to think of it as a tool in class, not a toy they have to put away? Do you remember the old clicker systems from 4 or 5 years ago?  You could embed questions in a PowerPoint that students would answer using these little handheld devices.  The clickers of today are cell phones.  You set up your poll questions on a web site, and as you present them (either on the web or in PowerPoint), students use there cell phones to respond, again, either by text or on a web site.

The one I've recommended is PollEverywhere. With PollEverywhere, you create poll questions on a web site or an app. They can be multiple choice, open-ended, or a clickable image. You can also use it to allow students to ask questions anonymously or to brainstorm ideas for problem solving. With a free account, you can create several polls but each poll can only accept up to 40 responses, and you can't track responses or participants.  You can upgrade to a paid account if you want more information about your users or large number of responses.

Another app that might be of interest is LiveBinders. Again, this is available as a website and as an app.  Livebinders is an aggregation tool that allows you store an organize a variety of resources from websites to videos to other files you upload. You can create a binder of organized information sources for students to use, or they could create their own to share with the class.  Again, they can access the binders you share with them via the web or via the app, but either way, they can use their smart phone.

I created a LiveBinder for cell phone use in the classroom. You can view it without signing up for LiveBinders.  You will find a binder with two tabs, one containing research on student cell phone use and one containing links to tools and other information about how to use the cell phone in the classroom.  The nice thing about a binder like that is I can update it as needed, and you can check back periodically to see if I've added anything. If you create your own account, you can add my binder to your shelf!

Students can create their own LiveBinder accounts and then collaborate on their own binders.  If you have a group research project, you can show them a sample binder, get them started on their own, and then they can create their own binders to share with you - all from their phone, if they want.

Another learning tool is Quizlet. Again, Quizlet is available as a website and as an app.  Remember the old flashcards we used to make when we were in school? Quizlet is the 21st century version of those.  You can create a list of questions for your students to test themselves on, and then they log into their Quizlet app on their phone to test themselves on their questions. Students can also make their own flashcards from which to study.

Finally, students can read their textbooks on their phones!  Coursesmart is an ebook provider from which students can buy or rent textbooks.  Faculty can get free access to books; I don't use paper books any more.  I can access all my textbooks and preview new ones either on the Coursesmart web site, on my iPad, or on my phone. the advantage of an ebook over a paper one is searching; I can search the ebook for a specific term and then tell students to look on a specific page for more information to their questions.  Can't do that with a paper book...

That is just the briefest taste of ways you can use cell phones as a learning tool, both in and out of the classroom. Don't forget to check back to my LiveBinder to see what else I find in the future!

Technique vs.Technology: Long Live the (Well-Designed) Lecture!

"Off with their heads!"

- Lewis Carroll, Alice's Adventures in Wonderland

---

30 years ago I started my first job as an instructional designer at NCR in Dayton, OH.  I was hired in part because of my ability to design and program computer-based training in Apple Pilot; NCR had their own version of the same language.

The first major project I worked on was training for key corporate accounting policies.  The corporation had eight accounting policies that accounted for about 80% of the accounting errors made, because corporate accountants misapplied the policies.  The company was 100 years old, and many of the policies had been in place since nearly its inception.  Unfortunately, they had been modified over the years, but they had never been rewritten.  They were literally cut and pasted together! That led to a confusing mishmash of sometimes conflicting wording that was just hard to understand.

The company had tried to develop training on these policies in the past.  Their most recent effort was an interactive video disc (it was the 80's and IVD was the next best thing...not so much), which was little more than a talking head reading the policies. Needless to say, this was not very effective. If I thought the policies were hard to read in print and understand, imagine how hard they were to listen to and understand.  Or stay awake. So they asked me to design computer-based training modules for the policies

Using good instructional design principles, I did a task analysis and a needs assessment and using Gagne's instructional events, came up with a modular design that started with an overview statement of the policy, a brief review of related policies, a detailed explanation of its application, examples of its proper application and examples of its improper application.  Each module ended with a quiz that the learners had to repeat until mastery and that would loop them back to the appropriate part of the training for remediation.  I worked with a couple of corporate accountants at NCR (including one with the last name Cashdollar...is there a better name for an accountant?)  who explained how the policies were supposed to be applied.  They also developed the overview statements and examples and had to check my explanations of the policy application in the training.

I finished the first four modules, and we had a checkpoint meeting.  I didn't hear anything for days, and then I was told to stop working on the project.  I couldn't imagine what was wrong!  It was great training, everything worked well, but maybe I took too many liberties and verged too far from the policies for the corporate stuffed shirts' comfort level. It was a long couple of weeks.

Then I finally heard from the accountants.  They loved the training.  They loved it so much, that they convinced their bosses that the policies needed to be rewritten from scratch following the same format I used in the training - policy statement, related policies, policy explanation, examples, non-examples. However, I had to wait until they could get that done before I could continue with the project.

I left the company before the project was done (I only stayed at NCR for 10 months).  However, someone else finished the project, and I heard months later that corporate loved the training, that their error rate was down significantly. Personally, I think it was because they finally wrote the policies in a way they could be understood.   But f they want to credit the training, hey, I'll take that.

The take-away here is not that the computer-based training was so great, but that the technique used to develop it was effective. Technology in and of itself is never the answer.  I'm not such a hardliner as Richard Clark, who likens educational technology used to deliver instruction to the truck delivering your groceries ("The best current evidence is that media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition."). He believed that any differences found in media comparison studies (where media-based instruction is compared to traditional lecture based instruction) could be attributed to greater effort in the design of media-based instruction, not the media itself.

I'm more on Kozma's side, who asserts that eventually media will affect instruction, once you find the right medium for the right instruction. I agree with Clark that good instructional design is the most important factor; well-designed "media-free" instruction will always trump poorly designed media-enhanced instruction. However, part of the instructional design process is finding the most effective means of instructional delivery, and sometimes that includes using media and technology.

Today, it's often argued that we do our students a disservice if we don't use technology.  After all, they use technology everyday, so we should, too.  Indeed, just today I saw a post in my Facebook feed from Inside Higher Ed with the quote, "We shouldn’t use the digital revolution to continue outdated forms of higher education, like the lecture." On the one hand, I agree with Clark, that using technology to deliver a lecture does not automatically make the lecture better.  But on the other, I disagree that the lecture is an outdated mode of instruction. After all, if you think about it, what is a Ted Talk, but a lecture?  Granted, it's often a lecture that uses technology, and you can watch it online...but it's still one-way communication, the sage on the stage as we say, and it is very, very popular. In fact, one of the most popular TedTalks, Susan Cain's The Power of Introverts, uses almost no technology at all (she wears a microphone since it was delivered in an auditorium, and, of course, it was recorded). So no, the lecture is not outdated.

What should be dead - but unfortunately isn't - is the idea that technology will solve all our problems. What technology does is give us options. It doesn't give us answers. In fact, technology often makes it HARDER to do our jobs well, because in addition to being subject matter experts who can tell people what we know, we also need to be media and technology experts - or at least be willing to work with those that are.  We need to be able to somehow know when it is and it isn't appropriate to use technology and how to best use it when we do.  For me, that's easier, since I am an instructional designer as well as a teacher.  It's what I've done for 30 years.

But for many teachers and professors, they rely on their textbook publishers or other providers to tell them what to do.  And that's a danger, because their ultimate goal is to make money.  Yes, it's nice to use the latest and greatest resource from wherever, but you still need to make sure it's appropriate for your students.  You still need to structure the use of that technology. You still need to ensure that students know what they are supposed to learn, know that they've learned it, and know how to use what they've learned in the future. *cough*Gagne*cough*

And that's what I did when I designed the Key Corporate Accounting Policies training at NCR, which is why it was effective.  The technology wasn't the difference, it was the technique used.  While I made the most of the capabilities of the technology to provide remediation as needed, it was good instructional design technique that made the real difference.  The technique not only resulted in well-design CBT but better written policies.

Keep that in mind when you are being told that this technology will solve all your problems and engage your students and improve their test scores.  You still need good techniques to use it effectively.  A talking head on video is no more effective than a talking head in a classroom.  It's still just a talking head. And the talking head in the classroom can adapt.