Khan Academy (from a Designer Point of View)

This post is a response to the post made by Jason Rosoff talking about the Khan Academy.

Hello Jason, first of all let me say that I really appreciate the job everyone is doing at Kahn Academy, I can obviously see that Mr. Sal Kahn is not only an amazing person, but he is wise when choosing the team he works with. Your passion and commitment really shows. Having said that, let's get to some constructive discussions,

Just a warning, I am not saying at any point things would be better if done the way I say, all suggestions and Comments I make on regards to the Academy are just opinions from my Design Point of view, Hopefully with a little back and forth brainstorming we can come up with great solutions.

The achivement system has become a game design element commonly used to help deal with replayability, it also helps motivate the user because they realize that they are being succesful. I think this system is very well implementeed at the Academy, I specially like the different types of awards, although I am not 100% sure about the amount of awards that are given out. basically each module gives out 2 achivements (for completing, and for answering fast). This could be ok, I'm just thinking that maybe they will lose value to the students, and if they do then kids might not be motivated by them, which would defeat their purpose.


The 10 streak system has definitely some positive aspects, and a couple of question marks as well. On the bright side, with a 10 streak you can notice how students are handling adversity, you can measure their effort. Some might just quit after failing a couple, while others might watch some videos and keep trying until they succeed. Another positive aspect is that with 10 questions various aspects can be analyzed, I'm not sure if there is any AI behind this or if it is random but for example when learning multiplication, with a series of questions you can test if the student understands aspects like multiplying by 0, by 10, 5, 100. So instead of the kid learning a couple of tables, with this variety they will test different aspects.

As you mentioned, loosing with a streak of 9 can be pretty frustrating, especially if a student has tryied for hours and maybe it was a slight misstype. Here is where my Game Design Lightbulb starts blinking, as far as the streak goes, maybe the doubt you have is because of the fact that there is no forgiveness to the user, mistakes can happen for many reasons, and the last thing you want is to make a mistake and completely ruin everything you have been working on, this probably the best explanation for the 3 lives system implemented by many games.


By adding a bit of forgiveness you let the players make a mistake, then if they make more than one they wont blame the system but themselves and this will motivate them to improve. With the no mistakes policy it could be seen as the systems fault. But with statistics this should be pretty nice to prove (I am jealous because with all that data it must be amazing seeing all the different patters and then seeing how your work affects them).


Another big question mark I see with this system is the assumption that when you answer 10 questions on a row you are then proficient on that module. Looking at one of the links you posted I noticed this card:

As you can see, this student passed a module but then struggled with the review, I'm not sure how long there is between each review, but this is definitely something to keep a close eye on. There is already a lot of research done by the Academy to determine what modules need to be learned before working on a particular module, so using that knowledge I would suggest a program that it is not a multiple choice answer or open answer, but an interactive module that lets you do step by step(sort of like the hint does it), then by allowing the kid show their work then it is easy to detect the reason why an answer was wrong, and then this would help detect gaps quicker. But this is easier said than done, so perhaps a less effective by easier test would be to create an "alert" system, so if the kid gets a question wrong, all the modules that are prerequisites get "alerted" then if a module gets "alerted" some 5 times then you can ask the student to do an emergency session of that particular module.

When you say kids are trying to game the game do you mean they are trying to find loopholes?


Now I'm going to throw out a bunch of ideas and I would love to hear your opinion on them:

1) Goals: Looking at the cards I noticed that the teacher was asking students to set their goals for the week. This sounds like a great idea, perhaps I wouldn't make it a mandatory weekly goal, let the students choose their goal and timetable but there are three great benefits of goals. first they motivate the player to want to "Study" longer hours, secondly, The software can track effort and confidence by looking at the goals being set by the student. and then there is also the fact that the teacher will have even another statistic that will quickly detect weak points on a kids knowledge.

2) Challenges: these are a bit different from the current set of challenges, these challenges would mostly be determined with the help of AI and statistics, and basically they would be challenging a student to accomplish a number of modules in order to receive a prize. students that are currently easily progressing through the levels would get a larger amount of levels needed to complete the challenge. This would be designed to motivate all students equally. By making shorter goals, the struggling kid would be able to succeed more often, thus getting more motivated.

3) Collaboration: There is nothing more motivating than competing with or against a friend. This component could be implemented on special challenges that required teams of students to compete against the clock to pass a certain amount of modules.


4) Class Awards: Again competition is a great motivator, so perhaps outputting to the teacher stats like the student that spent the most time watching videos, the one that answered the most questions, the student that gained the highest amount of modules. All these stats would not only motivate students to get them but at the same time it would allow for students with all different skill levels to win.

5) Personalization: The sims is one of the biggest successes of personalization, with the ability to create your own character many people get emotionally attached to their virtual player. Also this Avatar can be utilized at different points of the childrens progress.



6) Unlocks: unlockables are perhaps on of the most satisfying elements of games. When people get this "secret" content they feel really special. But obviously locking educational modules is not a good idea, all content needs to be worldly available, so here is where the use of avatars can help with the unlocks, perhaps students can get gear or something cool for their avatars only when they reach locked areas.

7) The last of my observations (for now) is that as of now there is a monotonus flow of the game. All modules have a video and a set of questions, but after doing 15 modules someone might not be motivated to keep playing, but there could be a simple game or activity that get's unlocked after playing a set amount of modules, this would create a resting period for the user. These stoppages are really helpful to keep away from constant challenge. An example of this is the surfing level on Mario Galaxy. This was simply a race, not too many stars were awarded but it gave the player a break from entering galaxies and collecting a bunch of stars.


Well, I hope you found this post interestingly enough to keep the discussion rolling. But I do want to say that if there is anything you ever need as far as testing, prototyping, commenting, or any other thing, you can count on me, I trully believe that the Khan Academy will help millions of students worldwide, and I want to extend my hard work and passion to help as much as possible.

thanks for reading