Personalized and adaptive learning has been described as the so-called holy grail of education. The idea is not new, though its technological instantiation is getting increased attention. In a well-funded education system, personalized instruction happens when guided by a teacher as each students strengths and weaknesses and knowledge gaps are known. However, when classrooms start to exceed 20+ students, some type of mediating agent is needed in order to address knowledge gaps as it becomes impossible for a teacher to be aware of what is happening with each learner. So, while the human educator is the original (and best) personalized learning system, the current funding constraints and other resource challenges have raised the need for alternative approaches to make sure that each learner is receiving support reflective of her needs.
Many of the personalized learning systems now available begin with an articulation of the knowledge space – i.e. what the learner needs to know. What the learner knows is somewhat peripheral and is only a focal point after the learner has started interacting with content. Additionally, the data that is built around learner profiles is owned by either the educational institution or the software company. This isn’t a good idea. Learners should own the representation of what they know.
Last year, I posted on personalized learner knowledge graphs. Since then, I’ve been working with several colleagues to refine and develop this idea. Embedded below is a summary of our recent thinking on what this would look like in practice. Personal Learning Graph (PLeG – pronounced ‘pledge’ (acronyms are hard)) is intended as a response to how work and life are changing due to technology and the importance of individuals owning their own learning representation.
I know some people don’t like the whole “battle” idea in my book, and I get why it isn’t always applicable. But sometimes, it really does feel that way. In what could become a regular feature, if I could be bothered, I thought I’d do a quick round-up of stories that really emphasise the battle (or struggle if you prefer) aspect of open education currently.
The battle for language: This story that the University of Guelph trademarked the term “OpenEd” has largely resolved itself now. Understandably most of us who have worked in Open Ed for years were outraged, particularly when Guelph then aggressively pursued BC Campus over its use of the term. Brian Lamb and Clint Lalonde both captured this sense of outrage. Eventually, Guelph backtracked and climbed down. I won’t dwell on how misguided the attempt was in the first place, but rather just highlight that this shows that “Open” has market value now, and that commercial interests will seek to control what that means.
The battle for money: I could pick a similar story every month, but this CBC piece comparing the profits publishers make with the dire straits of many university libraries caught my attention. The researcher found that “the five largest, for-profit academic publishers now publish 53 per cent of scientific papers in the natural and medical sciences – up from 20 per cent in 1973. In the social sciences, the top five publishers publish 70 per cent of papers.” That’s a lot of control we’ve ceded to them. Make no mistake, we, as academics, messed up here and lost control over our own content and knowledge dissemination. A similar story in that Russian libraries lost access to Springer journals because they were unable to pay the fees.
The battle for ownership: Thanks to my OU colleague Simon Knight for flagging this. Potentially a change in European copyright laws might see the loss of the “freedom of panorama”. That is, you can take photos of public buildings without breaching copyright. As Simon highlighted, there is an implication for OERs here that include photos of public buildings. It’s supposed to allow non-commercial use, but that can be a grey area (eg if you are sharing them on a commercial site such as Facebook, Slideshare, that might count as commercial use). It’s one of those detailed legal arguments that might come to nothing, but equally we might found someone being prosecuted for by some over-zealous claimant. It also adds in another potential barrier, fear factor and layer of confusion for educators who just want to create a learning resource.
The battle to share: Colombian student Diego Gomez faces a potential prison sentence of 4-8 years for sharing an academic article he liked on Scribd. He didn’t make any money from it, he just thought others would benefit from reading it. The author sued for economic damage and the full weight of the law kicked in. Even if you think it was wrong of Gomez to share, the response seems massively disproportionate – it is an example of a legal system designed for one use, coming smack into the digital world and then floundering around like a bully. These type of confrontations will become more frequent.
When you view these, it’s hard not to frame it as a battle, one to make sure openness stays open, and that it isn’t closed down or thwarted for other uses. Or maybe I’m just paranoid and see it everywhere…
George Siemens, Dragan Gasevic and Shane Dawson have produced an excellent report with this title. I think it’s a very ambitious and also very timely thing to do. They synthesise the research of distance, blended and online learning to provider an analysis of the benefits and issues for each. As nearly all universities offer one or more versions of these forms of learning, it is very useful to have a report to start from. As we’ve often voiced in the OER field, there is a lot of research published that is of questionable quality, and in order to make good decisions we need to be drawing on sound evidence.
So, I applaud their efforts and what I offer here is by way of an addendum, not a major criticism. I have two points I’d add to the report, both of which arise from my Open University experience. I fully appreciate that in wanting to produce a readable report they can’t give the detailed history of distance education. But I’d like to add the following two items for consideration:
i) The discussion of distance education seems to be focused on correspondence tuition and then jumps straight from here to interactive, online modes. Anyone who has worked at the OU becomes sensitive to this, so maybe it doesn’t matter to others, but I think it’s worth highlighting the Supported Open Learner model developed by the OU (and then successfully replicated across the globe). This has a range of elements all specifically designed to aid the distance learner, including course material designed to be studied individually, a part-time tutor allocated for support (by face to face tutorials, phone, online, etc), a regional centre support system, summer schools, use of different media and assessment constructed to be a feedback and progression mechanism. I stress it because many universities and online providers still haven’t discovered this rich support mechanism. I expect one will reinvent it soon, amongst much fanfare, but the point is that different elements have greater significance for different students. Think of it like reversing a car: you use side mirrors, rearview mirror, reverse sensor, look over your shoulder. All those elements are useful. I feel the report rather brushed over the significance of this in a rush to get to blended learning.
ii) The report states that distance learning has high retention. This seems odd, and makes me wonder what version of distance ed is being considered here. Distance ed is not synonymous with open education, but it has often been used as a means by which open education can be realised. One of the things about open education is that it doesn’t have high retention rates. Just as MOOC developers are now discovering, if you have open entry, it makes comparison with filtered entry difficult. MOOC providers are also making claims that traditional metrics of completion rates are not as applicable. This has always been the case for truly open education. Many open ed students come in, try one or two courses, and then leave the system, quite satisfied. They have got what they wanted and they never intended to gain a degree. This is why funding systems based solely on whole course completion are a disaster if you care about social mobility, inclusion, or open education. So to claim that distance learning has high retention seems a bit at odds with some of the reality experienced.
Apart from that, thanks George, Dragan and Shane, I really did enjoy reading it, and apologies if I’ve misinterpreted anything here.
I keep hearing rumours about Athabasca University dying or at least being on its deathbed. I guess stories like this don’t help: AU taskforce releases sustainability report. This article was picked up by Tony Bates, who states: “So Athabasca University is now in the same position as the Greek government, except it doesn’t have the EU, the IMF, or the Germans to look to for help – just the Alberta government, which itself has been fiscally devastated by the collapse of oil prices.”
I’m conflicted by Tony’s response. He has forgotten more about digital learning than most of us will ever know. He has a global view of the sector and has been in the trenches as a leader. His diagnoses of how AU’s problems came about resonates with the discussion that I have heard. Unfortunately, Tony also adds needless rhetoric to a situation that has qualitatively changed with the new government in Alberta. His comments don’t reflect the new Alberta context.
I don’t know all the behind the scenes discussions relating to this report. I don’t know the specific intent of the sustainability report. My impression from what I’ve read (see: Athabasca University’s Hostage Crisis and Athabasca University facing insolvency, Alberta government may have to step in) is that this is a political game (request for more funding) that is being played in the public sphere.
It’s amply clear that governments are divesting from public education. The defining challenge of our time is inequality. Any sufficiently advanced and civilized society should ensure a) healthcare is available to all and b) an education is available to all. Education is not the goal. The goal is a populace is able to improve their position in life and to live life on the terms of success that they define for themselves. Education is the best way that we have of doing this today.
The education system we are building today is failing to enable opportunities and is instead a system of hardening power structures and socio-economic positions. President Truman anticipated this, nearly 70 years ago:
If the ladder of educational opportunity rises high at the doors of some youth and scarcely rises at the doors of others, while at the same time formal education is made a prerequisite to occupational and social advance, then education may become the means, not of eliminating race and class distinctions, but of deepening and solidifying them
Publicly funded online universities (such as AU, OU, OUNL) have to date been the most successful systems in enabling educational access to learners who do not fit the traditional learner profile. While a number of traditional universities have recently started using the rhetoric of targeting and supporting under-represented students, open universities have been doing it since the 1960s (and some systems prior to that).
I hope that the new Alberta provincial leaders, and counterparts globally, would recognize and support the revolutionary and real life impact that open universities have had on the quality of life of many learners. It’s discouraging to see that at the exact point where many state and provincial leaders around the world start to recognize the need for improving education access, those systems that have been serving this mission for 50 years risk being cut off at the knees by limited vision and appropriate government support. Fortunately, early indications suggest that the government in Alberta is starting to listen: “”I take this situation seriously,” Sigurdson [Advanced Education Minister] told Metro. “The Alberta government is ready to work with the university and help it become more sustainable.”"
At the start of the digital, networked revolution there were lots of books about new business models. Most were, let’s face it, rubbish. But there were some salient points that came out amidst all the hyperbole. I think Weinberger’s concept of filtering on the way out instead of filtering on the way in, is a good example.
Anyway, now that internet models have settled a bit I’ve been thinking that the next phase might be around what openness offers. I circulate in different overlapping communities: OERs, open access journals, MOOCs, open textbooks. I’ve noticed a common theme emerging which you could label the “open flip”. Briefly stated, it is that money shifts from purchasing copyrighted resources to production of open ones.
Cable Green, speaking of open textbooks, says we have lots of money in education, we’re just really bad at spending it. His claim is that the cost savings for schools buying books is considerable, once you make this shift. Similarly, for open access journals, there is a good argument to stop buying journals, but instead start producing them ourselves. Or we stop buying elearning content and produce OERs.
There are other areas where this might be applicable too, beyond education. For instance, currently we spend billions on purchasing drugs from large pharmaceutical. An open flip would see that money spent on producing drugs that are then openly licensed so production is cheap. I don’t know enough about big pharma to know if the economics would work out in this instance, but the point is it is an approach that could be considered now.
The digital, networked infrastructure is the substratum that allows this to happen, but it is open licensing that adds the final ingredient. I think we will see variations of the open flip across many disciplines as the intersection of these three elements opens up new approaches. Often we have become so accustomed to existing models that they seem like the only way to realize the desired goal, but we have an opportunity to reconsider where money is allocated in the chain now, and there may be more effective ways of spending it.
With apologies to David’s 5Rs of reuse…
Whenever a new technology, or approach, or technology driven approach arises, the claims for it are often varied, ranging from student emancipation, to cost saving, to complete revolution of the higher education system. It often seems that nearly all of the early years of a technological development are spent arguing about what exactly it can help with, what problem it is solving. In this post I am taking a purely pragmatic approach, in that I am going to suggest that for any tech development to be taken up long term, it needs to solve some specific concerns of universities. Now, I fully accept that learning takes place outside of universities, or your goal might be to completely destroy that system. That may well be so, but that is probably a different argument. And similarly there are deeper perspectives than this one which address issues such as learner emotion, deep learning, etc. But my argument here is if you think an ed tech development has value, then a good strategy is to make an argument based on these pragmatic lines and recognise the context within which it is operating.
In an increasingly competitive higher education system, what is it that senior management at higher education institutions are concerned with? I guess the base line might be economic survivability, but if we take a level of abstraction above the purely financial, then I would argue that most good vice chancellors, provosts, presidents etc are legitimately concerned about three areas, as they seek to pursue their overall mission of educating people:
Now consider any recent tech development in the light of these three Rs: learning analytics, MOOCs, OERs, learning design, VLEs, etc. Quite often we have made confused claims against all three, or ignored these in favour of revolutionary rhetoric (“MOOCS will democratise education for all!”) or more abstract potential (“open education creates better citizens”). These may be true in the long run, but more practically it is useful to make specific claims against one or more of these Rs, and then set about conducting research which can verify this. It may be less exciting, but ultimately more useful if we can do this.
Let’s take OER as an example. Our work with the OER Research Hub has found that many students are using OERs before they take up formal study, so are trialling subjects. And others are using OERs to supplement their study whilst in formal education. We need some further work to get evidence on this: what is the conversion rate from studying OERs to formal study? How can this transition be helped effectively? Does using OERs in formal study lead to greater retention of students?
I would propose that answering such questions against one or all three of the Rs should be an aim for any new ed tech development once it moves beyond the experimental stage, if it is to be adopted widely in higher ed.
I’ve been working with several colleagues on arranging the upcoming Digital Learning Research Network (dLRN) conference at Stanford, October 16-17, 2015. The call for papers is now open. We are looking for short abstracts – 250 words – on topics of digital learning. The deadline is May 31. Our interest is to raise the nuance and calibre of the discussion about education in a digital era; one where hype and over-promising the power of technology has replaced structured interrogation of the meaning of changes that we are experiencing. We have a great lineup of speakers confirmed and are expanding the list rapidly. The conference will include social scientists, activists, philosophers, researchers, and rabble rousers. It will be an intentionally eclectic mix of people, institutions, and ideas as we explore the nodes that are weaving the network of education’s future. Representation from the following research organizations has already confirmed from: Stanford, Smithsonian, University of Michigan, University of Edinburgh, Columbia University, CMU, state systems (Georgia, California, Texas, and Arkansas), and SRI.
Join us for what will be a small (max 150 people) and exciting exploration of a) what education is becoming, b) who we (as learners, activists, and academics) are, and c) where these two intersect in forming the type of learning system that will enable us to create the type of society that we want for future generations.
From the call:
Learning introduces students to practices of sensemaking, wayfinding, and managing uncertainty. Higher education institutions confront the same experiences as they navigate changing contexts for the delivery of services. Digital technologies and networks have created a new sense of scale and opportunity within global higher education, while fostering new partnerships focused on digital innovation as a source of sustainability in volatile circumstances. At the same time, these opportunities have introduced risks in relation to the ethics of experimentation and exploitation, emphasizing disruption and novelty and failing to recognise universities’ long-standing investment in educational research and development.
(T-shirt available from Fake Elsevier)
This post is a plea really to academics to not surrender rights, or the promise of openness so readily. I completely understand that I am in a privileged position, and it’s easy for me as a prof to say “only publish in open access”, or “share your stuff openly”. But it’s a different story if you are a new researcher, and after one of those ever more elusive permanent positions. But even so, I am often surprised at just how readily academics acquiesce to bad deals, particularly with regards to publishing.
I have frequently heard “I would love to publish open access, but in my field I can’t”. Or “I tried, but they said no.” And that is it, there is no attempt to find an alternative journal, to negotiate the embargo period with a publisher, to offer any form of push back. Academics frequently underestimate their power I think. As I mentioned a few years ago, I took the open access oath (only reviewing and writing for open access). It is a remarkably effective step. Our labour (offered freely to journals) is all we have, but the system requires it to operate.
Similarly, I have heard academics state that they would like to develop an online profile, but it would be frowned upon by their boss. Academic life seems to me to be increasingly precarious, and this climate of doubt and uncertainty can be abused. You have to tow the line ever more to get, or stay in a job. But academics are remarkably good at fighting against attacks on the integrity of their discipline. I don’t feel that they have become accustomed to thinking of their labour and outputs in the same way. So my plea is this – push back a bit. Ask the question about open access, refuse to do the review, start a modest (non-job threatening) online profile. Each time we acquiesce quietly makes it harder next time.
In response to my previous post, a throwaway analogy about Eddie Murphy and MOOCs, Rolin Moe produced this post. It is probably the best response to one of my posts I’ve ever had. I respond in kind here.
In the article The Eddie Murphy MOOC Mystery – Part 1, Moe (2015) responds to the author’s critique of MOOCs through the lens of Eddie Murphy. The Moe article is undoubtedly a significant contribution to the MOOC-Murphy literature in its analysis of the three core claims in the original article. However, in its analysis it fails to take into account the temporal element of the Murphy film rating, instead ordering them by Box office and Rotten Tomatoes rating. It is the author’s contention that consideration of this temporal element will further reveal the utility of the Murphy metaphor.
Using the IMDB rating for each of Eddie Murphy’s films, we can plot them against year of release. This gives the scatter plot shown in Figure 1:
What this reveals is that (ignoring the outlier of Best Defense), Murphy’s career commenced with a series of highly successful, highly rated movies. By the late 1980s however, the Murphy success formula had reached its peak, and we witness a period of stagnation, and general decline through the 1990s. In the 2000s, the picture becomes more complex. There is a form of renaissance (a murphaissance if you will), particularly aided by the Shrek franchise. However, this is more than matched by an underlying general long trend of decline. For every Shrek, there are two Norbits.
Researchers in educational technology have often made reference to the Gartner Hype Cycle. The author proposes that the Murphy Cycle offers a more robust framework for analysing trends. Not only is it grounded in empirical data (unlike the Gartner model), but it also offers a more nuanced picture. It has a similar peak and trough, but then a more mixed pattern of usefulness amidst a general sea of poor application.
The author also contends that it provides researchers with more meaningful reference points than those used in Gartner. Teams can talk about being in “the Beverley Hills Cop moment”, or realise “this is our Pluto Nash.” It also offers a degree of hope – for after the Klumps, comes Shrek.
File under: pointless things that occur to you while walking the dog.
My daughter came across Beverley Hills Cop on TV the other day, and then worked her way through a range of Eddie Murphy films. She quickly discovered what the rest of us learnt back in the 80s. Most Eddie Murphy films are not very good. Murphy has undeniable screen presence, and when he’s not on screen these films are just interminable. But for a few years anything with Murphy in it was a guaranteed hit. I came to the conclusion that what happened was that a lot of average scripts had sat around without any real backing, because they weren’t very original. But stick Eddie Murphy in it, and you’ve got box office success on your hands. It is hard to imagine why on earth anyone would make The Golden Child if it didn’t have Murphy headlining it. Or 48 Hours which was just another mismatched cop film. But with Murphy in it, they become something else. A lot of them aren’t really comedies either. They’re just films that have Eddie Murphy in. This is not a critique of Murphy as an actor, but I think he paid the price for this in the end. That “anything with Murphy in it is great” attitude wore thin by Beverley Hills Cop 3, and by the time you got to Pluto Nash, it was insulting.
So, what does this have to do with MOOCs? The ‘stick Murphy in it’ attitude of studio bosses back in the 80s seems to me rather akin to the ‘stick a MOOC in it’ attitude I’ve encountered with research bids, or discussions around innovative teaching. I have joked that I’ve dusted off all my rejected research bids and replaced “OER” with “MOOC”. It’s not quite true, but there’s an element of the Golden Child script about it all. At the moment, even with the MOOC backlash hitting, funders, governments, journalists – they all want a bit of MOOC action in there. To extend the Murphy-MOOC analogy, then I think MOOCs will pay the price for this high coverage. A good MOOC proposal will be rejected because they were of their time.
Actually, watching all those 80s Murphy films made me think there is a MOOC in there somewhere about 1980s action films and what they reveal about social attitudes of the time (it’s a LOT by the way).
Stephen Downes responds to my previous post: “I said, “the absence of a background in the field is glaring and obvious.” In this I refer not only to specific arguments advanced in the study, which to me seem empty and obvious, but also the focus and methodology, which seem to me to be hopelessly naïve.”
Stephen makes the following points:
1. George has recanted his previous work and is now playing the academic game
2. Research as is done in the academy today is poor
3. Our paper is bad.
Firstly, before I respond to three points, I want to foreground an interesting aspect of Stephen’s dialogue in this post. I’m going to call it “academic pick-up artist” strategy (i.e. tactics to distract from the real point of engagement or to bring your target into some type of state of emotional response). I first encountered this approach by the talented Catherine Fitzpatrick (Prokofy Neva) during CCK08. Here’s how it works: employ strategies that are intended to elicit an emotional response but don’t quite cross over into ad hominen attacks. The language is at times dismissive, humorous, and aggressive. In Stephen’s case, he uses terms such as: hopelessly naïve, recant his previous work, a load of crap, a shell game, a con game, trivial, muddled mess, nonsense. These flamboyant terms have an emotional impact that is not about the research and don’t advance the conversation toward resolution or even shared understanding. I’ll try to avoid responding in a similar spirit, but I’ll admit that it is not an easy temptation to resist.
Secondly, Stephen makes some statements about me personally. He is complimentary in his assessment of me as a person. I have known Stephen since he did a keynote in Regina in 2001. I’ve followed his work since and have greatly valued his contributions to our field and his directness. I count him as a friend and close collaborator. I enjoy differences of opinion and genuinely appreciate and learn from his criticism. (do a “George Siemens” search on OLDaily – he has provided many learning opportunities for me).
Stephen says a few things about my motivations that require some clarification, specifically that I am trying to make an academic name for myself and that I am recanting previous work. I honestly don’t care about making an academic name for myself. I am motivated by doing interesting things that have an impact on access to learning and quality of learning for all members of society. I am a first in family degree completer – as an immigrant and from low socio-economic status. There are barriers that exist for individuals in this position: psychologically, emotionally, and economically. Higher education provides a critical opportunity for people to move between the economic-social strata of society. When access is denied, society becomes less equitable and hope dims. My interest in preparing for digital universities is to ensure that opportunities exist, equity is fostered, and that democratic and engaged citizenry are fostered. The corporatization of higher education is to be resisted as values of “profit making” are at often in conflict with values of “equity and fairness”. I want my children to inherit a world that is more fair and more just than what my generation experienced.
I will return later to Stephen’s assertion that I am recanting previous work.
1. George has recanted his previous work and is now playing the academic game
With academic pickup artistry and my motivations foregrounded, I’ll turn to Stephen’s assertions.
It has in recent years been engaged in a sustained attack on the very idea of the MOOC and alternative forms of learning not dependent on the traditional model of the professor, the classroom, and the academic degree. It is resisting, for good reason, incursions from the commercial sector into its space, but as a consequence, clinging to antiquated models and approaches to research.
This get at the heart of views that Stephen and I have discussed on numerous occasions. I believe in the value of the professoriate. In this instance, he is Illich to my Friere. As I interpret Stephen’s work, he would like to see all learning opportunities and control shift to the individual and sees limited value in the higher education system that is as much about preserving faculty positions as it is about preserving the academy. Stephen and I both resist commercialization of education but vary in how we want to see the university of the future. Stephen wants a university model without universities. This comes, I believe, from his unfortunate experiences in doing his phd where his supervisory panel played a hard heavy hand in determining what is and isn’t research that they valued. I’m sure his experience isn’t unique.
Faculty can be stunning idiots when it comes to preserving and perpetuating their egos. The pursuit of knowledge and advocacy for equity often takes a seat to ego and the goal building a faculty “mini me” who is expected to pick up a research stream done by a panel or department and toe the line. In contrast to Stephen’s views, I love universities. I want a future of more, not less, universities. Universities are not perfect, but they are the best model that we currently have to enable individuals to improve their position in life and a power structure that exists to counter and comment on the corporate and government power structures. Can these goals be realized by networks of individuals (i.e. the second superpower)? If the world was populated with primarily Stephens, then it might be possible. For many people, however, education is not a goal in itself, but rather a means to employment. Systems are needed to preserve and perpetuate the highest ideals of society. If left to chance, then the views of the most aggressive will become the norm. While society slept, many of the wealthiest were busy creating a tax system that preserved their resources and created inequity. In the past, unions existed to serve as an organizing structure to advocate for the rights of individual works. Stephen would argue that we could today do this organizing and democracy preserving work through networks. I agree that networks are important, but argue that institutions are a type of network that has been configured to better meet these needs. Some structure is needed. Perhaps not as much as we see today in universities, but a minimum level or organization is required in order to provide learning opportunities to society’s disenfranchised. Simply giving people access is not enough. Social, scaffolded, and structured support is needed.
Perhaps as a result, part of what Siemens has had to do in order to adapt to that world has been to recant his previous work… This recantation saddens me for a variety of reasons. For one this, we – Siemens and myself and others who were involved in the development of the MOOC – made no such statements. In the years between 2008, when the MOOC was created, and 2011, when the first MOOC emerged from a major U.S. university, the focus was on innovation and experimentation in a cautious though typically exuberant attitude.
I haven’t recanted my previous work. Stephen displays a linearity of thought, of cause/effect, that confuses me. I see the world in networked structures. Learning is about network making at neuronal, conceptual, and external levels. Knowledge is networked. The history of ideas is networked. I don’t see a “one or the other” approach to research, to corporate involvement in education, or to learning in general. Instead, I see 3D lattice-like network structure that have multiple dimensions and connections between those dimensions.
Siemens has moved over to that camp, now working with EdX rather than the connectivist model we started with… Again, these rash and foolish statements [from Agarwal] are coming from a respected university professor, a scion of the academy, part of this system Siemens is now attempting to join.
I disagree with this statement, largely because I have privileged access to my own thinking. In this instance, and at least one prior when I did a talk at Online Educa many years ago and he stated that I had become fully corporate, Stephen is putting me in a box. Nobody puts George in a box! I am part of the academy in terms of employment. I am part of the academy by nature of grant writing and research. I am part of the academy in terms of publishing with my peers. But I am not only a one-dimensional entity. I did not take a traditional academic route. My publication history is not typical. Many of my citations come from open public works rather than traditional publications. To say that I have recanted prior work is simply not true. I am bringing my previous work into a different context – one that allows for networks and university structures to exist. Stephen is doing something similar with his work with LPSS. Has he sold out to the corporate oil and gas sector?
The inclusion of the Chronicle article as part of Stephen’s comments makes this a more complex discussion. We are now not only looking at what Stephen feels is a bad report, but that my professional ambitions are now being interpreted through a Chronicle piece. My criticism here, and something that was not clear in the Chronicle article, is about the academy’s embrace of MOOCs. Stephen takes the “we” personally, whereas he was never the intended target of the “we”. I would love to see all media interviews and recordings posted fully with articles such as this. My use of “we” in the above quote is problematic. By “we”, I was speaking about education/hypesters/corporate entities like Udacity/Coursera. This is something that Rolin Moe also asks about.
And what is key here is that he [George, over here, still in a box] does not believe our work was based in research and evidence… He says nice things about us. But he does not believe we emphasize research and evidence.
I was making an argument that didn’t come off clearly. This is perhaps a similar failing to Stephen’s previous assertions that his work is about “making” not only reporting. I don’t believe he meant it in the way that others interpreted it. What Stephen was saying there, and I’m saying here, is that there is an approach to work (in my case research and in his case writing software) that produces hope for desirable outcomes rather than despair at seeing a seemingly inevitable techno-solutionist outcome. I’m not denying that Stephen does research. But he has placed himself in a difficult position: he doesn’t want the institution of higher education but he wants to be seen by people in the academy as someone who does the same type of work as they do. Stephen defines himself as a philosopher. His papers reflect this spirit. He doesn’t frequently subject his ideas to the traditional peer review that defines academic research (for obvious reasons – he doesn’t trust or feel that process has much value). His writing is open and transparent, however, so anyone could engage and critique if they were so inclined.
2. Research as is done in the academy today is poor
The comments above aren’t a direct engagement yet with our paper. In the second half of this post, Stephen expands on his primary concerns which are about educational research in general.
Why is this evidence bad? The sample sizes are too small for quantificational results (and the studies are themselves are inconsistent so you can’t simply sum the results). The sample is biased in favour of people who have already had success in traditional lecture-based courses, and consists of only that one teaching method. A very narrow definition of ‘outcomes’ is employed. And other unknown factors may have contaminated the results. And all these criticisms apply if you think this is the appropriate sort of study to measure educational effectiveness, which I do not.
Educational research is often poorly done. Research in social systems is difficult to reduce to a set of variables and relationships between those variables. Where we have large amounts of data, learning analytics can provide insight, but often require greater contextual and qualitative data. Where studies, such as Bonnie Stewart’s recent PhD, are qualitative, criticism against size can be leveraged. These are both unfair in that no single node represents the whole knowledge network. Research is a networked process of weaving together results, validating results, refuting results, and so on. It is essentially a conversation that happens through results and citations. The appeal to evidence is to essentially state that opinions alone are not sufficient. The US Department of Education has a clear articulation of what they will count as evidence for grants. It’s a bit depressing, actually, a utopia for RCTs. While Stephen says our evidence is poor, he doesn’t provide what he feels is better evidence. Where, outside of peer-reviewed articles and meta-studies, can academics, administrators, and policy makers find support and confidence to make decisions (the stated intent the introduction of our report)? What is our foundation for making decisions? If the foundation is opinions and ideas without evidence, than any edtech startup’s claim is equally valid to researchers, bloggers, and reformers. Where is the “real research being performed outside academia” and what are the criteria for calling that activity research, but what’s going on in the academy, and funded by NSF, JISC, OLT, SSHRC, as being largely trivial?
Stephen then makes an important point and one that needs to be considered that the meta-studies that we used are “hopelessly biased in favour of the traditional model of education as practiced in the classrooms where the original studies took place.” This is a significant challenge. How do we prepare for digital universities when we are largely duplicating classrooms? Where is the actual innovation? (I’d argue much of it can be fore in things like cmoocs and other technologies that we address in chapter 5 of the report). Jon Dron largely agrees with Stephen and suggests that a core problem exists in the report in that it is a “view from the inside, not from above.”
I need to reflect more on Jon’s and Stephen’s insight about research rooted in traditional classrooms and the suitability of assessing that against a networked model of education and society.
3. Our paper is bad
At this stage, Stephen turns to the paper itself. Short answer: he doesn’t like it and it’s a trivial paper. The list of what he doesn’t like is rather small actually.
At this stage of reviewing his post, I’m left with the impression that much of Stephen’s complaint about our paper is actually a discussion with himself: The Stephen that disagreed with his phd supervisory committee and the Stephen that today has exceeded the impact of members on that committee through blogging, his newsletter, presentations, and software writing. Our paper appears to be more of a “tool to think with” and enable Stephen to hold that discussion with his two selves, effectively Stephen of today affirming that the Stephen in front of the phd committee made the right decision – that there are multiple paths to research, that institutions can be circumvented and that individuals, in a networked age, have control and autonomy.
Stephen next statement is wrong: “With a couple of exceptions, these are exactly the people and the projects that are the “edtech vendors” vendors Siemens says he is trying to distance himself from. He has not done this; instead he has taken their money and put them on the committee selecting the papers that will be ‘representative’ of academic research taking place in MOOCs.”
The names listed were advisors on the MOOC Research Initiative – i.e. they provided comments and feedback on the timelines and methods. They didn’t select the papers. The actual peer review process included a much broader list, some from within the academy and some from the outside.
They do not have a background in learning technology and learning theory (except to observe that it’s a good thing).
In my previous post, I stated that we didn’t add to citations. We analyzed those that were listed in the papers that others submitted to MRI. Our analysis indicated that popular media influenced the MOOC conversation and the citations used by those who submitted to the grant. Many had a background in education. George Veltsianos shares his recent research:
Our tests showed that the MOOC literature published in 2013-2015 differed significantly from the MRI submissions: our corpus had a greater representation of authors from Computer Science and the Gašević et al., corpus had a greater representation of authors from Education and Industry. In other words, our corpus was less dominated by authors from the field of education than were the MRI submissions. One of Downes criticisms is the following: “the studies are conducted by people without a background in education.” This finding lends some support to his claim, though a lot of the research on MOOCs is from people affiliated with education, but to support that claim further one could examine the content of this papers and identify whether an educational theory is guiding their investigations.
He goes on to say that the MOOC conversation has changed and that greater interdisciplinarity now exists in research.
Stephen and I have had variations of the conversation above many times. Sometimes it has centred on views of what is acceptable knowledge. At other times, on the role of academics and knowledge institutions in networks. Some discussions have been more political. At the core, however, is a common ground: an equitable society with opportunities for all individuals to make the lives that they want without institutions (and faculty in this case) blocking the realization of those dreams. We differ in how to go about achieving this. I value the legacy of universities and desire a future where they continue to play a valuable role. Stephen imagines a future of greater individual control, less boundaries, and no universities. Fundamentally, it’s a difference of how to achieve a vision that we both share.
In my previous post, I mentioned the release of our report Preparing for the Digital University. Stephen Downes responds by saying “this is a really bad study”. He may be right, but I don’t think it is for the reasons that he suggests: “What it succeeds in doing, mostly, is to offer a very narrow look at a small spectrum of academic literature far removed from actual practice”. This resulted in a Twitter exchange about missing citations and forgotten elearning history. Rolin Moe responded by saying that the history that we included in our citation analysis of MOOCs was actually the one that most non-elearning folks follow “depending on lens, Friedman Pappano & Young are more representative of who’s driving EdTech conversation”.
We took two approaches in the report: one a broad citation analysis of meta-studies in distance, online, and blended learning. This forms the first three chapters. While we no doubt missed some sources, we addressed many of the most prominent (and yes, prominence is not a statement of quality or even impact). In the fifth chapter, we evaluated the citations based on the MOOC Research Initiative, which received close to 300 submissions. We only analyzed the citations – we didn’t add to them or comment on their suitability. Instead, our analysis reflects the nature of the dialogue in academic communities. In this regard, Stephen’s criticism is accurate: the narrative missed many important figures and many important developments.
The heart of the discussion for me is about the nature of educational technology narrative. At least three strands of discourse exist: the edtech hypesters, the research literature in peer reviewed publications, and the practitioner space. These are not exclusive spaces as there is often overlap. Stephen is the most significant figure in elearning. His OLDaily is read by 10′s of thousands or readers daily – academics, students, companies. His work is influential not only in practice, as his Google Scholar profile indicates. Compare his citations with many academics in the field and it’s clear that he has an impact on both practice and research.
Today’s exchange comes against the backdrop of many conversations that I’ve had over the past few weeks with individuals in the alt-ac community. This community, certainly blogs and with folks like Bonnie Stewart, Jim Groom, D’Arcy Norman, Alan Levine, Stephen Downes, Kate Bowles, and many others, is the most vibrant knowledge space in educational technology. In many ways, it is five years ahead of mainstream edtech offerings. Before blogs were called web 2.0, there was Stephen, David Wiley, Brian Lamb, and Alan Levine. Before networks in education were cool enough to attract MacArthur Foundation, there were open online courses and people writing about connectivism and networked knowledge. Want to know what’s going to happen in edtech in the next five years? This is the space where you’ll find it, today.
What I’ve been grappling with lately is “how do we take back education from edtech vendors?”. The jubilant rhetoric and general nonsense causes me mild rashes. I recognize that higher education is moving from an integrated end-to-end system to more of an ecosystem with numerous providers and corporate partners. We have gotten to this state on auto-pilot, not intentional vision.
When technology drives education, a number of unwelcome passengers are included: focus on efficacy over impact, metrics of management, reductionist thinking, etc. To sit at the table with academics and corporate players is essentially to acquiesce to capital as a driving and motivating factor. Educators have largely been out maneuvered, as indicated by the almost luddite interpretation by media to any resistance by faculty and teachers. We can’t compete through capital at this table. So instead we have to find an additional lever for influence.
One approach is to emphasize loosely coupled networks organized by ideals through social media. This is certainly a growing area of societal impact on a number of fronts including racism, sexism, and inequality in general. In education, alt-ac and bloggers occupy this space.
Another approach, and one that I see as complimentary and not competitive, is to emphasize research and evidence. At the decision making table in universities and schools, research is the only lever that I see as having comparable capacity to capital in shaping how decisions are made and how values are preserved. This isn’t to discount social networked organization or alt-ac. It is to say, however, that in my part of the world and where I am currently in my career/life, this is the most fruitful and potentially influential approach that I can adopt.
We’ve released a new report: Preparing for the Digital University: a review of the history and current state of distance, blended, and online learning (.pdf).
The report is an attempt to reposition the narrative of digital learning away from “look, my cool new technology does this” to something more like “here’s what we know from research and here’s what we can extrapolate”. Innovation is a bunnies and kittens type of concept – who could possibly oppose it? Sometimes new is not better, especially when it impacts the lives of people. Remember the failure of Udacity and San Jose State University project? Even passing familiarity with research in learning sciences could have anticipated the need for scaffolded social support. Instead, a large number of at-risk-students had yet another blow delivered to their confidence as learners, further entrenching negative views of their capability to success in university. This is bad innovation. It hurts people while it gains media accolades and draws VC funding. With our report, we are hoping to address exactly this type of failure by providing a research lens on how technology and learning are related in various contexts.
Five articles are included in the report and provide an overview of research literature, while a final article looks at future technology infrastructure :
- Distance education
- Blended learning
- Online learning
- MOOC research
- Future learning technology infrastructures
From the introduction:
It is our intent that these reports will serve to introduce academics, administrators, and students to the rich history of technology in education with a particular emphasis of the importance of the human factors: social interaction, well-designed learning experiences, participatory pedagogy, supportive teaching presence, and effective techniques for using technology to support learning.
The world is digitizing and higher education is not immune to this transition. The trend is well underway and seems to be accelerating as top universities create departments and senior leadership positions to explore processes of innovation within the academy. It is our somewhat axiomatic assessment that in order to understand how we should design and develop learning for the future, we need to first take a look at what we already know. Any scientific enterprise that runs forward on only new technology, ignoring the landscape of existing knowledge, will be sub-optimal and likely fail. To build a strong future of digital learning in the academy, we must first take stock of what we know and what has been well researched.
Audrey Watters does a fantastic job of debunking the myth around the concept of the factory school, or industrialised education model. I see this mentioned almost as often as ‘education is broken’, and it is a close ally of ‘education hasn’t changed in 100 years‘. The basic line is that we have an education system that was designed for an industrial age and we are now in a post-industrial age, ergo, that education system is faulty.
I think the first thing to do is what Audrey has done so magnificently, which is to really dig into the historical perspective, and demonstrate why the assumptions underlying that stance are just wrong. Another approach is to examine why the lack of change that is touted is wrong also. So view this piece as a little sibling to Audrey’s foundational work.
When people suggest that schooling is the same as in the industrial change, as Morrissey would have it, this is true, and yet it’s false. I want to explore both the elements that are true (and why that’s not a bad thing) and those that are incorrect, by way of an analogy. Imagine that it was commonly stated as fact that “reading hasn’t changed since the time of Dickens”. To take the true aspect first, we could take a photo of someone reading a book (maybe even a Dickens novel), printed on paper, sitting in a chair in front of a fire. If you could go back in time you could show this to Dickens and he would declare that indeed, reading has not changed (he’d also be very pleased that people were still reading his novels). The first question to ask then, is why would you want reading to change? Why is an absence of change deemed a bad thing? Reading a book is a pretty good way to convey an idea, a story, and an enjoyable, enriching thing to do. That it hasn’t changed significantly in 150 years is testament to its value, not a sign of its weakness.
The next aspect is to look at why this statement is, while true, also incorrect. There are undoubtedly core similarities between reading now and in Dickens’ time, but there are also very significant differences. For example there have been significant changes in:
So any statement that merely says nothing has changed would not recognise that reading in 2015 is a very different experience to what it was in 1840.
If we now return to the industrial schooling argument we see a similar pattern. Firstly, there are significant similarities, so the statement is true in some respects. If you looked at education now and in the 1900s there are some things you would recognise: we send children to a central place, we group them by age and ability, we have teachers. As with reading, this unchanging aspect might be because it’s a good thing. Whenever I hear people state that they want to revolutionise (or do away with) the school system, I am struck by their lack of a viable alternative. If you want to educate all children in your country, regardless of motivation, ability, parental engagement, etc then you need a robust system. If you completely started from scratch tomorrow, my bet is you would end up creating something that didn’t look dissimilar to a schooling system. So the absence of change so deplored by many may indicate that viable alternatives are not available.
The second aspect is to consider what is wrong with the statement. As with the Dickens example, it actually ignores many significant changes, including:
When you take these into account, the schooling of children in 2015 is nothing like that of kids in 1915. Now this is not to suggest that there aren’t significant changes we can make within the schooling system. The Finnish approach is often cited as having a better attitude to assessment, curriculum, grouping, pedagogy, etc. And too often the education system is subject to the whims of whoever is the education minister (for example, the disastrous Michael Gove, who seemingly did want a 1915 system). And this is what irritates me about the industrial schooling argument – as so often with tech driven approaches, it demands wholesale revolution, instead of focusing on doing practical changes within the system which would actually be useful.
I’m learning that if you call something existing by a new name, or if you get some press, you can discover well defined concepts and claim them as your own. Today’s example: Arizona State and edX Will Offer an Online Freshman Year, Open to All
The project, called the Global Freshman Academy, will offer a set of eight courses designed to fulfill the general-education requirements of a freshman year at Arizona State at a fraction of the cost students typically pay, and students can begin taking courses without going through the traditional application process… Students who pass a final examination in a course will have the option of paying a fee of no more than $200 per credit hour to get college credit for it.
So, for $200 a credit hour ($600 for a 3-credit course), you may well pay more than you would at a small college. The fees charged then are not innovative or game changing. The idea of open access? Oh, well the OU started that in the 1960′s: Brief History of OU.
The only innovation here? Marketing & PR.
Once systems like ASU, who have launched some innovative ideas over the past decade, start looking at what has been done in education and what is known about learning, and then launch a legitimately new idea, rather than playing a PR game, we may have the prospect of substantial educational change.
I went to an excellent presentation from Cable Green yesterday about the K12 OER Collaborative. The project is aiming to get states to some of the money they currently spend on buying text books from publishers to produce open ones. He highlighted very forcibly what a crappy deal we currently have in that books are often very old (because they can’t afford to update), children are not allowed to do anything useful like take notes in them (because they have to be passed on), and if you lose one, the parents have to pay to replace it (which results in some parents telling their kids not to bring the book home). And on top of this, it’s really expensive.
So what they did was for a fraction of the cost currently allocated to purchase books they put out a call for companies to create new ones, but crucially, these would be openly licensed. This means that a) the digital copy is free, b) the state owns the rights so can update and adapt as they want and c) they can match specifically to common core. While the big publishers boycotted the call, many smaller ones responded, as did university departments. A million dollars (say) may not mean much to Pearson, but to a small company it’s a decent sum of money, even if there is no further revenue had then on sales.
The finances are truly staggering here, at the moment the state (he was talking about Washington state) can afford to update two books a year. When you consider the range of subjects and the age ranges, that means a lot of set books are out of date before it’s their turn to be updated. For the same money to update 2 books a year, using the open approach they could create open textbooks for ALL subjects. And these would of course, be usable across the whole of the US, not just in one state. And they would have money to pay people to regularly update the books. And they’d still have change left over.
When this is laid out you realise, that much like the academic publishing model, the current system was devised when ownership resided with the physical artefact. It now looks ludicrous. I do think we will look back in years to come and think “how did we let it go on for so long?”. I don’t know what the figures are for buying UK textbooks for schools, or how the process works, but the same approach would surely work here. In the US the figure is $8 billion nationwide, and the K12 OER project reckons it could do it all for around $30 million. Imagine what that extra money might be spent on in education.
It reinforced to me an obvious point, but one that bears repeating – ownership is key here. The real reason education boards spend millions of dollars in buying textbooks is not because the publishers have specialised technology or skills anymore. It is because they own the rights to the content. Once you break that link, then all sorts of possibilities open up.
Here is Cable’s slidedeck:
I’m at OEGlobal this week and attended a session from Athabasca University Library this morning. They were talking about how they have gathered together open access resources under the subjects for their uni, and also gathered in open resources from elsewhere. You can access this open access collection at their site. I think more libraries should do this, prioritising open resources so everyone can access them.
But what struck me in their presentation was that MOOCs were quite a significant driver in doing this. For many university libraries collecting open access resources doesn’t really matter as the fee paying students will have access to those resources anyway (if the library can afford to pay for them). And so there is no real driver for educators to focus on OA above other resources. But when people started creating MOOCs, this breaks down – your open learners won’t have that privileged library access, so any resources you use must be open.
This is similar to the manner in which social media drives open access also. What it highlights is that openness in any form begets openness. So while we may sometimes bemoan that MOOCs themselves are not really open (in the sense of openly licensed), they do form part of a larger system, which helps drive openness. I expect you’d all realised this long ago, hadn’t you?
Last week was the OER15 conference here in Wales. I was the co-chair along with Haydn Blackey. While my view may be somewhat biased, I think it was a great success. We had great sessions, everything worked well, the venue was marvellous and the sun was out in Cardiff. If you haven’t been to the Uk OER conference before, I recommend getting along to Edinburgh next year for OER16. I was, as is so often the case, reminded very forcibly of how enthusiastic and engaging the open ed community are.
The theme of the conference was “Mainstreaming OER”. I suggested in the opening remarks, that it wasn’t the case that OER are already mainstream practice, but that they now stand on the cusp of it. After 13 years or so of development, a global community has been developed who are focused on OERs, open textbooks and open education in general. But the next stage is to move into the mainstream. There is almost nowhere else left to go now. That transition may not be successful, and it isn’t inevitable, but it is the next phase we need to attempt, in order to realise much of the ambition that underpinned the OER movement.
Often conference themes are rather vague, and don’t really bear any resemblance to the actual sessions. They’re rather like having a theory of parenting – you think it will go one way, and reality trundles along regardless of your interventions. But I feel that the theme of mainstreaming OER was really very relevant to the content of the conference. All of the keynotes explicitly addressed it, and in all the sessions I attended, participants made it a key thread in their work.
This caused me to muse somewhat on the nature of change (especially in higher education). I read Elizabeth Kolbert’s The Sixth Extinction the other week, and she talks about the nature of extinction and its time frame. Darwin and others believed species go extinct very slowly (the winding down of a natural selection process). But of course, we discovered that sometimes extinctions happen quickly, caused by major events (the dinosaur slaying K-T event being the most famous). As Kolbert puts it, “conditions on earth change only very slowly, except when they don’t”. Or as paleontologist David Roup sums it up, evolution is “long periods of boredom interrupted occasionally by panic.”
Related to this, my colleague Simon Horrocks pointed me to the French historical tradition of la longue duree. This argues that we shouldn’t focus on the big events in human history, but rather on longer cycles. While we tend to talk of significant battles and revolutions, the ideas or regimes these have overthrown persist for much longer. This is in line with the theme of my book, that having had the initial victory, it is actually now that direction is determined.
Which brings me back to the theme of OER15. I think change in higher ed has some resemblance to the evolutionary pattern (although over much shorter timescales) – change happens very, very slowly, and then very, very quickly. At the same time there are also longer patterns of change beneath this. For example, one might argue that MOOC hysteria was an example of one of those moments of panic. But this occurs within longer cycles – for example, the trend towards openness might be one, but so are much more fundamental practices such as knowledge construction, autonomy, critical thought, etc. I would suggest that silicon valley and the media are almost exclusively focused on those moments of panic, but ignore the equally important longer processes. In terms of OER then, I would argue we need to embrace both – be prepared for the long haul, but ready to react when the rapid change comes.
Here is a nice playlist of all the keynotes from OER15, and also an overview video:
At the Hewlett Grantees meeting in San Francisco, David Wiley made a very good argument that we need to focus on specific problems that OERs can address and solve those. I think this is part of the mainstreaming process – at the start of the movement there are grand claims and big visions. These are necessary to get it going, but over time and with further investment the focus becomes more practical. So, reducing the cost of textbooks for students in higher ed is one such specific problem. We can show how OERs (in the form of Open textbooks) can achieve this, we can implement an approach to solve it, and we can measure it.
We also discussed whether there are universal problems which OER can solve. David suggested that actually problems often look superficially similar, but there is such variety in context that they are actually very different problems. The situation in North America is different to that in Europe, and that in the UK is different to that in France, and that in K12 is different to higher ed, etc.
I would contend that there are some problems which, if not quite universal, are similar enough to be of interest to a wide range of people. If we take the original premise that we need to focus on specific problems, then the next stage is to find problems of sufficient interest. Here are some which our OER Research Hub findings point to, but these are just some suggestions, and will undoubtedly be influenced by my higher ed, northern hemisphere perspective, so I’d love to hear more:
There are obviously more, but that wouldn’t be a bad set of problems to both solve and to investigate fully. But I definitely feel that these targeted benefits allied with appropriate research is what is required in OER now.
I am at the Hewlett Grantees meeting in Sausalito this week, and last night they showed the film The Ivory Tower, in order to provoke discussion around what relevance OER had to the issues raised in the film. I’d seen it before, on a plane, and it had vaguely irritated me, but it was interesting to see it again last night, when it really irritated me.
I think a documentary film about how we fund higher education is an interesting thing to do, but this one jumps around all over the place. It suggests that the fault of high education costs lies with the university. It is not a film about how society funds higher education. For instance it only looks at the US. If you were interested in the topic of higher ed funding you would look at other countries with different models. As I’ve said before, if you make higher education a market, you shouldn’t then criticise universities for behaving in a perfectly logical way to succeed in that market. The film makes a big play on universities having climbing walls and fancy buildings, but if these attract students and money in a competitive market, then that they are inevitable. It doesn’t take the next step and make the discussion about funding in general, but rather says we should look at what universities are doing and whether education is now a good investment.
It also offers some of the alternatives that were popular a few years back, including UnCollege and, of course, MOOCs. The whole MOOC section just seems deeply embarrassing now. There is a definite ‘these will sweep away unis’ feeling, and they give the pre-pivot Thrun full rein. No-one making a documentary in 2015 would present MOOCs in such a light (which is not to fall into Good vs Evil Unicorn territory, not to say you couldn’t have an interesting doc about them). And this I think is the problem – for OERs they need to avoid getting caught up in any of the rhetoric that will date quickly. Instead, as David Wiley likes to propose, focus on particular problems and solve those. OERs don’t need to mean the end of university, but they might help with the high cost of textbooks. OERs don’t need to create an UnCollege program, but they can help students pick the right course by studying before they choose, and then help them complete by supplementing study when they’re in university. And so on. These benefits aren’t as glamorous and may not get you a documentary made, but they are actually useful.