My career has been spent in education and in this blog I say a few words about how theory has / has not played a role for me and to offer a partial resolution to some of the challenges I have encountered. I start by saying I have always found the idea of social theory perplexing  and over the course of my academic career I can point to phases in the way I have thought about theory. Recently I felt I have reached a partial resolution to the problems I have faced.
Theory as redundant … or not
My early career was spent teaching. Much of the theoretical work about schools and schooling that I had enjoyed in my own degree  turned out to be unhelpful to me as a teacher. Instead I found a home in action research as it was pragmatic, optimistic and began by addressing problems of practice. I was aware that literature was helpful and action research was itself a theory about change but broader sociological theory was remote from practice and complacent. It would not address my problems and my concerns.
As I shifted role from school into academic departments I started worrying more about explanatory frameworks. For example there was still far too much of ‘associations between variables and learning outcomes’ (loosely and a tad lazily characterized as positivism) without too much discussion as to why these variables should be related and if so in which direction the supposed association worked. In my field I was particularly critical of quasi experimental research on the impact of ICT (of which there was and remains a great deal) as it lacked the theoretical explanation as to why ICT should or shouldn’t make a difference. Perhaps this theoretical gap would have been less obvious if the evidence was overwhelmingly in support of impact, but it was not. In respect to ICT there was a further question which was: why are policymakers so keen on ICT in education? This suggested that theory about policy making might be useful even if such work was rarely accessed by practitioners.
Off the peg theory
I began to be interested in social theory. I had a desire to theorise and like many others I tended to borrow theory – principally the social constructivist psychology of Vygotsky, though, and more ambitiously, wider concepts in heavyweight thinkers such as Habermas. I began to find examples of theory in the education literature that interested me now I was sensitised to it. The key problem was not undertheorisation but the use of ‘off the peg’ theoretical frameworks. We (I think the ‘we’ here are some of my colleagues in the teacher education research community) were unduly reverent towards theory, summed up for me when people would describe themselves as ‘a social constructivist …’ rather than …someone who ‘found it useful to look at learning at times through a social constructivist lens’. I could see why this happened. Few of us had the time to really look in detail at theories or theorists and we could gloss over the difficulties. For example, I worked through Vygotksy’s ‘Mind and Society’ – often used to support the idea of peer collaborative learning, something that I was interested in. You could see why Vygotsky was evoked but if you looked harder there was in fact very little in his work directly about peer collaboration . However it felt too overwhelming to go beyond the surface and creatively adapt from what was given.
What after all was theory?
Theory was tied up for me in the idea of cause and effect, but I had read widely enough to see that this was perhaps only half the case or not the case at all. I was also confused as to whether action research was a theory and / or in what sense grounded theory was a theory.
A partial resolution
I was able to get some internal funding for a project looking at how we could better teach theory in social research. This evolved into a resource for research students on theory and theorising. The best thing about the project was that it involved talking to students and colleagues in different fields about their work. It provided some partial resolution to the difficulties I have outlined above.
Theory was not redundant
Whatever doubts and qualifications the people I spoke to felt about theory they all argued you had to do something that took you beyond the local and sought a general significance. Furthermore without theory you were not explaining what was happening. This did not necessarily mean cause and effect explanation, explanation could as easily be focused on the actions and consequences of actions. But you needed to explain. I came away strengthened in the idea that theory was important.
What after all was theory?
Colleagues could talk very clearly about the role of theories in their work – though a lot of this was tacit. Very few could say here are four or five definitions of theory let me take you through them. I spoke to people who saw theory as cause and effect associations; models of behaviour; ethnographic description of social phenomena; conceptual clarification; and approaches to theorising such as action research and grounded theory. I had spent too long looking for a single take on theory: you could talk about theory in broad terms but you could not unite different traditions and ideas in one all encompassing version. The challenge was to say what we meant by theory and show how this applied in our work and I learnt to stop worrying so much about theory. This did not of course mean that anything goes – there is a responsibility to step back and report on limitations in our field but to do this without preconceptions as far as that is possible. For example educational theory has had a traditional concern with practice and this is its obvious strength but also a limitation. In contrast other disciplines were remote from practice – this is their curse and their opportunity.
Off the peg theory
The problem of uncritically borrowing theory could be addressed, at least in part, by separating out theory (concepts, frameworks, models) from theorising (acts, strategies, ways of thinking) . Theorising, rather than “Theory’ is underdeveloped . If we knew more about theorising we might, for example, gain greater confidence in our judgments and be critical users of theory. Many people I spoke to described experiencing insights (for some best described as ‘aha’ moments) when ‘things fell into place’, for example when several cases of similar behavior could be grouped under a larger concept or when the relationship between two kinds of actions became clear. If we think of theorising as a creative activity then we might apply theory creatively.
My final thought is that it is possible to celebrate rather than run away from the idea of subjectivity in the process of theorisation. Social research might be as much art as science but it is a special kind of artistry based upon an intense, the only word I can use I am afraid is, engagement with both literature and data. In my experience it is striking that natural scientists and mathematicians, or at least those who have been interested in reflecting on the process of research in these fields, are not only willing but enthusiastically embrace intuition and indeed the aesthetics of theorisation. As famously expressed by Polanyi :
The affirmation of a great scientific theory is in part an expression of delight. The theory has an inarticulate component acclaiming its beauty, and this is essential to the belief that the theory is true. No animal can appreciate the intellectual beauties of science.
I am not sure we would talk about theorisation in the same way in social research, but we could. Perhaps scientists feel more willing to romanticise theory as the popular conception (whether right or wrong) is that their work concerns hard facts and the data they are working with have an objective quality. In contrast in social research the data will always be questioned on ground of reliability, validity or whatever; to admit to subjectivity in theorisation just seems to be one more step in a shaky process.
References I found strong echoes in Kiley, M. (2015) ‘I didn’t have a clue what they were talking about’: PhD candidates and theory, Innovations in Education and Teaching International, 52:1, 52-63.  I have in mind books from a sociology perspective such as ‘Learning to Labour’ Paul Willis Paul Corrigan’s Schooling the Smash Street Kids. Both good books but not written for a practitioner audience.  Something I recently picked up in the introduction to Van der Veer, R., & Valsiner, J. (1994). The Vygotsky Reader. Oxford, England: Blackwell.  Krauze helpfully sets out several associations made with theory in Krauze, M. (2016) The meanings of theorizing, The British Journal of Sociology, 67: 1, 23-29.  cf Swedberg, R. (2016). Before theory comes theorizing or how to make social science more interesting. The British Journal of Sociology, 67(1), 5-22.  Not surprisingly C Wright Mills work on the sociological imagination is offered repeatedly as an example of doing theory, as more recently does Umberto Eco on ‘How to Write a Thesis’. Good as these contributions are they, perhaps the recurring references to both illustrate the very restricted range of other reported experiences.  This final section is taken from my own blog at https://mickhammond.wordpress.com/blog-posts/   Polanyi, M. (1983) Personal Knowledge. Toward A Post-Critical Philosophy, London: Routledge & Kegan Paul, pp. 133-135.