Using the Safety Culture Discussion Cards to help understand textual data

‘What we call our data are really our own constructions of other people’s constructions of what they and their compatriots are up to’ (Geertz, 1973)

Probably the most common approach to trying to understand safety culture is via safety climate questionnaires, usually comprising a set of items with a Likert-scale to indicate the level of agreement with each item. Unfortunately, such questionnaires alone do little, if anything, to help understand the meanings that people ascribe to their values, beliefs and behaviour, and so do not explain why we do things, why we do things in the way that we do them, or the conflicts between what we say and what we do. To gain a deeper understanding, a qualitative, interpretive approach is more fruitful, not necessarily to supplant questionnaires, but at least to supplement them. Prior to interactive methods such as focus groups and interviews, one source of data from the questionnaire itself can be a useful starting point to an interpretive approach - the free-text comments written by the respondents.


I recently used the Safety Culture Discussion Cards to help analyse several hundred typed/written unstructured comments from a safety culture questionnaire - a fairly large amount of textual data. Many of the comments were several paragraphs long and referred to a variety of issues, and were mostly very interesting, well thought out and well-written. Making sense of rich textual data is never easy. But a common approach to understanding is via 'content analysis' (Krippendorff, 2004), or textual analysis. This often involves reading the text and applying a set of codes or categories to try to understand the data.

In this case, I decided to try to use the Safety Culture Discussion Cards to help code the data. The aim was to get a detailed understanding of the issues that questionnaire respondents were motivated to comment on - the specific issues, the way the writers related issues to each other, and the number of times that each issue was mentioned. An assumption was that issues mentioned more often by respondents reflect concerns that are important to them.

The cards cover most relevant aspects of safety culture but are (deliberately) not mutually exclusive, so this had to be kept in mind during the analysis. Prior to and during the coding, it was necessary to remove or combine cards as appropriate in order to achieve some satisfactory level of mutual exclusivity.



I started the analysis by reading all of the comments very carefully, and coding pieces of text within each comment using the eight elements of safety culture covered by the cards (Management Commitment; Resourcing; Just Culture, Reporting & Learning; Risk Awareness & Management; Teamwork; Communication; Responsibility; Involvement). Because a person's comment could cover all sorts of issues, it is not possible to apply just one element code to each comment. Even a particular sentence within a comment could cover two or more issues, such as 'Management Commitment' and 'Resourcing'. So at this stage, a sentence or paragraph could be coded using one or more elements.

The next stage was to re-read the comments and now apply more specific codes to the various pieces of text. The specific codes relate to the codes on the safety culture discussion cards, from 1a to 8e, noting also where the text was positive/favourable or negative/unfavourable in nature, or sometimes both. Since some of the cards overlap, where a piece of text could be coded using more than one card (and the cards could not reasonably be mutually exclusive) the codes were combined.

The final stage involved rechecking the use of the codes for each comment to ensure consistency and calculating the usage of each code. (An even more rigorous application of this method would involve having independent coders repeat the exercise with all or some of the text, as I and Amy Chung did when analysing comments relating to HF/Ergonomics practitioners' views on barriers to research application; see Chung and Shorrock, 2010.) This allowed the relative frequency of each issue to be determined, and gave an impression of the perceived pertinence of the various issues.

The frequency of each element as well as the top 20 issues were calculated. The quantitative data, combined with discussion of the actual content of the comments, added substantially to the data received from the Likert-scale standard questionnaire items.

A final interesting output from this exercise is the ability to the the cards to visualise the narratives in the comments by mapping the relationships between issues and the possible meanings emerging. This will be the subject of a different blog entry. The exercise also revealed a few issues that are not covered by the existing cards, as well as the issues that are covered by the cards but that were not mentioned at all by the commenters. This is useful feedback for the further development of the cards.

References

Geertz, C. (1973). The interpretation of cultures: Selected essays. Basic Books.
Krippendorff, K. (2004). Content analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
Chung, A.Z.Q. and Shorrock, S.T. (2011). The research-practice relationship in ergonomics and human factors - surveying and bridging the gap. Ergonomics, 54(5), 413-429.

Five questions about boredom, fatigue and vigilance

Below are five questions posed by a safety colleague, and the brief responses.

1. How different are boredom and fatigue?
Both affect our ability to pay attention - to notice something that may need attention - but they are different in terms of their causes and can occur completely independently. A person can be bored during a period of low activity, but not fatigued. Prolonged boredom, tends to result in fatigue, but so can high workload, lack of sleep or disruption to sleep patterns, or stress. Other than sleep or rest, there is little that you can do to manage fatigue effectively while on position, while more can be done to tackle boredom and stay in the loop. So preventing and managing fatigue is a key priority to ensure that people remain able to deal with unusual events.

2. Is low workload more dangerous then high workload?
Attention is stretched by both 'overload' and 'underload'. Both require hard work and can be stressful, particularly if there are safety consequences when something is missed. Which is more dangerous will depend on the situation and the person (for instance personality, experience and levels of stress and fatigue), but skilled professionals tend to cope better with higher workload up to the point of overload, when performance degrades more dramatically.

3. How to remain aware and vigilant for unusual situations?
Ask colleagues - people develop different visual and mental strategies that may not be obvious from the outside. But applied research using eye movement tracking gives some tips in terms of scanning. So-called "active scanning" can help to counteract degraded vigilance under low workload situations. With active scanning, people scan displays proactively in sequences or cycles depending on the traffic situation, linking specific information from different information sources. The scanning is more strategic, and helps to anticipate developing situations.

4. When are we most and least vigilant?
In a non-shiftwork environment we could highlight some times of day when we are least alert, especially during the very early morning hours, but shiftwork is a fact of life for many workers working a 24-hour operation. What we can say is that we are most vigilant when well rested, engaged and interested in the activity, not distracted (e.g. TV, radio, visitors) or preoccupied with other thoughts, well hydrated, and well supported by colleagues and supervisors.

5. Is the theoretical human performance knowledge adding value?
Yes, but not nearly as much as it should. So much is known about human performance that it seems that policy and practice are decades behind. But so much that is published is irrelevant to complex systems and activities, does not offer solutions, and technology and practices change fast and do not wait for research to catch up. Much theoretical knowledge in human factors comes from sterile experimental environments, normally focusing on one issue (e.g. vigilance) while 'controlling' (or ignoring) some of the most relevant real-life issues that interact to shape performance in the real world (e.g. motivation, risk, teamwork, supervision, background shift-fatigue). The hard part for practitioners is evaluating what aspects of the research are relevant, piecing them together and drawing out practical implications. With this in mind, the most directly useful human performance knowledge is gained by spending time with end users, listening to and observing them at work, and working with end users and other stakeholders to find solutions to human performance issues.