Developing a context-general self-report approach to measure three-level situation awareness

Bjørn Sætrevik

04_IMH_2013_2_Saetrevik

ORIGINAL PAPER

Developing a context-general self-report approach to measure three-level situation awareness

Bjørn Sætrevik

Department of Psychosocial Science, University of Bergen, Norway

Abstract

Background: Situation awareness (SA) is considered to be crucial for work in safety critical organisations, yet its precise definition and an agreed upon measurement approach have yet to emerge. SA is often measured as an operator’s overview of some specific parameters within a given work setting and a given time frame, an approach that entails both advantages and disadvantages. The current approach examines whether some aspects of SA relating to workplace safety can also be captured in a context-general inventory.

Material and methods: 166 offshore maritime personnel answered the SA inventory with 13 items describing the respondent’s typical cognitions concerning safety issues.

Results: Confirmatory factor analysis of response patterns showed that the internal pattern among the items reflected the three level structure predicted by the leading theoretical model. Strengths and weaknesses of the inventory itself, as well as the approach in general are discussed, and future research directions are outlined.

Conclusions: It appears feasible to measure aspects of SA in a context-general inventory, though additional adjustment and validation is required.

(Int Marit Health 2013; 64, 2: –71)

Key words: situation awareness, safety, questionnaire development, confirmatory factor analysis

Bjørn Sætrevik, Faculty of Psychology, University of Bergen, Christies gate 12, 5015 Bergen, Norway, e-mail: satrevik@gmail.com

INTRODUCTION

The term situation awareness (SA) refers to the knowledge an individual or a group of individuals have about the context they operate in. SA has been defined as “accessibility of a comprehensive and coherent situation representation which is continuously being updated in accordance with the results of recurrent situation assessments” [1]. Essentially, SA relates to knowing what is going on around you [2], and is a concept that appears to resonate with practitioners trying to enhance safety and efficiency in a variety of work contexts. In particular within the field of industrial safety, where an incorrect or insufficient comprehension of the situation can lead to large-scale accidents, SA has received extensive research attention, and is often the focus of workplace interventions. In a survey among offshore installation managers [3], lack of care and attention was cited as one of the main causes of the accidents. In interviews with offshore drill crews, Sneddon et al. [4] found that isolation, fatigue and stress were assumed by the operators to decrease SA, and character change was seen as an indicator of reduced SA.

The dominant view of SA in research and among practitioners has come to be Endsley’s model of SA [5] as composed of three hierarchical levels of knowledge about the environment. In this model, level 1 SA consists of perceiving the relevant factors of the environment, level 2 consists of synthesising the information from level 1 into a coherent and comprehensive view of the situation, and level 3 consists of being able to use information from level 1 and 2 in order to project what the environment will look like in the near future. In a maritime setting, instruments and charts may give you the ship’s position and course, location of the coastline and other traffic (level 1), from which you create a mental image of the ship’s course and speed relative to the surrounding area (level 2), and you can tell that a collision will occur if the ship continues on its current course (level 3). For Endsley, SA formed the basis for decision-making, and the actions decided on would influence the environment and thus feed back into the system. In an analysis among offshore drill accidents [4], 67% of the incidents that were said to be caused by human error were classified as a failure at the level of perception (level 1 SA), 20% were classified as a failure at the level of comprehension (level 2 SA) and 13% were classified as a failure at the level of projection (level 3 SA).

MEASURING SITUATION AWARENESS

A number of different approaches to measure individual SA has been suggested [6–8]. Indeed, it may be argued that too many SA measures have already been proposed, and that further research should focus on validating the existing measures rather than suggesting new ones [6]. Two main classes of SA measures are subjective and objective SA measures. In subjective measures, respondents are asked to give their own assessment of the degree to which they have an overview of their context. One of the most frequently used subjective SA measures is the situation awareness rating technique (SART) developed by Taylor [9]. In this approach, responders are queried for their understanding of the situation and the demand and supply of cognitive resources. Subjective measures are typically given during or immediately after a simulation or training exercise, so it is clear for the respondents which setting and which time period the questions refer to. Subjective SA measured during task performance (i.e. “freeze probes”) will inevitably intrude on and influence the primary task performance. Further, subjective SA measures depend on the respondents’ recall of and insight into their own cognitive states. Endsley [10] has argued that subjective SA measures may actually show the respondent’s confidence in their SA, rather than the mental representations of interest.

In objective SA measures, the respondent’s knowledge of the environment is compared to a “ground truth”. In the situation awareness global assessment technique (SAGAT) developed by Endsley [11], an in-depth cognitive task analysis is used to develop specific questions adapted to the current setting. A simulation or exercise is frozen at regular intervals and the operator’s knowledge of the situation is assessed based on the correspondence between responses and pre-defined correct answers. In a military exercise, a soldier may be asked where the enemies are (level 1), what they are doing (level 2), and where they are heading (level 3), and the answers are compared to the researcher’s knowledge of the enemies’ actual behaviour in the exercise. The quantitative analysis of situational awareness approach (QUASA) [12] consists of true/false queries about factual relationships, where each answer is also given a confidence rating, and in this way combines objective and subjective approaches.

Thus most subjective and objective SA measures are necessarily tied to cognitive processes or products that arise in a given context or situations, and some researchers would argue that the concept of SA must necessarily be measured in a context-specific way. One downside of using context-specific measures is that findings in one study cannot easily be generalised to apply to other domains. Further, different studies examining SA and its theoretical implications are difficult to compare between various work settings or to include in meta-analyses. If context-general measures of SA could be used, this may have advantages in terms of data collection efficiency and generalisability. A brief inventory measuring SA aspects independently of context can be included in the large-scale data collections to get a sense of employees’ confidence in their ability to safely handle a variety of relevant situations in their work. Such measures can in turn be compared to actual accident rates or recorded near-misses. In settings like the maritime industries where registered accidents may be relatively infrequent, but of critical importance, such an approach can be applied broadly and correlated to the actual incident rates. This may be preferable to a more in-depth measure of SA in smaller samples, which has only a vague relationship to actually occurring incidents. A survey approach makes it easy to compare SA between data collected in different work settings within an organisation, or between different organisations. This is not to argue that all aspects of SA can be captured in such a measure, and context-specific SA measures will remain necessary to study causes, effects, temporal and interpersonal variation in SA.

THE CURRENT STUDY

In an attempt to relate SA to broader concepts in cognitive and organisational psychology, the current research examines whether SA can be measured in a survey approach, where context-general items aim to tap into some of the cognitive aspects related to SA across the domains. A long-term aim of this approach will be to implement context-general SA in larger motivational models of safety behaviour. The research questions for the current study were whether SA can be reliably measured by using a context-general survey, and whether three separate factors corresponding to Endsley’s three levels [5] could be identified. Two confirmatory factor analyses (CFA) were applied to examine whether the responses to the 13 SA items correspond to a structure of three factors corresponding to three levels the items are intended to measure, or whether it corresponds to a one factor solution. If Endsley’s model is correct, the response pattern should show better fit with a model where the items are structured around the three levels, than with a model where all the items are structured on the same level.

MATERIAL AND METHODS

INSTRUMENTS DEVELOPMENT

A deductive approach was used to suggest a pilot inventory of eighteen items, six items to measure each level of Endsley’s three-stage model of SA [5]. The items thus asked the responders how they are able to perceive relevant information (level 1), understand what was going on (level 2) and anticipate future developments (level 3). The instructions asked responders to relate the questions to what cognitions they “usually or typically” had in their work, and the items used wording that could be relevant across different work settings. The items were scored on a five-point Likert scale, ranging from 1, “Completely disagree” to 5, “Completely agree.”

The pilot inventory was tested on a sample of 66 naval cadets after they completed an unfamiliar team task in a 2–3 hour-long computer simulation of navigating a naval vessel in a conflict situation. They were asked to respond to the items based on their experiences throughout the simulation, and were also invited to comment on the phrasing of the items. Quantitative analysis of the pilot study (correlations and exploratory factor analysis, not included in this paper), led to the removal of 5 items. Qualitative analysis of responders’ feedback led to rephrasing of the remaining items. To fit our research interest, the inventory was further rephrased to focus on the safety in job execution rather than on performance (Table 1).

In the inventory used in the current study, 4 items were related to the perception of safety issues, which were intended to reflect level 1 of the Endsley’s SA model [5] (e.g. item 10: “Some of the information I need to assess safety is presented in a way that makes it difficult to understand”). Five items were related to comprehending the safety situation, intended to reflect level 2 of the model (e.g. item 6: “I know which information is relevant for safety and which information is not relevant for safety”). Four items were related to projecting the safety situation into the near future, intended to reflect level 3 (e.g. item 1: “I notice when an unsafe situation is about to arise at my workplace”). An English version of the inventory can be found in Table 1.

Table 1. Text of the 13 items of the proposed context-general situation awareness inventory, with the corresponding level in Endsley’s [5] model added in parentheses. Negatively phrased items are marked with an asterisk. Mean and standard deviation (SD) from 166 Norwegian marine personnel are shown

Item number	Item text	Mean	SD
1	I notice when an unsafe situation is about to arise at my workplace (level 3)	4.28	0.75
2	I sometimes lose track of information relevant for maintaining safety in my work (level 1)*	4.02	1.00
3	It’s hard to know which consequences my actions have for safety (level 2)*	4.39	0.84
4	I sometimes lose track of safety due to receiving too much. information at the same time (level 1)*	3.93	1.04
5	I plan ahead in order to handle various adverse incident that may arise (level 3)	4.40	0.70
6	I know which information is relevant for safety and which information is not relevant for safety (level 2)	4.33	0.67
7	It is impossible to predict what will happen during an adverse incident (level 3)*	2.81	1.01
8	I know how to act to maintain safety (level 2)	4.48	0.59
9	I feel confident that I know how to deal with the various adverse incidents that may arise (level 2)	4.20	0.60
10	Some of the information I need to assess safety is presented in a way that makes it difficult to understand (level 1)*	3.23	1.09
11	I usually know what’s going to happen next with regards to safety (level 3)	3.44	1.01
12	The information I need to assess safety is easily available (level 1)	3.70	0.86
13	I know which situations in my work involves higher risk than others (level 2)	4.64	0.53

DATA COLLECTION

Data were collected in a large-scale survey on work safety and psychosocial environment factors for personnel in an international shipping fleet. The surveys were distributed by company contacts during the vessels’ voyages, and the responders were asked to complete the survey before the end of their stay on-board, and return it by mail directly to the research institution using provided addressed envelopes. Forms were distributed in both Norwegian and English versions, and the responders could choose which version to return. The surveys were distributed to 817 crew members on 31 different vessels. 594 questionnaires were returned, yielding a response rate of 73%. As language and other cultural issues were suspected confounders among the English version responders, as well as the Norwegian speaking part of the crew holding more safety critical positions on board, only data from the Norwegian questionnaires (from Norwegian crew and officers) were included in the analysis. The final study sample comprised 166 responders with an average age of 42.1 ± 10.7 years, of whom 97.6% were male and 93.4% had a permanent position in the company. Compared to all the returned questionnaires, higher ranking officers were overrepresented in the Norwegian sample, with 75.3% of the respondents having ranks of captain (22.3%), master (22.9%), chief engineer (14.5%) or chief officer (15.7%).

DATA ANALYSIS

The items with negative wording (items 2, 3, 4, 7 and 10) had their scores inverted, so that higher scores on all items indicated a higher degree of SA. Mean values for SA and safety climate subscales were calculated. The dataset was examined for applicability of CFA. A CFA with estimation of means and intercepts was performed in Amos [13], using a model where each of the survey items were associated with a factor corresponding to the SA level predicted by the theory. In order to test the applicability of the model against a null hypothesis, a CFA was also performed using a model with all items associated with a single SA factor.

RESULTS

Mean values and standard deviations for responses on the 13 SA items are given in Table 1. Cronbach’s alpha between the items was 0.74, indicating good reliability. The Kaiser-Meyer-Olkin measure of sampling adequacy was at 0.828, and Bartlett’s test for sphericity yielded an approximate Chi-square of 465.5, significant at p < 0.001, both of which indicate that the data are suitable for factor analysis.

A CFA was performed with all 13 items modelled to one of the three theoretical SA factors, perception, comprehension and projection, as indicated in Table 1 and Figure 1. The analysis showed good fit measures, with Chi-square = 86.94, df = 62 (Chi-square/df = 1.4), p-value for model = 0.02, CFI = 0.936, TLI = 0.906, RMSEA = 0.049, with upper boundary for two-sided 90% confidence interval at 0.072 and a PCLOSE value at 0.495, indicating that the data have a close fit with the model (n = 166). The standardised estimates for factor loadings can be found in Figure 1. Due to a low number of items on each factor, average inter-item correlation was used rather than Cronbach’s alpha for subfactor levels. Average inter-item correlations showed 0.261 for level 1 SA items, 0.237 for level 2 SA items and 0.127 for level 3 SA items. Exploring alternative models where items were removed or moved to other factors did not significantly improve model fit or factor loadings.

Figure 1. Standardised factor loading estimates in confirmatory factor analysis of 13 context-general items intended to measure Endsley’s three level model of situation awareness [5]

Another CFA with all 13 items modelled to a single SA factor was also performed. This analysis showed overall weaker fit indices that did not meet the traditional thresholds for good model fit: Chi-square = 116.03, df = 65 (Chi-square/df = 1.78), p-value for model = 0.001, CFI = 0.869, TLI = 0.817, RMSEA = 0.069, with upper boundary for two-sided 90% confidence interval at 0.089 and a PCLOSE value at 0.066.

DISCUSSION

The current approach developed 13 inventory items in a theoretical approach based on Endsley’s model for SA [5]. A CFA of the inventory responses from a sample of Norwegian marine offshore personnel against a measurement model where each item was mapped onto three factors representing the expected three levels of SA showed good indices for the model’s fit. A competing model with only one SA factor did not meet the threshold criteria for good fit with the data. This indicates that the pattern of responses to the queried items did indeed reflect the structure predicted by Endsley’s model. This can be taken to indicate that despite questions being raised regarding the SA model’s internal logic [e.g. 14, 15], the model appears to provide meaningful structure to the responses in the current sample. However, the finding that responses structure around the three intended factors does not necessarily imply that they represent three separate cognitive structures or processes, as it could also be due to commonalities in the phrasing of the questions, or due to the responders’ culturally imbued expectancies.

The average inter-correlations between each factor indicated that the factors may be causally related, as would be expected from the theoretical model. Admittedly, the inter-correlation within each factor, and in particular for level 3 SA could have been higher. There is also a high correlation between level 2 and level 3 in the CFA. This may reflect an ongoing theoretical discussion of whether level 2 and level 3 in Endsley’s model can really be separated on a cognitive level [15].

Some of the inventory items showed below optimal factor loadings. This is particularly true for item 7, which was mapped to level 3 and was phrased “When an adverse situation arises, I find it hard to know how it will play out” in the data collection. In the CFA results, this item stands out as having a particularly low and negative factor loading, and one could argue for removing this item from the inventory. However, the item is intended to tap onto a potentially important aspect of SA, as mental simulation of future outcomes is thought to be crucial to problem solving. Further, removing the item would leave only 3 items for the level 3 factor, while level 1 and 2 have 4 and 5 items, respectively. Thus, in order to achieve better factor loading in future data collections I would recommend rephrasing the item to “It is impossible to predict what will happen during an adverse incident” to make the item’s referent easier to understand and to have a clearer directional prediction from SA. Due to low factor loadings, rephrasing or removal could also be considered for items 10, 11 and 12.

Whether the current approach could be a valid measure of SA to some extent depends on whether one allows that some aspects of SA may be context-general. If SA is associated with cognitive capabilities, one may expect SA to show some extent of consistency across different situations, in the same way that general mental ability has been shown to be a good predictor of job performance in different domains, in particular for complex jobs [16, 17]. Further, many theoretical models attempting to understand naturalistic decision-making or factors that predict safe behaviour, tend to focus on individual differences rather than situations. An example of this is the model proposed by Eid et al. [18] for how safe behaviour is determined by individual factors such as the operator’s psychological resources, opinion of the closest leader’s leadership style and assessment of the safety climate. Central to this and other models of safety, is the concept of “safety climate” [19, 20], an individual variable that measures how the respondent experiences the safety relevant attitudes and actions of oneself and others. If it is of interest to incorporate some aspects of SA into the current models of how individual and situational factors influence safety, it would make sense to measure SA on the same level and through the same methods as the other factors in the model. Thus the current approach to measure some aspects of SA in context-general self-report surveys could be of value.

A context-general SA inventory like the one suggested in the current study would be expected to have the same weaknesses as other self-rating techniques of SA, such as SART [9] and QUASA [12], e.g. to be subject to the respondent’s recall and that the measurement may be confounded by the performance feedback. As opposed to other self-report measures, the current approach does not collect responses immediately after a given event or exercise. On one hand, this may decrease the extent to which the responses represent specific cognitive events, and may increase the impact of factors such as confidence and attitudes. On the other hand, asking the respondent to generalise across different experiences may be argued to represent a more reliable measure of the typical work setting and be less subject to recall artefacts.

The suggested approach presents an efficient and economical way to measure some aspects of SA, as it does not require specific exercises to be arranged. This allows one to perform simultaneous data collection for a large number of respondents (e.g. companywide) in their natural environment. As a practical application, the inventory could be used to identify sailors or subgroups of sailors with low scores on context-general SA, which could indicate that the personnel were concerned or dissatisfied with their own ability to handle safety aspects, or with their work environment’s allowances for enabling safety-related SA. This could serve as a warning signal for the organisation to adapt the safety management systems to mitigate the problem. Further, the SA measure can easily be combined with survey measurement of other relevant background variables and outcome variables. This may contribute to bridging the concept of SA with safety culture [21] in survey measures to allow for extensive modelling of several factors of interest, e.g. in a structural equation model. A similar approach has recently been suggested by Sneddon et al. [22], where a context-general inventory for “work situation awareness” has items concerning vigilance and paying attention, and is associated with stress, fatigue and sleep quality.

As mentioned above, context-specific approaches to measure SA will still be the preferable approach for in-depth analysis of SA mechanisms and affordances for specific settings. In order to determine causal factors behind SA, experimental studies should be conducted, where individual, team or environment factors are manipulated and objective SA is measured with context-specific SAGAT-like instruments [11], where the individual’s factual knowledge about the constructed scenario is tested [e.g. 23].

CONCLUSIONS

The current study suggests that it may be useful to measure some aspects of SA with a context-general inventory. While there may be a theoretical misalignment between the current measurement approach and more conventional approaches to SA, there are potential advantages in allowing large-scale data collections and comparisons between different domains and samples. The current analysis identified some potential weaknesses in the inventory construction (see above). On-going research is in the process of testing an adjusted version of the inventory, and correlating it with various predictors and outcomes. Optimally, context-general approaches should be tested against more conventional SA measures in field settings, and preferably against objective SA measures or measures of safety outcome. A developed and validated context-general SA inventory could be used in larger research efforts to investigate to which extent and through which mechanism e.g. the sailor’s personality, the ship-owner’s safety management system and the captain’s leadership style influence the individual SA.

ACKNOWLEDGEMENTS

I would like to thank all the responders to the questionnaire for their participation, and Jørn Hetland and Sigurd W. Hystad for their assistance in the early analysis work. This research was supported by the Norwegian Research Council’s PETROMAKS Program (grant number: 189521).

REFERENCES