July 2000 Experiment at NPMOC, San Diego
|
Experiment Overview
|
|
Experiment Overview
|
|
EXPERIMENT GOALS
|
|
Based on the experience of the first METOC human systems experiment, which was conducted in San Diego in July 1999, a follow-up experiment was planned for July 2000. A key difference was the inclusion of ship-based forecasters. Additionally, a task analysis provided by a civilian weather forecaster (i.e., a subject matter expert) was included in the plan.
|
|
EXPERIMENT HYPOTHESIS
|
|
Although the analysis of the data collected during the 1999 experiment was still ongoing, the following hypotheses were devised to help focus the 2000 experiment.
It was hypothesized that:
|
|
- The production of an air strike brief can be described by a workflow or description of sequenced events
- The forecast can be improved through the use of certain computer tools
- The forecaster develops a qualitative mental model from which a quantitative forecast is constructed
|
|
Experiment Narrative
|
|
Three teams of forecasters and technicians were observed building a mission planning brief and then later a strike briefing for simulated strikes on Port Angeles, WA and the bridge from the mainland to Whidbey Island, WA. A total of six sessions were conducted. In addition, an "expert" from the center at NPMOC was observed preparing a strike brief. Two of the teams were from the U.S.S. Constellation, which was in port in San Diego following a recent deployment. One team was from the METOC center at NPMOC. The NPMOC Science Officer played the role of Intel by providing pre-scenario briefings and on-call support during the exercise. Another NPMOC officer played the role of "regional center" for weather related questions typical of requests received at a center such as NPMOC. He also played the role of chief scientist with respect to questions concerning the use of numerical METOC models and TDAs.
At the conclusion of each session, the forecasters presented their brief to the test conductors and answered questions about their brief and the methods used to construct the brief. Between 12 and 24 hours later they returned for a recall session, where they were asked to present the brief from memory without the aid of the presentation slides prepared during the session. The planning document used for the experiment is available in Appendix D. Each trial had four participants. The Intel Officer gave the description of a strike scenario. The first three trials had the same strike scenarios. The last three trials had another scenario. During the experiment an extra trial performed by the expert forecaster was added. He gave the most productive session.
|
|
Notes and Lessons Learned
|
|
GENERAL COMMENTS
|
|
- The professional experience level of METOC personnel varied widely. The forecasters in this experiment had between 6 to 12 years of METOC experience, while the technicians had 2 to 3.5 years of experience.
- All teams were able to follow the strike scenario that was provided to them.
- Forecasters and technicians work as a team, not independently. Forecasters provide forecast information for TDA input. Very few interactions occurred with the regional center (even with encouragement).
- Technicians were not familiar with the application programs (e.g., TAWS) and setup. Forecasters had to tutor them.
- Conversion of units caused a lot of problems.
|
|
COMMENTS ON METHODOLOGY AND PROCEDURES
|
|
- Some of the forecasters wore eye-tracking headgear. Forecasters stopped for a break every 45 to 60 minutes so they could rest their eyes.
- Two note takers (one for the forecaster, one for the technician), and two MacSHAPA recorders were used for real-time task performance recording.
- Three videotape recorders were used. Forecaster's screen was captured by the eye-tracker videotape. Technician screen and interaction with the forecaster were captured on two individual video recorders.
- The Intel and regional centers were available during all trials.
- Product templates were provided to all forecasters.
- Observers were struck by the magnitude of the problems associated with unit conversions (nautical miles and meters/feet/yards; temperature degrees; latitude/longitude decimal degree to minutes) and time conversions (local to GMT). Every team was affected by the problem. Some did not recognize that the required inputs needed conversion.
|
