The following article originally appeared as a feature story on the NOAA Fisheries, Northwest Fisheries Science Center website
Want to keep eating wild fish? Great, but to prevent our delicious seafood resources from becoming overfished, we must ensure that we manage based on accurate assessments of their population size. To improve this science, known as fisheries stock assessment, experts from all over the world gathered at the Southwest Fisheries Science Center in La Jolla, California. The subject: best practices for dealing with scenarios in which multiple data sources give conflicting evidence, which leads to stock assessment models that cannot give one clear answer for stock status. This was the 3rd annual workshop hosted by the Center for the Advancement of Population Assessment Methodology (CAPAM), an organization designed to improve fishery stock assessment methods and provide education and training in stock assessment modeling. The conversation became technical very quickly, and many unique phrases were uttered, such as when Dr. Chris Francis said “John Schnute comes up with a wonderful likelihood…” More broadly, conflicting evidence is an issue that all scientists contend with: how to give a prediction given the presence of some data (either particular data points or entire datasets) that our models don’t explain well? In some cases these might be true outliers, warped by observation error (e.g., a mistake occurred when writing down the value of the observation), but it also could be that our models are misspecified (i.e., they contain incorrect assumptions, typically over-simplifications).
Thus, the fundamental questions are whether we (1) need a more complex model with increased flexibility to fit all of the data? (2) Should we throw out some of the data? (3) Or perhaps just put less stock (pun intended) in data which we consider to be a misrepresentation? From a philosophical perspective, data are “true” and models are “wrong” (because by definition they are simplifications of reality). This notion indicates that it would not be proper to get rid of data (2) that aren’t well explained by the model, because, as Dr. Mark Maunder pointed out, you will still get the wrong answer if you have the wrong model. Making models more flexible (1) opens up the question of whether they become too complex to be interpretable, and also increases the risk of statistical overfitting. Solution 3 refers to the idea of data weighting, wherein a user can tell a model to try to focus on explaining some data sources more than others. In fish stock assessment in particular, the question is often how to reconcile information on the numbers of individuals at each age (age composition) and the total numbers of individuals (abundance indices). Since the goal of stock assessment is to estimate population size, Dr. Chris Francis and others suggested giving less weight to the age composition data than the abundance data, proposing that this is “right-weighting” rather than “down-weighting.” Ideally, we wouldn’t need to use data weighting, but in stock assessment we are considering this approach because we have a limited understanding of the statistical characteristics of the data arising from how we collect fish.
One method of getting to the root of the problem, choosing the wrong model, is interpreting the output of multiple models and incorporating this into the ultimate prediction. This is called ensemble modeling, which has been shown to be quite successful in fields like hurricane prediction. Dr. Rick Methot and Dr. Ian Stewart are proponents of this approach, but admit that we need to come up with better ways to combine the model predictions into a single answer.
Taking a step back, if model misspecification and data conflicts are a general problem, why haven’t statisticians given us the answer? Well, part of the problem is that fisheries stock assessment has taken a particular form and set of approaches that make it unlike other areas of statistical ecology. This is due in part to legacy effects of those that influenced the field before us, but also the fact that the field has coevolved with the management system (as noted by Felipe Hurtado-Ferro and Dr. Rick Methot, among others). That is, the approaches and specific tools that are acceptable for informing management have been selected by highly rigorous peer review committees. The review process is much more extensive than that for publication of a scientific paper. Some speculate that this leads to a conservative approach, where the bar for using new techniques to inform management is extremely high, and the evolution of the field occurs primarily through consensus among a small group of experts. Fortunately, CAPAM is functioning to get all these experts in one place, along with students and other interested parties, to discuss some of the more philosophical questions about what is most important to delivering management advice, along with best practices given the current state of knowledge.
The workshop setting: sweeping views of La Jolla and the Pacific Ocean from the Southwest Fisheries Science Center. Photo by Jeff Reeder, https://swfsc.noaa.gov/
Anyone can try stock assessment! Open source tools for stock assessment modeling
Cole Monnahan gave a demonstration of an R package ss3sim, freeware that simplifies the process of setting up and running simulations of Stock Synthesis, the massive stock assessment modeling framework developed by Dr. Rick Methot. ss3sim relies on another R package designed to implement Stock Synthesis models, r4ss, for which Alan Hicks showed some of the features, including tidy results output with figures and captions.
The bottom line for stock assessment modelers:
- Consider the reasoning outlined here by Dr. Mark Maunder and Dr. Kevin Piner when model diagnostics indicate that you have a fitting problem.
- If you decide to use data weighting, do not use the McAllister and Ianelli (1997) method based on the arithmetic mean. Instead, use Francis’ method TA1.8 (Francis 2011; the extension of this method to age-length keys produces the least bias) because it accounts for correlations among data, or use an extension of McAllister and Ianelli employing the harmonic mean.
- Data weighting will not account for biases due to model misspecification (per Dr. Andre Punt), so consider other solutions. For example, Drs. Jim Thorson and Anders Nielsen propose using mixed-effects models with time-varying “nuisance” variables like recruitment, selectivity, growth or natural mortality modeled as a random effect.
CAPAM is a joint effort between the University of California San Diego/Scripps Institution of Oceanography/Cooperative Institute for Marine Ecosystems and Climate (UCSD/SIO/CIMEC); National Oceanic and Atmospheric Administration Fisheries/Southwest Fisheries Science Center (NOAA Fisheries/SWFSC); Inter-American Tropical Tuna Commission (IATTC); with funding from the NOAA National Marine Fisheries Service Assessment Methods Working Group; International Seafood Sustainability Foundation (ISSF) and the Pfleger Institute of Environmental Research (PIER). Find out more at http://www.capamresearch.org/