Labeled images of emerged salmonids in a riverine environment

Abstract

Objectives

These data enable the development of machine learned models to detect unintended passage of salmonids over in-stream barriers. Such models are key to fully characterizing the effectiveness of selective passage systems, as they detect and quantify fish passage which occurs outside of the intended transit passage and selection mechanism. These data were used to construct custom surveillance tools for FishPass (https://www.glfc.org/fishpass.php), a 20-year restoration project to provide selective up- and down-stream passage of desirable fishes while simultaneously blocking or removing undesirable fishes.

Data description

The datasets contain over 2300 annotated images of emerged salmonids collected in a natural riverine environment. The images stem from surveillance video collected during 2022 and 2023 fall runs of several pacific salmonid species introduced to the Laurentian Great Lakes on the Boardman (Ottaway) River in Traverse City, MI, USA. In addition to images of fully emerged salmonids, datasets are provided containing images of partially emerged salmonids, fully submerged fish, and other wildlife present in a riverine environment. The environmental conditions represented by most of the images were clear or partly cloudy. These datasets could be used to develop custom object detection models for emerged fish in riverine environments.

Objective

Man-made dams and weirs segment waterways and pose challenges to aquatic species which must traverse down- or up-stream to reach their spawning habitat [1]. While these structures may not have been designed as control measures for invasive species, such as sea lamprey (Petromyzon marinus) or bighead carp (Hypophthalmichthys nobilis), they are often vital to such efforts [2, 3]. For other aquatic species, the disruption posed by dams and weirs are at odds with fisheries management aims, creating a connectivity conundrum [4]. Selective passage has been proposed as one potential means of achieving species specific connectivity goals.

The overall effectiveness of a selective passage system will depend highly on the context of fish passage including what species are present and what selection mechanisms are deployed. The assessment of passage opportunities of all species, not just those that are considered desirable for passage, is a fundamental aspect of selective passage. Surveillance of both intended and unintended passage provides a more complete characterization of selective passage systems and supports other fisheries management tasks. Surveillance cameras are a common tool for capturing high-resolution images and videos of fish movement at barriers. When deployed long-term, the volume and continual delivery of data generated by surveillance systems do not lend themselves to manual review and are often managed by machine learning [5].

Presented here is a labeled dataset of emerged and jumping salmonids in a riverine environment. It was collected to develop a pipeline to detect unintended passage at FishPass. This dataset can be used to train deep convolutional neural networks or other machine learned object detection models to detect unintended passage. In contrast to existing datasets such DeepFish [6], Labeled Fishes in the Wild [7], or other salmonid datasets [8], this dataset contains images of salmonids partially or fully emerged from water and collected in situ.

Data description

Video recordings were captured fall 2022 and 2023 at a weir installed on the Boardman (Ottaway) River in Traverse City, MI, USA. The capture device was a custom micro factor computer running Frigate [9], a network video recorder (NVR), and utilized four IP cameras (Amcrest Outdoor PoE 8MP). Figure 1 indicates the field of view for cameras during the 2023 collection. The video recordings were stored at 10 frames per second with a resolution of 3840 pixels by 2160 pixels. The dates for collection were selected to coincide with fall migratory runs of several pacific salmonid species including steelhead rainbow trout (Oncorhynchus mykiss), chinook salmon (Oncorhynchus tshawytscha), and coho salmon (Oncorhynchus kisutch). Since all three salmonids are introduced to the Laurentian Great Lakes, distinguishing beyond the family level, Salmonidae, was not required.

Fig. 1

Field of view for cameras deployed during the 2023 collection

A two-staged, bootstrapped process was used to extract and label frames containing emerged salmonids. Initially, collected video was filtered using GMM-based background subtraction [10] to limit manual processing to frames with significant motion. This greatly reduced the number of videos that had to be processed. Images were created for frames containing emerged salmonids and these images were annotated using Label Studio [11]. Each fish visible above the river surface was labeled regardless of whether the complete fish was showing or if only some feature of the fish had surfaced (e.g., fin, snout, dorsal). Additionally, multiple annotations for an image were recorded if multiple fish were present in the image. All annotations were stored as text files in YOLO label format. These initial annotations were used to train an object detector to locate emerging salmonids from surveillance videos [12].

After an initial dataset was created, additional object detectors were developed to automatically identify and label additional instances of emerged salmonids from the surveillance records. This increased the number and variety of the images of emerged salmonids. From this larger pool of labeled images, several curated datasets were created through targeted sampling. Dataset 1 contains clear images of emerged salmonids which completely emerged above the water’s surface and were captured during the daylight hours. Dataset 2 contains images of emerged salmonids that were captured in poor weather conditions, such as during the night or during inclement weather. Dataset 3 contains images of emerged fins, snouts, or dorsals of salmonids. These images may be useful for detecting the presence of fish below the surface of the water but as the fish do not fully exit the water, they are unlikely to indicate movements which could lead to unintended passage of a barrier. Dataset 4 contain images of salmonids swimming below the surface of water and dataset 5 contains unannotated images. Dataset 6 contains short videos with jumping fish. The size of each dataset is listed in Table 1. Figure 2 is a composite image showing an example of each dataset.

Fig. 2

Composite image showing clockwise, from the top, left corner examples from datasets 1, 2, 5, 3, 3, and 4

The images and annotations from datasets 1 and 2 can be used to train and evaluate deep convolutional neural networks performing object detection. Figure 3 illustrates an annotation for an image. Datasets 4 and 5 provide negative examples which can aid in refining or further evaluating models. The videos in dataset 6 could be used to evaluate object detectors for real-time inference.

Fig. 3

Composite image showing raw image of jumping fish (top) and the same image with bounding box superimposed (bottom)

Limitations

As data was gathered over two collection periods spanning only a few weeks, it was difficult to curate images across a variety of environmental conditions (e.g., overcast skies, rain, wind). The typical weather during data collection was clear or partly cloudy. This lack of variety in environmental conditions biases the dataset towards clearer weather conditions and as such may not accurately represent the behavior of fish under different weather conditions. Additionally, the dataset includes only a few images that were collected during the non-daylight hours. The cameras employed for collection did have a NightVision mode but the quality of the image and field of view were very limited in this mode.

Table 1 Overview of datasets