What is SpSeg?

'SpSeg' (short for 'Species Segregator) is a Machine-learning tool for species-level segregation of camera-trap images originating from wildlife census and studies. SpSeg is currently trained specifically for the Central Indian landscape .

SpSeg is part of Microsoft's MegaDetector ecosystem and works as a second-step to species level segregation of camera-trap images. The approach in MegaDetector pipeline is to first classify the images into Animal, Person, Vehicle and Blank, followed by a fine level classification of Animal images into useful taxonomic classes.

📝 Check out this detailed overview of the approach in a Medium article and an explainer by Dan Morris on why this two-step process.

SpSeg model identifies an Indian fox within the bounding box.

SpSeg models

We initially trained different CNN architecture models from keras (see the results below). We found two models performing well - SpSeg_x.hdf5 and SpSeg_r152v2.hdf5 - and have tested those on various field datasets. SpSeg_x.hdf5 is an Xception model that achieved 88.9% test accuracy, and SpSeg_r152v2.hdf5 is a ResNet152v2 model that achieved 89.2% test accuracy. Find the models from a publicly shared SpSeg_Models Google Drive folder.

Results of initial trained models

The models in different architectures were trained for 100 ephocs each with the same training and test dataset.

Architecture	avg top-1 acc	Architecture	avg top-1 acc
ResNet152v2	89.2%	ResNet101	88.8%
InceptionResNetv2	89.2%	ResNet50	88.5%
ResNet101v2	89.1%	ResNet50v2	87.5%
Xception	88.9%	ResNet152	82.0%

Training data

SpSeg models can currently segregate 36 commonly encountered species (or taxonomic class) in camera-trap surveys in the Eastern Vidarbha Landscape, Maharashtra, India. A maximum of 5000 images were randomly selected for each species from our image dataset to train the models. The models were trained with 70% of the selected data, while 15% images were used for validation and remaining 15% were used for testing. Therefore, species with more than 5000 images in our image dataset (table below) have better accuracy, except sexually dimorphic species like Nilgai.

Species	Scientific name	Image set	Species	Scientific name	Image set
00_barking_deer	Muntiacus muntjak	7920	18_langur	Semnopithecus entellus	12913
01_birds	Excluding fowls	2005	19_leopard	Panthera pardus	7449
02_buffalo	Bubalus bubalis	7265	20_rhesus_macaque	Macaca mulatta	5086
03_spotted_deer	Axis axis	45790	21_nilgai	Boselaphus tragocamelus	6864
04_four_horned_antelope	Tetracerus quadricornis	6383	22_palm_squirrel	Funambulus palmarum & Funambulus pennantii	1854
05_common_palm_civet	Paradoxurus hermaphroditus	8571	23_indian_peafowl	Pavo cristatus	10534
06_cow	Bos taurus	7031	24_ratel	Mellivora capensis	5751
07_dog	Canis lupus familiaris	4150	25_rodents	Several mouse, rat, gerbil and vole species	4992
08_gaur	Bos gaurus	14646	26_mongooses	Urva edwardsii & Urva smithii	5716
09_goat	Capra hircus	3959	27_rusty_spotted_cat	Prionailurus rubiginosus	1649
10_golden_jackal	Canis aureus	2189	28_sambar	Rusa unicolor	28040
11_hare	Lepus nigricollis	8403	29_domestic_sheep	Ovis aries	2891
12_striped_hyena	Hyaena hyaena	2303	30_sloth_bear	Melursus ursinus	6348
13_indian_fox	Vulpes bengalensis	379	31_small_indian_civet	Viverricula indica	4187
14_indian_pangolin	Manis crassicaudata	1442	32_tiger	Panthera tigris	9111
15_indian_porcupine	Hystrix indica	5090	33_wild_boar	Sus scrofa	18871
16_jungle_cat	Felis chaus	4376	34_wild_dog	Cuon alpinus	7743
17_jungle_fowls	Includes Gallus gallus, Gallus sonneratii & Galloperdix spadicea	4760	35_indian_wolf	Canis lupus pallipes	553

How to Use SpSeg?

The best place to start is the User Guide of SpSeg, which covers the process in details.

SpSeg model (either SpSeg_x.hdf5 or SpSeg_r152v2.hdf5) reads the results of MegaDetector and classifies the animal images into species (or a defined biological taxonomic level). Therefore, both the models (detector and classifier) are required to run simultaneously on the images.

📝 You can find more about how you can run MegaDetector on your images from this official MegaDetecor GitHub page.

⚠️ Keep in mind that the SpSeg is developed with MDv4. Although MDv5 has been launched, at this moment we recommend to stick with MDv4. We shall integrate SpSeg tools with MDv5 in future updates.

Setting up the tools

These instructions are quite similar to the instruction for MegaDetector installation, where more details could be found. For technical details check the official websites of the mentioned software.

Step 1. Installation

Download and install Anaconda (or Miniconda). Installing Git is optional, but recommended to keep your repo updated. The latest NVIDIA drivers need to be checked if you are using a GPU.

Step 2. Access the detector and classifier models

Download MegaDetector model v4.1.0 and SpSeg models and keep at an accessible location. These instructions assume that the models are downloaded to a folder called c:\spseg_models.

Step 3. Clone git repo and set Python environment

There is no need to setup a separate MegaDetector environment, which is incorporated in the codes here. We are giving the instructions for Windows machines only, which is expected to work on Linux machines in a similar way. The environment is not tested on Mac.

Open Anaconda Prompt and make sure you have not opened an Anaconda Powershell Prompt or Windows Command Prompt. It should look something like this with the name of environment in the parenthesis:

Run the following commands one by one to download SpSeg repo from GitHub:

mkdir c:\git
cd c:\git
git clone https://github.com/bhlab/SpSeg

The commnads above are suggestions. git command will work at any directory and you can download GitHub repo wherever you find it convinient. Alternatively, you can download the SpSeg repo as zip folder and unzip it to a desired location.

⚠️ If your computer does not have a CUDA-supported GPU--> Go to the local SpSeg repo folder and open environment_SpSeg.yml file in notepad, change the dependency tensorflow-gpu to tensorflow and save the file.

Next, you need to enter where SpSeg repo is located using cd command:

cd c:\git\spseg
conda env create --file environment_SpSeg.yml

You are all set to run SpSeg! 🎉

Camera-Trap image data organization

We assume that your camera-trap data was organized in either of the following directory structures:
dataDir/Station
dataDir/Station/Camera
The first structure arrives when a single camera was placed at the station, while the second structure arrives when A and B cameras were placed facing each other. The directory before Station ID (dataDir) could include site ID, range ID and/or block ID.

Running SpSeg

Next time when you open Anaconda Prompt, first you need to activate SpSeg environment:

conda activate SpSeg

and then locate the SpSeg tools folder.

cd c:\git\spseg\tools

Which should look something like this:

The main tool to use SpSeg models is run_spseg.py

Once you are in the right setup (environment and tools folder), the following command calls detector and classifier models and runs over all the images:

python run_spseg.py "C:\spseg_models\md_v4.1.0.pb" "D:\dataDir" --recursive --spseg_model " C:\spseg_models\SpSeg_r152v2.hdf5" "D:\dataDir\data_output.json"

"D:\dataDir" is the location (local or on network server) where your organized camera-trap images are stored. You can set the path and name of the output file at "D:\dataDir\data_output.json".

The above command returns a .json directory for each image with detector tags for animal, person or vehicle, and species tags for animals (both the tags with a confident value).

💡 The image paths are retuned as absolute, which creates mismatches during further processing of the data specially if processing is carried out on a different system. Therefore, the best practice is to get the image paths as relative and keep the output file within image folder.

Relative paths in the output can be returned by setting the flag for it:

python run_spseg.py "C:\spseg_models\md_v4.1.0.pb" "C:\dataDir" --recursive --spseg_model " C:\spseg_models\SpSeg_ r152v2.hdf5" "C:\dataDir\data_output.json" -- output_relative_filenames

How to use the output to process the images?

AI-tools are not 100% accurate. Therefore, a workflow is required to process the output and correct for the errors. Currently, there are two approaches to process the output and finalize the identification/segregation of the images for further analyses-

Using Timelapse to review the images (only for Microsoft Windows systems),
Using an Image file manager system like Windows File Explorer, ExifPro, IrfanView or FastStone.

Detailed instructions on these workflows are provided in the User Guide of SpSeg.

Further development

We have in pipeline- 1. integration with MegaDetector version 5. 2. developement of DigiKam based workflow, 3. Further refinement of the SpSeg models (particularlly for dimorphic ungulate species and species with small dataset), and 4. Addition of more species, e.g. we missed out on the elephant in the current models.

So stay tuned!

Also, if you have any suggestion for additional species or would like to add to training dataset, reach out to us (bh@wii.gov.in) to Participate!

Citation

Cite the use of SpSeg models as:

Shrotriya, Shivam; Guthula, Venkanna Babu; Mondal, Indranil; and Habib, Bilal (2022). SpSeg User's Guide, version 1.0. TR No. 2022/29. Wildlife Institute of India, Dehradun, India.

And, don't forget to cite MegaDetector (without which SpSeg won't even exist):

Beery, Sara; Morris, Dan; and Yang, Siyu (2019). Efficient Pipeline for Camera Trap Image Review. arXiv preprint arXiv:1907.06772

License

SpSeg models and tools are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license. Find details on the terms of the license here.

Dr Bilal Habib's lab (BHlab) at Wildlife Institute India, Dehradun, India is a collaborator in the development, evaluation and use of MegaDetector model. Development of SpSeg was supported by Microsoft AI for Earth (Grant ID:00138001338). We thank Mahrastra Forest Department for funding long term research in the state of Maharashtra. Kathan Bandopadhyay helped in preparing training dataset. The researchers and volunteers are thanked for their time for reviewing and testing the tools in real workfow.