Newsletter of the Biological Survey of Canada (Terrestrial Arthropods)

Comments on error rates in insect identifications

Steve Marshall
Department of Environmental Biology, University of Guelph, Guelph, ON N1G 2W1

In recent years there has been growing appreciation of the pressing need to document Canada’s biodiversity. Surveys of terrestrial arthropods have been launched by national and provincial parks, federal and provincial departments concerned with ecosystem integrity (e.g. Canadian Forest Service, Fisheries and Oceans, Agriculture and Agri-Food Canada, Environment Canada, provincial resource ministries etc.), non-governmental organisations, and individual researchers. Inventory work is generally carried out over the short term, with results closely tied to lists of species collected and identified during the project. Identifications might be done by a single individual, a group, or by a network of cooperating systematists; surveys might cover one or a few taxa (butterflies or ground beetles, for example), or might try to do the most complete stock-taking possible in a given protected area.

Previous briefs by the Biological Survey of Canada have addressed general issues of sampling and sample processing, but have not examined the problem of error rates in identification. It is widely appreciated that misidentifications of potential pests can have serious economic implications, as demonstrated by recent, widely publicized incidents in which long-horned wood boring beetles were misidentified. In one case, a collector misidentified a native species as an exotic species that is not established in Canada but represents a serious threat to the forest industry. The misidentification was perpetuated by the electronic media, and only after considerable effort was the error corrected. In another case, a recently introduced long-horned beetle was misidentified as a native species, an error that was not recognised until the new introduction emerged as a pest a decade later. These misidentifications of relatively large, economically important insects in well-known groups are important in themselves, but they also point to the probability of large numbers of misidentifications going unrecognized in the long lists of species which are at the heart of ongoing biodiversity studies.

The proliferation of species lists based on identifications by non-specialists is an inevitable result of the taxonomic impediment imposed by Canada’s shrunken pool of professional systematists. It is important that agencies involved with the planning of biodiversity work be aware of the magnitude of error rates under different sets of circumstances, and make long-term plans to mitigate the effect of identification errors.

Ideally, specimens collected during regional surveys would be carefully prepared (using the guidelines in earlier briefs from the Biological Survey of Canada), competently sorted to family, then sent to taxonomic specialists with ready access to a national insect collection. Determinations would be assembled into an authoritative species list followed by collaborative efforts to database the records, interpret patterns from the accumulated determinations, and to curate the material for future study and reference. This rosy scenario is being played out to some degree thanks to the willingness of a few key systematists to identify good material in their areas of interest, but the onus is increasingly on the individuals doing the survey work to handle their own identifications. We can assume that identifications of sawflies done by a taxonomist specialising in sawflies will be correct, but what about the growing majority of identifications that now must be done by non-specialists? What is the error rate?

Assessing Error Rates

Some idea of the general error rate in identifications by experienced undergraduate biology students can be garnered by looking at the performance of students in advanced entomology courses with a collection/identification requirement, such as the fourth year insect systematics course at University of Guelph. Students in this course make insect collections during the summer, then spend their laboratory periods during the fall semester identifying the specimens in their collections plus supplemental material. Students have to identify all insects to at least the family level, and they are expected to develop a practical efficiency in using the excellent family keys in a standard textbook. Open book tests early in the course show that students inexperienced in keying have an error rate of over 50%, and even at the end of the course there is a significant error rate in keying to the family level.

This illustrates an obvious, but often overlooked, point of importance when considering the problem of error rates in survey work. Keys are tools that require training and experience for proficient use; the first time someone uses an unfamiliar tool the results are likely to be poor. Identification of insects usually depends on key use, and error rates will depend on the quality of the key, the intrinsic difficulty of the key, the experience of the user, and the key user’s opportunity to check and confirm the results. These are all difficult things to quantify, but it is still worthwhile to consider how each factor might influence error rates and how this information should affect studies of biodiversity.

1) Availability of, and differences between, keys
Many taxa are virtually unidentifiable because of the complete lack of keys; in other cases the keys are scattered throughout the literature and are of varying quality. In many cases there are no keys because the taxa involved still require revision and species, or even genera, remain undescribed or at least ill-defined.

The quality of published keys varies from almost foolproof to almost useless. Some keys are badly written or just don’t work, but in most cases differences between keys reflect intrinsic differences in the taxonomy of different groups. For example, good keys to fly genera of North America are readily available in the Manual of Nearctic Diptera (McAlpine, 1981, 1987) and keys to all beetle species of northeastern North America are available in Downie and Arnett (1996). Each book has “easy” keys (quick to use, low error rate) and “difficult” keys (very time-consuming, high error rate). For example, the key to Coccinellidae in The Beetles of Northeastern North America is within the reach of a beginning student, but I find the key to Cerambycidae to be considerably more difficult. The Cerambycidae of northeastern North America, however, are easily identified to species using a recent photographic identification guide (Yanega, 1996).

Error rates go down, and identification efficiency goes up, when richly illustrated regional guides are available. This applies even to relatively easily identified groups such as tiger beetles. For example, it takes considerable experience to correctly identify Ontario tiger beetles using a key written for the entire country, but it is easy to identify an Ontario tiger beetle from a set of photographs and notes restricted to the relatively small provincial fauna. Even those groups that lend themselves to richly illustrated guides, however, are prone to significant error rates if the guides are used without appropriate skills. A well-known case in point is provided by the recent CNF (Canadian Nature Federation) survey of Lady Beetles of Canada, in which the number of obvious misidentifications submitted by amateurs (combined with the absence of voucher specimens or photographs) compromised the results of the survey.

It must be remembered, then, that even for those taxa for which keys are available (and species-level keys are not available for many taxa), keys differ widely in intrinsic difficulty, quality, and scope. These differences alone make it impossible to quantify an overall error rate in general insect survey work.

2) Differences in skill level and level of experience
A certain level of training and experience is necessary to develop skills in identification and, all other things being equal, a university student who has had a fourth-year course in insect classification, as well as other entomology courses, will be better qualified to undertake inventory work than a student lacking such training. Of course, all other things are never equal, and individuals vary widely in their taxonomic abilities. Of the hundreds of students (with similar experience and training) who have worked with the University of Guelph Insect Collection, dozens have been “gifted” taxonomists who made few identification errors, whereas others developed only limited identification skills.

Academic qualifications are not the only predictor of an individual’s ability to correctly identify insects. An individual with the right combination of experience, enthusiasm, energy and innate ability, can make considerable progress on a regional inventory in a single season, while an otherwise highly qualified individual without good identification abilities can waste a lot of time (and precious resources) making very little progress. Agencies and organisations who hire personnel lacking the requisite qualifications to undertake inventory work would be well advised to treat the expenditure on salaries as money invested in training, not in the generation of useful results.

3) Availability of reference collections
The most important factor influencing error rate is the availability of well-identified reference collections. Although photographic guides and well-illustrated regional keys are available for increasing numbers of insect groups, in the vast majority of instances the only available identification tools are keys of varying levels of quality. Most keys have ambiguous or difficult couplets, and the diagnosis of the terminal taxon is rarely explicit enough for the user to confidently confirm the determination. Well-identified regional reference collections are essential for this purpose, and can greatly reduce identification error. A common approach is to key out a specimen, compare it to a reference collection, re-key the specimen if it does not match up, and repeat the process until a good match is made between a reference specimen and the specimen in hand. Error rates on the first pass (identification attempts made without recourse to a reference collection) often are very high. Of course, the same need to confirm identifications can be met in some groups by reference to good collections of photographs, and in some groups experts are available and willing to check over determinations by colleagues and students. In general, however, a well-identified regional reference collection is needed to reduce error rates in generic or specific keying to an acceptable level.

Conclusions

While the many variables discussed above preclude general statements like “error-rate in species identifications by non specialists exceed 50%”, it is worthwhile to emphasize that a very high error rate is likely to result when either good reference collections or collaborating specialists are absent. Errors are not necessarily a serious problem in a survey in which all specimens are databased and housed in a permanent insect collection, because they can be checked and corrected at a later date. In considering the interim results of survey and inventory work, the abilities and experience of the identifier must be considered along with the level of difficulty and quality of available keys and associated resources for each individual taxon for the area under study.