ArbNation Join free

How reliably do arborists judge decay and stem failure? Inter-assessor consistency studies

Studies of credentialed arborists rating the same trees find wide spread in risk ratings and in the work recommended. In a 2017 study the assessor was four times more likely than the tree to shape the final rating.

Checked against the sources at the bottom of this page on October 15, 2026. Rules, fees and pay change: the source has the last word.

This page draws on two AUF papers and a set of conference slides. It is not training and does not teach the work. The research comes from the University of Florida team of Ryan Klein and Andrew Koeser, and it appears in Arboriculture & Urban Forestry (AUF), ISA's journal.

The assessor and the tree

The 2019 AUF literature review by Klein and colleagues screened about 1,000 articles and kept 150. It found significant variability in the final determinations of risk assessors and of arborists using the common methods. The authors read this as inconsistency that best management practices and training programs have not yet resolved. They add that personal bias probably explains some of it.

The 2023 AUF paper, discussed below, summarises an earlier study by Koeser and Smiley (2017). In it nearly 300 arborists did basic (Level 2) assessments of three trees. The assessor was four times more likely to affect the final risk rating than the characteristics of the tree. Arborists with experience and credentials were four times more likely to recommend retaining and monitoring a tree rather than removing it than those without. Trained assessors also gave lower likelihood of impact, lower likelihood of failure and lower final ratings.

The 2023 reproducibility paper

Klein, Koeser and co-authors published "Evaluating the Reproducibility of Tree Risk Assessment Ratings Across Commonly Used Methods" in AUF in November 2023. They sent videos of urban trees to arborists and asked for assessments of three trees each. Of 450 arborists chosen from public directories, 60 replied, a 13.3 percent response rate. Of those, 28 held the ISA Tree Risk Assessment Qualification (TRAQ), 21 held Quantified Tree Risk Assessment (QTRA) training, and 11 had neither.

Risk ratings given to the same three trees, 2023 AUF paper
RatingTRAQ credential (82 assessments)ISA BMP, no TRAQ (32 assessments)
Low14%31%
Moderate38%31%
High43%25%
Extreme5%13%

The QTRA group used different terms, so the paper makes no direct comparison. Of 62 QTRA assessments, most were broadly acceptable, a risk of harm of 1 in 1,000,000 or less. The rest split nearly evenly between tolerable at 24 percent and unacceptable at 23 percent.

Tests of variability found no method more reproducible than another, with a lowest P-value of 0.251. The mitigation recommended did differ. Arborists without risk assessment training recommended removal half the time, more than the QTRA group did (P = 0.040). QTRA-trained arborists were more likely than the TRAQ group to recommend monitoring (P = 0.037).

The study also asked the ISA BMP and TRAQ arborists to rate likelihood of failure over one year and over three years. Ratings rose with the longer timeframe, but their variability did not (lowest P-value 0.161).

Decay and levels of assessment

Slides by Andrew Koeser of the University of Florida, hosted by the Arboricultural Association, report two further experiments. In the first, 153 southern live oaks and 86 laurel oaks were checked for internal decay. For some species a basic visual assessment by a trained arborist with minimal experience agreed well with a resistance recording drill.

In the second, 70 arborists assessed five trees at several levels, from limited visual to advanced. Raising the level of assessment often made the ratings more variable, not less. Ratings were highest after basic visual and mallet assessment, similar to the drill. They tended to be lower after limited visual assessment and after sonic tomography. The slides conclude that equipment can give precise estimates of decay and occupancy, but without defendable thresholds or decision rules risk assessments stay variable.

The slides also cover the target. In a video study, risk ratings correlated with actual occupancy (r = 0.29), and the correlation rose to 0.62 once the arborists saw the traffic data. The time of day at the site changed the ratings (P below 0.001).

Limits of the evidence

The 2023 sample was small, with 60 replies, and assessors rated videos, not trees standing in front of them. The slides are a conference presentation and print few numbers. Levels of assessment are compared in the guide to tree risk assessment levels.

Sources