Is AI smarter than a certified landscaper?

For our experiment we chose the International Society of Arboriculture (ISA) Certified Arborist Exam, which some consider to be the hardest landscaping exam. For our AI chatbot, we chose Perplexity.ai’s pro version. We used a publicly available practice exam, with an answer key to grade Perplexity.ai’s performance. The publicly available exam was a practice exam so the results might be a little bit different from if we were testing the AI against the real live exam.

The ISA Certified Arborist exam has ten unique knowledge areas, ranging from soil management to safe work practices. It is a 200 question multiple choice exam, with 3.5 hours (210 minutes) to answer all the questions. So a test taker has just over a minute for each question (gulp).

The questions have different weightings, meaning that some questions count for more points than others. This means that the passing grade changes from year to year, but usually a score of 70-76% should be enough to pass.

Perplexity.ai comfortably passed the exam with a score of 79.5%. And it did the entire exam in less than a minute!

But that isn’t the end of the story: what we found really surprising is how AI scored in the various knowledge areas:

Perplexity.ai got it’s best grades on the factual questions, specifically, questions such as tree identification and selection. This makes sense: there is a ton of information about these subjects on the internet, and most of it is pretty accurate from our experience (who hasn’t used a plant identification app?).

On the other hand, Perplexity.ai failed questions that required field context and experience: it logged its lowest scores in soil management, installation and establishment, and tree protection.

The takeaway from all this is that AI is a tremendous technology, it passed the hardest landscaping test in less than a minute. But it isn’t perfect, it failed questions that required experience and specialized knowledge. And if you had followed it blindly while taking care of your trees, you could have a very sorry (and non code-compliant) landscape.

So for now, until the bots get even better, make sure to only use Artificial Intelligence as a supplement to your own experience!