Evaluation of Computer Vision Systems and Applications to Estimate Trunk Cross-Sectional Area, Flower Cluster Number, Thinning Efficacy and Yield of Apple
by Luis Gonzalez Nieto, Anna Wallis, Jon Clements, Mario Miranda Sazo, Craig Kahlke, Thomas M. Kon, and Terence L. Robinson
Edited by Mike Basedow, the full original article can be accessed here.
Precision crop load management of apple requires measuring trunks, counting flowers, fruitlets, or fruit at various times during the year to guide management decisions. While previous studies have found ways to incorporate these manual measurements into helpful orchard tools, time has often been a limiting factor in the adoption of these practices. There are an increasing number of tech companies working in agriculture that are incorporating these models into their portfolio. Our hopes are that the time required to collect the necessary data can be minimized so that more growers can have actionable information to manage their orchard more precisely.
To get a sense of how accurate these current tools are, a study was recently led by researchers from four states. Researchers compared different commercial computer vision systems to estimate trunk cross-sectional area (TCSA) to assess optimal crop load per tree, flower cluster number to guide bloom thinning practices, thinning efficacy models to determine how much additional thinning is required, and yield estimation to assist with hand thinning and harvest planning decisions. These studies evaluated tools from Orchard Robotics and Pometa across 23 orchards in four states. Experiments were conducted in 2021, 2022, and 2023 in commercial apple orchards in Massachusetts, Michigan, New York, and North Carolina.
Orchard Robotics uses a proprietary camera system, and Pometa (previously Farm Vision) uses a cell phone camera system. TCSA and flower counting tools were evaluated with the Orchard Robotics camera in full orchard rows. Flower counting, fruit set estimation, and yield estimation were evaluated with the Pometa phone app system. These companies’ systems were ground truthed against manual hand measurements.
A summary of each of the tests follow:
Trunk Cross Sectional Area (TCSA) was evaluated in 2023 with the Orchard Robotics vison system in Geneva, New York. The Orchard Robotics vision system takes a photo of each single tree and detects the visible portion of the trunk (between graft union and first branches), and then measures the trunk diameter in the middle of the section that it can detect and calculates the TCSA. The ground-truthed TCSA was measured at 30 cm above the graft union. A tape measure was used for manual measurements of trunk circumference and converted to square centimeters (cm2) cross-sectional areas.
While there was positive correlation between the manual and Orchard Robotic measurements, the vision system tended to underestimate the TCSA in comparison with the manual measurements in all four trials (Figure 1). One explanation for the difference may be that the vision system calculated the TCSA using the average of all the visible trunk, while the manual measurements were always measured at 30 cm above the graft union.
The value of using the TCSA as a measure of a tree’s capacity to carry fruit is likely very good when the trees are young; however, when trees fill their allotted space at maturity, the TCSA continues to increase, but the yield is constant. Thus, the optimum crop load based on TCSA needs to be adjusted down each year as the TCSA continues to increase, but the productive capacity of the tree remains constant. With proper adjustment, the TCSA measurements obtained from Orchard Robotics could likely quantify the variability in tree size in an orchard to allow differential crop loads based on the trees’ capacity.
Blossom counts were evaluated in New York in 2023. Manual and computer vision estimates of the number of flower clusters per tree were carried out at the pink stage. The Pometa vision system was evaluated in a commercial orchard on ‘Gala’ and ‘NY1’. Pometa cluster counting was a beta product in 2023. Five representative trees were selected per block, which were scanned using a video function on a compatible cellular phone according to the company’s directions. Two videos were taken of each tree (one on each side), with a total of ten videos. Videos were uploaded to the Pometa website for data processing. After 24 h, results of the processed videos were available and were downloaded.
The Orchard Robotics vision system was evaluated in a commercial orchard of ‘Gala’ and ‘Fuji’. The Orchard Robotics vision system was evaluated using manufacturer recommendations. At full bloom, one side of the row (the well-exposed, sunny side) was scanned with the camera system mounted on an orchard vehicle at 8 km/h. All scans were uploaded automatically from the camera to the Orchard Robotics website for processing. After 24 h, the results of the processed videos were available on the company’s website. However, it was necessary to record the manual counts at the same time as the evaluation of the representative 5 trees to calibrate the system.
Both vision systems evaluated showed a positive linear correlation between estimated number of flowers per tree, measured with the vision systems and manual counts. Overall, both vision systems were accurate with their estimation of flower clusters per tree, except for ‘NY1′, in which Pometa underestimated the number of blossoms per tree (Figure 2).
The Orchard Robotics system consistently provided accurate counts when properly calibrated with manual counts to account for occluded flower clusters. The Pometa system showed variability in the results, with one block showing accurate counts and another with large differences between the manual flowers counted and those detected with the vision system. However, both blocks showed a high linear correlation, and it is possible that the overall error could be reduced with a correction factor based on manual counts of representative trees. It appears that both systems could be used to manage blossom thinning with precision. For example, Penzel, et al. considered estimates of flower intensity as essential to manage mechanical flower thinning. In addition, these systems could be used to estimate the best timing to initiate the pollen tube growth model. The geo-referenced information on flower cluster density can be used to create accurate heat maps to guide variable smart sprayers to manage the variable rate application of chemical thinning agents to match chemical dosage with flower cluster density on each tree.
Thinning efficacy trials were carried out in 13 commercial orchards in 2022 in New York, Massachusetts, Michigan, and North Carolina. ‘Gala’, ‘Fuji’, and ‘Honeycrisp’ cultivars were evaluated with Pometa scans with an iPhone, and compared with the MaluSim app.
Greene et al. developed the fruit growth rate model to predict the efficiency of thinning and to inform decisions on the necessity of more thinning sprays. The model uses the growth rate of tagged and measured fruitlets to predict the percentage that will thin off. A mobile phone app (MaluSim) was developed by Francescatto and Robinson (unpublished) to automate the calculations necessary to run this model, which improved the efficiency and accuracy of this method, although it still requires manual measurements of fruit diameters. Manual measurements require about two hours of work per orchard on each date. At least two measurements are required to estimate fruit per tree.
The Pometa vision system uses the same concept of the fruit growth rate model but uses computer vision to measure the fruit diameters. This system utilizes a cell phone camera to capture videos of five representative trees. The videos are processed by a proprietary algorithm to determine which fruitlets will abscise. The time required to carry out all measurements is about 10 min per block.
There were significant positive linear relationships between predicted fruit per tree early in the season (June, post-thinning) and final fruit harvested (Figure 3). Both methods (MaluSim and Pometa) overestimated the number of fruits at harvest over a range of different crop loads. When the final numbers of fruit at harvest were lower than 200 fruit/tree, the prediction of both systems was more accurate. That is, the numbers of fruit at harvest were similar to those predicted by each model 7 days after spraying the chemical thinning product. However, when the numbers of fruit at harvest were higher than 200 fruit/tree, both systems overestimated the numbers of fruit at harvest. The predictions of final fruit numbers per tree of both models were correlated. However, when the numbers of estimated fruit were lower than 200 fruit/tree, the Pometa system overestimated more than the MaluSim app. On the other hand, when the numbers of predicted fruit were higher than 200 fruit/tree, both systems showed the same prediction of numbers of fruit.
Our results suggest that both systems (MaluSim and Pometa) showed accurate predictions of the numbers of fruit at harvest when the final fruit numbers were lower than 200 fruit/tree. However, our results suggest that, when the numbers of fruit at harvest were higher than 200 fruit/tree, both the MaluSim and the Pometa systems overestimated the numbers of fruit at harvest. The overestimation of fruit numbers per tree is likely due to the higher natural fruit drop after the chemical thinning window has passed. This is typically attributed to a carbohydrate deficit in June, resulting in “June drop”. Despite the overestimation of final fruit numbers when the crop load is high, our results show that both the manual model (MaluSim) and the computer vision system of Pometa are correlated, and the estimates with the computer vison system are very close to the estimates with manual measurements. Thus, the vision system with a cell phone camera will likely become a preferred method to estimate thinning efficacy in the future, since it requires less time and provides easy interpretation of results. This will allow growers to make more informed thinning decisions between each thinner application, allowing them to assess if additional thinner applications are necessary.
Yield estimation trials were carried out in 11 commercial orchards in Michigan, New York, and North Carolina in 2021 and 2022. ‘Evercrisp’, ‘Granny Smith’, ‘Gala’, ‘Fuji’, and ‘Honeycrisp’ cultivars were evaluated with the Pometa system of yield estimation using iPhone scans and then compared with manual counts of fruit numbers at harvest.
There was a significant correlation between fruit yield estimation late in the season (August/September, immediately before harvest) with the Pometa vision system and final yield at harvest (Figure 4). The estimation was accurate when the final fruit numbers were fewer than 75 fruit per tree. When the numbers of fruit at harvest were higher than 75 fruit per tree, the vision system underestimated the final yield, except in one plot that overestimated the yield. Moreover, as the final fruit numbers per tree increased, the error in the estimate was higher.
Estimation of harvest provides essential information to apple orchard managers, storage operators, and fruit marketers. For automated harvests, yield maps will be required. Apple yield estimation with a smartphone offers the advantages of low cost, quick access, and simple operation. Our results suggest that the use of smartphones to carry out yield estimation produced accurate results, until the numbers of fruit exceeded 75 fruit per tree. When the numbers of fruit at harvest were higher than 75 fruit per tree, the Pometa vision system underestimated the final yield because there was significant fruit occlusion caused by foliage. The Pometa company is working on an occlusion model to improve the yield estimation with tree forms that have higher occlusion of fruit.
Conclusions
The use of the vision systems to estimate TCSA, bloom intensity, thinning efficiency, and yield generated accurate predictions and usable data for the growers. The vision systems could measure the TCSA for each tree to estimate the optimum fruit numbers for each tree. However, the vision system underestimated the TCSA in comparison with the manual measurements because the measurements with the vision system were carried out in a different place on the trunk. Both vision systems evaluated could estimate the numbers of flower clusters per tree accurately. This information when geo-referenced with each tree could allow precise management of flower-thinning spray applications. The fruit growth rate model using the MaluSim app and the Pometa system showed accurate predictions of thinning efficacy when the crop load was lower than 200 fruit/tree. However, our results suggest that, when the numbers of fruit at harvest were higher than 200 fruit/tree, both the MaluSim and the Pometa systems predicted higher numbers of fruit at harvest than were actually harvested. The vision system of Pometa with cell phones needed less time than the MaluSim method and provided easy interpretation of the results. The yield estimation with Pometa was accurate up to a crop load of 75 fruit per tree. However, when the numbers of fruit at harvest were higher than 75 fruit per tree, the Pometa vision system underestimated the final yield because there was significant fruit occlusion caused by foliage. Currently, Pometa is working with a new occlusion model to improve the yield estimation with tree forms with high occlusion of fruit.
Overall, precision crop load management with smartphone-based systems offers advantages such as low cost, quick access, and simple operation. However, the vision systems with special cameras and geo-referencing of each tree offer tree-specific information that will allow precise crop load management of each tree.