Review on greenhouse sweet pepper robotic harvester

Introduction

Digital farming is an modern new technology practice such as sensors, big data analysis and robotics, for shifting from tedious operations to automated process (Shamshiri et al., 2018) [20] . Due to lack of skilled workforce and high labor costs (Comba et al., 2010) [6] , harvesting sweet pepper (Capsicum) is a labor-intensive task that demands to shifting from manual harvesting to automated harvesting. Nowadays in world, million tons of Capsicum are still hand-picked every year in open-fields and greenhouses. And researchers and farmers are facing challenges to produce more food by use of less land, in order to meet the demands of predicted population about 9.8billion in 2050 (King et al., 2017) [12] . Therefore, development of an automated robotic harvesting considered as an alternative method for greenhouses (Mann et al., 2016) [14] . It also plays an vital role in improve the interacts between human, machine, and plants (González et al., 2009) [8] . And also prevent musculoskeletal disorders in manual harvesting in greenhouses has motivated for replacement of human labor by robots for picking the sweet pepper (Shamshiri et al., 2018) [19] . In this only focused on published literature for automated sweet pepper harvesting have an vision control with an advanced image processing techniques (Hemming et al., 2014) [10] , with the help of an customized platform design software introduce to an support robots for precision in agriculture operation with the help of robot components in figure.01 (Jensen et al., 2014 [11] ). In modern greenhouses, there is a growing need to automate labor due to the decreasing availability of skilled workers willing to perform repetitive tasks in challenging greenhouse environments. This labor shortage, combined with rising costs, is putting significant pressure on the competitiveness of the industry. Current robotic harvesting systems for fruits and vegetables, while promising, still face challenges. Despite decades of research and development, success rates for these robots remain low. For example, prototype systems have demonstrated harvest success rates of around 66%, with cycle times averaging 33 seconds per fruit. However, these figures are far from being commercially viable, as the market demands machines that are both highly accurate and fast.

That said, robots with lower performance levels can still offer value by supporting human labor, even if they aren't perfect. The main challenge lies in bridging the gap between the current technological readiness of these robots and the requirements for widespread use in commercial settings. Most existing solutions lack robustness, fail to handle errors effectively, and do not integrate well with postharvest logistics. The European Union-funded SWEEPER project aimed to develop, test, and validate a robotic harvesting solution for sweet peppers in real-world conditions. The project investigated various scientific and technical challenges, conducting extensive field tests over four weeks in a commercial greenhouse, with different growing rows and sweet pepper varieties. The research explored whether harvesting performance depended on the pepper variety and crop conditions, and integrated the robot into the greenhouse logistics using a mobile platform to traverse rows autonomously. Despite intensive R&D efforts, fully automated harvesting robots have made limited progress. Of approximately 50 systems developed worldwide, none have been commercially successful yet. Agricultural robots face significant hurdles due to the unpredictable, dynamic nature of farming environments. Each crop and task requires tailored solutions, and while companies are making strides with products like strawberry and tomato harvesters, most research remains in the early stages. Moreover, many tests have been conducted under controlled conditions rather than in real-world environments, making it difficult to assess the robots' true performance.

Methods and Methodology

Field tests on harvesting robots are often limited in scope, testing only a small number of fruits or focusing on a narrow range of conditions. This variability complicates the development of reliable performance indicators. To accurately evaluate these robots, long-term testing in a variety of conditions and crop types is essential.

System designed for sweet pepper robotic harvester

A robotic harvester generally comprises several crucial components that work together to achieve efficient harvesting. Among these components are an eye-in-hand system utilized for visual servo control, as outlined by Mehta et al. (2016) [15] and Barth et al. (2016) [4] . This system plays a pivotal role in guiding the robot's movements with precision. The harvester also includes a manipulator, which is responsible for positioning and maneuvering, and an end effector designed specifically for grasping sweet p e p p e r s . T h e e n d e f f e c t o r o f t e n f e a t u r e s a s p e c i a l i z e d gripper design, as discussed by Murphy (2000) [16] a n d Eizicovits et al. (2016) [7] . To ensure optimal performance, motion planning algorithms, such as those developed by Bac et al. (2016) [1] , are employed to analyze and refine the robot's actions. Addressing these components and their interactions is crucial for developing a fully functional harvesting robot, as emphasized by Sandini et al. (1990).

Various robotic systems designed for sweet pepper harvesting, including the CROPS robot (Bontsema et al., 2015) [5] , the Harvey robot (Lehnert et al., 2017), and the SWEEPER robot (Ringdahl et al., 2019), integrate different functionalities and components tailored to their specific designs. These systems face three primary challenges in their development: (A) detection of the peppers, (B) selection of the appropriate grasping method, and (C) effective manipulation of the peppers. The typical process for harvesting sweet peppers includes several stages: scanning the environment, detecting the crop, selecting the grasp method, attaching to the crop, and finally detaching the crop. This process is illustrated in Figure 03, which provides a detailed overview of each stage involved in the robotic harvesting system. [15] . Bontsema et al., 2015 [5] 5 megapixel color cameras and a Time of Flight camera Direct linear transformation (DLT) algorithm 2.

Lehnert et al., 2017 [13] Intel Realsense SR300 RGB-D Kinect Fusion algorithm 3.

Ringdahl et al., 2019 [17] 5250RE RGB camera with DT20HI displacement measurement laser sensor Standard color segmentation algorithms

Pepper Detection

Sweet peppers from the 3D scene are divided utilizing shading data and restricted by fitting a 3D parametric model to the portioned purposes of sweet pepper. A key function of the system is the detection of fruits to be harvested. For successful operation, it is essential to accurately determine the three-dimensional (3D) location of each fruit. The chosen solution utilizes an RGB-D camera that simultaneously captures both color and depth information. Specifically, the Fotonic F80 camera (Fotonic Inc., Sweden) employs time-of-flight technology for depth measurements and uses a single sensor to capture both RGB and depth, ensuring fully synchronized channels. This camera has demonstrated robustness in varying outdoor lighting conditions, with a signal-to-noise ratio suitable for agricultural applications in greenhouses (Ringdahl et al., 2019). Using this camera, along with a custom-built LED illumination system, RGB images of the plant were captured from both an overview distance and close range. Detectability depended on both the imaging distance and the tilt of the sensing system (Hemming, Ruizendaal, Hofstee, & van Henten, 2014; Kurtser & Edan, 2018) [24,25] , so the sensors and illumination rig were consistently tilted upwards. To enable high frame-rate operation, a shape and color-based detection algorithm was implemented (Arad et al., 2019) [26] . Achieving color constancy, a critical factor in the algorithm's performance, was done by applying a filter to ensure even light distribution across the image. Prior to operation, the algorithm was calibrated using several sample images (2-3 images), and the user was prompted to select the target fruit. Based on the selected areas, the algorithm automatically set color thresholds for detection using statistical analysis. Calibration was required only once per pepper variety. Other methods to automatically determine thresholds were also explored, such as those developed by Ostovar, Ringdahl, and Hellström (2018) [9] . After calibration, the algorithm scanned each image for regions that matched the target color thresholds, refining the detections by eliminating those that fell outside predefined size limits.

Grasp Selection

An candidate grasp poses were computed with the help by utilizing the segmented sweet pepper point cloud. The manipulator rotates around the stem so that the stem is positioned behind the fruit as viewed by the camera. To achieve this, a method was developed to estimate the angle at which the fruit is positioned around the stem. Initially, the images were semantically segmented using a deep learning approach (Barth, IJsselmuiden, Hemming, & Van Henten, 2017, 2018) [28,29] to identify the fruit and stem. From these segmented images, the centers of the plant parts were calculated. The detected stem regions were processed using a Canny edge detector, and a Hough transform was applied to detect straight lines representing the stem's path.

A 200 × 200 pixel rectangular search region was then defined around the stem's center to locate the fruit. The largest connected region within this search area was identified as the fruit. To refine the position, a 125-pixel circular mask was applied around the center of mass of the fruit region, and the refined center of mass was selected as the final fruit center. The corresponding stem center was calculated on the same horizontal plane Once the centers of the fruit and stem were identified from one viewpoint, the position of the fruit around the stem (angle) was determined, using the known average distance between the fruit center and stem (4 cm) and the distance between the camera and the fruit surface (20 cm). However, since this method could not distinguish whether the fruit was in front of or behind the stem, a second viewpoint was required to confirm the correct angle.

The method was tested on 45 greenhouse images, where the actual angles between the fruit and stem were known. In 73% of the cases, the error was less than 25°, meeting the accuracy requirement set by the end effector's mechanical constraints (Barth, Hemming, & Van Henten, 2019) [30] .

Pepper Attachment

The end effector grips the sweet pepper before the fruit is detached. All end effectors are connected to a robotic arm or hand, as described in Table 02 below. [5] 9DOF manupulator 2. Lehnert et al., 2017 [13] 6DOF Universal Robotics UR5 3. Ringdahl et al., 2019 [17] 6DOF manipulator Fanuc LR Mate 200iD

Pepper Detachment

Detaching the pepper from the plant without causing harm. The sweet peppers are removed using a cutter, assisted by the end effector, as shown in Table 03 below. [5] Lip knife Lip type Scissor Fin type 2. Lehnert et al., 2017 [13] Oscillating blade Decoupled mechanism type 3. Ringdahl et al., 2019 [17] Oscillating blade Holder type

Platform Design software

The robot was implemented using a traditional ROS (Robot Operating System) programming framework, which has been employed in the development of agricultural robots, as detailed by Hellström et al. (2013) [9] . The system is constructed with a hybrid robot architecture that incorporates a state machine to execute a flowchart-based approach. This architecture allows for the integration of various components and processes, facilitating the advanced functionality and operational efficiency of the agricultural robot. The use of ROS programming, coupled with a state machine structure, provides a robust foundation for enhancing the performance and capabilities of the robot in agricultural applications.

Discussion

According to the experiment conducted by Bontsema et al.

(2015) [5] , the CROPS robot achieved a success rate of only 2 to 6 percent when harvesting sweet peppers from an unmodified crop. However, by simplifying the crop environment-specifically, by removing clusters of sweet peppers and obstructing leaves-the success rate for harvesting improved significantly, reaching up to 33 percent. The average cycle time for picking a fruit with this approach was recorded as 94 seconds.

In a different study by Lehnert et al. (2017) [13] , the Harvey robot harvester for sweet peppers featured a newly designed end-effector. This design led to successful harvesting of 58 percent and 42 percent of sweet peppers in two separate field trials, respectively, with varieties Claire and Redject. These figures account for both successful attachment and detachment of the peppers. The average time taken for harvesting in these trials was between 35 and 40 seconds. Furthermore, Ringdahl et al. (2019) [17] assessed two distinct approach strategies for sweet pepper harvesting using the SWEEPER robot. Their findings indicated that a single approach strategy was successful in 84 percent of cases, whereas a multiple approach strategy achieved success in only 49 percent of the cases. The average picking time was notably reduced to 24 seconds, as detailed in Table 04 below. [17] 1. Single approach strategy 84% 24 2. Multiple approach strategy 49%

All these robots are harvesting sweet pepper successfully, but still now these robotic harvester are not commercialized for the fresh market (bac et al., 2015) [3] . Every year, millions of tons of sweet pepper (Capsicum) are harvested manually in both open fields and greenhouses. This reliance on manual labor is largely due to a shortage of skilled workers and the increasing cost of labor, which has driven the agricultural industry to explore alternative solutions. One such solution is the development of robotic harvesters that can automate the process of sweet pepper harvesting in greenhouses. However, early robotic systems have faced challenges in achieving high success rates in harvesting.

For example, the CROPS robot, a harvesting robot developed for sweet pepper, has demonstrated limited success. When tested in unmodified crops, its harvest success rate was reported to be between only 2% and 6%. This low efficiency highlights the difficulties of designing robots that can operate effectively in complex agricultural environments where factors like plant structure and fruit positioning are variable. On the other hand, more advanced robotic systems, such as the Harvey robot, have been designed with improvements like a novel end-effector, which has enhanced its ability to harvest sweet peppers more efficiently. When tested on different sweet pepper varieties, Harvey was able to successfully harvest 58% of the Claire variety and 42% of the Redject variety. These success rates account for both the attachment and detachment of the peppers, indicating that while the robot still has room for improvement, it shows promise in automating the harvesting process. Another notable robot, SWEPPER, was evaluated using two different harvesting approach strategies: a single approach and a multiple approach strategy. The results of this evaluation showed that the single approach strategy had a significantly higher success rate of 84%, compared to the 49% success rate of the multiple approach strategy. This suggests that simplifying the robot's actions during the harvesting process may lead to better outcomes. In terms of efficiency, the average time required for robotic harvesting of sweet peppers varies significantly between different robotic systems. For instance, SWEPPER demonstrated an average picking time of 24 seconds per pepper, whereas the CROPS robot required an average of 94 seconds per pepper. This wide range in picking times further underscores the challenges and potential in the development of robotic harvesters. Despite these challenges, robotic harvesting systems hold great potential for addressing some of the major issues faced by growers today, such as labor shortages and rising costs.

As robotic technology continues to advance, it is expected that these systems will become more efficient and reliable, ultimately transforming the way sweet peppers and other crops are harvested in the future.

Conclusion

The majority of failures in the robotic harvesting of sweet peppers occur during the attachment stage. To improve the success rate of robotic harvesting, removing leaves and handling multiple fruits can increase the effectiveness by up to 27 percent. The time required for robotic harvesting varies significantly, with the average picking duration ranging from 24 seconds for the SWEPPER model to 94 seconds for the CROPS model. This variability highlights the potential of robotic harvesters to address several pressing issues faced by growers today, including labor costs, labor availability, and concerns related to food safety and quality. By overcoming these challenges, robotic harvesters could significantly enhance the efficiency and effectiveness of sweet pepper harvesting.

References

1. Bac CW, Roorda T, Reshef R, Berman S, Hemming J, van Henten EJ. Analysis of a motion planning problem for sweet-pepper harvesting in a dense obstacle environment. Biosyst Eng. 2016;146:85-97.
2. Bac CW, Henten EJ, Hemming J, Edan Y. Harvesting robots for high-value crops: state-of-the-art review and challenges ahead. J Field Robot. 2014;31(6):888-911.
3. Barth R, Hemming J, van Henten EJ. Design of an eyein-hand sensing and servo control framework for harvesting robotics in dense vegetation. Biosyst Eng. 2016;146:71-84.
4. Bontsema J, Hemming J, Pekkeriet E, Saeys W, Edan Y, Shapiro A, et al. CROPS: Clever robots for crops. Eng Technol. 2015;1(1):1-11.
5. Comba L, Gay P, Piccarolo P, Ricauda Aimonino D. Robotics and automation for crop management: trends and perspective. In: International Conference Ragusa SHWA2010. 2010. p. 471-478.
6. Eizicovits D, van Tuijl B, Berman S, Edan Y. Integration of perception capabilities in gripper design using graspability maps. Biosyst Eng. 2016;146:98-113.
7. R, Rodríguez F, Sánchez-Hermosilla J, Donaire JG. Navigation techniques for mobile robots in greenhouses. Appl Eng Agric. 2009;25(2):153-165.
8. Hellström T, Ringdahl O. A software framework for agricultural and forestry robots. Ind Robot Int J. 2013;40(1):20-26. Available from: https://doi.org/10.1108/01439911311294228
9. Hemming J, Bac W, van Tuijl B, Barth R, Bontsema J, Pekkeriet E, et al. A robot for harvesting sweet-pepper in greenhouses. Proc Int Conf Agric Eng. 2014; p. 6-10.
10. Jensen K, Larsen M, Nielsen SH, Larsen LB, Olsen KS, Jørgensen RN. Towards an open software platform for field robots in precision agriculture. Robotics. 2014;3(2):207-234.
11. King A. Technology: The future of agriculture. Nature. 2017;544.
12. Lehnert C, English A, McCool C, Tow AW, Perez T. Autonomous sweet pepper harvesting for protected cropping systems. IEEE Robot Autom Lett. 2017;2(2):872-879.
13. Mann MP, Zion B, Shmulevich I, Rubinstein D, Linker R. Combinatorial optimization and performance analysis of a multi-arm cartesian robotic fruit harvester-extensions of graph coloring. J Intell Robot Syst. 2016;82(3-4):399-411.
14. Mehta SS, MacKunis W, Burks TF. Robust visual servo control in the presence of fruit motion for robotic citrus harvesting. Comput Electron Agric. 2016;123:362-375.
15. Ringdahl O, Kurtser P, Edan Y. Evaluation of approach strategies for harvesting robots: case study of sweet pepper harvesting. J Intell Robot Syst. 2019;95(1):149164.
16. Sandini G, Buemi F, Massa M, Zucchini M. Visually guided operations in greenhouses. In: IEEE International Workshop on Intelligent Robots and Systems (IROS'90'). 1990. p. 279-285.
17. Shamshiri RR, Hameed IA, Karkee M, Weltzien C. Robotic harvesting of fruiting vegetables: a simulation approach in V-REP, ROS and MATLAB. In: Proceedings in Automation in Agriculture-Securing Food Supplies for Future Generations. 2018. InTech.
18. Shamshiri RR, Weltzien C, Hameed IA, Yule IJ, Grift TE, Balasundram SK, et al. Research and development in agricultural robotics: a perspective of digital farming. Int J Agric Biol Eng. 2018;11(4):1-14.
19. Lehnert C, English A, McCool C, Tow AW, Perez T. Autonomous sweet pepper harvesting for protected cropping systems. IEEE Robot Autom Lett. 2017;2(2):872-879.
20. Bac CW, van Henten EJ, Hemming J, Edan Y. Harvesting robots for high-value crops: state-of-the-art review and challenges ahead. J Field Robot. 2014;31(6):888-911.
21. Ringdahl O, Hellström T, Liljebäck P. Performance evaluation of a commercial robotic harvesting system for sweet peppers. Robot Auton Syst. 2019;116:13-24.
22. Hemming J, Ruizendaal J, Hofstee JW, van Henten EJ. Fruit detectability analysis for different camera positions in sweet-pepper harvesting. Sensors. 2014;14(4):6032-6044.
23. Kurtser P, Edan Y. A sweet pepper grasping approach for crop-robot interaction. Biosyst Eng. 2018;174:4556.
24. Arad B, Vitzrabin I, Edan Y, Berman S. Color and shape-based fruit detection algorithms for a robotic sweet-pepper harvesting system. Comput Electron Agric. 2019;162:338-345.
25. Ostovar A, Ringdahl O, Hellström T. Automatic detection and segmentation of sweet peppers for robotic harvesting. Comput Electron Agric. 2018;146:327-335.
26. Barth R, IJsselmuiden J, Hemming J, van Henten EJ. Data synthesis methods for semantic segmentation in agriculture: a Capsicum annuum dataset. Comput Electron Agric. 2017;144:284-296.
27. Barth R, IJsselmuiden J, Hemming J, van Henten EJ. Synthetic bootstrapping of convolutional neural networks for semantic plant part segmentation. Biosyst Eng. 2018;169:57-67.
28. Barth R, Hemming J, van Henten EJ. Optimising data synthesis for semantic segmentation in agriculture. Robot Auton Syst. 2019;115:49-61.