Automation: opportunities for adoption in agriculture

Take home messages

Automation is a stepping-stone on the path to autonomy.
Automation reduces risk by increasing precision and repeatability.
Opportunities for farm automation are here now.

Background

Gains in productivity are being made through increased control of inputs and through precise timing of management operations, indicating a potential use case for increased integration of automation and autonomous systems on-farm.

Original equipment manufacturers (OEMs), such as John Deere and Case IH, are primarily developing machinery with an increasing level of automation leading to full autonomy.

However, there is a lack of consumer available functionality which provides active perception and decision making to on-farm machinery. This limits the potential of the automated technologies to situations where an operator can be present, limiting operations both by the number of automation capable and equipped machines and by the number of available skilled operators.

While automated systems are widely used in agriculture today, a focus will need to be on bringing active perception and intelligent decision-making tools to automated systems, allowing for fully autonomous systems to be viable and actively adopted in agriculture. This will allow the removal of the operator, enabling operations to be completed with increased efficiency.

Fundamentals of automation for agriculture

Defining automation to autonomy

The initial step towards on-farm autonomous machinery is automation. Automation allows a machine to complete a series of functions automatically. This differs to autonomy where the machine must complete a task without human assistance while having sufficient skill to complete that task in changing environments. This difference is illustrated when looking at a machine which can be operated using an automatic guidance system on pre-defined tracks and a machine which can follow those pre-defined tracks, identify a perceived obstacle and make a decision regarding the interaction of the current operation and that obstacle (Figure 1).

Figure 1. A) An automatic machine requiring human input to make a decision and avoid an obstacle. B) An autonomous machine identifying an obstacle and making a decision regarding interaction with that obstacle.

Figure 1. A) An automatic machine requiring human input to make a decision and avoid an obstacle. B) An autonomous machine identifying an obstacle and making a decision regarding interaction with that obstacle.

Autonomy is driven by perception which can be broken down into two groups:

Locative Perception where the machine knows something about its surroundings and its place in those surroundings.
Task Perception where the machine knows something about what it is doing and how well this task is being done.

Machine objectives

The key features which enable tractor autonomy are defined in Baillie et al. (2020, 2017) and include the automated systems which work together to provide an autonomous solution, such as automated tractor guidance systems, variable rate technology (VRT), sensing and perception systems and in-field communication.

Currently, OEMs and third-party equipment manufacturers do not offer commercial autonomous navigation perception or sensing options for integration into their existing automation technology stacks. These options are presently limited to expensive, single-use technologies from high-tech start-ups. However, perception technologies currently exist within OEM and third-party technology stacks for other uses and these companies are actively investing in and exploring autonomous use cases, as evidenced by John Deere’s recent acquisition of Bear Flag Robotics (John Deere, 2021a) and John Deere, Case New Holland (CNH), and Kubota’s development of autonomous tractor concepts (Case IH, 2021a; John Deere, 2021b; Wilson, 2020) (Figure 2).

Figure 2. Kubota X autonomous tractor (Topspeed Media, 2021).

Figure 2. Kubota X autonomous tractor (Topspeed Media, 2021).

Current state-of-the-art

Currently, the majority of OEM and third-party provider offerings revolve around automated guidance systems and VRT, with some expansion into machine vision and perception as operator assistance tools. All major OEMs provide an option for automated tractor guidance through GPS and RTK correction systems (Case IH, 2021b; CLAAS, 2018a; Deutz Fahr, 2021; Fendt, 2021; John Deere, 2021c) as well as VRT application (Case IH, 2021c; CLAAS, 2018b; John Deere, 2021d). However, there is limited product development of perception technologies by OEMs for active obstacle avoidance. Current perception technologies developed by OEMs are focused on in-row wheel guidance and harvester optimisation.

Locative perception

CNH provides a range of sensing and perception technologies through their technology stacks with a focus on in-crop row guidance and harvester optimisation. AFS RowGuide™ provides mechanical deflection-based row guidance (Case IH, 2021d), while Cruise Cut provides laser-based harvester guidance through the detection of the crop edge (Case IH, 2021b).

CLAAS’s range of sensing and perception technologies have also been developed primarily around in-crop row guidance and harvest optimisation. CLAAS provides three separate sensor-based automatic steering options for their harvesters, AUTO PILOT, CAM PILOT and LASER PILOT (CLAAS Group, 2021). CLAAS AUTO PILOT uses two mechanical deflection-based sensors to determine the position of maize rows relative to the harvester. CLAAS CAM PILOT builds on this using a stereo camera to control automatic steering in grass harvesting operations. Finally, CLAAS LASER PILOT uses a LiDAR to guide the harvester along the cut edge of cereal crops.

John Deere provides sensing options for in-crop guidance through AutoTrac^TM Vision. AutoTrac^TM Vision uses a single frame-mounted camera using machine vision to keep sprayer wheels in between crop rows (John Deere, 2021e).

SAME Deutz-Fahr (SDF) developed Driver Extended Eyes to assist the operator in navigating machine blind spots. This technology uses an intelligent control unit to actively monitor up to six cameras, extending the view of the driver to the left and right and automatically preventing tractor movement if a human obstacle is detected by cameras (SDF Group, 2021). SDF also uses their machine vision camera technology to provide automatic trailer hitch coupling using environmental recognition to control tractor movement. These cameras actively identify the position of an implement in relation to the tractor and the control system calculates the best path for coupling to the implement, automatically performing all necessary movements (SDF Group, 2021).

Raven Precision, recently acquired by CNH Industrial (Raven Precision, 2021a), has also developed in-row guidance systems and offers the VSN® Visual Guidance Sensor, a stereo vision camera-based machine vision system which can navigate crop rows up to full canopy closure (Raven Precision, 2021b). Raven’s AutoBoom® XRT uses radar and optical obstacle detection to maintain optimal spray height (Raven Precision, 2021c).

Raven Precision is on the leading edge of full on-farm autonomy with their OMNiDRIVE^TM and OMNiPOWER^TM systems. Raven’s OMNiDRIVE^TM system is an autosteer system which allows an operator to monitor and control a driverless tractor from the cab of the harvester (Raven Precision, 2021d). The autonomous tractor can return to a predetermined unloading area when the grain cart is full. Raven’s OMNiPOWER^TM is a self-propelled power platform which allows fully autonomous control of field tasks which can be remotely commanded and supervised. However, OMNiPOWER^TM requires specially made implements and currently only supports a nutrient applicator spreader and a VRT spray system (Raven Precision, 2021e).

Task perception

CNH developed AFS Harvest Command for Case IH machines which provides automatic adjustments to the harvester in real-time as conditions change (Case IH, 2021e). CNH also provides IntelliFill^TM, a 3D camera-based system to detect trailer edges and monitor forage harvester filling on New Holland machines (New Holland Agriculture, 2021).

CLAAS offers variable rate harvesting control and automatic trailer filling using a 3D camera to detect trailer edges and monitor and adjust the upper grain discharge chute during cart fill (CLAAS, 2021).

John Deere also provides sensing options for automatic grain-cart filling and real-time spot spraying. John Deere’s Active Fill Control controls the harvester discharge chute similar to CLAAS’ system, enabling automated cart filling (John Deere, 2021f). John Deere’s See & Spray^TM Select uses an integrated camera and machine vision algorithms to automatically detect green plants within fallow ground, triggering a chemical application (Burwood-Taylor, 2017).

Future opportunities

As OEMs prepare for the future, they are looking towards technologies such as big data, artificial intelligence, unmanned aerial vehicles (UAVs) and electrification. CNH has produced two autonomous tractor concept vehicles to date. One tractor was a futuristic concept cab-less Case IH while the New Holland NH^Drive concept resembles a retrofit tractor with a cab which allows operator control. Both CNH vehicles use AI, GPS/RTK, LiDAR, radar and RGB camera sensors to run autonomously. John Deere has developed both a concept autonomous one axle electric-drive tractor and an autonomous electric-drive sprayer, both concepts were unveiled at Agritechnica 2019. The electric-drive sprayer is fully autonomous with a 30-foot boom, 560-litre spray tank, independent four-wheel steering and tracks to reduce ground pressure (Francis, 2019). John Deere has indicated that they will consider improvements to battery life, speed and autonomous filling (Real Agriculture, 2019). John Deere has actively pursued these battery improvements as evidenced by their recent acquisition of majority ownership in Kreisel Electric, a high-density, high-durability battery manufacturer (John Deere, 2021g). Autonomous filling was revealed at Agritechnica 2019 on John Deere’s semi-autonomous tractor concept as part of their Automation to Autonomy core technology stack for the future of farming (John Deere, 2021b). The spray tank on this machine is capable of being filled completely automatically to reduce operator exposure to pesticides. John Deere’s autonomous electric one axle tractor concept removes operating emissions and greatly reduces noise levels (John Deere, 2021b). John Deere has also developed a Command Cab, a future tractor cab concept which John Deere views as an intermediate step from automation to autonomy, integrating big data to provide a mobile command centre for the farm (John Deere, 2021b). Finally, John Deere has announced the development of a bolt-on package of hardware and software that uses machine learning to create a fully autonomous tractor which will begin production in late 2022 (Vincent, 2022). If successful, John Deere may have the first widespread, OEM developed autonomous tractor kit on the market.

Multiple manufacturers are also developing autonomous UAV spray systems. John Deere is developing an autonomous swarm drone sprayer concept. The drones are equipped with John Deere’s weed identification technologies and each carry a 10.6L spray tank which is filled automatically at a field boundary station (John Deere, 2021b). Kubota is also developing pesticide spraying drones, integrating environmental data into their current decision system software and aim to optimise planting patterns through AI decision systems (Kubota, 2021). Kubota has developed a three-step roadmap to autonomy (Kubota, 2021). This revolved around Kubota’s Agri Robo Tractor launch in 2017, followed by the Agri Robo Combine Harvester in 2018 under the plan. Kubota further plans to implement unmanned operation across multiple fields and enable remote monitoring of equipment travel on farm roads. While Case IH has not yet signalled anything around the development of UAV spray systems, they are actively collaborating with DroneDeploy® to provide a drone package which monitors crop health for rapid decision-making (Case IH, 2021f).

Discussion

Current OEM automation technologies enable the majority of features which are required for full machine autonomy. However, they are limited by lack of perception integration options. Major OEMs are developing autonomous concepts which incorporate AI-driven perception systems (namely LiDAR, radar, stereo cameras). Additionally, these perception technologies exist within the OEMs for other uses, such as increasing operator safety or reducing fatigue.

To determine how sophisticated autonomy enabling technologies, such as perception, need to be an operating design domain (ODD) needs to be defined. This ODD specifies what the machine will be tasked for and therefore informs the technical requirements. Currently, there is an opportunity to better define specific applications for autonomy and the ODD with the OEMs. This will allow OEMs to refine the use case and the cost benefit to farmers.

In addition to the technical impediments to autonomy, there are other barriers and, in particular, the safe operation of this equipment and where the various responsibilities exist for the technology users, dealers, service providers and manufacturers. The code of practice for autonomous machinery, led by Grain Producers Australia in collaboration with industry partners , a world first, has been developed to better define these responsibilities (GPA, TMA, SPAA, 2021).

The integration of AI-driven machine perception for navigation by OEMs into their technology stacks and the development of ODDs within these OEMs will likely be the catalyst to allow for the uptake of true on-farm automation. This will, in turn, bring regulatory and legislative attention to on-farm automation technologies and allow for regulatory requirements to begin to be seriously discussed. The Code of Practice has helped put Australian agriculture on the front food in this regard. The gradual implementation of electrification will also inform the pathway to autonomy.

Conclusion

True on-farm autonomous vehicles are limited in use as they are often developed by high-tech start-ups and are targeted towards the specialty or high-value crop markets. OEM driven machine autonomy is limited by lack of AI-driven perception integration options for autonomous navigation and the lack of defined ODDs within these OEMs. With current GPS autosteer, end-of-run sequence automation and communication capabilities, current OEM and third-party options can be integrated to make a machine which can run fully autonomously. However, OEMs put safety mechanisms in place to ensure operator presence as the current machines lack perception integration. These same OEMs are currently developing autonomous concepts but have yet to make them available to consumers, instead focusing on the development of integrated technology stacks, progressively increasing machine automation. Further hurdles to full autonomy include the lack of a direct path to market and the lack of standard regulatory requirements. An AI-driven perception technology, likely developed by an OEM or major third-party provider after the definition of an ODD, which is compatible with current technology stacks, will be the catalyst for the transition from on-farm automation to true autonomy. These developments are beginning to be seen with John Deere’s announcement of a commercial autonomous kit for their 8R series tractor in January 2022 (Vincent, 2022). John Deere have defined the ODD for the specific use case, operating a chisel plow in US broadacre cropping systems and notably connected to the farmer via a smart phone to provide supervision. It is expected that the ODD will be developed as the capabilities of the technology advance and incorporate other use cases, including adoption of that technology in Australia.

Acknowledgements

The research undertaken as part of this project is made possible by the significant contributions of growers through both trial cooperation and the support of the GRDC, the author would like to thank them for their continued support. We would like to acknowledge Sugar Research Australia and the Cotton Research and Development Corporation who have contributed funding to the previous work undertaken by the University of Southern Queensland’s Centre for Agricultural Engineering.

Review of Technologies, Regulations and Operating Standards for Field Based Autonomous Agricultural Machinery

CPA Code of Practice

References

Baillie C, Lobsey, C, McCarthy C, Antille D, Thomasson J, Xu Z, Sukkarieh S (2017) Developments in autonomous tractors. Grains Research & Development Corporation

Baillie C, Torrance L, Long D, Pothula A, Brett P, Humpal J (2020) Review of technologies, regulations and operating standards for field based autonomous agricultural machinery. Grains Research & Development Corporation

Burwood-Taylor L (2017) John Deere acquires ‘see & spray’ robotics startup Blue River Technology for $305m. Ag Funder News

Case IH (2021a) Autonomous farming

Case IH (2021b) Guidance and steering

Case IH (2021c) Section & rate control

Case IH (2021d) AFS RowGuide^TM

Case IH (2021e) AFS harvest command^TM

Case IH (2021f) Case IH UAV

CLAAS (2018a) Automatic steering systems

CLAAS (2018b) Variable-rate drilling | base fertiliser application | liming

CLAAS (2021) Operator assistance systems

CLAAS Group (2021) Satellite-based GNSS steering systems