INVITA and AGFEML – Monitoring and extending the value of NVT trials

Take home message

Utilising digital technologies in national variety trials can be used to

• Provide verifiable records of trials through the season
• Assess trial quality and spatial variability at different stages of the season for different traits
• Improve confidence in estimates of yield performance.

Image analytics applied to phone, drone or ‘dashcam’ cameras have potential in research and production fields to quantify variation in plant, head count and other metrics and to map spatial variability in these measures across trials and fields.

For growers, we anticipate that these technologies will

• Improve the utility and prediction of variety performance in NVT to help growers choose varieties
• Be more accessible to growers and consultants via services offered by NVT contractors who have been trained via INVITA in use of UAVs and GPS tools
• Support commercial availability of spatial ‘counting’ methods in consultant and on-tractor imaging systems that will in future augment technologies like scouting, satellite mapping and yield mapping.

Aims

This paper overviews initial results from two complementary projects which started in 2020..

INVITA (INnovations in Variety Testing in Australia - UOQ2003-011RTX), in which UQ partners with CSIRO and WU (Wageningen University, The Netherlands), monitors the quality of national variety trials (through use of drone and phone camera based surveys) and aims to improve the utilisation of environment and observation data (drone imagery, weather data, satellite monitoring) in the process of prediction of variety performance. AGFEML (AGriculture Feature Extraction and Machine Learning - UOQ2002-008RTX) is a project that has worked with Arvalis (France) and the University of Tokyo to develop machine-learning image analysis techniques to accurately count wheat and sorghum heads in research and production fields using images from phone cameras and aerial UAVs (Unpiloted Autonomous Vehicles) . AGFEML is a pilot project in the GRDC Machine Learning program of research that was initiated in 2019 and aims to quantify spatial variation in the field as indicated by the changes in head density measured by imagery. The project has prototyped machine-learning cameras to be able to count heads in real-time, for example on a tractor ‘dash-cam’ type setup.

Background

The INVITA project was initiated by GRDC to leverage upon the $12M INVITE (INnovations in Variety Testing in Europe) investment by the EU Commission which began in 2018/19. INVITE involves a series of research activities to improve the process of variety testing across multiple EU countries and is led by INRAe (the French National Institute for Agriculture and Environment). UQ partnered with CSIRO and with Wageningen University (a leading partner in INVITE) to develop INVITA in Australia to build on findings in INVITE and to co-develop measurement and analysis technologies for the GRDC NVT.

Over the last 15 years or so, NVT has developed into one of the largest public variety testing programs in the world and provides Australian growers with timely information about performance which has been assured through investment in high quality experiment design, data cleaning and statistical analysis. INVITE and INVITA both have activities that aim to utilise additional phenotyping information (i.e., plant observations) using drone-based imaging, phone camera data collection, weather monitoring and satellite remote sensing in further improving performance prediction (Smith et al., 2021). In Australia, spatial field variation and year-to-year and location-to-location variation in weather have always been major potential sources of uncertainty in research experiments and these technologies aim to partially accommodate and account for spatial and temporal variation effects on crop growth and yield.

NVT and most plant breeding trials typically measure most traits (such as grain yield) at the plot level (i.e., one value per 7 x 2m plot area), and they account for field spatial effects using the methods developed and implemented by SAGI in the annual analyses of NVT. Imaging methods, especially from drones provide sub-plot resolution (<1 to 20cm pixel resolution) and can be used any time in the season. To date, most analytics from UAV images have been based around inferring crop cover and canopy height. In AGFEML we have particularly focused on improving methodologies to be able to ‘count’ heads of wheat and sorghum using phone and UAV cameras. Hence AGFEML outcomes contribute directly into INVITA in the first instance, with potential applications in other domains.

Methods

INVITA

INVITA data collection began in 2020 using only the main season wheat variety trials. INVITA has three major activities – data augmentation (collecting additional data using satellites, drones, weather stations etc), data analytics (statistical methods) and simulation and machine learning to interpret relationships among sensing and environment measurements and relations to NVT.

In 2020, we

• Established contracts with NVT trial service providers (TSPs) to include extra plots and organise additional data collection including drone training and GPS data collection
• Augmented data collection at up to 100 wheat main season variety trial sites, including satellite data and at 55 sites, additional measurements collected by UAV, high-resolution satellite (<1m res), phone cameras, biomass sampling, Greenseeker measurements, an IoT (internet of things) camera, canopy temperature sensor, as well as estimates of harvest index. We received a total of 344 UAV flights from the service providers, across 84 different sites. A total of 133 229 plot photos were uploaded across 58 sites.  Manual observations were recorded in spreadsheets for 43 sites
• Developed data management pipeline for largely automated processing of datasets (including UAV data via commercial partner) and establishment of data checking and filtering protocols
• Coordinated and initiated historical analyses of NVT wheat datasets with research partners (Wageningen University Research) and demonstrated capability to spatially account for variability in grain yield associated with early season scores and/or UAV derived data (e.g., fractional ground cover).

Table 1 shows the types of data and methods used by the INVITA project in NVT sites.

Table 1. Summary of data types, collection and spatial and temporal resolution

¹https://play.google.com/store/apps/details?id=lt.noframe.fieldsareameasure&hl=en_US&gl=US , https://apps.apple.com/gb/app/gps-fields-area-measure/id1123033235

The map (Figure 1) shows the distribution of trials and data collection for the 2021 INVITA measurements, overlaid on NVT trials. Trial  outlines were collected using the GPS Fields Area Measure App which allowed us to find trials and extract satellite data as well as to plan UAV flight missions etc. Intensive measurements were taken in 46 wheat main season trial sites (cameras, GoannaAg canopy temp sensors), with at least one UAV flight  conducted at approximately 80 sites. See Figure 2 for examples of field camera setup, in-season images and a trial image for NDVI of a reference trial with NVT entries which was grown at UQ Gatton in 2020. The field camera allows us to trace ground cover and phenology (e.g., flowering date) over the season via image analysis. In 2021, another 113 trials of wheat and other crops (barley, canola, chickpea, faba bean, field pea, lentil, lupin, oats) had at least one UAV flight planned. Sentinel-2 satellite data (10m resolution) were collected for all NVT crops at all sites, with approximately 55 sites monitored by high-resolution satellites (~ 30 cm pixel resolution). High-res satellites (Figure 1) may allow us to replace or augment UAV data as we work out how to potentially utilise findings from INVITA into future NVT operations. Regarding historical NVT, we have assembled all Sentinel-2 data back to 2016, as well as LandSat and Planet data as far as available. Due to issues in locating NVT trials, we have also developed a machine-learning assisted approach to ‘find’ the NVT trials in the satellite imagery.

Figure 1. Overview of main wheat trials and (in colour) the level of measurements being collected for INVITA in 2021 winter season. Example of different satellite resolutions in 2020 WMaA20BEVE6

Drone imagery from NVT trials is uploaded to a database and processed to generate images like that in Figure 3 which shows the variation in NDVI signal late in the season.  Here the red plots are in grain filling and the later-planted crops are still green.

The UAV and plot imagery have been further processed to estimate crop cover and crop height through the season. The aim is to analyse these data to see what they show about early season spatial variability, as well as whether these types of traits are related to the performance of varieties. We report on some of those outcomes in the results, although the main purpose of this paper is to discuss the way these technologies are being used to improve research trials and their availability to contractors for use in breeding and agronomy applications.

We have also begun developing analyses of simulations that are created from NVT trials. For these we use the APSIM model, measured weather data and satellite imagery. These are used to ‘tune’ APSIM in order to estimate soil parameters at the NVT site. INVITA has used NVT data to check predictions of flowering date in conjunction with models being developed by the GRDC National Phenology Initiative (ULA00011) and this information will allow us to create seasonal patterns of stress indices for drought, high temperature, frost etc and the occurrence of these in each NVT. Later in the project, such indices will inform statistical models that may be used to predict variety performance in relation to different patterns of stress, but this will take some validation before it would become available in NVT.

Figure 2. Early season 4G camera images from 2020 NVT with camera and spectral sensor shown on left and an example of camera photo masked to provide an estimate of ground cover from phone, field camera or UAV

Figure 3. Example of NDVI per plot data collected from analysis of a single UAV drone flight at UQ Gatton. This image is stitching together multiple images taken by a drone in a ‘lawnmower’ pattern that takes about 30 minutes per ha at this high resolution.

AGFEML

Machine learning (ML) technologies allow us to do some amazing things. For example, ML methods can now count objects efficiently from imagery and video, e.g. recognising and counting the heads of people in an airport. In this project, we have adapted these types of technologies to count ‘heads’ of wheat and sorghum. With our partners in Arvalis (France) and U Tokyo, we undertook several activities related to ‘head counting’. The first was to work with multiple universities and institutes to create the ‘Global Wheat Head Dataset (GWHD)’ and establish an online ‘competition’ (led by U Saskatchewan and coordinated by Arvalis) on the ‘Kaggle’ website for internet teams to count wheat heads (Figure 4). This had a great response (> 2000 teams) as did another competition in 2021 on the AICrowd (https://www.aicrowd.com/challenges/global-wheat-challenge-2021) website (>2500 teams) and provided rapid insight into what kind of expertise could inform the development of an analytics pipeline for the counting of wheat heads. This pipeline was designed to work using phone or ground images taken by researchers or contractors in NVT trials. There is also the potential to use such images in applications related to scouting for agronomic problems like heat and frost damage to heads.

Figure 4. The 1st GWHD competition on Kaggle https://www.kaggle.com/c/global-wheat-detection which attracted 2245 teams

Figure 5. UAV and ground platform photos for testing of wheat head counting. Data from France and from Australia (INVITA trial at Gatton)

The Arvalis team then developed models using the GWHD and applying the best methods and ideas from the competitions. Two contrasting methods (FasterRCNN and SFC2Net) were tested on a set of wheat head images (Figure 5) that had been collected as ground photos in two locations in France and in the INVITA trial at UQ Gatton.

The second major activity was to explore automation of sorghum head counting from UAV images. In this work, we wanted to establish a robust pipeline that would work well in diverse environments (Figure 5). Counting plant heads can be harder than counting human heads in a crowd – images of crops (populations of plants) have a much more uniform ‘style’ with most of the heads looking similar as well as the background looking similar. Hence, we need to train our system with multiple sets of images from different ‘domains’ (e.g., taken on different days or different times of development). One method we use for this is called GAN (Generalised Adversarial Networks) which were only invented in about 2014 (see here for some examples https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/). This is the same method that can be used to turn images of one animal into another or to ‘replace’ a person with a different person in an internet video – sometimes called ‘deep fakes’. After developing a robust method of head counting for sorghum, we also tested the method on wheat head datasets that had been collected in ground photo images using a machine-learning camera. In the presentation of this paper, we will show some of the results from the open-source machine-learning camera which we have utilised to demonstrate real-time counting of wheat heads in the field.

Figure 6. Analysis pipeline for counting sorghum heads from UAV images

Results

INVITA mapping spatial variation

For trials in 2020 and 2021, multiple UAV flights have been analysed to estimate the fraction of ground cover in NVT trials at different times of the season. These data are derived by extracting plot data from UAV data similar to that in Figure 3. Data for each plot are combined with design information and analysed using spatial statistical modelling like ASREML or SPATs (Rodriguez-Alvarez et al., 2020). An example is shown in Figure 7 for a range of ground cover from 0.2 to 0.7 early in season.

Figure 7. Spatial analysis of ground cover estimate for wheat main season trial, 2 July 2020. The lower left image shows the spatial trend which has been found in the data and has been adjusted in the estimates of the genotype means (the ‘BLUEs’)

These analyses of ground cover using a UAV provide a more objective measure of the within and between plot variation compared to visual scores, and we have shown that these ground cover estimates are relatively accurate and repeatable. Two questions of interest are:

• Designing criteria to make early-season decisions regarding trial progress, e.g. in situations of extreme spatial variability due to soil issues, rainfall events, crop emergence etc can we utilise these data to inform whether a trial should be abandoned early so that resources can be focused on other higher quality trials?
• Can these measures of ground cover provide an early-season indicator of yield? In Figure 8 we show genetic correlations between early season ground cover and final yield for 30 trials in 2020. In general, these correlations are positive and sometimes neutral, but in several trials the correlations were negative, i.e., early high ground cover was associated with lower final yield. In terms of agronomy, these negative correlations may be related to interactions with seasonal water and nutrient supply e.g., in a situation where rainfall is poor during the season, high early vigour can exhaust soil water supply and result in haying off and reduced grain yield. We are further investigating the seasonal conditions for these contrasting trials to try to better determine why/how negative correlations occur and their relationship to seasonal and soil conditions.

Figure 8. Map of Australia showing the genotypic correlations between yield and early season ground cover (GC1) (DAS < 60) for 30 trials in 2020. Colour shading indicates the strength and direction of the correlation, i.e. positive (blue) genetic correlation of yield and ground cover means that genotypes with better ground cover early in season had a better final yield.

Figure 9. Comparison of UAV (RGB <1cm resolution), Airbus (0.3m), Planet (3m) and Sentinel-2 (10m) images of trial WMaA20BENC6 during the mid-vegetation stage. The plot boundaries overlaid on these images show the limitation of Sentinel-2, in terms of spatial resolution.

Given results from the UAV imaging research, we can initially conclude that spatial analysis offers potential for trial monitoring and identifying sources of error that may impact on estimated of variety performance in trials. A challenge of using UAVs in the NVT is simply the cost and time required to make frequent visits to remotely-located trials. Hence, another aspect of INVITA research is to look at how spatial data from satellites may be utilised, especially to infill changes in spatial patterns between UAV flights. The cost of a seasonal set of higher-resolution satellite images (approx. once/month) is similar to the cost of a single UAV flight and processing. We are currently working on analyses of UAV and satellite data collected on the same dates and rescaling the different images to determine how much of the detailed spatial data in UAV images can be inferred through analysis of satellite images. This will determine how we can best manage the value of using UAV and satellite imaging techniques in the in-season monitoring of NVT trials.

INVITA tracking seasonal variation and weather

The simulation component of INVITA utilises the phenology models of APSIM to estimate the flowering time of trials and genotypes within trials. NVT trials are distributed over an extraordinary range of sites with many being several hours drive from locations of trial contractors (Figure 10). In this part of INVITA we aim to model the flowering time of NVT trials, and especially the genotypes if possible, learning from the outcomes of the GRDC National Phenology Initiative project led by James Hunt at LaTrobe. Our analysis of >21 000 flowering observations (Figure 11) shows that we now have good confidence in being able to predict trial flowering dates using weather data from nearby stations or recorded at NVT sites. This will allow us to characterise the likelihood that frost, heat or drought events were experienced at NVT sites and how these may have interacted with different varieties. The aim here is to have a clearer understanding of when weather events should be informing decisions around the utility of specific trials, e.g., were some genotypes particularly disadvantaged.

Figure 10. Analysis of 21 000 flowering observations across 2015 to 2020 in 310 trials at 129 locations. Dots show how many observations were used from each historical NVT location

Figure 11. Comparison of prediction of wheat flowering time using parameterisations of two types of APSIM phenology models. This shows we can predict flowering time of most trials and genotypes quite well from weather data alone.

AGFEML wheat head counting from ground images

Of the many activities undertaken in 2021, we report on two significant results here. The first is the result of the application of machine-learning models to count wheat heads in images taken by cameras over the top of field plots. The types of models tested and the image augmentation methods used were inspired by the GWHD competitions described earlier and summarised by David et al., 2020 and 2021. The images were taken using the same techniques in sites in France and in Australia (at a copy of the northern NVT which was grown at UQ Gatton). In these images, we had a plastic tubing frame of about 50 x 50 cm that was used as a boundary, and we counted all of the heads we could see, at the time of taking an image above the plants. The Arvalis team then took two models which had been trained on the GWHD (>150 000 labelled wheat heads from many different trials and locations and conditions) and made independent tests.

Table 2. Results from applying two different machine-learning models trained on the GWHD and tested on independent wheat head datasets in France and Australia (Gatton) (Where rRMSE = root mean square deviation; rBIAS = relative bias; and R² = the correlation coefficient)

The results in UQ Gatton (Table 2, Figure 12) were good across a large range (20 to 100 heads in the 0.25m² image with r2 of 0.71 or 0.86) and demonstrated that we should be able to take such images in NVT trials during early to mid grain-filling and be able to obtain reasonable estimates of head density. The object-based model (Faster-RCNN) tends to under-estimate the head number while the density-based model (SFC2Net) is generally more precise. The research team is working to determine issues around how/when the models are most suitable so that we might be able to automatically process ground photo imagery from NVT to obtain this data. The reason for interest in head number density is that our current yield analysis measures yield and grain size, so we can determine grain number per unit area, but do not have any measure of head number per area. Estimates of head number per unit area can inform us about which situations (soil + weather season) interact with traits like tillering (which increases head number per unit area) and how the balance of crop ‘investment’ in tillers can benefit or penalise potential yield for that situation.

Figure 12. Performance results from independent testing of two head counting algorithms (RCNN and SFC²Net) on quadrat counts of wheat heads in France and Australia (Gatton)

AGFEML sorghum head counting from UAV images

In the sorghum component of AGFEML, we assembled various datasets including those from UQ and from collaborators in a US DoE project based at Purdue University in the USA. These sorghum images all came from UAV datasets (Figure 13). By applying the GAN pipeline we described in Figure 6, we ‘converted’ UQ images into fake images by applying the ‘style’ from Purdue images. In Figure 13, it can be seen that in the ‘fake’ sorghum images in the 2^nd column have heads in the same positions as in the ‘real’ images’. We then put these ‘fake’ images back into the machine-learning model and train it to recognise these sorghum heads which look quite different to the originals. This greatly improves the model so that we only have an error of about 2 heads in 50, even when we only use 100 images to train the model (Figure 14). Training the model on both ‘real’ and ‘fake’ images makes it work much better than training only on ‘real’ images.

Figure 13. CutGAN 'fake' images generated using UQ image + Purdue ‘style’ (sorghum) and USaskatchewan + UTokyo ‘style’ (wheat). Note how the heads are in the same position in the ‘fake’ images as in the ‘real’ images. So now the ‘fake’ images can retrain the model in a new style.

Figure 14. Performance of sorghum head counting model trained on Original or Original+Synthetic images for different size datasets. Use of fake images makes model training work better

Using a modified version of the sorghum head-counting model, we developed a ‘rapid’ processing pipeline for a drone flight of 90 x 500m in size (Figure 15). In this pipeline, we can process each image from a drone and identify all heads within an image, and then assign automatically detect the rows in the image. This allows mapping of head count for every row and identification of gaps within rows which indicated problems with planter or in-season effects. The result is a detailed head density map and analysis of variation in head density for comparison to soil and yield maps.

Figure 15. Head and row detection in UAV images (top) and fitted map of head density for entire 90 x 500m field (below). These analyses can be generated from UAV images without full mosaic processing and are viable for computation in the field.

We took our sorghum head-counting model and then applied it to wheat datasets, using the GAN technique again to train the model on both ‘real’ and ‘fake’ images like those in Figure 13 (right). When we then implemented this model into a ‘machine-learning camera’, we could walk through field plots and take photos and obtain counts of all heads in the image as we recorded a 4K video on the camera. This demonstrates that it should be possible to develop a camera system that can be carried by a consultant (e.g., looking at head damage in wheat) or potentially installed on tractors to monitor head density in field conditions (Figure 16).

Figure 16. Real time 4K still image processing to count wheat heads using a machine-learning camera

Acknowledgements

INVITA is a large research project that runs at NVT sites and includes researchers from CSIRO (BZ) and WU (FvE, MB, DB-K, JR, JH). We greatly appreciate the work of the NVT trial service providers who have been contracted to undertake additional measurements on NVT and adjacent trials and install and maintain sensors to enable this project. AGFEML has been a collaboration with numerous researchers in France (Arvalis: BdS, SM, ED; INRAe: FB) and U Tokyo (WG).

The research undertaken as part of this project is made possible by the significant contributions of growers through both trial cooperation and the support of the GRDC, the author would like to thank them for their continued support.

References

David E, Madec S, Sadeghi-Tehran P, Aasen H, Zheng B, Liu S, Kirchgessner N, Ishikawa G, Nagasawa K, Badhon MA, Pozniak C, de Solan B, Hund A, Chapman SC, Baret F, Stavness I and Guo W (2020). Global Wheat Head Detection (GWHD) Dataset: A Large and Diverse Dataset of High-Resolution RGB-Labelled Images to Develop and Benchmark Wheat Head Detection Methods [WWW Document]. Plant Phenomics. https://doi.org/10.34133/2020/3521852

David E, Serouart M, Smith D, Madec S, Velumani K, Liu S, Wang X, Pinto F, Shafiee S, Tahir ISA, Tsujimoto H, Nasuda S, Zheng B, Kirchgessner N, Aasen H, Hund A, Sadhegi-Tehran P, Nagasawa K, Ishikawa G, Dandrifosse S, Carlier A, Dumont B, Mercatoris B, Evers B, Kuroki K, Wang H, Ishii M, Badhon MA, Pozniak C, LeBauer DS, Lillemo M, Poland J, Chapman S, de Solan B, Baret F, Stavness I and Guo W. (2021) Global Wheat Head Detection 2021: An Improved Dataset for Benchmarking Wheat Head Detection Methods. Plant Phenomics Volume 2021, Article ID 9846158, 9 pages https://doi.org/10.34133/2021/9846158

James, C, Gu, Y, Chapman SC, Guo W, David E, Madec S, Potgieter A and Eriksson A 2021. Domain Adaptation for Plant Organ Detection with Style Transfer. Digital Image Computing: Techniques and Applications (DICTA) 28 Nov to 1 Dec 2021, Gold Coast, Queensland, Australia. http://dicta2021.dictaconference.org/full_program.html

Rodriguez-Alvarez MX, Boer M, Eilers P and van Eeuwijk F (2020) SpATS-package: Spatial analysis of field trials with splines.

Smith DJ, Potgieter AB and Chapman SC (2021) Scaling up High-throughput Phenotyping for abiotic stress selection in the field. Theoretical and Applied Genetics 134: 1845-1866 https://doi.org/10.1007/s00122-021-03864-5

Velazco JG, Rodríguez-Álvarez MX, Boer MP, Jordan DR, Eilers PHC, Malosetti M and van Eeuwijk FA (2017) Modelling spatial trends in sorghum breeding field trials using a two-dimensional P-spline mixed model. Theoretical and Applied Genetics, 130, 1375-1392.

Contact details

Prof Scott Chapman
The University of Queensland
UQ Gatton Campus, Warrego Highway, Gatton 4343, QLD
Ph: 0417 076 787
Email: scott.chapman@uq.edu.au

GRDC Project Code: UOQ2003-011RTX, UOQ2002-008RTX,