基于图像和卷积神经网络的蝴蝶兰种苗生长势评估.pdf

第 36卷 第 9期 农 业 工 程 学 报 V ol 3 6 N o 9 2019年 5月 Transactions of the Chinese Society of Agricultural Engineering May 2020 185 Image based assessment of growth vigor for Phalaenopsis aphrodite seedlings using convolutional neural network Zhu Fengle 1 Zheng Zengwei 1 2 1 Intelligent Plant Factory of Zhejiang Province Engineering Lab Zhejiang University City College Hangzhou 310015 China 2 School of Computer image processing convolutional neural network seedling fine tuning feature visualization assessment doi 10 11975 j issn 1002 6819 2020 09 021 CLC number Documents code A Article ID 1002 6819 2020 09 0185 10 Zhu Fengle Zheng Zengwei Image based assessment of growth vigor for Phalaenopsis aphrodite seedlings using convolutional neural network J Transactions of the Chinese Society of Agricultural Engineering Transactions of the CSAE 2020 36 9 185 194 in English with Chinese abstract doi 10 11975 j issn 1002 6819 2020 09 021 http www tcsae org 朱逢乐 郑增威 基于图像和卷积神经网络的蝴蝶兰种苗生长势评估 J 农业工程学报 2020 36 9 185 194 doi 10 11975 j issn 1002 6819 2020 09 021 http www tcsae org 0 Introduction Phalaenopsis commonly known as moth orchid are among the most valuable and popular potted flowering plants and cut flowers due to their outstanding floral Received date 2020 01 18 Accepted date 2020 04 14 Foundation item Natural Science Foundation of Zhejiang Province China LGN20F020003 Biography Zhu Fengle Lecturer research interests digital agriculture plant phenotyping image processing machine learning Email zhufl Corresponding author Zheng Zengwei Professor research interests internet of things wireless sensor network location based service digital agriculture pervasive computing Email zhengzw appearance longevity and year round availability 1 Nowadays Phalaenopsis has been widely cultivated as ornamental flowering plants thanks to recent improvements in the propagation and cultivation techniques as well as in the efficiency of supply chains 2 Large scale potted Phalaenopsis production with considerable commercial returns is now taking place in various regions around the world such as the Netherlands United States Germany southern China etc In the United States 75 of all orchids purchased are Phalaenopsis 1 The modern propagation of Phalaenopsis is generally carried out through tissue culture under laboratory conditions Then the further cultivation process of the tissue culture seedlings in the greenhouse is divided into three stages vegetative cultivation spike induction and 农业工程学报 http www tcsae org 2020年 186 flowering 3 The vegetative growth is the basis of high quality flower production Phalaenopsis seedlings in this stage play an important role in the whole production chain affecting the final economic benefits of flower growers Phalaenopsis seedlings must be healthy or mature enough before they can be moved to a separate greenhouse for the second stage of being induced to spike 4 Thus growers are deeply concerned about the growth vigor status healthy or weak of seedlings at the end of vegetative cultivation each seedling must be assessed individually to pick out the healthy ones The growth vigor of Phalaenopsis seedlings is assessed in many aspects primarily considering the plant phenotypic morphology including the number of leaves the leaf span the morphology of leaves if the new leaf is larger than the previous one 4 the height of plants etc Currently the assessment of growth vigor mainly relies on manual inspection for each Phalaenopsis seedling which may vary with the inspector s experience Measuring the morphological properties of seedlings by instruments is an alternative 5 but this approach is labor intensive and time consuming limiting its feasibility for large scale greenhouse production Therefore research on the application of a fast and non destructive approach to assessing the growth vigor of seedlings is important for the Phalaenopsis industry to improve productivity while alleviating labor costs Visible light images e g Red Green Blue RGB with commercial digital cameras or cell phones are highly affordable and beneficial in evaluating the phenotypic growth traits of plants in a non destructive and high throughput manner The targeted task in the present study belongs to plant level phenotyping in the controllable environment e g greenhouse according to the comprehensive review of multi scale plant phenotyping 6 Related publications with RGB image on the growth vigor of plants seedlings focused on extracting the phenotyping features derived from image analysis such as projected leaf area convex hull area plant height and width plant aerial density etc 2 7 13 Linear regression analyses were commonly carried out to correlate the image extracted features and measured morphological and physical traits including biomass leaf area height 7 10 11 Besides the linear regression Back Propagation Neural Network BPNN was also adopted in the study of flask seedlings of Phalaenopsis to calibrate 6 measured physical properties with 16 image features 2 Statistical analysis was performed to examine the relation between the growth of Phalaenopsis leaves in terms of calculated leaf area and the greenhouse environmental factors to identify the optimal cultivation conditions for Phalaenopsis 8 A more recent study approached the plant growth problem by classifying the growth stages of bean plants using fuzzy logic based on features extracted from RGB images 13 The above researches have been conducted to explore the feasibility of assessing the growth vigor of seedlings using conventional image processing techniques based on hand crafted features user defined prior features extracted from images followed by statistical analysis or machine learning algorithm The feature extraction procedure in the traditional image analysis involves time consuming trial and error steps and its effectiveness may depend on the experience of the data scientist 14 Recent advances in deep learning have remarkably impacted the machine learning domain and have been reported to achieve state of the art performance on many computer visions tasks 15 16 One of the fundamental advantages of deep learning lies in the automatic hierarchical feature extraction process via learning a stack of multiple linear and nonlinear layers before performing decision making Thereby deep learning models can leverage raw data trained in an end to end approach without the requirement to manually design a suitable feature extractor 17 Over the past few years major research efforts on computer vision focused on convolutional neural networks commonly referred to CNNs due to their capability to hierarchically abstract representations with local operations 15 The objective of the present study was the image based assessment of growth vigor for Phalaenopsis seedlings focusing on their morphological classification using CNN Through the research this was the first study using deep learning models to explore its feasibility for the growth assessment of orchid seedlings from a commercial greenhouse in an end to end manner Different CNN architectures VGG ResNet Inception v3 coupled with various training mechanisms were evaluated Considering that the target task focused on the morphological classification of individual seedlings in greenhouse conditions with complex image backgrounds adding images depicting the holistic plant morphology with the segmented background as augmenting samples might boost the learning performance of models Hence additional seedlings images were acquired in a controlled laboratory environment which was used to augment the classification models in two approaches Moreover feature maps were visualized to reveal the learning process of the CNN model 1 Material and methods A general overview process for seedlings growth assessment was illustrated in Fig 1 The whole procedure mainly included image acquisition image preprocessing the establishment of baseline models and augmented models Fig 1 A general overview process for seedlings growth assessment 第 9期 朱逢乐等 基于图像和卷积神经网络的蝴蝶兰种苗生长势评估 187 An overview of the greenhouse dataset and laboratory dataset including number proportion and purpose was displayed in Table 1 The detailed procedure of dataset acquisition and splitting for the model establishment was provided in the following sections Table 1 Overview of the greenhouse dataset and laboratory dataset Number of seedling images Number of image sets Dataset Total Healthy Weak Training Validation Test Purpose Greenhouse dataset 1840 900 940 920 50 552 30 368 20 Baseline models Augmented models Laboratory dataset 960 480 480 672 70 288 30 Augmented models Note Number in the represents the proportion of the whole image set 1 1 Image acquisition Seedlings of Phalaenopsis aphrodite subsp formosana Zi Shan were grown in translucent plastic pots in a commercial greenhouse located in Hangzhou China In August 2019 seedlings reaching their minimum growth time of vegetative cultivation were manually assessed by experienced botanists for their growth status labeling each one as healthy or weak The manual labeling was primarily concerned with the plant phenotypic morphology healthy seedlings were characterized with larger total leaf area larger averaged leaf area and larger leaf area for newer leaves than older ones RGB images were acquired using the camera of iPhone XS with a spatial resolution of 3 024 4 032 pixels Phalaenopsis aphrodite seedling was asymmetric in shape its leaves were grown mainly along the long axial direction with overlapping in the short axial direction Oblique view from the top facing the long axial direction provided the best site for observing the morphological characteristics of whole seedlings Thus each seedling was imaged from the two opposite long axial directions with the diagonal field of view of approximate 45 yielding two oblique view images Image collection was first conducted for 450 healthy seedlings and 470 weak seedlings in greenhouse conditions captured with seedlings on the cultivation bed producing 1 840 images 900 for healthy seedlings 940 for weak seedlings The camera was mounted on a movable tripod situated plumb 0 6 m over the cultivation bed The temperature and relative humidity of the greenhouse was maintained at 28 5 33 4 and 69 3 86 7 respectively suitable for the vegetative growth of Phalaenopsis aphrodite The collection of greenhouse dataset was taken across four different days 9 00 12 00 and 14 00 17 00 per day with varying circumstances and weather conditions Hence image background was complex due to the presence of some leaves from surrounding seedlings different illumination conditions various greenhouse settings e g cultivation bed water pipeline and irrelevant objects e g gloves shoes Since the task under consideration was a morphological classification problem for individual plants images of holistic seedlings without complex background might help the learning process of deep learning models Besides the greenhouse dataset a portion of the labeled seedlings was randomly selected for image acquisition under a controlled environment using similar imaging angle and distance as above regarding the laboratory dataset 480 images for healthy seedlings 480 images for weak seedlings in this study The controlled environment mainly referred to a uniform background and consistent lighting intensity without obvious shadows to facilitate background segmentation in the processing of laboratory dataset Fig 2 showed representative images in the greenhouse dataset and laboratory dataset with obviously increased complexity in the image background in the former dataset a Weak seedling in greenhouse b Healthy seedling in greenhouse c Weak seedling in laboratory d Healthy seedling in laboratory Fig 2 Representative images of weak and healthy seedlings in greenhouse dataset and laboratory dataset 1 2 Image preprocessing and basic data augmentation For all models in this study the following image preprocessing and basic data augmentation were implemented All captured images were first resized to 604 806 pixels and cropped to 604 604 pixels by removing the bottom part of the image to improve the efficiency of image processing while eliminating unnecessary background information To conform to the input requirement of CNNs three basic image preprocessing steps were then carried out for all images Firstly images were resized to 224 224 pixels size for ResNet and VGG 299 299 for Inception v3 Secondly all pixel values were divided by 255 to be compatible with the network s initial values Finally normalization was performed for each channel to improve training efficiency 18 Current CNNs heavily rely on learning and very large labeled datasets to achieve high performance 15 In the case of an insufficient dataset which often happens in plant science a very useful and practical approach is to augment the dataset The basic and most widely employed data 农业工程学报 http www tcsae org 2020年 188 augmentation technique is geometric transformations in studies regarding plants seedlings 18 21 The following geometric transformations were adopted as basic data augmentation techniques in this study horizontal axis flipping slight rotations between 15 15 and Gaussian noise injection These applied transformations could occur in images captured under practical greenhouse conditions while not much altering the holistic morphology of seedlings in images Only training images were subjected to the basic data augmentation in an online manner Specifically the train set was not augmented before training and during training transformations were applied to each training image at each epoch 22 The online augmentation advantaged over offline augmentation in terms of saving storage space 1 3 Model architectures The three basic CNN architectures being investigated in this work concerning the growth assessment of seedlings from their images were the following i VGG 23 ii ResNet 24 iii Inception v3 25 The basic building blocks of a classical CNN included a convolution layer a nonlinearity or rectification layer a normalization layer and a pooling layer 26 VGG was the first deep CNN architecture focusing on the investigation of increasing network depth using very small convolution filters 23 VGG16 the 16 weight layers version of the network was applied in this study ResNet stood for the residual network highlighted in its residual learning via the use of skip connections 24 The 34 layer ResNet was employed The Inception v3 was an improved inception architecture consisting of stacked inception modules in which convolution operations at various scales and spatial pooling happen in parallel 25 For all these three CNN architectures the final layer was a fully connected layer with Softmax output To adapt to the present classification task the final layer of each network architecture was reshaped to maintain the same number of inputs as before and to have the same number of outputs as the number of classes in the prese