No.1 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサー)

タイトル 
非線形パーセプトロンを用いた相互学習

著者 
斎藤 大輔、原 一之

誌名 
Journal of Artificial Intelligence and Soft Computing Research

出版者 
Walter de Gruyter GmbH

巻号頁 
5/ 1, 7177

出版日 
2015/04/30

ISSN 
20832567

DOI 
10.1515/jaiscr20150020

URL 

概要 
We propose a mutual learning method using nonlinear perceptron within the framework of online learning and have analyzed its validity using computer simulations. Mutual learning involving three or more students is fundamentally different from the twostudent case with regard to variety when selecting a student to act as the teacher. The proposed method consists of two learning steps: first, multiple students learn independently from a teacher, and second, the students learn from others through mutual learning. Results showed that the mean squared error could be improved even if the teacher had not taken part in the mutual learning.

機関リポジトリ 


No.2 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(第一著者)

タイトル 
Theoretical analysis of learning speed in gradient descent algorithm replacing derivative with constant

著者 
Kazuyuki Hara amd Kentaro Katahira

誌名 
Information Processing Society of Japan Transactions on Mathematical Modeling and Its Applications

出版者 
日本情報処理学会

巻号頁 
6/ 3, 100105

出版日 
2013/12/01

ISSN 
03875806

DOI 
10.11185/imt.9.61

URL 

概要 
In online gradient descent learning, the local property of the derivative term of the output function can slowly converge. Improving the derivative term, such as by using the natural gradient, has been proposed for speeding up the convergence. Beside this sophisticated method, we propose an algorithm that replaces the derivative term with a constant and show that this greatly increases convergence speed when the learning step size is less than 2.7, which is near the optimal learning step size. The proposed algorithm is inspired by linear perceptron learning and can avoid locality of the derivative term. We derived the closed deterministic differential equations by using a statistical mechan ics method and show the validity of theoretical results by comparing them with computer simulation solutions. In real problems, the optimum learning step size is not given in advance. Therefore, the learning step size must be small. The proposed method is useful in this case.

機関リポジトリ 


No.3 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(第一著者)

タイトル 
Statistical Mechanics of NodePerturbation Learning for Nonlinear Perceptron

著者 
Kazuyuki Hara, Kentaro KATAHIRA, Kazuo OKANOYA, and Masato OKADA

誌名 
Journal pf the physical society of Japan

出版者 

巻号頁 
82, 054001

出版日 
2013/04/01

ISSN 

DOI 
10.7566/JPSJ.82.054001

URL 

概要 
Nodeperturbation learning is a type of statistical gradient descent algorithm that can be applied to problems where the objective function is not explicitly formulated, including reinforcement learning. A nonlinear perceptron has several advantages over a linear perceptron, such as the ability to use nonlinear outputs, learnability, storage capacity, and so forth. However, nodeperturbation for a nonlinear perceptron has yet to be analyzed theoretically. In this paper, we derive a learning rule of nodeperturbation learning for a nonlinear perceptron within the framework of REINFORCE learning and analyze the learning behavior by using statistical mechanical methods. From the results, we found that the signal and crosstalk terms of the order parameter Q have different forms for a nonlinear perceptron. Moreover, the increase in the generalization error with increasing number of outputs is less than for a linear perceptron.

機関リポジトリ 


No.4 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(第一著者)

タイトル 
Statistical Mechanics of Online Ensemble Teacher Learning through a Novel Perceptron Learning Rule

著者 
Kazuyuki Hara, Seiji Miyoshi

誌名 
Journal of the physical society of Japan

出版者 

巻号頁 
81, 06400216

出版日 
2012/06/15

ISSN 

DOI 
10.1143/JPSJ.81.064002

URL 

概要 
In ensemble teacher learning, ensemble teachers have only uncertain information about the true teacher, and this information is given by an ensemble consisting of an infinite number of ensemble teachers whose variety is sufficiently rich. In this learning, a student learns from an ensemble teacher that is iteratively selected randomly from a pool of many ensemble teachers. An interesting point of ensemble teacher learning is the asymptotic behavior of the student to approach the true teacher by learning from ensemble teachers. Perceptron learning with a margin is identical to the perceptron learning rule when the margin is zero and identical to the Hebbian learning rule when the margin is infinity. Thus, this rule connects the perceptron learning rule and the Hebbian learning rule continuously through the size of the margin. From the results, we show that by setting a margin of κ>0, the effect of an ensemble appears and becomes significant when a larger margin κ is used.

機関リポジトリ 


No.5 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Statistical Mechanics of Online Nodeperturbation Learning

著者 
Kentaro Katahira, Kazuo Okanoya, Masato Okada

誌名 
Information Processing Society of Japan Transactions on Mathematical Modeling and Its Applications

出版者 

巻号頁 
4/ 1, 7281

出版日 
2011/01

ISSN 

DOI 

URL 

概要 
Nodeperturbation learning (NPlearning) is a kind of statistical gradient descent algorithm that estimates the gradient of an objective function through application of a small perturbation to the outputs of the network. It can be applied to problems where the objective function is not explicitly formulated, including reinforcement learning. In this paper, we show that nodeperturbation learning can be formulated as online learning in a linear perceptron with noise, and we can derive the differential equations of order parameters and the generalization error in the same way as for the analysis of learning in a linear perceptron through statistical mechanical methods. From analytical results, we show that crosstalk noise, which originates in the error of the other outputs, increases the generalization error as the output number increases.

機関リポジトリ 


No.6 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Statistical Mechanics of OnLine Mutual Learning with Many Linear Perceptrons

著者 
Yoichi Nakayama, Seiji Miyoshi, Masato Okada

誌名 
Journal of the Physical Society of Japan

出版者 

巻号頁 
78/ 11, 114001

出版日 
2009/11

ISSN 

DOI 
10.1143/JPSJ.78.114001

URL 

概要 
We propose a new mutual learning using many weak learners (or students) which converges into the identical state of Bagging that is kind of ensemble learning, within the framework of online learning, and have analyzed its asymptotic property through the statistical mechanics method. Mutual learning involving three or more students fundamentally differs from the twostudent case with regard to the variety of selecting a student to act as teacher. The proposed model consists of two learning steps: many students independently learn from a teacher, and then the students learn from others through the mutual learning. In mutual learning, students learn from other students and the generalization error is improved even if the teacher has not taken part in the mutual learning. We demonstrate that the learning style of selecting a student to act as teacher randomly is superior to that of cyclic order by using principle component analysis.

機関リポジトリ 


No.7 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning

著者 
Takahiro Yamada

誌名 
Journal of the Physical Society of Japan

出版者 

巻号頁 
77/ 2, 024005

出版日 
2008/02

ISSN 

DOI 
10.1143/JPSJ.77.024005

URL 

概要 
We propose an optimization method of mutual learning which converges into the identical state of optimum ensemble learning within the framework of online learning, and have analyzed its asymptotic property through the statistical mechanics method. The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. In mutual learning, students learn from each other and the generalization error is improved even if the teacher has not taken part in the mutual learning. However, in the case of different initial overlaps between teacher and students, a student with a larger initial overlap tends to have a larger generalization error than that of before the mutual learning. To overcome this problem, our proposed optimization method of mutual learning optimizes the step sizes of two students to minimize the asymptotic property of the generalization error. Consequently, the optimized mutual learning converges to a generalization error identical to that of the optimal ensemble learning.

機関リポジトリ 


No.8 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Statistical Mechanics of Mutual Learning with a Latent Teacher

著者 
Masato Okada

誌名 
Journal of the Physical Society of Japan

出版者 

巻号頁 
76/ 1, 014001

出版日 
2007/01

ISSN 

DOI 
10.1143/JPSJ.76.014001

URL 

概要 
We propose a mutual learning with a latent teacher within the framework of online learning, and have analyzed its dynamical behavior through the statistical mechanics method. The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. A teacher is not used in the mutual learning, so we refer to the teacher as a latent teacher. Linear perceptrons are used as the teacher and student. Our analytical results show that the overlaps between teacher and students become larger through mutual learning. In addition, we found that the mutual learning converges into the bagging of the ensemble learning scheme. We also show that a student with a larger initial overlap for mutual learning transiently passes through a state of parallel boosting in the ensemble learning in the slow learning rate limit. We have concluded that mutual learning is able to mimic the integration mechanism of bagging and parallel boosting.

機関リポジトリ 


No.9 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Ensemble Learning of Linear Perceptrons: OnLine Learning Theory

著者 
Masato Okada

誌名 
Journal of the Physical Society of Japan

出版者 

巻号頁 
74/ 11, 2966–2972

出版日 
2005/11

ISSN 

DOI 
10.1143/JPSJ.74.2966

URL 

概要 
We analyze ensemble learning including the noisy case where teacher or student noise is present. Linear perceptrons are used as teacher and student. First, we analyze the homogeneous correlation of initial weight vectors. The generalization error consists of two parts: the first term depends on the number of perceptrons K and is proportional to 1=K, the second does not depend on K in the first case. In the inhomogeneous correlation of initial weight vectors case, the weighted average could be optimized to minimize the generalization error. We found that the optimal weights do not depend on time without student noise, while the optimal weights depend on time and become 1=K with student noise.

機関リポジトリ 


No.10 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(第一著者以外)

タイトル 
Analysis of ensemble learning using simple perceptron based on online learning theory

著者 
Seiji Miyoshi, Masato Okada

誌名 
Physical Review E

出版者 

巻号頁 
71/ 32A, 036116111

出版日 
2005/03/15

ISSN 

DOI 
10.1103/PhysRevE.71.036116

URL 

概要 
Ensemble learning of K nonlinear perceptrons, which determine their outputs by sign functions, is discussed within the framework of online learning and statistical mechanics. One purpose of statistical learning theory is to theoretically obtain the generalization error. This paper shows that ensemble generalization error can be calculated by using two order parameters, that is, the similarity between a teacher and a student, and the similarity among students. The differential equations that describe the dynamical behaviors of these order parameters are derived in the case of general learning rules. The concrete forms of these differential equations are derived analytically in the cases of three wellknown rules: Hebbian learning, perceptron learning and AdaTron learning. Ensemble generalization errors of these three rules are calculated by using the results determined by solving their differential equations. As a result, these three rules show different characteristics in their affinity for ensemble learning, that is “maintaining variety among students.” Results show that AdaTron learning is superior to the other two rules with respect to that affinity.

機関リポジトリ 


No.11 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Online learning through simple perceptron learning with a margin

著者 
Masato Okada

誌名 
Neural Networks

出版者 

巻号頁 
17, 215223

出版日 
2004/03

ISSN 

DOI 
10.1016/j.neunet.2003.08.005

URL 

概要 
We analyze a learning method that uses a margin k a la Gardner for simple perceptron learning. This method corresponds to the perceptron learning when k ¼ 0; and to the Hebbian learning when k!1: Nevertheless, we found that the generalization ability of the method was superior to that of the perceptron and the Hebbian methods at an early stage of learning. We analyzed the asymptotic property of the learning curve of this method through computer simulation and found that it was the same as for perceptron learning. We also investigated an adaptive margin control method.

機関リポジトリ 


No.12 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(コレスポンディングオーサーかつ第一著者)

タイトル 
Training Data Selection Method for Generalization by Multilayer Neural Networks

著者 
Kenji Nakayama

誌名 
IEICE Trans. Fundamentals.

出版者 

巻号頁 
E81/ 3, 374381

出版日 
1998/03

ISSN 

DOI 

URL 

概要 
A training data selection method is proposed for multilayer neural networks (MLNNs). This method selects a small number of the training data, which guarantee both generalization and fast training for the MLNNs applied to pattern classification. The generalization will be satisfied using the data locate close to the boundary of the pattern classes. However, if these data are only used in the training, convergence is slow. This phenomenon is analyzed in this paper. Therefore, in the proposed method, the MLNN is first trained using some number of the data, which are randomly selected(Step 1). The data, for which the output error is relatively large, are selected. Furthermore, they are paired with the nearest data belong to the different class. The newly selected data are further paired with the nearest data. Finally, pairs of the data, which locate to the boundary, can be found. Using these pairs of the data, the MLNNs are further trained (Step 2). Since, there are some variations to combine Step 1 and 2, the proposed method can be applied to both offline and online training. The proposed method can reduce the number of the training data, at the same time, can hasten the training. Usefulness is confirmed through computer simulation.

機関リポジトリ 


No.13 

掲載種別 
研究論文（学術雑誌）

単著・共著区分 
共著(第一著者)

タイトル 
MultiFrequency Signal Classification by Multilayer Neural Networks and Linear Filter Methods

著者 
Kenji Nakayama

誌名 
IEICE Transaction of Fundamentals of Electronics, Communications and Computer Science

出版者 

巻号頁 
E80/ 5, 894902

出版日 
1997/12/05

ISSN 

DOI 

URL 

概要 
This paper compares signal classification performance of multilayer neural networks (MLNNs) and linear filters (LFs) ,both methods are compared based on frequency selective performance. The signals to be classified contain several frequency components. Furthermore, effects of the number of the signal samples are investigated. In this case, the frequency information may be lost to some extent. From practical viewpoint, computational complexity is also limited to the same level in both methods. IIR and FIR filters are compared. When the number of the input samples is strictly limited, the signal vectors are widely distributed in the multidimensional signal space. In this case, signal classification by the LF method cannot provide a good performance. Because, they are designed to extract the frequency components. On the other hand, the MLNN method can form class regions in the signal vector space with high degree of freedom. When the number of the signal samples is not so limited, both the MLNN and LF methods can provide the same high classification rates. In this case, since the signal vectors are distributed in the specific region, the MLNN method has some convergence problem, that is local minimum problem. The LFs can suppress wideband noise by using very highQ filters. However, the MLNN method can be also robust. Rather, it is a little superior to the LF method when the computational load is limited.

機関リポジトリ 


