論文
公開件数:13件

No.1
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサー)
タイトル 非線形パーセプトロンを用いた相互学習
著者 斎藤 大輔、原 一之
誌名 Journal of Artificial Intelligence and Soft Computing Research
出版者 Walter de Gruyter GmbH
巻号頁 5/ 1, 71-77
出版日 2015/04/30
ISSN 2083-2567
DOI 10.1515/jaiscr-2015-0020
URL
概要 We propose a mutual learning method using nonlinear perceptron within the framework of online learning and have analyzed its validity using computer simulations. Mutual learning involving three or more students is fundamentally
different from the two-student case with regard to variety when selecting a student to act as the teacher. The proposed method consists of two learning steps: first, multiple students learn independently from a teacher, and second, the students learn from others through mutual learning. Results showed that the mean squared error could be improved even if the teacher had not taken part in the mutual learning.
機関リポジトリ

No.2
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(第一著者)
タイトル Theoretical analysis of learning speed in gradient descent algorithm replacing derivative with constant
著者 Kazuyuki Hara amd Kentaro Katahira
誌名 Information Processing Society of Japan Transactions on Mathematical Modeling and Its Applications
出版者 日本情報処理学会
巻号頁 6/ 3, 100-105
出版日 2013/12/01
ISSN 03875806
DOI 10.11185/imt.9.61
URL
概要 In on-line gradient descent learning, the local property of the derivative term of the output function can slowly converge. Improving the derivative term, such as by using the natural gradient, has been proposed for speeding up the convergence. Beside this sophisticated method, we propose an algorithm that replaces the derivative term with a constant and show that this greatly increases convergence speed when the learning step size is less than 2.7, which is near the optimal learning step size. The proposed algorithm is inspired by linear perceptron learning and can avoid locality of the derivative term. We derived the closed deterministic differential equations by using a statistical mechan- ics method and show the validity of theoretical results by comparing them with computer simulation solutions. In real problems, the optimum learning step size is not given in advance. Therefore, the learning step size must be small. The proposed method is useful in this case.
機関リポジトリ

No.3
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(第一著者)
タイトル Statistical Mechanics of Node-Perturbation Learning for Nonlinear Perceptron
著者 Kazuyuki Hara, Kentaro KATAHIRA, Kazuo OKANOYA, and Masato OKADA
誌名 Journal pf the physical society of Japan
出版者
巻号頁 82, 054001
出版日 2013/04/01
ISSN
DOI 10.7566/JPSJ.82.054001
URL
概要 Node-perturbation learning is a type of statistical gradient descent algorithm that can be applied to problems where the objective function is not explicitly formulated, including reinforcement learning. A nonlinear perceptron has several advantages over a linear perceptron, such as the ability to use nonlinear outputs, learnability, storage capacity, and so forth. However, node-perturbation for a
nonlinear perceptron has yet to be analyzed theoretically. In this paper, we derive a learning rule of node-perturbation learning for a nonlinear perceptron within the framework of REINFORCE learning and analyze the learning behavior
by using statistical mechanical methods. From the results, we found that the signal and cross-talk terms of the order parameter Q have different forms for a nonlinear perceptron. Moreover, the increase in the generalization error with
increasing number of outputs is less than for a linear perceptron.
機関リポジトリ

No.4
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(第一著者)
タイトル Statistical Mechanics of On-line Ensemble Teacher Learning
through a Novel Perceptron Learning Rule
著者 Kazuyuki Hara, Seiji Miyoshi
誌名 Journal of the physical society of Japan
出版者
巻号頁 81, 064002-1-6
出版日 2012/06/15
ISSN
DOI 10.1143/JPSJ.81.064002
URL
概要 In ensemble teacher learning, ensemble teachers have only uncertain information about the true teacher, and this information is given by an ensemble consisting of an infinite number of ensemble teachers whose variety is sufficiently rich. In this learning, a student learns from an ensemble teacher that is iteratively selected randomly from a pool of many ensemble teachers. An interesting point of ensemble teacher learning is the asymptotic behavior of the student to approach the true teacher by learning from ensemble teachers. Perceptron learning with a margin is identical to the perceptron learning rule when the margin is zero and identical to the Hebbian learning rule when the margin is infinity. Thus, this rule connects the perceptron learning rule and the Hebbian learning rule continuously through the size of the margin. From the results, we show that by setting a margin of κ>0, the effect of an ensemble appears and becomes significant when a larger margin κ is used.
機関リポジトリ

No.5
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル Statistical Mechanics of On-line Node-perturbation Learning
著者 Kentaro Katahira, Kazuo Okanoya, Masato Okada
誌名 Information Processing Society of Japan Transactions on Mathematical Modeling and Its Applications
出版者
巻号頁 4/ 1, 72-81
出版日 2011/01
ISSN
DOI
URL
概要 Node-perturbation learning (NP-learning) is a kind of statistical gradient descent
algorithm that estimates the gradient of an objective function through application
of a small perturbation to the outputs of the network. It can be applied
to problems where the objective function is not explicitly formulated, including
reinforcement learning. In this paper, we show that node-perturbation learning
can be formulated as on-line learning in a linear perceptron with noise, and we
can derive the differential equations of order parameters and the generalization
error in the same way as for the analysis of learning in a linear perceptron
through statistical mechanical methods. From analytical results, we show that
cross-talk noise, which originates in the error of the other outputs, increases
the generalization error as the output number increases.
機関リポジトリ

No.6
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル Statistical Mechanics of On-Line Mutual Learning with Many Linear Perceptrons
著者 Yoichi Nakayama, Seiji Miyoshi, Masato Okada
誌名 Journal of the Physical Society of Japan
出版者
巻号頁 78/ 11, 114001
出版日 2009/11
ISSN
DOI 10.1143/JPSJ.78.114001
URL
概要 We propose a new mutual learning using many weak learners (or students) which converges into the
identical state of Bagging that is kind of ensemble learning, within the framework of on-line learning,
and have analyzed its asymptotic property through the statistical mechanics method. Mutual learning
involving three or more students fundamentally differs from the two-student case with regard to the
variety of selecting a student to act as teacher. The proposed model consists of two learning steps: many
students independently learn from a teacher, and then the students learn from others through the mutual
learning. In mutual learning, students learn from other students and the generalization error is improved
even if the teacher has not taken part in the mutual learning. We demonstrate that the learning style of
selecting a student to act as teacher randomly is superior to that of cyclic order by using principle
component analysis.
機関リポジトリ

No.7
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル Optimization of the Asymptotic Property of Mutual Learning Involving an Integration Mechanism of Ensemble Learning
著者 Takahiro Yamada
誌名 Journal of the Physical Society of Japan
出版者
巻号頁 77/ 2, 024005
出版日 2008/02
ISSN
DOI 10.1143/JPSJ.77.024005
URL
概要 We propose an optimization method of mutual learning which converges into the identical state of optimum ensemble learning within the framework of on-line learning, and have analyzed its asymptotic property through the statistical mechanics method. The proposed model consists of two learning steps: two students independently learn from a teacher, and then the students learn from each other through the mutual learning. In mutual learning, students learn from each other and the generalization error is improved even if the teacher has not taken part in the mutual learning. However, in the case of different initial overlaps between teacher and students, a student with a larger initial overlap
tends to have a larger generalization error than that of before the mutual learning. To overcome this problem, our proposed optimization method of mutual learning optimizes the step sizes of two students to minimize the asymptotic property of the generalization error. Consequently, the optimized mutual learning converges to a generalization error identical to that of the optimal ensemble learning.
機関リポジトリ

No.8
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル Statistical Mechanics of Mutual Learning with a Latent Teacher
著者 Masato Okada
誌名 Journal of the Physical Society of Japan
出版者
巻号頁 76/ 1, 014001
出版日 2007/01
ISSN
DOI 10.1143/JPSJ.76.014001
URL
概要 We propose a mutual learning with a latent teacher within the framework of on-line learning, and have
analyzed its dynamical behavior through the statistical mechanics method. The proposed model consists
of two learning steps: two students independently learn from a teacher, and then the students learn from
each other through the mutual learning. A teacher is not used in the mutual learning, so we refer to the
teacher as a latent teacher. Linear perceptrons are used as the teacher and student. Our analytical results
show that the overlaps between teacher and students become larger through mutual learning. In addition,
we found that the mutual learning converges into the bagging of the ensemble learning scheme. We also
show that a student with a larger initial overlap for mutual learning transiently passes through a state of
parallel boosting in the ensemble learning in the slow learning rate limit. We have concluded that mutual
learning is able to mimic the integration mechanism of bagging and parallel boosting.
機関リポジトリ

No.9
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル Ensemble Learning of Linear Perceptrons: On-Line Learning Theory
著者 Masato Okada
誌名 Journal of the Physical Society of Japan
出版者
巻号頁 74/ 11, 2966–2972
出版日 2005/11
ISSN
DOI 10.1143/JPSJ.74.2966
URL
概要 We analyze ensemble learning including the noisy case where teacher or student noise is present.
Linear perceptrons are used as teacher and student. First, we analyze the homogeneous correlation of
initial weight vectors. The generalization error consists of two parts: the first term depends on the number
of perceptrons K and is proportional to 1=K, the second does not depend on K in the first case. In the
inhomogeneous correlation of initial weight vectors case, the weighted average could be optimized to
minimize the generalization error. We found that the optimal weights do not depend on time without
student noise, while the optimal weights depend on time and become 1=K with student noise.
機関リポジトリ

No.10
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(第一著者以外)
タイトル Analysis of ensemble learning using simple perceptron based on online learning theory
著者 Seiji Miyoshi, Masato Okada
誌名 Physical Review E
出版者
巻号頁 71/ 3-2A, 036116-1-11
出版日 2005/03/15
ISSN
DOI 10.1103/PhysRevE.71.036116
URL
概要 Ensemble learning of K nonlinear perceptrons, which determine their outputs by sign functions,
is discussed within the framework of online learning and statistical mechanics. One purpose of
statistical learning theory is to theoretically obtain the generalization error. This paper shows
that ensemble generalization error can be calculated by using two order parameters, that is, the
similarity between a teacher and a student, and the similarity among students. The differential
equations that describe the dynamical behaviors of these order parameters are derived in the case of
general learning rules. The concrete forms of these differential equations are derived analytically in
the cases of three well-known rules: Hebbian learning, perceptron learning and AdaTron learning.
Ensemble generalization errors of these three rules are calculated by using the results determined
by solving their differential equations. As a result, these three rules show different characteristics
in their affinity for ensemble learning, that is “maintaining variety among students.” Results show
that AdaTron learning is superior to the other two rules with respect to that affinity.
機関リポジトリ

No.11
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル On-line learning through simple perceptron learning with a margin
著者 Masato Okada
誌名 Neural Networks
出版者
巻号頁 17, 215-223
出版日 2004/03
ISSN
DOI 10.1016/j.neunet.2003.08.005
URL
概要 We analyze a learning method that uses a margin k a la Gardner for simple perceptron learning. This method corresponds to the perceptron
learning when k ¼ 0; and to the Hebbian learning when k!1: Nevertheless, we found that the generalization ability of the method was
superior to that of the perceptron and the Hebbian methods at an early stage of learning. We analyzed the asymptotic property of the learning
curve of this method through computer simulation and found that it was the same as for perceptron learning. We also investigated an adaptive
margin control method.
機関リポジトリ

No.12
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(コレスポンディングオーサーかつ第一著者)
タイトル Training Data Selection Method for Generalization by Multilayer Neural Networks
著者 Kenji Nakayama
誌名 IEICE Trans. Fundamentals.
出版者
巻号頁 E81/ 3, 374-381
出版日 1998/03
ISSN
DOI
URL
概要 A training data selection method is proposed for multilayer neural networks (MLNNs). This method selects a small number of the training data, which guarantee both generalization and fast training for the MLNNs applied to pattern classification. The generalization will be satisfied using the data locate close to the boundary of the pattern classes. However, if these data are only used in the training, convergence is slow. This phenomenon is analyzed in this paper. Therefore, in the proposed method, the MLNN is first trained using some number of the data, which are randomly selected(Step 1). The data, for which the output error is relatively large, are selected. Furthermore, they are paired with the nearest data belong to the different class. The newly selected data are further paired with the nearest data. Finally, pairs of the data, which locate to the boundary, can be found. Using these pairs of the data, the MLNNs are further trained (Step 2). Since, there are some variations to combine Step 1 and 2, the proposed method can be applied to both off-line and on-line training. The proposed method can reduce the number of the training data, at the same time, can hasten the training. Usefulness is confirmed through computer simulation.
機関リポジトリ

No.13
掲載種別 研究論文(学術雑誌)
単著・共著区分 共著(第一著者)
タイトル Multi-Frequency Signal Classification by Multilayer Neural Networks and Linear Filter Methods
著者 Kenji Nakayama
誌名 IEICE Transaction of Fundamentals of Electronics, Communications and Computer Science
出版者
巻号頁 E80/ 5, 894-902
出版日 1997/12/05
ISSN
DOI
URL
概要 This paper compares signal classification performance of multilayer neural networks (MLNNs) and linear filters (LFs) ,both methods are compared based on frequency selective performance. The signals to be classified contain several frequency components. Furthermore, effects of the number of the signal samples are investigated. In this case, the frequency information may be lost to some extent. From practical viewpoint, computational complexity is also limited to the same level in both methods. IIR and FIR filters are compared. When the number of the input samples is strictly limited, the signal vectors are widely distributed in the multi-dimensional signal space. In this case, signal classification by the LF method cannot provide a good performance. Because, they are designed to extract the frequency components. On the other hand, the MLNN method can form class regions in the signal vector space with high degree of freedom. When the number of the signal samples is not so limited, both the MLNN and LF methods can provide the same high classification rates. In this case, since the signal vectors are distributed in the specific region, the MLNN method has some convergence problem, that is local minimum problem. The LFs can suppress wide-band noise by using very high-Q filters. However, the MLNN method can be also robust. Rather, it is a little superior to the LF method when the computational load is limited.
機関リポジトリ