Combining Multiple Pairwise Neural Networks Classifiers: A Comparative Study

Olivier Lezoray, Hubert Cardot

Abstract

Classifier combination constitutes an interesting approach when solving multiclass classification problems. We review standard methods used to decode the decomposition generated by a one-against-one approach. New decoding methods are proposed and are compared to standard methods. A stacking decoding is also proposed and consists in replacing the whole decoding by a trainable classifier to arbiter among the conflicting predictions of the binary classifiers. Substantial gain is obtained on all datasets used in the experiments.

References

  1. Furnkranz, J.: Round robin classification. Journal of Machine Learning Research 2 (2002) 721-747
  2. Furnkranz, J.: Pairwise classification as an ensemble technique. In: European Conference on Machine Learning (ECML). (2002) 97-110
  3. Lezoray, O., Cardot, H.: A neural network architecture for data classification. International Journal of Neural Systems 11 (2001) 33-42
  4. Price, D., Knerr, S., Personnaz, L.: Pairwise neural network classifiers with propabilistic outputs. In: Advances in Neural Information Processing Systems (NIPS). Volume 7., MIT Press (1995) 1109-116
  5. Kressel, U.: Pairwise classification and support vector machines. In: Advances in Kernel Methods, Support Vector Learning. MIT Press (1999)
  6. Ou, G., Murphey, Y., Feldkamp, A.: Multiclass pattern classification using neural networks. In: International Conference on Pattern Recognition (ICPR). Volume 4. (2004) 585- 588
  7. Rifkin, R., Klautau, A.: In defense of one-vs-all classi.cation. Journal of Machine Learning Research 5 (2004) 101-141
  8. Hastie, T., Tibshirani, R.: Classification by pairwise coupling. The annals of Statistics 26 (1998) 451-471
  9. Tax, D., Duin, R.: Using two-class classifiers for multiclass classification. In: International Conference on Pattern Recognition (ICPR). Volume 2. (2002) 124-127
  10. Lezoray, O., Fournier, D., Cardot, H.: Neural network induction graph for pattern recognition. Neurocomputing (2004) 257-274
  11. Allwein, E., Schapire, R., Singer, Y.: Reducing multiclass to binary: a unifying approach for margin classifiers. Journal of Machine Learning Research 1 (2000) 113-141
  12. Platt, J., Cristianini, N., Shawe-Taylor, J.: Large margin dags for multiclass classification. In: Advances in Neural Information Processing Systems (NIPS). Volume 12., MIT Press (2000) 547-553
  13. Moreira, M., Mayoraz, E.: Improved pairwise coupling classification with correcting classifiers. In: European Conference on Machine Learning (ECML), Springer-Verlag (1998) 160-171
  14. Hsu, C.W., Lin, C.J.: A comparision of methods for multi-class support vector machines. IEEE Transactions on Neural Networks 13 (2002) 415-425
  15. Mayoraz, E., Alpaydin, E.: Support vector machines for multi-class classification. In: International Work conference on Artificial Neural Networks. Volume 2. (1999) 833-842
  16. F. Tahahashi, S.A.: Optimizing directed acyclic graph support vector machines. In: Artificial Neural Networks in Pattern Recognition (ANNPR). (2003)
  17. Lu, B.L., Ito, M.: Task decomposition and module combination based on class relations: A modular neural network for pattern classification. IEEE Transaction on Neural Networks 10 (1999) 1244-1256
  18. Cardot, H., Lezoray, O.: Graph of neural networks for pattern recognition. In: International Conference on Pattern Recognition (ICPR). Volume 2. (2002) 124-127
  19. Campbell, C.: Constructive Learning Techniques for Designing Neural Network Systems. San Diego: Academic Press (1997)
  20. Kwok, T.Y., Yeung, D.Y.: Constructive algorithms for structure learning in feedforward neural networks for regression problems. IEEE Trans. on Neural Networks 8 (1997) 630- 645
  21. Friedman, J.: Another approach to polychotomous classification. Technical report, Dept. of statistics, Stanford University (1996)
  22. Crammer, K., Singer, Y.: On the learnability and design of output codes for multiclass problems. Machine Learning 47 (2002) 201 - 233
  23. Klautau, A., Jevtic, N., Orlitsky, A.: Combined binary classifiers with applications to speech recognition. In: International Conference on Spoken Language Processing (ICSLP). (2002) 2469-2472
  24. Klautau, A., Jevtic, N., Orlitsky, A.: On nearest neighbor error-correcting output codes with application to all-pairs multiclass support vector machnies. Journal of Machine Learning Research 4 (2003) 1-15
  25. Ko, J., Kim, E., Byun, H.: Improved n-division output coding for multiclass learning problems. In: International Conference on Pattern Recognition (ICPR). Volume 3. (2004) 470- 473
  26. Phetkaew, T., Kijsirikul, B., Rivepiboon, W.: Reordering adaptive directed acyclic graphs: an improved algorithm for multiclass support vector machines. In: International Joint Conference on Neural Networks (IJCNN). Volume 2. (2003) 1605- 1610
  27. Vural, V., Dy, J.G.: A hierarchical method for multi-class support vector machines. In: International Conference on Machine Learning (ICML). (2004)
  28. Savicky, P., Furnkranz, J.: Combining pairwise classifiers with stacking. In: Intellignent Data Analysis (IDA). (2003)
  29. Wolpert, D.: Stacked generalization. 'Neural Networks 5 (1992) 241-260
  30. Zhou, Z.H.: Nec4.5: Neural ensemble based c4.5. IEEE Transactions on Knowledge and Data Engineering (2003)
  31. S. Hettich, C.B., Merz, C.: UCI repository of machine learning databases. Technical report, University of California, Irvine, Dept. of Information and Computer Sciences (1998)
  32. Quinlan, J.: C4.5 : programs for machine learning. Morgann Kauffman, San Mateo (1993)
Download


Paper Citation


in Harvard Style

Lezoray O. and Cardot H. (2005). Combining Multiple Pairwise Neural Networks Classifiers: A Comparative Study . In Proceedings of the 1st International Workshop on Artificial Neural Networks and Intelligent Information Processing - Volume 1: ANNIIP, (ICINCO 2005) ISBN 972-8865-36-8, pages 52-61. DOI: 10.5220/0001193700520061


in Bibtex Style

@conference{anniip05,
author={Olivier Lezoray and Hubert Cardot},
title={Combining Multiple Pairwise Neural Networks Classifiers: A Comparative Study},
booktitle={Proceedings of the 1st International Workshop on Artificial Neural Networks and Intelligent Information Processing - Volume 1: ANNIIP, (ICINCO 2005)},
year={2005},
pages={52-61},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0001193700520061},
isbn={972-8865-36-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 1st International Workshop on Artificial Neural Networks and Intelligent Information Processing - Volume 1: ANNIIP, (ICINCO 2005)
TI - Combining Multiple Pairwise Neural Networks Classifiers: A Comparative Study
SN - 972-8865-36-8
AU - Lezoray O.
AU - Cardot H.
PY - 2005
SP - 52
EP - 61
DO - 10.5220/0001193700520061