Xuan Zhang

• Du, Yuhao; Cao, Liqin; Xiong, Haiyang; Gao, Lin; Zhang, Xuan & Zhong, Yanfei (2026). DBMSA: A Unified Framework for Infrared Small Target Detection via Dual-Branch CNN-ViT and Multistrategy Adversarial Training. IEEE Transactions on Geoscience and Remote Sensing. ISSN 0196-2892. 64. doi: 10.1109/TGRS.2026.3723654. Full text in Research Archive Show summary

  Infrared small target detection (IRSTD) faces multiple challenges, including weak target signals, variable target scales, severe background noise, and insufficient model robustness. Although existing deep learning-based methods have achieved improvements in detection accuracy, they still struggle to balance local and global feature representations and exhibit limited resistance to complex adversarial noise, which constraints detection performance. To address these issues, this article proposes dual-branch CNN-ViT architecture and multistrategy adversarial training (DBMSA), a unified framework for IRSTD. The proposed approach first designs a dual-branch attention transmission network (DBATNet) architecture and introduces a multistrategy adversarial training paradigm. The DBATNet architecture integrates the local feature extraction capability of CNNs with the global dependency modeling strength of ViTs. The introduced feature transfer unit (FTU) facilitates intelligent cross-branch feature fusion through a dynamic attention mechanism, thereby improving detection performance and applicability under varying background complexity. Furthermore, we propose a multistrategy fast gradient sign method (FGSM)–based adversarial example generation and retraining pipeline, which jointly incorporates standard FGSM, adaptive perturbation–based FGSM, and atmospheric scattering–based FGSM to construct diverse adversarial examples for model retraining, resulting in enhanced robustness against complex interference. Experimental results demonstrate that, under standard training conditions, the proposed DBATNet achieves detection performance that is superior to or at least comparable with current state-of-the-art methods. After multistrategy adversarial training, DBMSA achieves the best detection performance on the SIRST-Aug, IRSTD-1k, and NUAA-SIRST datasets, and shows clear improvements compared with its counterpart before adversarial training. The DBMSA achieves F1 and normalized Inter...

• Belbachir, Nabil; Brokstad, Rebecca Jane Cox; Hernández, Julio; Jochemsen, Arnoud; Liu, Na & Mohn, Kristin Greve-Isdahl [Show all 10 contributors for this article] (2026). A Clinical Hygiene Ruler on the Basis of Hyperspectral Imaging and Machine Learning: A Proof-of-Concept Model for SARS-CoV-2 Virus Detection. SN Computer Science. ISSN 2662-995X. 7(5). doi: 10.1007/s42979-026-05101-6. Full text in Research Archive Show summary

  This paper explores the possibility of applying hyperspectral imaging and machine learning in detecting infectious coronavirus in the indoor environment. Currently, the polymerase chain reaction (PCR) test is the leading technique to detect the existence of SARS-CoV-2 virus in humans (serum or nasalpharyngelal specimens). While the test is a gold standard in detection of virus in humans, positive PCR results do not necessarily mean infectious virus is detected, as dead virus can trigger positive results in the nucleic acid amplification test. This paper proposes a novel method which is based on measuring live cells being infected by infectious virus, and has the potential to directly predict the infectiousness of the targeted virus within 48 hours. The infection status was measured by a hyperspectral camera. A Partial Least Square - Discriminant Analysis (PLS-DA) model was trained on the hyperspectral images to do prediction. We believe this method opens a new avenue to detecting infectious virus in an indoor environment and can be potentially applied in SARS-CoV-2 detections on surfaces, hence used to aid in protection control measures and detecting virus contamination.

• Zhou, Hao; Xu, Wentao; Zhang, Xuan; Yazidi, Anis & Yu, Ke (2026). Temporal Conditional Score Network for Multivariate Time Series Anomaly Detection. IEEE Internet of Things Journal. ISSN 2327-4662. 13(13), p. 28165–28180. doi: 10.1109/JIOT.2026.3681780. Full text in Research Archive Show summary

  Time series anomaly detection is a critical technique for secure monitoring in cyber-physical systems (CPSs). Most existing methods learn a normal data profile and identify anomalies as deviations from this profile. Recently, score-based models, which estimate the gradient of the log data probability density, have shown promise in improving data distribution estimation due to their strong generative capabilities. However, their application to time series anomaly detection in CPSs remains underexplored, with key challenges in network architecture design and the accuracy-efficiency trade-off yet to be addressed. In this paper, we propose the Temporal Conditional Score Network (TCSN), a novel framework that jointly estimates conditional scores and detects anomalies based on reconstruction error. Our approach enhances a U-Net backbone with temporal attention skip connections, preserving detailed spatio-temporal features at both variable and time-step levels. Additionally, signature matrix embedding is introduced to improve sensitivity to anomalous patterns. We further establish connections between score-based and reconstruction-based anomaly measurements, leading to a more efficient and accurate detection strategy. To ensure practical applicability, we design a full-cycle deployment pipeline that integrates data sanitization, denoising score matching–based training, hyperparameter tuning, and proactive model updates, enhancing scalability and long-term stability. Extensive experiments on public datasets demonstrate that TCSN consistently outperforms state-of-the-art baselines, achieving up to 0.2155 improvement in composition F1 score while maintaining strong robustness under varying noise levels.

• Cao, Liqin; Zhao, Yuan; Zhang, Fan; Zhang, Xuan; Liu, Ju Hua & Zhong, Yanfei (2025). MDTNet: Partial transformer with degradation-aware module for restoring old photos with multiple degradations. Neurocomputing. ISSN 0925-2312. 616. doi: 10.1016/j.neucom.2024.128866. Full text in Research Archive Show summary

  Old photos preserve the memories of the old days and have special meanings in life. However, they often suffer from severe and complex degradations, such as cracks, dirt, and noise. Old photo restoration is thus a meaningful but challenging task. Most of the current deep learning methods to repair corrupted old photos focus on one single type of degradation or limited number of degradations. The lack of proper old photo datasets makes the digital restoration problem even more challenging. In this paper, we propose MDTNet, which is a novel transformer-based method for restoring old photos with multiple degradations. MDTNet is an efficient end-to-end transformer architecture with only a single encoder and a decoder to complete the restoration. Specifically, a novel partial transformer encoder is designed with partial multi-head self-attention to extract local-global context information from the valid masked area, which represents the unbroken area. In addition, a degradation-aware module is proposed in the decoder to automatically learn different degradation information, to improve the effect of multiple degradation restoration. Moreover, a synthetic old photo dataset — the SynOld dataset — was built, which simulates the multiple degradations of real old photos. Experimental results show that the proposed method outperforms the state-of-the-art methods on both public datasets and the custom-developed dataset. The code is available at https://github.com/YuanZhaoc/MDTNet.

• Zhengguang, Song; Zhijiang, Li; Liqin, Cao; Lei, Jiao & Zhang, Xuan (2025). GC3: Grouped convolutional color constancy. Pattern Recognition. ISSN 0031-3203. 158. doi: 10.1016/j.patcog.2024.110957. Full text in Research Archive Show summary

  Computational color constancy is an ill-posed problem since different combinations of objects and illumination may appear the same color in an image. Most methods produce a single illuminant estimate and cannot explain the potential ambiguity between possible illuminant solutions. In this work, a grouped convolutional color constancy (GC) method is proposed. It approaches the computational color constancy as a grouped regression problem, dealing with ambiguity by generating multiple possible illuminant solutions. The GC consists of two components: fast feature extraction (FFE) and grouped illuminant regression (GIR). FFE uses small convolution kernels and rapid down-sampling scheme for quick feature extraction, while GIR employs grouped convolution layers for feature grouping regression, which estimates the illuminant solution and its probability for each group with two separate branches. The final illumination estimation is obtained by averaging all illuminant solutions, weighted by the normalized probability for each group. The proposed method yields competitive results on the ColorChecker dataset and achieves the state-of-the-art accuracy on the NUS and Cube+ datasets. Moreover, when compared with the state-of-the-art methods, our proposed approach exhibits superior time efficiency, making it the most efficient method available.

• Cao, Liqin; Gao, Lin; Wan, Yuting; Zhang, Xuan; Gao, Lyuzhou & Zhong, Yanfei (2024). Thermal infrared hyperspectral image band selection based on dynamic metric-guided particle swarm optimization for quantitative analysis of potash. International Journal of Remote Sensing. ISSN 0143-1161. 45(12), p. 4075–4094. doi: 10.1080/01431161.2024.2358545. Full text in Research Archive
• Granmo, Ole-Christoffer; Andersen, Per-Arne; Lei, Jiao; Zhang, Xuan; Blakely, Christian Dallas & Berge, Geir Thore [Show all 7 contributors for this article] (2023). Learning Minimalistic Tsetlin Machine Clauses with Markov Boundary-Guided Pruning, 2023 International Symposium on the Tsetlin Machine (ISTM). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9798350344776. doi: 10.1109/ISTM58889.2023.10454914. Full text in Research Archive
• Abeyrathna, Kuruge Darshana; Abouzeid, Ahmed Abdulrahem Othman; Bhattarai, Bimal; Giri, Charul; Glimsdal, Sondre & Granmo, Ole-Christoffer [Show all 11 contributors for this article] (2023). Building Concise Logical Patterns by Constraining Tsetlin Machine Clause Size. In Elkind, Edith (Eds.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. AAAI Press. ISSN 9781956792034. p. 3395–3403. doi: 10.24963/ijcai.2023/378. Full text in Research Archive Show summary

  Tsetlin machine (TM) is a logic-based machine learning approach with the crucial advantages of being transparent and hardware-friendly. While TMs match or surpass deep learning accuracy for an increasing number of applications, large clause pools tend to produce clauses with many literals (long clauses). As such, they become less interpretable. Further, longer clauses increase the switching activity of the clause logic in hardware, consuming more power. This paper introduces a novel variant of TM learning – Clause Size Constrained TMs (CSC-TMs) – where one can set a soft constraint on the clause size. As soon as a clause includes more literals than the constraint allows, it starts expelling literals. Accordingly, oversized clauses only appear transiently. To evaluate CSC-TM, we conduct classifcation, clustering, and regression experiments on tabular data, natural language text, images, and board games. Our results show that CSC-TM maintains accuracy with up to 80 times fewer literals. Indeed, the accuracy increases with shorter clauses for TREC, IMDb, and BBC Sports. After the accuracy peaks, it drops gracefully as the clause size approaches a single literal. We fnally analyze CSC-TM power consumption and derive new convergence properties.

• Omslandseter, Rebekka Olsson; Jiao, Lei; Zhang, Xuan; Yazidi, Anis & Oommen, John (2022). The Hierarchical Discrete Pursuit Learning Automaton: A Novel Scheme with Fast Convergence and Epsilon-Optimality. IEEE Transactions on Neural Networks and Learning Systems. ISSN 2162-237X. 35(6), p. 8278–8292. doi: 10.1109/TNNLS.2022.3226538. Full text in Research Archive Show summary

  Since the early 1960s, the paradigm of learning automata (LA) has experienced abundant interest. Arguably, it has also served as the foundation for the phenomenon and field of reinforcement learning (RL). Over the decades, new concepts and fundamental principles have been introduced to increase the LA’s speed and accuracy. These include using probability updating functions, discretizing the probability space, and using the “Pursuit” concept. Very recently, the concept of incorporating “structure” into the ordering of the LA’s actions has improved both the speed and accuracy of the corresponding hierarchical machines, when the number of actions is large. This has led to the $\epsilon$ -optimal hierarchical continuous pursuit LA (HCPA). This article pioneers the inclusion of all the above-mentioned phenomena into a new single LA, leading to the novel hierarchical discretized pursuit LA (HDPA). Indeed, although the previously proposed HCPA is powerful, its speed has an impediment when any action probability is close to unity, because the updates of the components of the probability vector are correspondingly smaller when any action probability becomes closer to unity. We propose here, the novel HDPA, where we infuse the phenomenon of discretization into the action probability vector’s updating functionality, and which is invoked recursively at every stage of the machine’s hierarchical structure. This discretized functionality does not possess the same impediment, because discretization prohibits it. We demonstrate the HDPA’s robustness and validity by formally proving the $\epsilon$ -optimality by utilizing the moderation property. We also invoke the submartingale characteristic at every level, to prove that the action probability of the optimal action converges to unity as time goes to infinity. Apart from the new machine being $\epsilon$ -optimal, the numerical results demonstrate that the number of iterations required for convergence is significantly reduced for the HDPA, when compared to the state-of-the-art HCPA scheme.

• Omslandseter, Rebekka Olsson; Jiao, Lei; Zhang, Xuan; Yazidi, Anis & Oommen, John (2022). The Hierarchical Discrete Learning Automaton Suitable for Environments with Many Actions and High Accuracy Requirements. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 13151, p. 507–518. doi: 10.1007/978-3-030-97546-3_41. Full text in Research Archive Show summary

  Since its early beginning, the paradigm of Learning Automata (LA), has attracted much interest. Over the last decades, new concepts and various improvements have been introduced to increase the LA’s speed and accuracy, including employing probability updating functions, discretizing the probability space, and implementing the “Pursuit” concept. The concept of incorporating “structure” into the ordering of the LA’s actions is one of the latest advancements to the field, leading to the ϵ-optimal Hierarchical Continuous Pursuit LA (HCPA) that has superior performance to other LA variants when the number of actions is large. Although the previously proposed HCPA is powerful, its speed has a handicap when the required action probability of an action is approaching unity. The reason for this slow convergence is that the learning parameter operates in a multiplicative manner within the probability space, making the increment of the action probability smaller as its probability becomes close to unity. Therefore, we propose the novel Hierarchical Discrete Learning Automata (HDPA) in this paper, which does not possess the same impediment as the HCPA. The proposed machine infuse the principle of discretization into the action probability vector’s updating functionality, where this type of updating is invoked recursively at every depth within a hierarchical tree structure and we pursue the best estimated action in all iterations through utilization of the Estimator phenomenon. The proposed machine is ϵ-optimal, and our experimental results demonstrate that the number of iterations required before convergence is significantly reduced for the HDPA, when compared with the HCPA.

• Lei, Jiao; Zhang, Xuan; Granmo, Ole-Christoffer & Abeyrathna, Kuruge Darshana (2022). On the Convergence of Tsetlin Machines for the XOR Operator. IEEE Transactions on Pattern Analysis and Machine Intelligence. ISSN 0162-8828. 45(5), p. 6072–6085. doi: 10.1109/TPAMI.2022.3203150. Full text in Research Archive Show summary

  The Tsetlin Machine (TM) is a novel machine learning algorithm with several distinct properties, including transparent inference and learning using hardware-near building blocks. Although numerous papers explore the TM empirically, many of its properties have not yet been analyzed mathematically. In this article, we analyze the convergence of the TM when input is non-linearly related to output by the XOR-operator. Our analysis reveals that the TM, with just two conjunctive clauses, can converge almost surely to reproducing XOR, learning from training data over an infinite time horizon. Furthermore, the analysis shows how the hyper-parameter T guides clause construction so that the clauses capture the distinct sub-patterns in the data. Our analysis of convergence for XOR thus lays the foundation for analyzing other more complex logical expressions. These analyses altogether, from a mathematical perspective, provide new insights on why TMs have obtained the state-of-the-art performance on several pattern recognition problems.

• Zhou, Hao; Yu, Ke; Zhang, Xuan; Wu, Guanlin & Yazidi, Anis (2022). Contrastive autoencoder for anomaly detection in multivariate time series. Information Sciences. ISSN 0020-0255. 610, p. 266–280. doi: 10.1016/j.ins.2022.07.179. Full text in Research Archive Show summary

  With the proliferation of the Internet of Things, a large amount of multivariate time series (MTS) data is being produced daily by industrial systems, corresponding in many cases to life-critical tasks. The recent anomaly detection researches focus on using deep learning methods to construct a normal profile for MTS. However, without proper constraints, these methods cannot capture the dependencies and dynamics of MTS and thus fail to model the normal pattern, resulting in unsatisfactory performance. This paper proposes CAE-AD, a novel contrastive autoencoder for anomaly detection in MTS, by introducing multi-grained contrasting methods to extract normal data pattern. First, to capture the temporal dependency of series, a projection layer is employed and a novel contextual contrasting method is applied to learn the robust temporal representation. Second, the projected series is transformed into two different views by using time-domain and frequency-domain data augmentation. Last, an instance contrasting method is proposed to learn local invariant characteristics. The experimental results show that CAE-AD achieves an F1-score ranging from 0.9119 to 0.9376 on the three public datasets, outperforming the baseline methods.

• Oommen, John; Zhang, Xuan & Lei, Jiao (2022). A Comprehensive Survey of Estimator Learning Automata and Their Recent Convergence Results. Lecture Notes in Networks and Systems. ISSN 2367-3370. 289, p. 33–52. doi: 10.1007/978-3-030-87049-2_2. Full text in Research Archive Show summary

  The pre-cursor field to Reinforcement Learning is that of Learning Automata (LA). Within this field, Estimator Algorithms (EAs) can be said to be the state-of-the-art. Further, the subset of Pursuit Algorithms (PAs), discovered by Thathachar and Sastry [34, 39], were the pioneering schemes. This chapter contains a comprehensive survey of the various EAs, and the most recent convergence results for PAs. Unlike the prior LA, EAs are based on a fundamentally distinct phenomenon. They are also the most accurate LA, converging in the least time. EAs operate on two vectors, namely, the action probability vector which is updated using responses from the Environment, and quickly-computed estimates of the reward probabilities of the various actions. The proofs that they are ε-optimal is thus very complex. They have to incorporate two rather snon-orthogonal phenomena, which are the convergence of these estimates and the convergence of the probabilities of selecting the various actions. For almost three decades, the reported proofs of PAs possessed an infirmity (or flaw), which we refer to as the claim of the “monotonicity” property. This flaw was discovered by the authors of [37], who also provided an alternate proof for a specific PA where the scheme’s parameter decreased with time. This paper first records all the reported EAs. It then reports a comprehensive survey of the proofs from a different perspective. These proofs have not required that the sequence of action probabilities of selecting the optimal action satisfies the property of monotonicity. On the other hand, whenever any action probability is close enough to unity, we require that the process jumps to an absorbing barrier at the next time instant, i.e., in a single step. By requiring such a constraint, these proofs invoke the weaker property, i.e., the submartinagale property of pm(t), to demonstrate the ε-optimality. We have thus proven the ε-optimality for the Absorbing CPA [49, 50], the Discretized PA [51, 52], and for the family of Bayesian PA [53], where the estimates are obtained by a Bayesian (rather than a Maximum Likelihood (ML)) process.

• Zhang, Xuan; Lei, Jiao; Granmo, Ole-Christoffer & Goodwin, Morten (2021). On the convergence of Tsetlin machines for the IDENTITY- and NOT operators. IEEE Transactions on Pattern Analysis and Machine Intelligence. ISSN 0162-8828. 44(10), p. 6345–6359. doi: 10.1109/TPAMI.2021.3085591. Full text in Research Archive
• Abeyrathna, Kuruge Darshana; Granmo, Ole-Christoffer; Zhang, Xuan & Goodwin, Morten (2020). Adaptive Continuous Feature Binarization for Tsetlin Machines Applied to Forecasting Dengue Incidences in the Philippines, 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781728125473. p. 2084–2092. doi: 10.1109/SSCI47803.2020.9308291. Full text in Research Archive
• Zhang, Xuan; Jiao, Lei; Oommen, John & Granmo, Ole-Christoffer (2019). A Conclusive Analysis of the Finite-Time Behavior of the Discretized Pursuit Learning Automaton. IEEE Transactions on Neural Networks and Learning Systems. ISSN 2162-237X. 31(1). doi: 10.1109/TNNLS.2019.2900639. Full text in Research Archive Show summary

  This paper deals with the finite-time analysis (FTA) of learning automata (LA), which is a topic for which very little work has been reported in the literature. This is as opposed to the asymptotic steady-state analysis for which there are, probably, scores of papers. As clarified later, unarguably, the FTA of Markov chains, in general, and of LA, in particular, is far more complex than the asymptotic steady-state analysis. Such an FTA provides rigid bounds for the time required for the LA to attain to a given convergence accuracy. We concentrate on the FTA of the Discretized Pursuit Automaton (DPA), which is probably one of the fastest and most accurate reported LA. Although such an analysis was carried out many years ago, we record that the previous work is flawed. More specifically, in all brevity, the flaw lies in the wrongly ``derived'' monotonic behavior of the LA after a certain number of iterations. Rather, we claim that the property should be invoked is the submartingale property. This renders the proof to be much more involved and deep. In this paper, we rectify the flaw and reestablish the FTA based on such a submartingale phenomenon. More importantly, from the derived analysis, we are able to discover and clarify, for the first time, the underlying dilemma between the DPA's exploitation and exploration properties. We also nontrivially confirm the existence of the optimal learning rate, which yields a better comprehension of the DPA itself.

• Abeyrathna, Kuruge Darshana; Granmo, Ole-Christoffer; Zhang, Xuan & Goodwin, Morten (2019). A Scheme for Continuous Input to the Tsetlin Machine with Applications to Forecasting Disease Outbreaks. In Wotawa, F (Eds.), Advances and Trends in Artificial Intelligence. From Theory to Practice.. Springer Nature. ISSN 9783030229993. p. 564–578. doi: 10.1007/978-3-030-22999-3_49. Full text in Research Archive Show summary

  In this paper, we apply a new promising tool for pattern classification, namely, the Tsetlin Machine (TM), to the field of disease forecasting. The TM is interpretable because it is based on manipulating expressions in propositional logic, leveraging a large team of Tsetlin Automata (TA). Apart from being interpretable, this approach is attractive due to its low computational cost and its capacity to handle noise. To attack the problem of forecasting, we introduce a preprocessing method that extends the TM so that it can handle continuous input. Briefly stated, we convert continuous input into a binary representation based on thresholding. The resulting extended TM is evaluated and analyzed using an artificial dataset. The TM is further applied to forecast dengue outbreaks of all the seventeen regions in Philippines using the spatio-temporal properties of the data. Experimental results show that dengue outbreak forecasts made by the TM are more accurate than those obtained by a Support Vector Machine (SVM), Decision Trees (DTs), and several multi-layered Artificial Neural Networks (ANNs), both in terms of forecasting precision and F1-score.

• Abeyrathna, Kuruge Darshana; Granmo, Ole-Christoffer; Zhang, Xuan; Lei, Jiao & Goodwin, Morten (2019). The regression Tsetlin machine: a novel approach to interpretable nonlinear regression. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. ISSN 1364-503X. doi: 10.1098/rsta.2019.0165. Full text in Research Archive
• Yazidi, Anis; Zhang, Xuan; Lei, Jiao & Oommen, John (2019). The Hierarchical Continuous Pursuit Learning Automation: A Novel Scheme for Environments With Large Numbers of Actions. IEEE Transactions on Neural Networks and Learning Systems. ISSN 2162-237X. 31(2). doi: 10.1109/TNNLS.2019.2905162. Full text in Research Archive Show summary

  Although the field of learning automata (LA) has made significant progress in the past four decades, the LA-based methods to tackle problems involving environments with a large number of actions is, in reality, relatively unresolved. The extension of the traditional LA to problems within this domain cannot be easily established when the number of actions is very large. This is because the dimensionality of the action probability vector is correspondingly large, and so, most components of the vector will soon have values that are smaller than the machine accuracy permits, implying that they will never be chosen . This paper presents a solution that extends the continuous pursuit paradigm to such large -actioned problem domains. The beauty of the solution is that it is hierarchical, where all the actions offered by the environment reside as leaves of the hierarchy. Furthermore, at every level, we merely require a two -action LA that automatically resolves the problem of dealing with arbitrarily small action probabilities. In addition, since all the LA invoke the pursuit paradigm, the best action at every level trickles up toward the root. Thus, by invoking the property of the “max” operator, in which the maximum of numerous maxima is the overall maximum, the hierarchy of LA converges to the optimal action. This paper describes the scheme and formally proves its $\epsilon $ -optimal convergence. The results presented here can, rather trivially, be extended for the families of discretized and Bayesian pursuit LA too. This paper also reports extensive experimental results (including for environments having 128 and 256 actions) that demonstrate the power of the scheme and its computational advantages. As far as we know, there are no comparable pursuit-based results in the field of LA . In some cases, the hierarchical continuous pursuit automaton requires less than 18% of the number of iterations than the benchmark $L_{R-I}$ scheme, which is, by all metrics, phenomenal .

• Yazidi, Anis; Zhang, Xuan; Lei, Jiao & Oommen, John (2018). The Hierarchical Continuous Pursuit Learning Automation for Large Numbers of Actions. In Lazaros, Iliadis (Eds.), Artificial Intelligence Applications and Innovations. Springer. ISSN 9783319920061. p. 451–461. doi: 10.1007/978-3-319-92007-8_38. Full text in Research Archive Show summary

  Although the field of Learning Automata (LA) has made significant progress in the last four decades, the LA-based methods to tackle problems involving environments with a large number of actions are, in reality, relatively unresolved. The extension of the traditional LA (fixed structure, variable structure, discretized, and pursuit) to problems within this domain cannot be easily established when the number of actions is very large. This is because the dimensionality of the action probability vector is correspondingly large, and consequently, most components of the vector will, after a relatively short time, have values that are smaller than the machine accuracy permits, implying that they will never be chosen. This paper pioneers a solution that extends the continuous pursuit paradigm to such large-actioned problem domains. The beauty of the solution is that it is hierarchical, where all the actions offered by the environment reside as leaves of the hierarchy. Further, at every level, we merely require a two-action LA which automatically resolves the problem of dealing with arbitrarily small action probabilities. Additionally, since all the LA invoke the pursuit paradigm, the best action at every level trickles up towards the root. Thus, by invoking the property of the “max” operator, in which, the maximum of numerous maxima is the overall maximum, the hierarchy of LA converges to the optimal action. Apart from reporting the theoretical properties of the scheme, the paper contains extensive experimental results which demonstrate the power of the scheme and its computational advantages. As far as we know, there are no comparable results in the field of LA.

• Lei, Jiao; Zhang, Xuan; Oommen, John & Granmo, Ole-Christoffer (2016). Optimizing channel selection for cognitive radio networks using a distributed Bayesian learning automata-based approach. Applied Intelligence - The International Journal of Research on Intelligent Systems for Real Life Complex Problems. ISSN 0924-669X. 44(2), p. 307–321. doi: 10.1007/s10489-015-0682-x. Full text in Research Archive
• Zhang, Xuan; Oommen, John & Granmo, Ole-Christoffer (2016). The design of absorbing Bayesian pursuit algorithms and the formal analyses of their ε-optimality. Pattern Analysis and Applications. ISSN 1433-7541. p. 1–12. doi: 10.1007/s10044-016-0535-1. Full text in Research Archive Show summary

  The fundamental phenomenon that has been used to enhance the convergence speed of learning automata (LA) is that of incorporating the running maximum likelihood (ML) estimates of the action reward probabilities into the probability updating rules for selecting the actions. The frontiers of this field have been recently expanded by replacing the ML estimates with their corresponding Bayesian counterparts that incorporate the properties of the conjugate priors. These constitute the Bayesian pursuit algorithm (BPA), and the discretized Bayesian pursuit algorithm. Although these algorithms have been designed and efficiently implemented, and are, arguably, the fastest and most accurate LA reported in the literature, the proofs of their ϵϵ-optimal convergence has been unsolved. This is precisely the intent of this paper. In this paper, we present a single unifying analysis by which the proofs of both the continuous and discretized schemes are proven. We emphasize that unlike the ML-based pursuit schemes, the Bayesian schemes have to not only consider the estimates themselves but also the distributional forms of their conjugate posteriors and their higher order moments—all of which render the proofs to be particularly challenging. As far as we know, apart from the results themselves, the methodologies of this proof have been unreported in the literature—they are both pioneering and novel.

• Zhang, Xuan; Oommen, John; Granmo, Ole-Christoffer & Lei, Jiao (2015). A formal proof of the e-optimality of discretized pursuit algorithms. Applied Intelligence - The International Journal of Research on Intelligent Systems for Real Life Complex Problems. ISSN 0924-669X. doi: 10.1007/s10489-015-0670-1. Full text in Research Archive
• Zhang, Xuan; Granmo, Ole-Christoffer; Oommen, John & Lei, Jiao (2014). A formal proof of the ε-optimality of absorbing continuous pursuit algorithms using the theory of regular functions. Applied Intelligence - The International Journal of Research on Intelligent Systems for Real Life Complex Problems. ISSN 0924-669X. 41(3), p. 974–985. doi: 10.1007/s10489-014-0541-1. Full text in Research Archive Show summary

  The most difficult part in the design and analysis of Learning Automata (LA) consists of the formal proofs of their convergence accuracies. The mathematical techniques used for the different families (Fixed Structure, Variable Structure, Discretized etc.) are quite distinct. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and within this family, the set of Pursuit algorithms have been considered to be the pioneering schemes. Informally, if the environment is stationary, their ε-optimality is defined as their ability to converge to the optimal action with an arbitrarily large probability, if the learning parameter is sufficiently small/large. The existing proofs of all the reported EAs follow the same fundamental principles, and to clarify this, in the interest of simplicity, we shall concentrate on the family of Pursuit algorithms. Recently, it has been reported Ryan and Omkar (J Appl Probab 49(3):795–805, 2012) that the previous proofs for ε-optimality of all the reported EAs have a common flaw. The flaw lies in the condition which apparently supports the so-called “monotonicity” property of the probability of selecting the optimal action, which states that after some time instant t 0, the reward probability estimates will be ordered correctly forever. The authors of the various proofs have rather offered a proof for the fact that the reward probability estimates are ordered correctly at a single point of time after t 0, which, in turn, does not guarantee the ordering forever, rendering the previous proofs incorrect. While in Ryan and Omkar (J Appl Probab 49(3):795–805, 2012), a rectified proof was presented to prove the ε-optimality of the Continuous Pursuit Algorithm (CPA), which was the pioneering EA, in this paper, a new proof is provided for the Absorbing CPA (ACPA), i.e., an algorithm which follows the CPA paradigm but which artificially has absorbing states whenever any action probability is arbitrarily close to unity. Unlike the previous flawed proofs, instead of examining the monotonicity property of the action probabilities, it rather examines their submartingale property, and then, unlike the traditional approach, invokes the theory of Regular functions to prove that the probability of converging to the optimal action can be made arbitrarily close to unity. We believe that the proof is both unique and pioneering, and adds insights into the convergence of different EAs. It can also form the basis for formally demonstrating the ε-optimality of other Estimator algorithms which are artificially rendered absorbing.

• Lei, Jiao; Zhang, Xuan; Granmo, Ole-Christoffer & Oommen, John (2014). A Bayesian Learning Automata-Based Distributed Channel Selection Scheme. In Moonis, Ali; Pan, Jeng-Shyang; Chen, Shyi-Ming & Horng, Mong-Fong (Ed.), Modern Advances in Applied Intelligence, 27th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2014,Kaohsiung, Taiwan, June 3-6, 2014, Part II. Springer. ISSN 9783319074559. p. 48–57. doi: 10.1007/978-3-319-07467-2_6. Full text in Research Archive Show summary

  We consider a scenario where multiple Secondary Users (SUs) operate within a Cognitive Radio Network (CRN) which involves a set of channels, where each channel is associated with a Primary User (PU). We investigate two channel access strategies for SU transmissions. In the first strategy, the SUs will send a packet directly without operating Carrier Sensing Medium Access/Collision Avoidance (CSMA/CA) whenever a PU is absent in the selected channel. In the second strategy, the SUs implement CSMA/CA to further reduce the probability of collisions among co-channel SUs. For each strategy, the channel selection problem is formulated and demonstrated to be a so-called “Potential” game, and a Bayesian Learning Automata (BLA) has been incorporated into each SU so to play the game in such a manner that the SU can adapt itself to the environment. The performance of the BLA in this application is evaluated through rigorous simulations. These simulation results illustrate the convergence of the SUs to the global optimum in the first strategy, and to a Nash Equilibrium (NE) point in the second.

• Zhang, Xuan; Oommen, John; Granmo, Ole-Christoffer & Lei, Jiao (2014). Using the Theory of Regular Functions to Formally Prove the ε -Optimality of Discretized Pursuit Learning Algorithms. In Moonis, Ali; Pan, Jeng-Shyang; Chen, Shyi-Ming & Horng, Mong-Fong (Ed.), Modern Advances in Applied Intelligence - 27th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2014, Kaohsiung, Taiwan, June 3-6, 2014, Proceedings, Part I. Springer. ISSN 9783319074672. p. 379–388. doi: 10.1007/978-3-319-07455-9_40. Full text in Research Archive Show summary

  Learning Automata (LA) can be reckoned to be the founding algorithms on which the field of Reinforcement Learning has been built. Among the families of LA, Estimator Algorithms (EAs) are certainly the fastest, and of these, the family of Pursuit Algorithms (PAs) are the pioneering work. It has recently been reported that the previous proofs for ε -optimality for all the reported al- gorithms in the family of PAs have been flawed 1. We applaud the researchers who discovered this flaw, and who further proceeded to rectify the proof for the Continuous Pursuit Algorithm (CPA). The latter proof, though requires the learning parameter to be continuously changing, is, to the best of our knowledge, the current best and only way to prove CPA’s ε -optimality. However, for all the algorithms with absorbing states, for example, the Absorbing Continuous Pursuit Algorithm (ACPA) and the Discretized Pursuit Algorithm (DPA), the constrain of a continuously changing learning parameter can be removed. In this paper, we provide a new method to prove the ε -optimality of the Discretized Pursuit Algorithm which does not require this constraint. We believe that our proof is both unique and pioneering. It can also form the basis for formally showing the ε -optimality of the other EAs with absorbing states.

• Zhang, Xuan; Lei, Jiao; Granmo, Ole-Christoffer & Oommen, John (2013). Channel selection in cognitive radio networks: A switchable Bayesian learning automata approach. In Correia, Luis M. (Eds.), 2013 IEEE 24th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781467362344. p. 2362–2367. Full text in Research Archive Show summary

  We consider the problem of a user operating within a Cognitive Radio Network (CRN) which involves N channels each associated with a Primary User (PU). The problem consists of allocating a channel which, at any given time instant is not being used by a PU, to a Secondary User (SU). Within our study, we assume that a SU is allowed to perform “channel switching”, i.e., to choose an alternate channel S times (where S +1 ≤ N) if the previous choice does not lead to a channel which is vacant. The paper first presents a formal probabilistic model for the problem itself, referred to as the Formal Secondary Channel Selection (FSCS) problem, and the characteristics of the FSCS are then analyzed. Thereafter, the paper proposes a fascinating solution to the FSCS problem by invoking the recently devised Bayesian Learning Automaton (BLA). The crucial advantage of the BLA is that unlike traditional Learning Automata (LA), it does not involve an action probability vector, but rather relies on “sampling” as per the a posteriori Bayesian estimates of the channel occupation probabilities. However, rather than utilize the BLA in the form that was earlier proposed, we shall extend it to the so-called Switchable Bayesian Learning Automaton (SBLA), which, indeed, attains the optimal solution in the overall composite action space. Apart from proposing the solution, the paper also contains detailed simulation results which demonstrate the power of the solution proposed.

• Zhang, Xuan; Granmo, Ole-Christoffer; Oommen, John & Lei, Jiao (2013). On Using the Theory of Regular Functions to Prove the Epsilon-Optimality of the Continuous Pursuit Learn- ing Automaton. In Ali, Moonis (Eds.), Recent Trends in Applied Artificial Intelligence. Springer. ISSN 9783642385766. p. 262–271. doi: 10.1007/978-3-642-38577-3_27. Full text in Research Archive Show summary

  There are various families of Learning Automata (LA) such as Fixed Structure, Variable Structure, Discretized etc. Informally, if the environment is stationary, their ε-optimality is defined as their ability to converge to the optimal action with an arbitrarily large probability, if the learning parameter is sufficiently small/large. Of these LA families, Estimator Algorithms (EAs) are certainly the fastest, and within this family, the set of Pursuit algorithms have been considered to be the pioneering schemes. The existing proofs of the ε-optimality of all the reported EAs follow the same fundamental principles. Recently, it has been reported that the previous proofs for the ε-optimality of all the reported EAs have a common flaw. In other words, people have worked with this flawed reasoning for almost three decades. The flaw lies in the condition which apparently supports the so-called “monotonicity” property of the probability of selecting the optimal action, explained in the paper. In this paper, we provide a new method to prove the ε-optimality of the Continuous Pursuit Algorithm (CPA), which was the pioneering EA. The new proof follows the same outline of the previous proofs, but instead of examining the monotonicity property of the action probabilities, it rather examines their submartingale property, and then, unlike the traditional approach, invokes the theory of Regular functions to prove the ε-optimality. We believe that the proof is both unique and pioneering, and that it can form the basis for formally demonstrating the ε-optimality of other EAs.

• Zhang, Xuan; Granmo, Ole-Christoffer & Oommen, John (2013). On incorporating the paradigms of discretization and Bayesian estimation to create a new family of pursuit learning automata. Applied Intelligence - The International Journal of Research on Intelligent Systems for Real Life Complex Problems. ISSN 0924-669X. 39(4), p. 782–792. doi: 10.1007/s10489-013-0424-x. Full text in Research Archive Show summary

  There are currently two fundamental paradigms that have been used to enhance the convergence speed of Learning Automata (LA). The first involves the concept of utilizing the estimates of the reward probabilities, while the second involves discretizing the probability space in which the LA operates. This paper demonstrates how both of these can be simultaneously utilized, and in particular, by using the family of Bayesian estimates that have been proven to have distinct advantages over their maximum likelihood counterparts. The success of LA-based estimator algorithms over the classical, Linear Reward-Inaction (L RI )-like schemes, can be explained by their ability to pursue the actions with the highest reward probability estimates. Without access to reward probability estimates, it makes sense for schemes like the L RI to first make large exploring steps, and then to gradually turn exploration into exploitation by making progressively smaller learning steps. However, this behavior becomes counter-intuitive when pursuing actions based on their estimated reward probabilities. Learning should then ideally proceed in progressively larger steps, as the reward probability estimates turn more accurate. This paper introduces a new estimator algorithm, the Discretized Bayesian Pursuit Algorithm (DBPA), that achieves this by incorporating both the above paradigms. The DBPA is implemented by linearly discretizing the action probability space of the Bayesian Pursuit Algorithm (BPA) (Zhang et al. in IEA-AIE 2011, Springer, New York, pp. 608–620, 2011). The key innovation of this paper is that the linear discrete updating rules mitigate the counter-intuitive behavior of the corresponding linear continuous updating rules, by augmenting them with the reward probability estimates. Extensive experimental results show the superiority of DBPA over previous estimator algorithms. Indeed, the DBPA is probably the fastest reported LA to date. Apart from the rigorous experimental demonstration of the strength of the DBPA, the paper also briefly records the proofs of why the BPA and the DBPA are ϵ-optimal in stationary environments.

• Zhang, Xuan; Granmo, Ole-Christoffer & Oommen, John (2012). Discretized Bayesian Pursuit – A New Scheme for Reinforcement Learning. In Jiang, He; Ding, Wei; Ali, Moonis & Wu, Xindong (Ed.), Advanced Research in Applied Artificial Intelligence : 25th International Conference 25th International Conference on Industrial Engineering and OtherApplications of Applied Intelligent Systems, IEA/AIE 2012 Dalian, China, June 9-12, 2012 Proceedings. Springer. ISSN 9783642310867. p. 784–793. doi: 10.1007/978-3-642-31087-4_79. Full text in Research Archive Show summary

  The success of Learning Automata (LA)-based estimator algorithms over the classical, Linear Reward-Inaction (LRI)-like schemes, can be explained by their ability to pursue the actions with the highest reward probability estimates. Without access to reward probability estimates, it makes sense for schemes like the LRI to first make large exploring steps, and then to gradually turn exploration into exploitation by making progressively smaller learning steps. However, this behavior becomes counter-intuitive when pursuing actions based on their estimated reward probabilities. Learning should then ideally proceed in progressively larger steps, as the reward probability estimates turn more accurate. This paper introduces a new estimator algorithm, the Discretized Bayesian Pursuit Algorithm (DBPA), that achieves this. The DBPA is implemented by linearly discretizing the action probability space of the Bayesian Pursuit Algorithm (BPA) [1]. The key innovation is that the linear discrete updating rules mitigate the counter-intuitive behavior of the corresponding linear continuous updating rules, by augmenting them with the reward probability estimates. Extensive experimental results show the superiority of DBPA over previous estimator algorithms. Indeed, the DBPA is probably the fastest reported LA to date

• Zhang, Xuan; Oommen, John & Granmo, Ole-Christoffer (2011). Generalized Bayesian pursuit: a novel scheme for multi-armed Bernoulli bandit problems. IFIP Advances in Information and Communication Technology. ISSN 1868-4238. 364, p. 122–131. Full text in Research Archive
• Zhang, Xuan; Granmo, Ole-Christoffer & Oommen, John (2011). The Bayesian Pursuit Algorithm: A New Family of Estimator Learning Automata. In Mehrotra, Kishan G.; Mohan, Chilukuri K.; Oh, Jae C.; Varshney, Pramod K. & Ali, Moonis (Ed.), Modern Approaches in Applied Intelligence24th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2011, Syracuse, NY, USA, June 28 – July 1, 2011, Proceedings, Part II. Springer. ISSN 9783642218262. p. 522–531. doi: 10.1007/978-3-642-21827-9_53. Full text in Research Archive Show summary

  he fastest Learning Automata (LA) algorithms currently available come from the family of estimator algorithms. The Pursuit algorithm (PST), a pioneering scheme in the estimator family, obtains its superior learning speed by using Maximum Likelihood (ML) estimates to pursue the action currently perceived as being optimal. Recently, a Bayesian LA (BLA) was introduced, and empirical results that demonstrated its advantages over established top performers, including the PST scheme, were reported. The BLA scheme is inherently Bayesian in nature, but it succeeds in avoiding the computational intractability by merely relying on updating the hyper-parameters of sibling conjugate priors, and on random sampling from the resulting posteriors. In this paper, we integrate the foundational learning principles motivating the design of the BLA, with the principles of the PST. By doing this, we have succeeded in obtaining a completely novel, and rather pioneering, approach to solving LA-like problems, namely, by designing the Bayesian Pursuit algorithm (BPST). As in the BLA, the estimates are truly Bayesian (as opposed to ML) in nature. However, the action selection probability vector of the PST is used for its exploration purposes. Also, unlike the ML estimate, which is usually a single value, the use of a posterior distribution permits us to choose any one of a spectrum of values in the posterior, as the appropriate estimate. Thus, in this paper, we have chosen a 95% percentile value of the posterior (instead of the mean) to pursue the most promising actions. Further, as advocated in [7], the pursuit has been done using both the Linear Reward-Penalty and Reward-Inaction philosophies, leading to the corresponding BPSTRP and BPSTRI schemes respectively. It turns out that the BPST is superior to the PST, with the BPSTRI being even more robust than the BPSTRP. Moreover, by controlling the learning speed of the BPST, the BPST schemes perform either better or comparable to the BLA. We thus believe that the BPST constitutes a new avenue of research, in which the performance benefits of the PST and the BLA are mutually augmented, opening up for improved performance in a number of applications, currently being tested.

• Zhang, Xuan; Oommen, John & Granmo, Ole-Christoffer (2011). Generalized Bayesian Pursuit: A Novel Scheme for Multi-Armed Bernoulli Bandit Problems. In Iliadis, Lazaros; Maglogiannis, Ilias & Papadopoulos, Harris (Ed.), Artificial Intelligence Applications and Innovations 12th INNS EANN-SIG International Conference,EANN2011 and 7th IFIP WG 12.5 International Conference, AIAI 2011 Corfu, Greece, September 15-18, 2011 Proceedings , Part II. Springer. ISSN 9783642239595. p. 122–131. doi: 10.1007/978-3-642-23960-1_16. Full text in Research Archive Show summary

  In the last decades, a myriad of approaches to the multi-armed bandit problem have appeared in several different fields. The current top performing algorithms from the field of Learning Automata reside in the Pursuit family, while UCB-Tuned and the ε-greedy class of algorithms can be seen as state-ofthe- art regret minimizing algorithms. Recently, however, the Bayesian Learning Automaton (BLA) outperformed all of these, and other schemes, in a wide range of experiments. Although seemingly incompatible, in this paper we integrate the foundational learning principles motivating the design of the BLA, with the principles of the so-called Generalized Pursuit algorithm (GPST), leading to the Generalized Bayesian Pursuit algorithm (GBPST). As in the BLA, the estimates are truly Bayesian in nature, however, instead of basing exploration upon direct sampling from the estimates, GBPST explores by means of the arm selection probability vector of GPST. Further, as in the GPST, in the interest of higher rates of learning, a set of arms that are currently perceived as being optimal is pursued to minimize the probability of pursuing a wrong arm. It turns out that GBPST is superior to GPST and that it even performs better than the BLA by controlling the learning speed of GBPST. We thus believe that GBPST constitutes a new avenue of research, in which the performance benefits of the GPST and the BLA are mutually augmented, opening up for improved performance in a number of applications, currently being tested.

View all works in NVA

• Zhang, Xuan; Indresand, Hege; Belbachir, Nabil & Fiksdal, Jan Robert (2024). Hyperspectral Imaging and Computer Vision Based Remote Monitoring of SO2 Emissions in Maritime Vessels. Full text in Research Archive

View all works in NVA