Highly comprehensive, detailed, and up-to-date overview of artificial intelligence and cybernetics, with practical examples and supplementary learning resources
Cybernetical Intelligence: Engineering Cybernetics with Machine Intelligence is a comprehensive guide to the field of cybernetics and neural networks, as well as the mathematical foundations of these technologies. The book provides a detailed explanation of various types of neural networks, including feedforward networks, recurrent neural networks, and convolutional neural networks as well as their applications to different real-world problems. This groundbreaking book presents a pioneering exploration of machine learning within the framework of cybernetics. It marks a significant milestone in the field’s history, as it is the first book to describe the development of machine learning from a cybernetics perspective. The introduction of the concept of “Cybernetical Intelligence” and the generation of new terminology within this context propel new lines of thought in the historical development of artificial intelligence. With its profound implications and contributions, this book holds immense importance and is poised to become a definitive resource for scholars and researchers in this field of study.
Each chapter is specifically designed to introduce the theory with several examples. This comprehensive book includes exercise questions at the end of each chapter, providing readers with valuable opportunities to apply and strengthen their understanding of cybernetical intelligence. To further support the learning journey, solutions to these questions are readily accessible on the book’s companion site. Additionally, the companion site offers programming practice exercises and assignments, enabling readers to delve deeper into the practical aspects of the subject matter.
Cybernetical Intelligence includes information on: - The history and development of cybernetics and its influence on the development of neural networks - Developments and innovations in artificial intelligence and machine learning, such as deep reinforcement learning, generative adversarial networks, and transfer learning - Mathematical foundations of artificial intelligence and cybernetics, including linear algebra, calculus, and probability theory - Ethical implications of artificial intelligence and cybernetics as well as responsible and transparent development and deployment of AI systems
Presenting a highly detailed and comprehensive overview of the field, with modern developments thoroughly discussed, Cybernetical Intelligence is an essential textbook that helps students make connections with real-life engineering problems by providing both theory and practice, along with a myriad of helpful learning aids.
Table of Contents
Preface xv
About the Author xix
About the Companion Website xxi
1 Artificial Intelligence and Cybernetical Learning 1
1.1 Artificial Intelligence Initiative 1
1.2 Intelligent Automation Initiative 4
1.2.1 Benefits of IAI 5
1.3 Artificial Intelligence Versus Intelligent Automation 5
1.3.1 Process Discovery 6
1.3.2 Optimization 7
1.3.3 Analytics and Insight 8
1.4 The Fourth Industrial Revolution and Artificial Intelligence 9
1.4.1 Artificial Narrow Intelligence 10
1.4.2 Artificial General Intelligence 12
1.4.3 Artificial Super Intelligence 13
1.5 Pattern Analysis and Cognitive Learning 14
1.5.1 Machine Learning 15
1.5.1.1 Parametric Algorithms 16
1.5.1.2 Nonparametric Algorithms 17
1.5.2 Deep Learning 20
1.5.2.1 Convolutional Neural Networks in Advancing Artificial Intelligence 21
1.5.2.2 Future Advancement in Deep Learning 22
1.5.3 Cybernetical Learning 23
1.6 Cybernetical Artificial Intelligence 24
1.6.1 Artificial Intelligence Control Theory 24
1.6.2 Information Theory 26
1.6.3 Cybernetic Systems 27
1.7 Cybernetical Intelligence Definition 28
1.8 The Future of Cybernetical Intelligence 30
Summary 32
Exercise Questions 32
Further Reading 33
2 Cybernetical Intelligent Control 35
2.1 Control Theory and Feedback Control Systems 35
2.2 Maxwell’s Analysis of Governors 37
2.3 Harold Black 39
2.4 Nyquist and Bode 40
2.5 Stafford Beer 42
2.5.1 Cybernetic Control 42
2.5.2 Viable Systems Model 42
2.5.3 Cybernetics Models of Management 43
2.6 James Lovelock 43
2.6.1 Cybernetic Approach to Ecosystems 43
2.6.2 Gaia Hypothesis 44
2.7 Macy Conference 44
2.8 McCulloch-Pitts 45
2.9 John von Neumann 47
2.9.1 Discussions on Self-Replicating Machines 47
2.9.2 Discussions on Machine Learning 48
Summary 48
Exercise Questions 49
Further Reading 50
3 The Basics of Perceptron 51
3.1 The Analogy of Biological and Artificial Neurons 51
3.1.1 Biological Neurons and Neurodynamics 52
3.1.2 The Structure of Neural Network 53
3.1.3 Encoding and Decoding 56
3.2 Perception and Multilayer Perceptron 57
3.2.1 Back Propagation Neural Network 59
3.2.2 Derivative Equations for Backpropagation 59
3.3 Activation Function 61
3.3.1 Sigmoid Activation Function 61
3.3.2 Hyperbolic Tangent Activation Function 62
3.3.3 Rectified Linear Unit Activation Function 62
3.3.4 Linear Activation Function 64
Summary 65
Exercise Questions 67
Further Reading 67
4 The Structure of Neural Network 69
4.1 Layers in Neural Network 69
4.1.1 Input Layer 69
4.1.2 Hidden Layer 70
4.1.3 Neurons 70
4.1.4 Weights and Biases 71
4.1.5 Forward Propagation 72
4.1.6 Backpropagation 72
4.2 Perceptron and Multilayer Perceptron 73
4.3 Recurrent Neural Network 75
4.3.1 Long Short-Term Memory 76
4.4 Markov Neural Networks 77
4.4.1 State Transition Function 77
4.4.2 Observation Function 78
4.4.3 Policy Function 78
4.4.4 Loss Function 78
4.5 Generative Adversarial Network 78
Summary 79
Exercise Questions 80
Further Reading 81
5 Backpropagation Neural Network 83
5.1 Backpropagation Neural Network 83
5.1.1 Forward Propagation 85
5.2 Gradient Descent 85
5.2.1 Loss Function 85
5.2.2 Parameters in Gradient Descent 88
5.2.3 Gradient in Gradient Descent 88
5.2.4 Learning Rate in Gradient Descent 89
5.2.5 Update Rule in Gradient Descent 89
5.3 Stopping Criteria 89
5.3.1 Convergence and Stopping Criteria 90
5.3.2 Local Minimum and Global Minimum 91
5.4 Resampling Methods 91
5.4.1 Cross-Validation 93
5.4.2 Bootstrapping 93
5.4.3 Monte Carlo Cross-Validation 94
5.5 Optimizers in Neural Network 94
5.5.1 Stochastic Gradient Descent 94
5.5.2 Root Mean Square Propagation 96
5.5.3 Adaptive Moment Estimation 96
5.5.4 AdaMax 97
5.5.5 Momentum Optimization 97
Summary 97
Exercise Questions 99
Further Reading 100
6 Application of Neural Network in Learning and Recognition 101
6.1 Applying Backpropagation to Shape Recognition 101
6.2 Softmax Regression 105
6.3 K-Binary Classifier 107
6.4 Relational Learning via Neural Network 108
6.4.1 Graph Neural Network 109
6.4.2 Graph Convolutional Network 111
6.5 Cybernetics Using Neural Network 112
6.6 Structure of Neural Network for Image Processing 115
6.7 Transformer Networks 116
6.8 Attention Mechanisms 116
6.9 Graph Neural Networks 117
6.10 Transfer Learning 118
6.11 Generalization of Neural Networks 119
6.12 Performance Measures 120
6.12.1 Confusion Matrix 120
6.12.2 Receiver Operating Characteristic 121
6.12.3 Area Under the ROC Curve 122
Summary 123
Exercise Questions 123
Further Reading 124
7 Competitive Learning and Self-Organizing Map 125
7.1 Principal of Competitive Learning 125
7.1.1 Step 1: Normalized Input Vector 128
7.1.2 Step 2: Find the Winning Neuron 128
7.1.3 Step 3: Adjust the Network Weight Vector and Output Results 129
7.2 Basic Structure of Self-Organizing Map 129
7.2.1 Properties Self-Organizing Map 130
7.3 Self-Organizing Mapping Neural Network Algorithm 131
7.3.1 Step 1: Initialize Parameter 132
7.3.2 Step 2: Select Inputs and Determine Winning Nodes 132
7.3.3 Step 3: Affect Neighboring Neurons 132
7.3.4 Step 4: Adjust Weights 133
7.3.5 Step 5: Judging the End Condition 133
7.4 Growing Self-Organizing Map 133
7.5 Time Adaptive Self-Organizing Map 136
7.5.1 TASOM-Based Algorithms for Real Applications 138
7.6 Oriented and Scalable Map 139
7.7 Generative Topographic Map 141
Summary 145
Exercise Questions 146
Further Reading 147
8 Support Vector Machine 149
8.1 The Definition of Data Clustering 149
8.2 Support Vector and Margin 152
8.3 Kernel Function 155
8.3.1 Linear Kernel 155
8.3.2 Polynomial Kernel 156
8.3.3 Radial Basis Function 157
8.3.4 Laplace Kernel 159
8.3.5 Sigmoid Kernel 159
8.4 Linear and Nonlinear Support Vector Machine 160
8.5 Hard Margin and Soft Margin in Support Vector Machine 164
8.6 I/O of Support Vector Machine 167
8.6.1 Training Data 167
8.6.2 Feature Matrix and Label Vector 168
8.7 Hyperparameters of Support Vector Machine 169
8.7.1 The C Hyperparameter 169
8.7.2 Kernel Coefficient 169
8.7.3 Class Weights 170
8.7.4 Convergence Criteria 170
8.7.5 Regularization 171
8.8 Application of Support Vector Machine 171
8.8.1 Classification 171
8.8.2 Regression 173
8.8.3 Image Classification 173
8.8.4 Text Classification 174
Summary 174
Exercise Questions 175
Further Reading 176
9 Bio-Inspired Cybernetical Intelligence 177
9.1 Genetic Algorithm 178
9.2 Ant Colony Optimization 181
9.3 Bees Algorithm 184
9.4 Artificial Bee Colony Algorithm 186
9.5 Cuckoo Search 189
9.6 Particle Swarm Optimization 193
9.7 Bacterial Foraging Optimization 196
9.8 Gray Wolf Optimizer 197
9.9 Firefly Algorithm 199
Summary 200
Exercise Questions 201
Further Reading 202
10 Life-Inspired Machine Intelligence and Cybernetics 203
10.1 Multi-Agent AI Systems 203
10.1.1 Game Theory 205
10.1.2 Distributed Multi-Agent Systems 206
10.1.3 Multi-Agent Reinforcement Learning 207
10.1.4 Evolutionary Computation and Multi-Agent Systems 209
10.2 Cellular Automata 211
10.3 Discrete Element Method 212
10.3.1 Particle-Based Simulation of Biological Cells and Tissues 214
10.3.2 Simulation of Microbial Communities and Their Interactions 215
10.3.3 Discrete Element Method-Based Modeling of Biological Fluids and Soft Materials 216
10.4 Smoothed Particle Hydrodynamics 218
10.4.1 SPH-Based Simulations of Biomimetic Fluid Dynamic 219
10.4.2 SPH-Based Simulations of Bio-Inspired Engineering Applications 220
Summary 221
Exercise Questions 222
Further Reading 223
11 Revisiting Cybernetics and Relation to Cybernetical Intelligence 225
11.1 The Concept and Development of Cybernetics 225
11.1.1 Attributes of Control Concepts 225
11.1.2 Research Objects and Characteristics of Cybernetics 226
11.1.3 Development of Cybernetical Intelligence 227
11.2 The Fundamental Ideas of Cybernetics 227
11.2.1 System Idea 227
11.2.2 Information Idea 229
11.2.3 Behavioral Idea 230
11.2.4 Cybernetical Intelligence Neural Network 231
11.3 Cybernetic Expansion into Other Fields of Research 234
11.3.1 Social Cybernetics 234
11.3.2 Internal Control-Related Theories 237
11.3.3 Software Control Theory 237
11.3.4 Perceptual Cybernetics 238
11.4 Practical Application of Cybernetics 240
11.4.1 Research on the Control Mechanism of Neural Networks 240
11.4.2 Balance Between Internal Control and Management Power Relations 240
11.4.3 Software Markov Adaptive Testing Strategy 242
11.4.4 Task Analysis Model 244
Summary 245
Exercise Questions 246
Further Reading 247
12 Turing Machine 249
12.1 Behavior of a Turing Machine 250
12.1.1 Computing with Turing Machines 251
12.2 Basic Operations of a Turing Machine 252
12.2.1 Reading and Writing to the Tape 253
12.2.2 Moving the Tape Head 254
12.2.3 Changing States 254
12.3 Interchangeability of Program and Behavior 255
12.4 Computability Theory 256
12.4.1 Complexity Theory 257
12.5 Automata Theory 258
12.6 Philosophical Issues Related to Turing Machines 259
12.7 Human and Machine Computations 260
12.8 Historical Models of Computability 261
12.9 Recursive Functions 262
12.10 Turing Machine and Intelligent Control 263
Summary 264
Exercise Questions 265
Further Reading 265
13 Entropy Concepts in Machine Intelligence 267
13.1 Relative Entropy of Distributions 268
13.2 Relative Entropy and Mutual Information 268
13.3 Entropy in Performance Evaluation 269
13.4 Cross-Entropy Softmax 271
13.5 Calculating Cross-Entropy 272
13.6 Cross-Entropy as a Loss Function 273
13.7 Cross-Entropy and Log Loss 274
13.8 Application of Entropy in Intelligent Control 275
13.8.1 Entropy-Based Control 275
13.8.2 Fuzzy Entropy 276
13.8.3 Entropy-Based Control Strategies 277
13.8.4 Entropy-Based Decision-Making 278
Summary 279
Exercise Questions 279
Further Reading 280
14 Sampling Methods in Cybernetical Intelligence 283
14.1 Introduction to Sampling Methods 283
14.2 Basic Sampling Algorithms 284
14.2.1 Importance of Sampling Methods in Machine Intelligence 286
14.3 Machine Learning Sampling Methods 287
14.3.1 Random Oversampling 288
14.3.2 Random Undersampling 290
14.3.3 Synthetic Minority Oversampling Technique 290
14.3.4 Adaptive Synthetic Sampling 292
14.4 Advantages and Disadvantages of Machine Learning Sampling Methods 293
14.5 Advanced Sampling Methods in Cybernetical Intelligence 294
14.5.1 Ensemble Sampling Method 295
14.5.2 Active Learning 297
14.5.3 Bayesian Optimization in Sampling 299
14.6 Applications of Sampling Methods in Cybernetical Intelligence 302
14.6.1 Image Processing and Computer Vision 302
14.6.2 Natural Language Processing 304
14.6.3 Robotics and Autonomous Systems 307
14.7 Challenges and Future Directions 308
14.8 Challenges and Limitations of Sampling Methods 309
14.9 Emerging Trends and Innovations in Sampling Methods 309
Summary 310
Exercise Questions 311
Further Reading 312
15 Dynamic System Control 313
15.1 Linear Systems 314
15.2 Nonlinear System 316
15.3 Stability Theory 318
15.4 Observability and Identification 320
15.5 Controllability and Stabilizability 321
15.6 Optimal Control 323
15.7 Linear Quadratic Regulator Theory 324
15.8 Time-Optimal Control 326
15.9 Stochastic Systems with Applications 328
15.9.1 Stochastic System in Control Systems 329
15.9.2 Stochastic System in Robotics and Automation 329
15.9.3 Stochastic System in Neural Networks 330
Summary 331
Exercise Questions 331
Further Reading 332
16 Deep Learning 333
16.1 Neural Network Models in Deep Learning 335
16.2 Methods of Deep Learning 336
16.2.1 Convolutional Neural Networks 337
16.2.2 Recurrent Neural Networks 340
16.2.3 Generative Adversarial Networks 342
16.2.4 Deep Learning Based Image Segmentation Models 345
16.2.5 Variational Auto Encoders 348
16.2.6 Transformer Models 350
16.2.7 Attention-Based Models 352
16.2.8 Meta-Learning Models 354
16.2.9 Capsule Networks 357
16.3 Deep Learning Frameworks 358
16.4 Applications of Deep Learning 359
16.4.1 Object Detection 360
16.4.2 Intelligent Power Systems 361
16.4.3 Intelligent Control 362
Summary 362
Exercise Questions 363
References 364
Further Reading 365
17 Neural Architecture Search 367
17.1 Neural Architecture Search and Neural Network 369
17.2 Reinforcement Learning-Based Neural Architecture Search 371
17.3 Evolutionary Algorithms-Based Neural Architecture Search 374
17.4 Bayesian Optimization-Based Neural Architecture Search 376
17.5 Gradient-Based Neural Architecture Search 378
17.6 One-shot Neural Architecture Search 379
17.7 Meta-Learning-Based Neural Architecture Search 381
17.8 Neural Architecture Search for Specific Domains 383
17.8.1 Cybernetical Intelligent Systems: Neural Architecture Search in Real-World 384
17.8.2 Neural Architecture Search for Specific Cybernetical Control Tasks 385
17.8.3 Neural Architecture Search for Cybernetical Intelligent Systems in Real-World 386
17.8.4 Neural Architecture Search for Adaptive Cybernetical Intelligent Systems 388
17.9 Comparison of Different Neural Architecture Search Approaches 389
Summary 391
Exercise Questions 391
Further Reading 392
Final Notes on Cybernetical Intelligence 393
Index 399
