“The discussion is firmly grounded in established art historical practices, such as close visual analysis and an understanding of artists’ working methods, and real-world examples demonstrate how computer-assisted techniques can complement traditional approaches.”
- Dr. Emilie Gordenker, Director of the Van Gogh Museum
The pioneering presentation of computer-based image analysis of fine art, forging a dialog between art scholars and the computer vision community
In recent years, sophisticated computer vision, graphics, and artificial intelligence algorithms have proven to be increasingly powerful tools in the study of fine art. These methods - some adapted from forensic digital photography and others developed specifically for art - empower a growing number of computer-savvy art scholars, conservators, and historians to answer longstanding questions as well as provide new approaches to the interpretation of art.
Pixels & Paintings provides the first and authoritative overview of the broad range of these methods, which extend from image processing of palette, marks, brush strokes, and shapes up through analysis of objects, poses, style, composition, to the computation of simple interpretations of artworks. This book stresses that computer methods for art analysis must always incorporate the cultural contexts appropriate to the art studies at hand - a blend of humanistic and scientific expertise. - Describes powerful computer image analysis methods and their application to problems in the history and interpretation of fine art - Discusses some of the art historical lessons and revelations provided by the use of these methods - Clarifies the assumptions and applicability of methods and the role of cultural contexts in their use - Shows how computation can be used to analyze tens of thousands of artworks to reveal trends and anomalies that could not be found by traditional non-computer methods
Pixels & Paintings is essential reading for computer image analysts and graphics specialists, conservators, historians, students, psychologists and the general public interested in the study and appreciation of art.
Table of Contents
List of Figures xxi
List of Tables xlv
List of Algorithms xlvii
Preface xlix
Lorenzo Lotto lviii
Giovanni Morelli and the birth of "scientific" connoisseurship lix
Overview lxi
Intended audience lxii
Prerequisites lxiii
Acknowledgements lxiv
1 Digital imaging 1
1.1 Introduction 1
1.2 Electromagnetic radiation and light 4
1.3 Interaction of electromagnetic radiation with art materials 7
1.4 Cameras and scanners 9
1.4.1 Cameras 10
1.4.2 Flatbed scanners 11
1.5 Parameters for image acquisition in the visible 12
Billy Pappas 13
1.5.1 Spatial resolution 15
1.5.2 Bit depth 16
1.5.3 Dynamic range and contrast 17
1.6 Reading digital images of art on-screen 18
1.6.1 Reading a digital image of Leonardo's La Bella Principessa 22
Leonardo da Vinci 22
1.7 Infrared photography and reflectography 25
1.8 Ultraviolet imaging 26
1.9 Multispectral and hyperspectral imaging 27
1.9.1 Hyperspectral imaging of the Archimedes Palimpsest 30
1.10 X-radiographic imaging 32
1.11 Fluorescence imaging 35
1.12 Capture of three-dimensional surfaces of art 37
1.12.1 Raking illumination 38
1.12.2 Reflectance transformation imaging (RTI) 40
1.12.3 Stereographic imaging 42
1.13 Optical coherence tomography (OCT) 43
1.14 Raman spectroscopic imaging and X-ray fluorescence imaging 45
1.14.1 Raman spectroscopic imaging (RSI) 45
1.14.2 X-ray fluorescence imaging (XRF) 46
1.15 Summary 47
1.16 Bibliographical remarks 49
2 Image processing 53
2.1 Introduction 53
2.2 Pixel-based image processing 57
2.3 Region-based image processing 61
2.3.1 Linear image processing 62
2.3.2 Nonlinear region-based image processing 63
2.3.3 Color quantization 64
2.3.4 Edge and line detection 69
2.3.5 Dilation and erosion 71
2.3.6 Skeletonization 72
2.4 Inpainting 72
2.5 Feature extraction 74
2.5.1 Keypoint extraction 75
2.5.2 Craquelure and crazing analysis 78
2.5.3 Computational tests for counterproofing by Jan van der Heyden 81
Jan van der Heyden 83
2.6 Segmentation 86
2.6.1 Deep nets for image segmentation 88
2.7 Geometric transformations 95
2.8 Chamfer transform and Chamfer distance 101
2.8.1 Tests for copying of Jan van Eyck's portraits of Niccolò Albergati 103
2.9 Discrete Fourier and wavelet transforms 111
2.9.1 Discrete Fourier transform (DFT) 111
2.9.2 Canvas support weave analysis 114
2.9.3 Discrete wavelet transform (DWT) 116
2.10 Compositing and integrating art images 118
2.10.1 Image compositing 118
2.10.2 Superresolution 119
2.11 Image separation 123
2.12 Summary 123
2.13 Bibliographical remarks 125
3 Color analysis 129
3.1 Introduction 129
3.2 Visible-light spectra and color appearance 132
3.3 Overview of human color vision 133
3.3.1 Properties of color descriptions 134
3.3.2 Opponent color processing and unique hues 137
3.3.3 Humanist descriptions of color 138
3.3.4 Spatial aspects of color perception 139
Josef Albers 140
3.3.5 Color and lightness constancy and brightness perception 141
3.3.6 Quantitative descriptions and additive color mixing 141
3.3.7 Representing artists' palettes 145
3.4 Physics of color in art materials 147
3.4.1 Pigments and color appearance 147
3.5 Representing color arising from mixing paints 151
3.5.1 Identifying pigments in artworks based on spectra 152
3.6 Digital rejuvenation of pigment colors 154
3.6.1 Digital rejuvenation of faded artworks 157
Georges Seurat 158
3.7 Digital cleaning of paintings 160
3.8 Summary 164
3.9 Bibliographical remarks 165
4 Brush stroke and mark analysis 171
4.1 Introduction 171
Cy Twombly 173
4.2 Analysis of printed lines and marks 175
Katsushika Hokusai 178
4.3 Inferring tools from marks 182
Sheila Waters 184
4.3.1 Analysis of brush strokes 185
4.3.2 Segmenting and isolating brush strokes computationally 187
4.3.3 Extracting opaque marks in multiple layers 189
Vincent Willem van Gogh 193
4.3.4 Visual evidence of authorship of Pollock's drip paintings 194
Jackson Pollock 195
4.3.5 Extracting layers of translucent brush strokes 195
4.4 Characterizing the shapes of strokes and marks 203
4.5 Global methods for inferring sequences of marks in paintings 206
4.6 Summary 208
4.7 Bibliographical remarks 208
5 Perspective and geometric analysis 211
5.1 Introduction 211
5.2 Projective geometry 214
5.2.1 The mathematics of projection 216
5.2.2 One-point, two-point, and three-point perspectives 222
5.2.3 Parallel or orthographic perspective in Asian art 223
5.3 Estimating the center of projection 224
5.3.1 Foreshortening and size comparisons of depicted objects 230
Piero della Francesca 231
5.3.2 Cross-ratio analysis 232
5.3.3 Estimating the center of projection from object sizes 234
5.4 Estimating geometric accuracy in artworks 235
5.4.1 Hans Memling's Flower Still-Life 235
Hans Memling 237
5.4.2 The carpet in Lorenzo Lotto's Husband and Wife 238
5.4.3 The chandelier in the Arnolfini Portrait 238
Jan van Eyck 243
5.4.4 Warping Andrea Mantegna's Lamentation of Christ to make consistent perspective 251
5.4.5 Dewarping the murals in Sennedjem's Tomb 252
5.4.6 Warping de Chirico's Ariadne to make consistent perspective 255
Giorgio de Chirico 256
5.4.7 Robert Campin and workshop's Mérode Altarpiece 257
Robert Campin 258
5.5 Slant anamorphic art 260
Ed Ruscha (Edward Joseph Ruscha IV) 260
5.5.1 Hans Holbein's The Ambassadors 263
Hans Holbein 263
5.6 Inferring depth from projected images 264
5.6.1 Computing a three-dimensional model from one perspective image 265
Masaccio 266
5.6.2 Computing a three-dimensional model from two perspective images 267
5.7 Summary 271
5.8 Bibliographical remarks 272
6 Optical analysis 275
6.1 Introduction 275
6.2 Reflection and refraction 277
6.3 Plane mirrors 278
6.3.1 Virtual image formation by plane mirrors 279
6.3.2 Depictions of plane mirrors in art 281
6.3.3 Diego Velázquez’s Las Meninas 283
Diego Velázquez 284
6.4 Convex spherical mirrors 288
6.4.1 Virtual image formation by convex spherical mirrors 290
6.4.2 Jan van Eyck’s Portrait of Giovanni Arnolfini and his Wife 292
6.4.3 Claude glass 297
6.4.4 Parmigianino’s Self-Portrait in a Convex Mirror 298
Parmigianino (Girolamo Francesco Maria Mazzola) 298
6.4.5 Hans Memling's Virgin and Child and Maarten van Nieuwenhove 304
6.4.6 Dewarping images in generalized cylindrical mirrors 308
6.5 Conical and cylindrical mirrors and anamorphic art 312
6.5.1 Conical mirror anamorphic art 313
6.5.2 Cylindrical mirror anamorphic art 317
6.6 Concave spherical mirrors 318
6.6.1 Virtual image formation by concave mirrors 320
6.6.2 Real image formation by concave mirrors 322
6.7 Converging lenses 323
6.7.1 Virtual image formation by converging lenses 325
6.7.2 Real image formation by convex lenses 327
6.8 Camera lucida and camera obscura 328
6.8.1 Camera lucida 328
6.8.2 Camera obscura 331
6.8.3 Depth of field, depth of focus, and blur spots 333
6.9 Optical projections and the creation of art 336
6.9.1 Jan van Eyck's Portrait of Giovanni Arnolfini and his wife 337
6.9.2 Caravaggio's Supper at Emmaus 342
6.9.3 Lorenzo Lotto's Husband and Wife 345
6.9.4 Johannes Vermeer's Lady at the Virginals with a Gentleman 349
Johannes Vermeer 349
6.9.5 Canaletto's Piazza San Marco 363
Canaletto (Giovanni Antonio Canal) 364
6.9.6 Photorealists 364
Philip Barlow 366
6.10 Refraction and nonimaging optics in art 366
6.10.1 Leonardo's Salvator Mundi 366
6.11 Summary 371
6.12 Bibliographical remarks 372
7 Lighting analysis 377
7.1 Introduction 377
7.2 Basic shadows 381
7.2.1 General classes of lighting analysis methods 383
7.3 Cast-shadow analysis 383
7.3.1 Illumination from two or more point-sources 388
7.3.2 Cast-shadow analysis under geometric constraints 388
7.4 Lighting information from highlights 389
7.4.1 Illumination direction from highlights on simple estimated shapes 393
7.5 The optics of diffuse reflections 394
7.6 Inferring illumination from plane surfaces 396
Georges de la Tour 398
7.7 Interreflection 400
7.8 Occluding-contour algorithms 401
7.8.1 Single-point occluding-contour algorithm 403
7.8.2 General occluding-contour algorithm 405
Caravaggio (Michelangelo Merisi da Caravaggio) 407
7.8.3 Lightfield occluding-contour algorithm 408
Garth Herrick 409
7.8.4 Theory of the lightfield occluding-contour algorithm 410
7.8.5 Application of the lightfield occluding-contour algorithm 415
7.9 Computer graphics for the analysis of lighting 418
7.9.1 Georges de la Tour's Christ in the Carpenter's Studio (model) 419
7.9.2 Johannes Vermeer's Girl with a Pearl Earring 421
7.9.3 René Magritte's The Menaced Assassin 422
7.9.4 Bidirectional reflectance distribution functions (BRDFs) 424
7.9.5 Caravaggio's The Calling of St. Matthew 425
7.10 Shape-from-shading algorithms 426
7.10.1 Shape-from-shading by deep neural networks 429
7.10.2 Shape-from-shading for estimating both illumination and depth 430
7.11 Integrating lighting estimates 433
7.11.1 Integrating one-dimensional lighting estimates 433
7.11.2 Integrating two-dimensional lighting estimates 436
7.12 Lighting analysis for dating depicted scenes 439
7.13 Summary 442
7.14 Bibliographical remarks 444
8 Object analysis 449
8.1 Introduction 449
8.2 Image-based object classification 452
8.2.1 Feature-based object recognition 452
8.3 Feature-based analysis of faces and bodies 454
8.3.1 Feature-based analysis of body pose 464
8.3.2 Feature-based analysis of head poses 466
8.4 Deep neural network-based object recognition 468
Jacques-Louis David 472
8.4.1 Transfer training 472
8.5 Summary 474
8.6 Bibliographical remarks 475
9 Style and composition analysis 477
9.1 Introduction 477
9.2 Automatic classification of style 480
9.3 Compositional balance 482
9.3.1 Computational balance of actors 485
9.4 Geometric properties of composition 486
9.4.1 Design in Piet Mondrian's Neoplastic paintings 487
Piet Mondrian 487
9.5 Analysis of trends and similarities in artistic style 497
9.5.1 Trends in landscape compositions 498
9.5.2 Large-scale trends in the development of style 502
9.5.3 Graph representations of stylistic similarities 503
9.6 Style transfer 505
9.6.1 Style transfer by deep networks 505
9.6.2 Rejuvenating tapestries 506
9.6.3 Coloration of black-and-white photographs of artworks 507
9.6.4 Style transfer for visualizing underdrawings 509
9.7 Recovering Rembrandt's complete The Night Watch 513
Rembrandt 514
9.8 Computational generation of images for art analysis 516
9.8.1 Computational recovery of lost artworks 518
9.9 Summary 521
9.10 Bibliographical remarks 522
10 Semantic analysis 525
10.1 Introduction 525
Jacques-Louis David 528
10.2 Semantics and visual art 534
10.2.1 Natural language processing and knowledge representation 536
10.3 Meaning through associations 538
10.3.1 Signifiers and signifieds 538
10.4 Semantics of color 544
10.5 Identifying saints by their attributes 546
Andrea del Verrocchio 549
10.6 Learning associations between signifiers and signifieds 550
Harmen Steenwijck 551
10.7 Meaning through artistic style 554
10.7.1 Context in the creation of meaning 556
10.8 Automatic image captioning and question answering 557
10.8.1 Image captioning 557
10.8.2 Automatic answering of questions about artworks 559
10.9 Meaning through shape relations and associations 563
Rogier van der Weyden 563
10.9.1 Recognizing meaning-bearing stories 565
Albrecht Dürer 567
10.10 Summary 568
10.11 Bibliographical remarks 569
Appendix 573
A Symbols, acronyms, and mathematical notation 573
A.1 Mathematical notation, definitions, and operations 573
A.2 Solving simultaneous linear equations 578
A.3 Lagrange optimization 579
A.4 Basis functions 580
A.5 Discrete Fourier analysis and synthesis 580
A.6 Discrete wavelet transform 582
A.7 Spherical harmonics 582
B Probability 584
B.1 Accuracy, precision, and recall 585
B.2 Conditional probability 585
B.3 The definition of information 586
B.4 Hidden Markov models (HMMs) 586
C Bayes' theorem and reasoning about uncertainty 588
C.1 Statistical independence 588
C.2 Maximum likelihood estimation 589
C.3 Bias and variance 591
C.4 Intersection over Union metric 592
D Deep neural networks 593
E Ray tracing and image formation in mirrors and lenses 596
E.1 Converging lenses 596
E.2 Diverging lenses 599
E.3 Mirrors 600
E.4 The focal length and radius of curvature of a spherical mirror 602
E.5 Spherical versus parabolic mirrors 603
F Resources 604
Epilog 607
Glossary 609
Bibliography 615
Figure credits 673
Timeline of artists 682
Index of artists 683
Index 687
About the book 713
