File System Forensics delivers comprehensive knowledge of how file systems function and, more importantly, how digital forensic tools might function in relation to specific file systems. It provides a step-by-step approach for file content and metadata recovery to allow the reader to manually recreate and validate results from file system forensic tools.
The book includes a supporting website that shares all of the data (i.e. sample file systems) used for demonstration in the text and provides teaching resources such as instructor guides, extra material, and more.
Written by a highly qualified associate professor and consultant in the field, File System Forensics includes information on: - The necessary concepts required to understand file system forensics for anyone with basic computing experience- File systems specific to Windows, Linux, and macOS, with coverage of FAT, ExFAT, and NTFS- Advanced topics such as deleted file recovery, fragmented file recovery, searching for particular files, links, checkpoints, snapshots, and RAID- Issues facing file system forensics today and various issues that might evolve in the field in the coming years
File System Forensics is an essential, up-to-date reference on the subject for graduate and senior undergraduate students in digital forensics, as well as digital forensic analysts and other law enforcement professionals.
Table of Contents
Preface xvii
Acknowledgements xxi
Part I Preliminaries 1
1 Introduction 3
1.1 What is Digital Forensics? 4
1.2 File System Forensics 5
1.3 Digital Forensic Principles 5
1.4 Digital Forensic Methodology 7
1.4.1 Preparation 8
1.4.2 Localisation/Preservation 8
1.4.3 Acquisition 8
1.4.4 Processing 9
1.4.5 Analysis 9
1.4.6 Reporting 9
1.4.7 Quality Assurance 10
1.4.8 Evidence Return 10
1.5 About This Book 10
1.5.1 Who Should Read This Book? 11
1.6 Book Structure 12
1.7 Summary 13
Exercises 13
Bibliography 14
2 Linux as a Forensic Platform 17
2.1 Open-Source Software 17
2.1.1 Advantages of Open-Source Software 19
2.1.2 Open Source ≠ Free 20
2.2 Open-Source Software in Digital Forensics 20
2.3 What is Linux? 21
2.3.1 The Anatomy of the Linux OS 22
2.3.2 Linux Distributions 27
2.3.3 A (very) Brief History of Linux 28
2.4 Using Linux 29
2.4.1 User Accounts 30
2.4.2 Basic Linux Commands 32
2.4.2.1 Navigating the File System 32
2.4.2.2 Getting Help 34
2.4.2.3 Viewing/Editing Text Files 34
2.4.2.4 Managing Directories 35
2.4.2.5 Redirection and Pipes 35
2.5 Linux as a Forensic Platform 36
2.5.1 Commands for Digital Forensics 36
2.5.1.1 Hashing 36
2.5.1.2 Hex Viewers 38
2.5.1.3 Archiving/Compression 39
2.5.1.4 The file Command 40
2.5.1.5 The strings Command 40
2.5.1.6 Text Searching with (e)grep 41
2.6 Summary 42
Exercises/Discussion Topics 42
Bibliography 43
3 Mathematical Preliminaries 45
3.1 Bits and Bytes 45
3.2 Number Systems 48
3.2.1 Notational Conventions 48
3.2.2 Decimal 48
3.2.3 Binary 49
3.2.4 Hexadecimal 50
3.2.5 Number Conversions 51
3.2.6 Number Conversion with Bash 51
3.2.7 Negative Numbers 53
3.2.8 Floating-Point Numbers 53
3.3 Representing Text 56
3.3.1 Ascii 56
3.3.2 Iso- 8859 57
3.3.3 Unicode 59
3.3.4 Utf- 8 60
3.3.5 Utf- 16 61
3.4 Representing Time 62
3.4.1 Unix Time 63
3.4.2 The Linux date Command 64
3.5 Endianness and Raw Data 64
3.6 Summary 66
Exercises 67
Bibliography 68
4 Disks, Partitions and File Systems 69
4.1 Disk Storage 70
4.1.1 Traditional Rotational Hard Drives 71
4.1.1.1 Optical Media 72
4.1.2 Flash Drives 73
4.1.3 Solid-State Drives 73
4.2 Partitions 74
4.2.1 Creating Partitions/File Systems on Linux 74
4.2.1.1 Mounting File Systems on Linux 77
4.2.2 Master Boot Record 78
4.2.3 GUID Partition Table 80
4.3 File Systems 83
4.3.1 File System Concepts 83
4.3.2 Comparison of File Systems 86
4.4 Acquisition of File System Data 88
4.4.1 Logical vs Physical Acquisition 88
4.4.2 Acquisition Under Linux 88
4.4.2.1 The dd Family 89
4.4.2.2 Expert Witness Format (EWF) 90
4.4.2.3 guymager 91
4.5 Analysis of File Systems 92
4.5.1 The Sleuth Kit 92
4.5.1.1 Determine the Partition Layout 93
4.5.1.2 Determine the File System Type 93
4.5.1.3 List the Files 94
4.5.1.4 Recover File Metadata 95
4.5.1.5 Recover File Content 95
4.5.1.6 Other TSK Commands 95
4.5.2 Data Carving 96
4.6 Summary 97
Exercises 97
Bibliography 98
Part II Windows File Systems 99
5 The FAT File System 101
5.1 On-Disk Structures 101
5.1.1 Layout 102
5.1.2 Volume Boot Record 102
5.1.3 File System Information (FSINFO) 102
5.1.4 File Allocation Table 104
5.1.5 Directory Entries 105
5.1.6 FAT Date and Time 108
5.1.7 Mapping Clusters to Sectors 109
5.2 Analysis of FAT 32 109
5.2.1 Creating FAT32 File Systems 109
5.2.2 Supplied FAT32 Image Files 110
5.2.3 FAT32 Manual Analysis 110
5.2.3.1 Process the VBR 111
5.2.3.2 Process the Root Directory 112
5.2.3.3 Process Sub-directories 113
5.2.3.4 Recover Metadata/Content 113
5.3 FAT32 Advanced Analysis 115
5.3.1 Deleted Files 116
5.3.2 The Volume Label 117
5.4 Summary 117
Exercises 118
Bibliography 118
6 The ExFAT File System 121
6.1 On-Disk Structures 121
6.1.1 Volume Boot Record 122
6.1.2 File Allocation Table 123
6.1.3 Directory Entries 125
6.1.3.1 Allocation Bitmap (Type: 0x81) 127
6.1.3.2 Up-Case Table (Type: 0x82) 128
6.1.3.3 Volume Label (Type: 0x83) 128
6.1.3.4 File (Type: 0x85) 129
6.1.3.5 Volume GUID (Type: 0xA0) 130
6.1.3.6 Stream Extension (Type: 0xC0) 130
6.1.3.7 Filename Extension 131
6.1.3.8 Other Directory Entries 132
6.2 Analysis of ExFAT 132
6.2.1 Creating ExFAT File Systems 132
6.2.2 Supplied ExFAT Image Files 132
6.2.3 ExFAT Manual Analysis 132
6.2.3.1 Step 1: Process the VBR 133
6.2.3.2 Step 2: Process the Root Directory 133
6.2.3.3 Step 3: Process Subdirectories 136
6.2.3.4 Step 4: Recover Metadata 137
6.2.3.5 Step 5: Recover Content 137
6.3 ExFAT Advanced Analysis 139
6.3.1 Long File Names 139
6.3.2 Deleted Files 140
6.3.3 Fragmented Files and Large Directories 141
6.4 Summary 142
Exercises 143
Bibliography 143
7 The NTFS File System 145
7.1 On-Disk Structures 146
7.1.1 $Boot 146
7.1.2 Indexes 147
7.1.3 Fixup Arrays 149
7.1.4 Time in NTFS 150
7.1.5 Master File Table 151
7.1.6 MFT Record Structure 152
7.1.6.1 MFT Record Header 152
7.1.6.2 Browsing Attributes 155
7.1.6.3 $STANDARD_INFORMATION (0x10) 155
7.1.6.4 $ATTRIBUTE_LIST (0x20) 156
7.1.6.5 $FILENAME (0x30) 156
7.1.6.6 $OBJECT_ID (0x40) 157
7.1.6.7 $SECURITY_DESCRIPTOR (0x50) 159
7.1.6.8 $VOLUME_NAME (0x60) 162
7.1.6.9 $VOLUME_INFORMATION (0x70) 162
7.1.6.10 $DATA (0x80) 163
7.1.6.11 $INDEX_ROOT (0x90) 163
7.1.6.12 $INDEX_ALLOCATION (0xA0) 165
7.1.6.13 $BITMAP (0xB0) 165
7.1.6.14 $REPARSE_POINT (0xC0) 166
7.1.6.15 $EA_INFORMATION (0xD0) and $EA (0xE0) 167
7.2 Analysis of NTFS 167
7.2.1 Creating NTFS File Systems 168
7.2.2 Supplied NTFS Image Files 168
7.2.3 NTFS Manual Analysis 168
7.2.3.1 Process $Boot 169
7.2.3.2 Recover $MFT 171
7.2.3.3 Process Directories 173
7.2.3.4 Recover File Metadata 177
7.2.3.5 Recover File Content 182
7.3 NTFS Advanced Analysis 185
7.3.1 Further File System Information 185
7.3.2 Deleted Files 186
7.3.3 Fragmented Files 187
7.3.4 Alternate Data Streams 190
7.3.5 Large MFT Records 191
7.4 Summary 194
Exercises 194
Bibliography 195
Part III Linux File Systems 197
8 The EXT2 File System 199
8.1 On-Disk Structures 200
8.1.1 The Superblock 201
8.1.2 The Block Group Descriptor Table 204
8.1.3 The Inode Table 205
8.1.3.1 Mode/Permissions 207
8.1.3.2 Inode Flags 208
8.1.3.3 Block Pointers 208
8.1.4 The Data and Inode Bitmaps 209
8.1.5 Locating an Inode 209
8.2 Analysis of ext 2 210
8.2.1 Creating ext2 File Systems 210
8.2.2 Supplied ext2 Image Files 210
8.2.3 Ext2 Manual Analysis 211
8.2.3.1 Process the Superblock 211
8.2.3.2 Map the Block Groups 213
8.2.3.3 Process Root Directory Inode 216
8.2.3.4 Process the Root Directory 217
8.2.3.5 Process Directories 219
8.2.3.6 Process Files 219
8.3 Ext2 Advanced Analysis 222
8.3.1 Fragmented Files 222
8.3.2 Links 223
8.3.3 Deleted Files 225
8.4 Summary 226
Exercises 226
Bibliography 227
9 The EXT3/EXT4 File Systems 229
9.1 Supplied Image Files 229
9.2 The ext3 File System 229
9.2.1 The Ext Journal 230
9.2.2 HTree Directory Indexing 237
9.3 The Ext4 File System 241
9.3.1 Large Inodes 241
9.3.1.1 Timestamps 241
9.3.2 Ext4 Data Storage 244
9.3.2.1 Extent-Based Storage 244
9.3.2.2 Inline Storage 248
9.3.2.3 Symbolic Links 248
9.3.3 File Deletion in Ext 4 249
9.3.4 Extended Attributes 252
9.3.5 Ext4 Block Group Descriptors 255
9.3.6 Flexible Block Groups 255
9.4 Summary 258
Exercises 259
Bibliography 260
10 The XFS File System 263
10.1 On-Disk Structures 264
10.1.1 Allocation Groups 264
10.1.2 Addressing 266
10.1.2.1 Inode Addressing 266
10.1.3 XFS B+ Trees 267
10.1.4 The Superblock 268
10.1.4.1 Locating Superblocks 268
10.1.5 XFS Signatures 271
10.1.6 XFS Inodes 271
10.1.7 Directories 273
10.1.8 Extents 274
10.1.9 Time in XFS 275
10.2 Analysis of XFS 275
10.2.1 Creating XFS File Systems 275
10.2.2 Supplied XFS Image Files 275
10.2.3 XFS Manual Analysis 276
10.2.3.1 Process the Superblock 276
10.2.3.2 Locate the Root Directory 277
10.2.3.3 Process the Root Directory 279
10.2.3.4 Process the Subdirectories 281
10.2.3.5 Recover File Content/Metadata 281
10.3 XFS Advanced Analysis 282
10.3.1 AG Free Space Management 283
10.3.1.1 AG Free List 285
10.3.2 AG Inode Management 286
10.3.3 Deleted Files 289
10.3.4 Extended Attributes 290
10.3.5 Links 291
10.3.6 The XFS Journal 292
10.4 Summary 300
Exercises 301
Bibliography 301
11 The Btrfs File System 303
11.1 On-Disk Structures 304
11.1.1 The Superblock 305
11.1.2 Btrfs Trees 305
11.1.3 Btrfs Tree Structure 307
11.1.3.1 Node Header Structure 307
11.1.3.2 Internal Node Structure 309
11.1.4 Btrfs Keys 309
11.1.5 Btrfs Items 310
11.1.6 Time in Btrfs 315
11.1.7 Logical and Physical Addressing 315
11.2 Analysis of Btrfs 317
11.2.1 Creating Btrfs File Systems 317
11.2.2 Supplied Btrfs Image Files 318
11.2.3 Btrfs Analysis Methodology 318
11.2.4 Manual Analysis of a Single Device File System 320
11.2.4.1 Process the Superblock 320
11.2.4.2 Process the CHUNK_ARRAY 321
11.2.4.3 Locate the CHUNK_TREE 322
11.2.4.4 Process the CHUNK_TREE 323
11.2.4.5 Locate the Root Tree 326
11.2.4.6 Locate the FS_TREE 327
11.2.4.7 Processing the FS_TREE 328
11.2.4.8 Process Directories 329
11.2.4.9 Recovering Metadata 335
11.2.4.10 Recovering File Contents 336
11.3 Btrfs Advanced Analysis 338
11.3.1 File Deletion 338
11.3.2 Analysis of Internal Nodes 342
11.3.3 Multiple Device Configuration 343
11.3.4 Subvolumes and Snapshots 346
11.4 Summary 350
Exercises 350
Bibliography 351
Part IV Apple File Systems 353
12 The HFS+ File System 355
12.1 On-Disk Structures 355
12.1.1 Forks 357
12.1.2 Time in HFS+ 357
12.1.3 Volume Header 358
12.1.4 B-Trees 358
12.1.5 Catalog File 362
12.1.6 HFS+ Permissions 363
12.1.7 Text Encoding 365
12.1.8 Extents Overflow File 365
12.1.9 Allocation File 366
12.1.10 HFS+ Journal 367
12.2 Analysis of HFS+ 369
12.2.1 Creating HFS+ File Systems 369
12.2.2 Supplied HFS+ Image Files 370
12.2.3 HFS+ Manual Analysis 370
12.2.3.1 Process the Volume Header 370
12.2.3.2 Locate the Catalog File 371
12.2.3.3 Process the Catalog B-Tree 373
12.2.3.4 Gather Metadata 377
12.2.3.5 Recover File Content 377
12.3 HFS+ Advanced Analysis 380
12.3.1 Deleted Files 380
12.3.2 Index Nodes 381
12.3.3 Fragmented Files 383
12.3.4 Links 387
12.4 Summary 390
Exercises 391
Bibliography 391
13 The APFS File System 393
13.1 On-Disk Structures 394
13.1.1 Time in APFS 394
13.1.2 Objects 394
13.1.3 B-Trees 396
13.1.4 Containers and Volumes 399
13.1.5 Container Superblock 400
13.1.6 Volume Superblock 402
13.1.7 Object Maps 404
13.1.8 File-Related Structures 405
13.1.8.1 File System Keys 406
13.1.8.2 Inode 407
13.1.8.3 Directory Record 408
13.1.8.4 Extent 410
13.1.9 Checkpoints 410
13.1.10 Other APFS Structures 412
13.2 Analysis of APFS 412
13.2.1 Creating APFS File Systems 412
13.2.2 Supplied APFS Image Files 413
13.2.3 APFS Manual Analysis 413
13.2.3.1 Process the Container Superblock 414
13.2.3.2 Process the Container Object Map 415
13.2.3.3 Process the Volume Superblock 418
13.2.3.4 Process the Volume Object Map 418
13.2.3.5 Process the File System Tree 419
13.3 APFS Advanced Analysis 425
13.3.1 Deleted Files 425
13.3.2 Checkpoint Recovery 426
13.3.3 Multi-Level B-Trees 427
13.3.4 Multiple Volumes 429
13.3.5 Extended Attributes 430
13.3.6 Links 431
13.4 Summary 433
Exercises 433
Bibliography 434
Part V The Future 435
14 Future Challenges in Digital Forensics 437
14.1 Challenges in Digital Forensics 437
14.1.1 Data Volume 438
14.1.2 Multi-Source Correlation 439
14.1.3 New File Systems 440
14.1.4 Encryption 440
14.1.5 Cloud Storage 441
14.1.6 Lack of Resources 441
14.1.6.1 Human Resources 441
14.1.6.2 Software Resources 442
14.1.6.3 Hardware Resources 442
14.1.7 Tool Validation/Datasets 443
14.1.8 Lack of Standardisation 444
14.1.9 Legal/Scientific Challenges 444
14.1.10 Presentation of Evidence 445
14.1.11 Human Error/Bias 446
14.2 Where Do We Go from Here? 447
14.2.1 Training/Education 448
14.2.2 Free Open-Source Software (FOSS) 448
14.2.3 Triage 449
14.2.4 Artificial Intelligence (AI) 449
14.2.5 Live Data Forensics 450
14.2.6 Legal Solutions 451
14.2.7 Data Set Development/Tool Testing 452
14.2.8 Standardisation 452
14.2.9 Information Sharing 453
14.2.10 Virtualisation 453
14.3 Summary 454
Bibliography 454
Index 457
