User:SilverbackNet/sandbox

File Allocation Table (FAT) is the name of a computer file system architecture and a family of industry standard file systems utilizing it.

File system types

Original 8-bit FAT

The original FAT file system (or FAT structure, as it was called initially) was designed and coded by Marc McDonald, based on a series of discussions between McDonald and Bill Gates. It was introduced with 8-bit table elements (and valid data cluster numbers 0x02 to 0xBF) in Microsoft's Stand-alone Disk BASIC-80 for a 8080-based successor of the NCR 7200 data-entry terminal with 8-inch (200 mm) floppy disks in 1977. In 1978, Stand-alone Disk BASIC was ported to the 8086 using an emulator on a DEC PDP-10, since no real 8086 systems were available at this time. The FAT file system was also utilized in Microsoft's MDOS/MIDAS, an operating system for 8080/Z80 platforms written by McDonald since 1979. The Stand-alone Disk BASIC version supported three FATs, whereas this was a parameter for MIDAS. Reportedly, MIDAS was also prepared to support 10-bit, 12-bit and 16-bit FAT variants. While the size of directory entries was 16 bytes in Stand-alone Disk BASIC, MIDAS instead occupied 32 bytes per entry.

Tim Paterson of Seattle Computer Products (SCP) was first introduced to Microsoft's FAT structure when he helped Bob O'Rear adapting the Stand-alone Disk BASIC-86 emulator port onto SCP's S-100 bus 8086 CPU board prototype during a guest week at Microsoft in May 1979. The final product was shown at Lifeboat Associates' booth stand at the National Computer Conference in New York on 4–7 June 1979, where Paterson learned about the more sophisticated FAT implementation in MDOS/MIDAS and McDonald talked to him about the design of the file system.

FAT12

Between April and August 1980, while borrowing the FAT concept for SCP's own 8086 operating system QDOS 0.10, Tim Paterson extended the table elements to 12 bits, reduced the number of FATs to two, redefined the semantics of some of the reserved cluster values, and modified the disk layout, so that the root directory was now located between the FAT and the data area for his implementation of FAT12, whereas the format used in Microsoft Stand-alone Disk BASIC's 8-bit file system precursor was not supported by QDOS. By August 1980, QDOS had been renamed into 86-DOS already. Starting with 86-DOS 0.42, the size and layout of directory entries was changed from 16 bytes to 32 bytes in order to add a file date stamp and increase the theoretical file size limit beyond the previous limit of 16 MiB. 86-DOS 1.00 became available in early 1981. Later in 1981, 86-DOS evolved into Microsoft's MS-DOS and IBM PC DOS. 86-DOS retained the capability to read previously formatted volumes with 16-byte directory entries up to at least SCP MS-DOS 1.25.

Originally designed as a file system for floppy disks, FAT12 used 12-bit entries for the cluster addresses in the FAT, which not only limited the maximum generally possible count of data clusters to 4078 (for data clusters 0x002 to 0xFEF) or in some controlled scenarios even up to 4084 (for data clusters 0x002 to 0xFF5), but made FAT manipulation tricky with the PC's 8-bit and 16-bit registers. (While MS-DOS and PC DOS support up to 4084 data clusters on FAT12 volumes in general, cluster value 0xFF0 is treated as additional end-of-chain marker on any FAT12 volume since MS-DOS/PC DOS 3.3, which also introduced the 0xF0 media descriptor value, therefore restricting the maximum practical number of data clusters to 4078 for compatibility purposes with these operating systems.)

The disk's size was stored and calculated as a 16-bit count of sectors, which limited the size to 32 MiB for a logical sector size of 512 bytes. FAT12 was used by several manufacturers with different physical formats, but a typical floppy disk at the time was 5.25-inch (130 mm), single-sided, 40 tracks, with 8 sectors per track, resulting in a capacity of 160 KiB for both the system areas and files. The FAT12 limitations exceeded this capacity by a factor of ten or more. (NB. The 32 MiB limit was later circumvented using logical sectored FATs with logical sector sizes larger than 512 bytes in some OEM versions of MS-DOS 3.x, but this fell into disuse when FAT16B became available with DOS 3.31, which supported 32-bit sector numbers and thereby further lifted the limits.)

By convention, all the control structures were organized to fit inside the first track, thus avoiding head movement during read and write operations, although this varied depending on the manufacturer and physical format of the disk. A limitation which was not addressed until much later (with FAT32) was that any bad sector in the control structures area, track 0, could prevent the disk from being usable. The DOS formatting tool rejected such disks completely. Bad sectors were allowed only in the file data area and were marked with the reserved value 0xFF7 in the FAT. They made the entire containing cluster unusable.

While 86-DOS supported three disk formats (250.25 KiB, 616 KiB and 1232 KiB with FAT IDs 0xFF and 0xFE) on 8-inch (200 mm) floppy drives, IBM PC DOS 1.0, released with the original IBM Personal Computer in 1981, supported only an 8-sector floppy format with a formatted capacity of 160 KiB (FAT ID 0xFE) for single-sided 5.25-inch floppy drives, and PC DOS 1.1 added support for a double-sided format with 320 KiB (FAT ID 0xFF). PC DOS 2.0 introduced support for 9-sector floppy formats with 180 KiB (FAT ID 0xFC) and 360 KiB (FAT ID 0xFD).

86-DOS 1.00 and PC DOS 1.0 directory entries included only one date, the last modified date. PC DOS 1.1 added the last modified time. PC DOS 1.x file attributes included a hidden bit and system bit, with the remaining six bits undefined. At this time, DOS did not support a hierarchical file system, which was still acceptable, given that the number of files on a disk was typically not more than a few dozen.

The PC XT was the first PC with a hard drive from IBM, and PC DOS 2.0 supported that hard drive with FAT12 (FAT ID 0xF8). The fixed assumption of 8 sectors per clusters on hard disks practically limited the maximum partition size to 16 MiB for 512 byte sectors and 4 KiB clusters.

The BIOS Parameter Block (BPB) was introduced with PC DOS 2.0 as well, and this version also added read-only, archive, volume label, and directory attribute bits for hierarchical sub-directories.

MS-DOS 3.0 introduced support for high-density 1.2 MiB 5.25-inch diskettes (media descriptor 0xF9), which notably had 15 sectors per track, hence more space for the FATs.

FAT12 remains in use on all common floppy disks, including 1.44 MiB and later 2.88 MiB disks (media descriptor byte 0xF0).

Initial FAT16

On 14 August 1984, IBM released the PC AT, which featured a 20 MiB hard disk and PC DOS 3.0. Microsoft introduced MS-DOS 3.0 in parallel. Cluster addresses were increased to 16-bit, allowing for up to 65,524 clusters per volume, and consequently much greater file system sizes, at least in theory. However, the maximum possible number of sectors and the maximum (partition, rather than disk) size of 32 MiB did not change. Therefore, although cluster addresses were 16 bits, this format was not what today is commonly understood as FAT16. A partition type 0x04 indicates this form of FAT16 with less than 65536 sectors (less than 32 MiB for sector size 512).

With the initial implementation of FAT16 not actually providing for larger partition sizes than FAT12, the early benefit of FAT16 was to enable the use of smaller clusters, making disk usage more efficient, particularly for large numbers of files only a few hundred bytes in size, which were far more common at the time.

MS-DOS 2.x hard disks larger than 15 MiB are incompatible with later versions of MS-DOS. A 20 MiB hard disk formatted under MS-DOS 3.0 was not accessible by the older MS-DOS 2.0 because MS-DOS 2.0 did not support version 3.0's FAT16. MS-DOS 3.0 could still access MS-DOS 2.0 style 8 KiB-cluster partitions under 15 MiB.

Logical sectored FAT

When hard disks grew larger and the FAT12 and FAT16 file system implementation in MS-DOS / PC DOS did not provide means to take advantage of the extra storage, several manufacturers developed their own FAT variants to address the problem in their MS-DOS OEM issues.

Some vendors (AST and NEC) supported eight, instead of the standard four, primary partition entries in their custom extended Master Boot Record (MBR), and they adapted MS-DOS to use more than a single primary partition.

Other vendors worked around the volume size limits imposed by the 16-bit sector entries and arithmetics by increasing the size of the sectors the file system dealt with, thereby blowing up dimensions.

These so-called logical sectors were larger (up to 8192 bytes) than the physical sector size (still typically 512 bytes) as expected by the ROM-BIOS INT 13h or the disk drive hardware. The DOS-BIOS or System BIOS would then combine multiple physical sectors into logical sectors for the file system to work with. These changes were transparent to the file system implementation in the DOS kernel, since on the file system's abstraction level volumes are seen as a linear series of logically addressable sectors, also known as absolute sectors (addressed by their Logical Sector Number (LSN), starting with LSN 0) independent of the physical location of the volume on the physical medium and its geometry. The underlying DOS-BIOS translated these logical sectors into physical sectors according to partitioning information and the drive's physical geometry.

The drawback of this approach was a less memory-efficient sector buffering and deblocking in the DOS-BIOS, thereby causing an increased memory footprint for the DOS data structures. Since older DOS versions were not flexible enough to work with these logical geometries, the OEMs had to introduce new partition IDs for their FAT variants in order to hide them from off-the-shelf issues of MS-DOS and PC DOS. Known partition IDs for logical sectored FATs include: 0x08 (Commodore MS-DOS 3.x), 0x11 (Leading Edge MS-DOS 3.x), 0x14 (AST MS-DOS 3.x), 0x24 (NEC MS-DOS 3.30), 0x56 (AT&T MS-DOS 3.x), 0xE5 (Tandy MS-DOS), 0xF2 (Sperry IT MS-DOS 3.x, Unisys MS-DOS 3.3 — also used by Digital Research DOS Plus 2.1). OEM versions like Toshiba MS-DOS, Wyse MS-DOS 3.2 and 3.3, as well as Zenith MS-DOS are also known to have utilized logical sectoring.

While non-standard and sub-optimal, these FAT variants are perfectly valid according to the specifications of the file system itself. Therefore, even if default issues of MS-DOS and PC DOS were not able to cope with them, most of these vendor-specific FAT12 and FAT16 variants can be mounted by more flexible file system implementations in operating systems such as DR-DOS, simply by changing the partition ID to one of the recognized types. Also, if they no longer need to be recognized by their original operating systems, existing partitions can be "converted" into FAT12 and FAT16 volumes more compliant with versions of MS-DOS/PC DOS 4.0-6.3, which do not support sector sizes different from 512 bytes, by switching to a BPB with 32-bit entry for the number of sectors, as introduced since DOS 3.31 (see FAT16B below), keeping the cluster size and reducing the logical sector size in the BPB down to 512 bytes, while at the same time increasing the counts of logical sectors per cluster, reserved logical sectors, total logical sectors, and logical sectors per FAT by the same factor.

A parallel development in MS-DOS / PC DOS which allowed an increase in the maximum possible FAT size was the introduction of multiple FAT partitions on a hard disk. To allow the use of more FAT partitions in a compatible way, a new partition type was introduced in PC DOS 3.2 (1986), the extended partition (EBR), which is a container for an additional partition called logical drive. Since PC DOS 3.3 (April 1987), there is another, optional extended partition containing the next logical drive, and so on. The MBR of a hard disk can either define up to four primary partitions, or an extended partition in addition to up to three primary partitions.

See also: Extended boot record

Final FAT16

Finally in November 1987, Compaq MS-DOS 3.31 (a modified OEM version of MS-DOS 3.3 released by Compaq with their machines) introduced what today is simply known as the FAT16 format, with the expansion of the 16-bit disk sector count to 32 bits in the BPB. Although the on-disk changes were minor, the entire DOS disk driver had to be converted to use 32-bit sector numbers, a task complicated by the fact that it was written in 16-bit assembly language. The result was initially called the DOS 3.31 Large File System. Microsoft's DSKPROBE tool refers to type 0x06 as BigFAT, whereas some older versions of FDISK described it as BIGDOS. Technically, it is known as FAT16B.

Since older versions of DOS were not designed to cope with more than 65535 sectors, it was necessary to introduce a new partition type for this format in order to hide it from pre-3.31 issues of DOS. The original form of FAT16 (with less than 65536 sectors) had a partition type 0x04. To deal with disks larger than this, type 0x06 was introduced to indicate 65536 or more sectors. In addition to this, the disk driver was expanded to cope with more than 65535 sectors as well. The only other difference between the original FAT16 and the newer FAT16B format is the usage of a newer BPB format with 32-bit sector entry. Therefore, newer operating systems supporting the FAT16B format can cope also with the original FAT16 format without any necessary changes.

If partitions to be used by pre-DOS 3.31 issues of DOS need to be created by modern tools, the only criteria theoretically necessary to meet are a sector count of less than 65536, and the usage of the old partition ID (0x04). In practice however, type 0x01 and 0x04 primary partitions should not be physically located outside the first 32 MiB of the disk, due to other restrictions in MS-DOS 2.x, which could not cope with them otherwise.

In 1988, the FAT16B improvement became more generally available through DR DOS 3.31, PC DOS 4.0, OS/2 1.1, and MS-DOS 4.0. The limit on partition size was dictated by the 8-bit signed count of sectors per cluster, which originally had a maximum power-of-two value of 64. With the standard hard disk sector size of 512 bytes, this gives a maximum of 32 KiB cluster size, thereby fixing the "definitive" limit for the FAT16 partition size at 2 GiB for sector size 512. On magneto-optical media, which can have 1 or 2 KiB sectors instead of 0.5 KiB, this size limit is proportionally larger.

Much later, Windows NT increased the maximum cluster size to 64 KiB, by considering the sectors-per-cluster count as unsigned. However, the resulting format was not compatible with any other FAT implementation of the time, and it generated greater internal fragmentation. Windows 98, SE and ME also supported reading and writing this variant, but its disk utilities did not work with it and some FCB services are not available for such volumes. This contributes to a confusing compatibility situation.

Prior to 1995, versions of DOS accessed the disk via CHS addressing only. When MS-DOS 7.0 / Windows 95 introduced LBA disk access, partitions could start being physically located outside the first ca. 8 GiB of this disk and thereby out of the reach of the traditional CHS addressing scheme. Partitions partially or fully located beyond the CHS barrier therefore had to be hidden from non-LBA-enabled operating systems by using the new partition type 0x0E in the partition table instead. FAT16 partitions using this partition type are also named FAT16X. The only difference, compared to previous FAT16 partitions, is the fact that some CHS-related geometry entries in the BPB record, namely the number of sectors per track and the number of heads, may contain no or misleading values and should not be used.

The number of root directory entries available for FAT12 and FAT16 is determined when the volume is formatted, and is stored in a 16-bit field. For a given number RDE and sector size SS, the number RDS of root directory sectors is RDS=ceil((RDE×32)/SS), and RDE is normally chosen to fill these sectors, i.e., RDE*32=RDS*SS. FAT12 and FAT16 media typically use 512 root directory entries on non-floppy media. Some third-party tools, like mkdosfs, allow the user to set this parameter.

FAT32

In order to overcome the volume size limit of FAT16, while at the same time allowing DOS real mode code to handle the format, Microsoft designed a new version of the file system, FAT32, which supported an increased number of possible clusters, but could reuse most of the existing code, so that the available conventional memory footprint was reduced by less than 5 KiB under DOS. Cluster values are represented by 32-bit numbers, of which 28 bits are used to hold the cluster number. The boot sector uses a 32-bit field for the sector count, limiting the FAT32 volume size to 2 TiB for a sector size of 512 bytes and 16 TiB for a sector size of 4,096 bytes. FAT32 was introduced with MS-DOS 7.1 / Windows 95 OSR2 in 1996, although reformatting was needed to use it, and DriveSpace 3 (the version that came with Windows 95 OSR2 and Windows 98) never supported it. Windows 98 introduced a utility to convert existing hard disks from FAT16 to FAT32 without loss of data. In the Windows NT line, native support for FAT32 arrived in Windows 2000. A free FAT32 driver for Windows NT 4.0 was available from Winternals, a company later acquired by Microsoft. Since the acquisition the driver is no longer officially available. Since 1998, Caldera's dynamically loadable DRFAT32 driver could be used to enable FAT32 support in DR-DOS. The first version of DR-DOS to natively support FAT32 and LBA access was OEM DR-DOS 7.04 in 1999. That same year IMS introduced native FAT32 support with REAL/32 7.90, and IBM 4690 OS added FAT32 support with version 2. Ahead Software provided another dynamically loadable FAT32.EXE driver for DR-DOS 7.03 with Nero Burning ROM in 2004. IBM PC DOS introduced native FAT32 support with OEM PC DOS 7.10 in 2003.

The maximum possible size for a file on a FAT32 volume is 4 GiB minus 1 byte or 4,294,967,295 (2³² − 1) bytes. This limit is a consequence of the file length entry in the directory table and would also affect huge FAT16 partitions with a sufficient sector size. Video applications, DVD images, large databases, and some other software easily exceed this limit.

The open FAT+ specification proposes how to store larger files up to 256 GiB minus 1 byte or 274,877,906,943 (2³⁸ − 1) bytes on slightly modified and otherwise backwards compatible FAT32 volumes, but imposes a risk that disk tools or FAT32 implementations not aware of this extension may truncate or delete files exceeding the normal FAT32 file size limit. Also, support for FAT32+ (and FAT16+) is limited to some versions of DR-DOS and not available in mainstream operating systems so far. (This extension is critically incompatible with the /EAS option of the FAT32.IFS method to store OS/2 extended attributes on FAT32 volumes.)

As with previous file systems, the design of the FAT32 file system does not include direct built-in support for long filenames, but FAT32 volumes can optionally hold VFAT long filenames in addition to short filenames in exactly the same way as VFAT long filenames have been optionally implemented for FAT12 and FAT16 volumes.

Two partition types have been reserved for FAT32 partitions, 0x0B and 0x0C. The latter type is also named FAT32X in order to indicate usage of LBA disk access instead of CHS. On such partitions, some CHS-related geometry entries in the EBPB record, namely the number of sectors per track and the number of heads, may contain no or misleading values and should not be used.

Extensions

Extended Attributes

OS/2 heavily depends on extended attributes (EAs) and stores them in a hidden file called "EA␠DATA.␠SF" in the root directory of the FAT12 or FAT16 volume. This file is indexed by two previously reserved bytes in the file's (or directory's) directory entry at offset 0x14. In the FAT32 format, these bytes hold the upper 16 bits of the starting cluster number of the file or directory, hence making it impossible to store OS/2 EAs on FAT32 using this method.

However, the third-party FAT32 installable file system (IFS) driver FAT32.IFS version 0.70 and higher by Henk Kelder & Netlabs for OS/2 and eComStation stores extended attributes in extra files with filenames having the string "␠EA.␠SF" appended to the regular filename of the file to which they belong. The driver also utilizes the byte at offset 0x0C in directory entries to store a special mark byte indicating the presence of extended attributes to help speed up things. (This extension is critically incompatible with the FAT32+ method to store files larger than 4 GiB minus 1 on FAT32 volumes.)

Extended attributes are accessible via the Workplace Shell desktop, through REXX scripts, and many system GUI and command-line utilities (such as 4OS2).

To accommodate its OS/2 subsystem, Windows NT supports the handling of extended attributes in HPFS, NTFS, FAT12 and FAT16. It stores EAs on FAT12, FAT16 and HPFS using exactly the same scheme as OS/2, but does not support any other kind of ADS as held on NTFS volumes. Trying to copy a file with any ADS other than EAs from an NTFS volume to a FAT or HPFS volume gives a warning message with the names of the ADSs that will be lost. It does not support the FAT32.IFS method to store EAs on FAT32 volumes.

Windows 2000 onward acts exactly as Windows NT, except that it ignores EAs when copying to FAT32 without any warning (but shows the warning for other ADSs, like "Macintosh Finder Info" and "Macintosh Resource Fork").

Cygwin uses "EA␠DATA.␠SF" files as well.

Long file names

One of the user experience goals for the designers of Windows 95 was the ability to use long filenames (LFNs—up to 255 UCS-2 code units long), in addition to classic 8.3 filenames (SFNs). For backward and forward compatibility LFNs were implemented as an optional extension on top of the existing FAT file system structures using a workaround in the way directory entries are laid out.

This transparent method to store long file names in the existing FAT file systems without altering their data structures is usually known as VFAT (for "Virtual FAT") after the Windows 95 virtual device driver.

In Windows NT, support for VFAT long filenames started from version 3.5.

Non VFAT-enabled operating systems can still access the files under their short file name alias without restrictions, however, the associated long file names may get lost, when files with long file names are copied under non VFAT-aware operating systems.

OS/2 added long filename support to FAT using extended attributes (EA) before the introduction of VFAT; thus, VFAT long filenames are invisible to OS/2, and EA long filenames are invisible to Windows, therefore experienced users of both operating systems would have to manually rename the files.

In order to support Java applications, the FlexOS-based IBM 4690 OS version 2 introduced its own virtual file system (VFS) architecture to store long filenames in the FAT file system in a backwards compatible fashion. If enabled, the virtual filenames (VFN) are available under separate logical drive letters, whereas the real filenames (RFN) remain available under the original drive letters.

Forks and Alternate Data Streams

The FAT file system itself is not designed for supporting Alternate Data Streams (ADS), but some operating systems that heavily depend on them have devised various methods for handling them in FAT drives. Such methods either store the additional information in extra files and directories (Mac OS), or give new semantics to previously unused fields of the FAT on-disk data structures (OS/2 and Windows NT).

Mac OS using PC Exchange stores its various dates, file attributes and long filenames in a hidden file called "FINDER.DAT", and resource forks (a common Mac OS ADS) in a subdirectory called "RESOURCE.FRK", in every directory where they are used. From PC Exchange 2.1 onwards, they store the Mac OS long filenames as standard FAT long filenames and convert FAT filenames longer than 31 characters to unique 31-character filenames, which can then be made visible to Macintosh applications.

Mac OS X stores resource forks and metadata (file attributes, other ADS) in a hidden file with a name constructed from the owner filename prefixed with "._", and Finder stores some folder and file metadata in a hidden file called ".DS_Store".

UMSDOS permissions and filenames

Further information: FAT filesystem and Linux

Early Linux distributions also supported a format known as UMSDOS, a FAT variant with Unix file attributes (such as long file name and access permissions) stored in a separate file called "--linux-.---". UMSDOS fell into disuse after VFAT was released and it is not enabled by default in Linux kernels from version 2.5.7 onwards.

Derivatives

FATX

FATX is a family of file systems designed for Microsoft's Xbox video game console hard disk drives and memory cards, introduced in 2001.

While resembling the same basic design ideas as FAT16 and FAT32, the FATX16 and FATX32 on-disk structures are simplified but fundamentally incompatible with normal FAT16 and FAT32 file systems, making it impossible for normal FAT file system drivers to mount such volumes.

The non-bootable superblock sector is 4 KiB in size and holds an 18 byte large BPB-like structure completely different from normal BPBs. Clusters are typically 16 KiB in size and there is only one copy of the FAT on the Xbox. Directory entries are 64 bytes in size instead of the normal 32 bytes. Files can have filenames up to 42 characters long using the OEM character set and be up to 4 GiB minus 1 byte in size. The on-disk timestamps hold creation, modification and access dates and times but differ from FAT: in FAT, the epoch is 1980; in FATX, the epoch is 2000. On the Xbox 360, the epoch is 1980.

exFAT

Main article: exFAT

exFAT is an incompatible file system, introduced with Windows Embedded CE 6.0 in November 2006 and brought to mainline Windows in Vista Service Pack 1. It is loosely based on the File Allocation Table architecture, but proprietary and protected by patents.

The MBR partition type is 0x07 (the same as used for IFS, HPFS, NTFS, etc.). Logical geometry information located in the VBR is stored in a format not resembling any kind of BPB. exFAT is intended for use on flash drives (such as SDXC and Memory Stick XC), where FAT32 is otherwise used.

It offers several benefits over FAT32 including breaking the 4 GiB file size limit of standard FAT32 (contrast FAT+ above), being more space-efficient for files smaller than 64 KiB on large volumes and, compared to light-weight implementations of FAT32 in DOS and some embedded systems, it can offer faster seeks if more than a few thousand files are stored in a single sub-directory, whereas FAT32 is typically faster than exFAT for larger files as used on digital cameras, camcorders and media players or when flash cards are used mainly for archival purposes.

Storage devices formatted as exFAT cannot exchange data with equipment not supporting the format. Most consumer electronics do not support exFAT, which requires acquisition of a commercial license from Microsoft, which rules out its legal distribution as part of open source operating systems.

Beginning in Windows Vista SP1, Microsoft's GUI and command-line format utilities now offer exFAT as an alternative to NTFS (and for smaller partitions, FAT16 and FAT32).

See Also

File Allocation Table