NIBFS The Non-Indexed Blob File System A High-Efficiency, Capability-Based, Storage System for Archival Websites

资源类型: 资源大小: 文档分类: 上传者:孙雯


【标题】NIBFS The Non-Indexed Blob File System A High-Efficiency, Capability-Based, Storage System for Archival Websites

【作者】 Jeremy T. Hilliker 

【摘要】Image archival is a popular and high profile Web 2.0 service. We have examined the problem domain of internet archival websites; particularly image hosting sites; to discover the usage and characteristics unique to their problem domain. We have used this knowledge to determine how these services’ needs differ from those offered by traditional posix filesystems; and then have constructed a tailored filesystem to better meet their needs. Particularly; our system reduces disk seeks caused by unnecessary meta-data and the domain’s long tailed access distribution. Our filesystem offers a 40 to 55% improvement in throughput over standard filesystems; which is significant since these services are I/O bound. Increasing I/O efficiency for these services allows them to serve more content with fewer resources; and to scale better. Introduction Web 2.0 websites are built to allow users to collaborate with each other; share information; and to create content [5]. One application of Web 2.0 is online photo albums and image hosting. These websites allow users to post images to their accounts to be shared with their peers or their audience. This feature has bene adopted by many social networking websites which allow users to post images to be viewed by members of their social circle. Some website place more or less of an emphasis on social networking or image hosting. Imageshack is strictly an image hosting site with no social component. Flickr is primarily an image archival site with a small social component (tagging; comments; friends; and groups). 4chan is an image message board with equal emphasis on image sharing and social interaction. Facebook is a social networking site with an image sharing component. Other sites such as Photobucket; Picasa; Blogger; Google Video; and Youtube provide differing mixes of social networking and content sharing and archival. Facebook is currently the largest social networking site on the internet; and the 5 most popular th website overall [6]. Though the primary function of Facebook does not appear to be image archival; it is nevertheless the largest image archival site on the internet with over 6.5 billion images (with 4 or 5 sizes each) occupying 540 terra-bytes of storage [2]. 475;000 images are served per second; (including 200;000 profile images per second) and 100 million images are uploaded per week. Facebook currently serves 99.8% of its profile requests and 92% of its photo requests through content distribution networks (CDNs) such as Akamai and Limelight [2]. This results in approximately 452;600 images served through CDNs per second; and 22;400 images served per second by their own servers. The cost of using CDNs to this extent is prohibitive; but Facebook has had to use them due to their inability to scale their own servers and services fast enough. They would like to reduce their reliance on CDNs to lower costs [2]. Facebook has 10;000 servers [3] and has had to borrow $100 million dollars to purchase more [4]. Facebook’s scale of image hosting (as well as the other image hosts) makes them an interesting research case to discover if novel approaches to archival storage in their usage model can offer significant improvement (in either space or throughput) over existing approaches. Problem Facebook began with the naive approach of storing their image files in a traditional filesystem served over NFS by clusters of NetApp servers. These servers became heavily I/O bound; with as may as 15 disk reads required to serve each request [2]. The high number of reads was due to the meta-data (data about data) used by traditional file systems. Every time a file is opened; the file’s name and path must be resolved to an i-node which acts as a bookmark for the file. Each directory (a component of the path) has an i-node; and that i-node maps to the directory’s contents on disk. When the final component of the path is resolved; the system gets an i-node which points to the file to be read. Only then can the