Information and documentation — WARC file format
信息和文档 - WARC文件格式
发布日期:
2017-08-14
ISO 28500:2017规定了WARC文件格式:
-存储来自主流互联网应用层协议(如HTTP、DNS和FTP)的有效载荷内容和控制信息;
-存储与其他存储数据链接的任意元数据(例如主题分类器、发现的语言、编码);
-支持数据压缩,维护数据记录完整性;
-存储来自采集协议的所有控制信息(例如请求头),而不仅仅是响应信息;
-存储与其他存储数据链接的数据转换结果;
-存储与其他存储数据链接的重复检测事件(在存在相同或基本相似资源的情况下减少存储);
-在不中断现有功能的情况下进行扩展;
-支持在需要时通过截断或分段来处理过长的记录。
ISO 28500:2017 specifies the WARC file format:
- to store both the payload content and control information from mainstream Internet application layer protocols, such as the HTTP, DNS, and FTP;
- to store arbitrary metadata linked to other stored data (e.g. subject classifier, discovered language, encoding);
- to support data compression and maintain data record integrity;
- to store all control information from the harvesting protocol (e.g. request headers), not just response information;
- to store the results of data transformations linked to other stored data;
- to store a duplicate detection event linked to other stored data (to reduce storage in the presence of identical or substantially similar resources);
- to be extended without disruption to existing functionality;
- to support handling of overly long records by truncation or segmentation, where desired.