WebMark® 2004 Benchmarking Rules
This document describes the rules governing publication of results derived from running the BAPCo WebMark 2004 benchmark. Any licensee who wishes to publish such results is obliged to adhere to all rules and regulations described in this document or other documents provided by BAPCo related to execution and reporting of WebMark 2004 benchmark results. BAPCo reserves the right to change the rules outlined in this document at any time. Licensees are encouraged to consult BAPCo or Futuremark websites at www.bapco.com or www.futuremark.com for the latest information regarding execution and publication rules. For more information please consult the Software License Agreement.Benchmark Execution
All benchmarks must be run using the WebMark 2004 Workload
Manager. This utility is an executable program that runs the WebMark
2004 benchmark. It also includes a module that generates the final benchmark
results for the system under test. For a detailed guide to the operation of the
Workload Manager see Using WebMark 2004.
All published results must be derived only from a complete and correct
Official Run of the benchmark. All tests must be performed without
modification to any part of the WebMark 2004 software.
The collective set of hardware and software used in generation of an official
WebMark 2004 result must be available to the public as defined by BAPCo's
availability criteria. The following specific rules apply:
Plug-in Software: WebMark 2004 must be executed using the provided plug-in software from BAPCo. Whereas licensees shall, for their own experimental curiosity, upgrade the provided plug-in's with new versions, they may not publish results from running WebMark 2004 in such a manner. BAPCo will, at appropriate times, evaluate and publish plug-in update packs that can be used to generate official WebMark 2004 results. BAPCo will designate plug-in update packs with specific version numbers. At that point, the version number used in generating an official WebMark 2004 result must be noted in the full disclosure report. Licensees must use the most up to date plug-in pack when running WebMark 2004. Full disclosure reports generated using older plug-in packs will be accepted for a period of 10 days after the release of the current plug-in pack. This exception applies to the most recent plug-in pack before the current plug-in pack only.
Operating System: The operating system used to generate official results must be an official, release-quality, publicly available, release from the original software vendor. Whereas it is highly recommended to use operating system releases listed on BAPCo/Futuremark websites, licensees are free to publish results generated from official WebMark 2004 runs executed on publicly available, release-quality operating systems only (available from the original vendor). Upgrade of specific drivers that are released and maintained by vendors other than the operating system vendor (e.g. graphics drivers, disk drivers, motherboard setting files, multimedia drivers, codecs) is allowed provided these drivers are publicly available, release-quality drivers from original vendors of such drivers. Licensees may also use operating system service packs, API updates (e.g. Direct X) and other such updates provided they are publicly available, release-quality software from the original vendor. Licensees should also consult license agreements from individual software vendors to ensure compliance with vendor benchmark restrictions.
Performance Enhancing Utilities: Use of performance enhancing utilities or techniques is allowed provided that the utility or technique is publicly available, release-quality software from the original vendor. The utility needs to be documented and such techniques should not compromise the integrity of the system under test or the benchmark. BAPCo shall reserve the right to review the usage of such tools or techniques and invalidate published results if need be.
Client and Server Hardware: The system hardware used to generate official WebMark 2004 results must be publicly available at the time of publication or within 60 days thereafter. The system must be purchasable as a complete platform from an Original Equipment Manufacturer within 60 days of publication. Alternatively, licensees may report results for systems which may not be available from an OEM but whose individual components are publicly available and purchasable from various vendors at the time of publication or within 60 days thereafter. Note that the 60-day grace period does not apply to system software. All software used must be publicly available and be of release-quality available from the original software vendor.
Network Connection: WebMark 2004 is designed such that it can be executed with various connection topologies and speeds. For evaluation projects whose goal is to measure the performance of the client hardware, it is recommended that the client connect to the server hardware via a private LAN. However, licensees may use various topologies and connection speeds for measuring the system performance provided all system parameters are disclosed. For rules governing disclosures refer to Full Disclosure Requirements.
Result Generation
All results used for publication must be generated using the Report Generator of the Workload Manager. The Report Generator automatically computes the benchmark results for the system under test. Note that unofficial results generated by the Workload Manager cannot be used for computing official WebMark 2004 results.Full Disclosure Report
A Full Disclosure Report (FDR) is required for all published results. This report includes all results generated by the Workload Manager as well as pertinent run-time information about the state of the system under test. The format and the information included in this report are determined by BAPCo. The Report Generator automatically produces FDRs based on valid test runs and stores them in ASCII format. Any additional information about the system under test can be added to the FDR.Availability
All Full Disclosure Reports submitted to BAPCo must include the availability of the tested system. The tested system must be available for purchase by the public either at the time of publication or within 60 days after publication. Note that the 60-day grace period does not apply to system software. All software used must be release-quality and made publicly available from the original software vendor.Publication
Any licensee may publish results. Licensees may either publish performance results relating to WebMark 2004 in a media of their choice and/or by submitting results for publication to www.bapco.com. A Full Disclosure Report generated by the Workload Manager must be filed with BAPCo before publication.FDR Processing
The submitted FDRs are processed by BAPCo and classified as:
Released; the results are released for publication as filed
Rejected; the results are rejected for publication as filed
The Released for publication classification is in no way a validation or an endorsement of the results by BAPCo. The submitting licensees are free to publish results as long as a complete FDR has been submitted to BAPCo except in the case of publishing results for competitive comparisons . BAPCo will perform frequent audits of the submitted FDRs and reserves the right to revoke the license granted to licensees who have not adhered to the regulations described in this or other WebMark 2004 documentation.
In the case where an FDR is returned to the licensee with rejected classification, the licensee must immediately stop publication of the rejected results until the matter is clarified by BAPCo.
Any publication of results must be accompanied by at least the model number plus any discrepancies between the configuration of the stated model number and the configuration of the system tested. The publication can use the single overall WebMark 2004 rating or the combination of the overall rating, scenario rating and Site rating. Licensees may not publish single or partial scenarios or Site ratings unless the overall WebMark 2004 rating is also included. (E.g. the Portal site Rating cannot be published without disclosing the Information Processing Scenario Rating and the WebMark 2004 overall rating)
The term WebMark is a registered trademark of the Business Applications Performance Corporation and may only be used in conjunction with performance metrics generated by the WebMark 2004 Workload Manager. Any other performance characterizations may be made but must be derived from the performance metrics generated by the Workload Manager. The terms WebMark and WebMark 2004 may not be used as the performance metric describing such characterizations.
Publication of Results on Non-OEM Platforms
BAPCo allows the publication of performance indices using WebMark 2004 for making "Non-OEM" platform comparisons. This includes performance comparisons on systems that are not shipped from an OEM as a complete platform. An example of such comparisons is when a graphics card manufacturer wishes to compare the performance of its graphics card to that of another manufacturer in an identical platform by swapping the cards and measuring the performance. Results published on Non-OEM platforms must be marked as Modified on the accompanying FDR and whenever appearing in publications of any kind.Publication of Results for Competitive Comparisons
Licensees may publish results on competitor systems as long as the performance results for the competitor have been published previously at www.bapco.com. In the case where competitor performance results are not available, the competitor will be notified by BAPCo and given fifteen working days to provide a counter-FDR for its own system or challenge the submitted FDRs (results). The submitted results may not be published until the fifteen-day waiting period has expired and all matters with respect to any counter-FDRs have been resolved by BAPCo. In the absence of a response from the competitor after fifteen days, the original publishing licensee may begin publication. Industry publications may publish results for making competitive comparisons without being subject to the above procedure. Publications are, however, subject to all other regulations set forth in this document.
