SYSmark® 2004 SE Benchmarking Rules

This document describes the rules governing publication of results derived from running the BAPCo® SYSmark® 2004 SE benchmark. Any licensee who wishes to publish such results is obliged to adhere to all rules and regulations described in this document or other documents provided by BAPCo related to execution and reporting of SYSmark 2004 SE benchmark results.

BAPCo reserves the right to change the rules outlined in this document at any time. Licensees are encouraged to consult the BAPCo website at www.bapco.com for the latest information regarding execution and publication rules. For more information please consult the Software License Agreement.

Benchmark Execution

All SYSmark 2004 SE benchmarks must be run using the provided SYSmark 2004 SE software program. The SYSmark 2004 SE program generates log files that include all pertinent run-time data. It also includes a module that generates the final benchmark results for the system under test. All results published may be taken only from a complete and correct execution of SYSmark 2004 SE benchmarks under the control of the SYSmark 2004 SE program. All tests must be performed without modification to any part of the SYSmark 2004 SE software program. The collective set of hardware and software used in generation of an official SYSmark 2004 SE result must be available to the public as defined by BAPCo availability criteria. The following specific rules apply:

Operating System: The operating system used to generate official results must be an official, release-quality, publicly obtainable release from the original software vendor. BETA and interim development releases of the operating system are not supported. Whereas it is highly recommended to use operating system releases listed on BAPCo website, licensees are free to publish results generated from official SYSmark 2004 SE runs executed on publicly available release-quality operating systems only (available from the original vendor). Upgrades of specific drivers that are released and maintained by vendors other than the operating system vendor (e.g. graphics drivers, disk drivers, motherboard setting files, multimedia drivers, codecs) are allowed provided they are publicly available release-quality drivers from actual original vendors of such drivers. Licensees may also use operating system service packs, API updates (e.g. DirectX*) and other such updates, provided they are publicly available release-quality software from the actual original vendor. Licensees should also consult license agreements from individual software vendors to ensure compliance with any vendor benchmark restrictions.

Performance-Enhancing Utilities: Use of performance-enhancing utilities or techniques is allowed provided that the utility or technique is publicly available release-quality software from the original vendor. The utility needs to be documented and such techniques should not compromise the integrity of the system under test or the benchmark. This documentation must be fully disclosed when publishing results and included with additional notation accompanying any Full Disclosure Report submitted to BAPCo. BAPCo shall reserve the right to review the usage of such tools or techniques and invalidate published results if need be.

Hardware: The system hardware used to generate official SYSmark 2004 SE results must be publicly available at the time of publication or within 60 days thereafter. The system must be purchasable as a complete platform from an Original Equipment Manufacturer within 60 days of publication. Alternatively, licensees may report results for systems which may not be available from an OEM but whose individual components are publicly available and purchasable from various vendors at the time of publication or within 60 days thereafter. Note that the 60-day grace period does not apply to system software. All software used must be publicly available, of release-quality, and available from the original software vendor at the time of submission.

Result Generation

All results used for publication must be generated by SYSmark 2004 SE and displayed in the SYSmark Results Viewer as part of a successful test run. SYSmark automatically computes the benchmark results for the system under test and displays them in the Results Viewer. Results from failed tests, even if generated by SYSmark 2004 SE, cannot be used to compute SYSmark results.

Full Disclosure Report

A Full Disclosure Report (FDR) must be generated for all published results (see Submitting an FDR for more information). The FDR includes results generated by the SYSmark 2004 SE program as well as pertinent run-time information about the state of the system under test. The format and the information included in this report are determined by BAPCo. SYSmark 2004 SE automatically produces FDRs based on valid test runs and stores them in ASCII text format. Additional information about the system under test can be added to the FDR.

Availability

All Full Disclosure Reports submitted to BAPCo must include information about the availability of the tested system. This should be included with additional notation accompanying any Full Disclosure Report submitted to BAPCo. The tested system must be available for purchase by the public either at the time of publication or within 60 days after publication. Note that the 60-day grace period does not apply to system software. All software used must be release-quality and made publicly available from the original software vendor at the time of submission.

Full Disclosure Report Publication

Any Licensee may publish results but only those results obtained through a Full Disclosure Report. Licensees may either publish performance results relating to SYSmark 2004 SE in a medium of their choice and/or by submitting results for publication on www.bapco.com.

It is considered ‘best practice’ to submit all Full Disclosure Reports generated by SYSmark 2004 SE with BAPCo prior to publication (see ‘Submitting an FDR’ below).

Licensees that do not file Full Disclosure Reports must make available to BAPCo upon its request, any published Full Disclosure Report within five business days. Full Disclosure Report records not submitted to BAPCo must be kept and maintained by the licensee until BAPCo retires the benchmark (see www.bapco.com for list of retired benchmarks). BAPCo reserves the right to revoke the license granted to licensees who fails to adhere to these terms.

At a minimum all licensees are strongly encouraged to submit at least one FDR per ‘platform’ as to benefit the community of BAPCo users, who regularly use published FDR data from BAPCo’s website for comparative and analytical purposes.

Submitting an FDR

The submitted FDRs are processed by BAPCo and classified as:

Released: the results are released for publication as filed
Rejected: the results are rejected for publication as filed

The Released classification is in no way a validation or an endorsement of the results by BAPCo. BAPCo will perform frequent audits of submitted FDRs and reserves the right to revoke the license granted to licensees who have not adhered to the regulations described in this or any other SYSmark documents.

In the case where a report which has been submitted or been requested is returned to the licensee “Rejected,” the licensee must immediately cease and/or publish a retraction of the rejected results until the matter is clarified.

Any publication of results must be accompanied by a complete system description, including at a minimum the OEM's system denomination and model number. In general any configuration changes between OEM stated configuration and actually tested configuration should be stated (also see Publication of Results on Non-OEM Platforms below).

In the case of notebooks, only results gathered with the system running on AC should be submitted as regular SYSmark FDRs. In the less frequent occurrence of a SYSmark 2004 SE FDR submission from a notebook on battery power, the run rules pertaining to MobileMark 2005 should also be applied in addition to these SYSmark run rules. Additionally, accompanying documentation to this effect should be submitted with the FDR.

The term "SYSmark" is a trademark of the Business Applications Performance Corporation (BAPCo) and may only be used in conjunction with performance and metrics generated by the SYSmark software program. Any other performance and characterizations may be made but must be derived from the performance metrics and generated by the SYSmark program. The terms "SYSmark" and "SYSmark 2004 SE" may not be used as the performance and metrics describing such characterizations.

BAPCo Benchmark Disclaimer: Business Applications Performance Corporation (BAPCo) is a non-profit, member organization of companies in the computing industry. Its mission is to create benchmark products that compare performance among computer systems, with particular view to business applications, as fairly and impartially as possible, in the light of the latest developments in technology and trends in usage. BAPCo products result from collaborative balancing and integration of technical input from participating members and experts, and reflect their professional judgment. We believe that BAPCo benchmarks are relevant and impartial measures for comparing general system performance on widely used business applications. BAPCo recognizes, however, that performance measures intrinsically involve subjective values, preferences, and priorities, which vary from user to user. BAPCo does not claim or warrant that the balances of system values, preferences, and priorities reflected in its benchmark products will coincide with those of any particular user.

Publication of Results on Non-OEM Platforms

BAPCo allows the publication of performance indices using SYSmark benchmark suites for making "non-OEM" platform comparisons. This includes performance comparisons on systems that are not shipped from an OEM as a complete platform, such as “white box” systems. An example of such a comparison is when a graphics card manufacturer wishes to compare the performance of its graphics card to that of another manufacturer in an identical platform by swapping the cards and measuring the performance differences. Results published on Non-OEM platforms must be marked as “Modified” on the accompanying FDR with all components fully specified, and whenever appearing in publications of any kind.

Publication of Results for Competitive Comparisons

Licensees may publish results on competitor systems as long as the performance results for the competitor have been published previously at www.bapco.com. In the case where competitor performance results are not available, the competitor will be notified by BAPCo and given fifteen working days to provide a counter-FDR for its own system or challenge the submitted FDRs (results). The submitted results may not be published until the fifteen-day waiting period has expired and all matters with respect to any counter-FDRs have been resolved by BAPCo. In the absence of a response from the competitor after fifteen days, the original publishing licensee may begin publication. Industry publications may publish results for making competitive comparisons without being subject to the above procedure. Publications are, however, subject to all other regulations set forth in this document.

*All brand and product names are trademarks or registered trademarks of their respective holders.