Introducing Blender Benchmark

Today we present the Blender Benchmark, a new platform to collect and display the results of hardware and software performance tests. With this benchmark we aim at an optimal comparison between system hardware and installations, and to assist developers to track performance during Blender development.

The benchmark consists of two parts: a downloadable package which runs Blender and renders on several production files, and the Open Data portal on blender.org, where the results will be (optionally) uploaded.

We’ve built the Blender Benchmark platform with maximum focus on transparency and privacy. We only use free and open source software (GNU GPL), the testing content is public domain (CC0), and the test results are being shared anonymized as public domain data – free for anyone to download and to process further.

We believe this is the best way to invite the Blender community to contribute the results of their performance tests, and create a world-class Open Dataset for the entire CG industry.

How does it work?

Users download the Benchmark Client and run one of the two benchmarks (‘quick’ or ‘complete’). The benchmark will gather information about the system, such as operating system, RAM, graphics cards, CPU model, as well as information about the performance of the system during the execution of the benchmark. After that, the user will be able to share the result online on the Blender Open Data platform, or to save the data locally.

In order to provide control over the data that is shared online, the benchmark result is first associated with the Blender ID of the user, and uploaded on mydata.blender.org, where the user will be able to redact and anonymize the parts containing personal information (Blender ID username and hostname). Currently this information is removed by default. No other personal information is collected.

Blender Open Data Architecture

Blender Open Data portal

In order to visualize, share and explore the data, we’ve built opendata.blender.org. The data hosted on the website is available under Public Domain, it is updated in near-realtime after every benchmark and it is easily processable and well documented.

While hosting Blender Benchmark results will be the initial purpose of the Open Data portal, we plan to host other data sets in the future. For example, information about Blender downloads, telemetry information, etc. Each data set published on the platform will adhere to our Open Data principles, and its collection will be clearly communicated.

Blender Open Data principles

As initial guideline for our definition of Blender Open Data, we were inspired by the Eight Principles of Open Data. We believe that Blender Open Data should be:

  • Complete. All public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations.
  • Primary. Data is as collected at the source, with the highest possible level of granularity, not in aggregate or modified forms.
  • Timely. Data is made available as quickly as necessary to preserve the value of the data.
  • Accessible. Data is available to the widest range of users for the widest range of purposes.
  • Machine processable. Data is reasonably structured to allow automated processing.
  • Non-discriminatory. Data is available to anyone, with no requirement of registration.
  • Non-proprietary. Data is available in a format over which no entity has exclusive control.
  • License-free. Data is not subject to any copyright, patent, trademark or trade secret regulation. Reasonable privacy, security and privilege restrictions may be allowed.
  • Online and Free. Information is not meaningfully public if it is not available on the Internet at no charge, or at least no more than the marginal cost of reproduction. It should also be findable.
  • Permanent. Data should be made available at a stable Internet location indefinitely and in a stable data format for as long as possible.

We take a privacy-conscious approach when handling Benchmark data.

Timeline

  • August 10th: public beta; a test run to verify if everything works. Send feedback to devtalk.blender.org.
  • September: first official release.

Credits

The project has been developed by the team at Blender: Brecht van Lommel, Dan MacGrath, Francesco Siddi, Markus Ritberger, Pablo Vazquez, Sybren Stüvel and Sergey Sharybin.This project was commissioned by Ton Roosendaal, chairman Blender Foundation.

share this story on