In my recent blog posts (part 1, part 2 and part 3) I have described in detail how to do micro benchmarking for Java and C/C++ with JMH and Hayai. I have presented a common execution approach based on Gradle. Everything was brought together in one single Gradle project structure, that keeps benchmarking infrastructure completely away from your production code and allows to build and execute both Java and C++ projects and benchmarks in one command. Git submodules might have caused some headaches, but this time we clarify some more details, I promise!
Back to my list. What we did so far, and what's next:
Today we want to tackle 6. and 8.
I've promised it in part 3 already. After we have isolated all the relevant code in very specific Gradle build files, the whole project was ready to move all the enabling code to dedicated plugins. To describe the Gradle plugin approach in one sentence: "Take your tasks and move them to a new project, where, with some minimal code and infrastructure overhead, you get a distributable plugin, which looks like all the others". Applied plugins are basically nothing else than tasks you define on-the-fly in your build.gradle!
Ok, to make it short, I would like to refer to the comprehensive Gradle Docs for details how to create plugins.
Here, I would like to show-case you the transformation and discuss the side effects. However, if you have never seen a Gradle plugin from the inside, take a closer look! You will quickly find similarities and pattern!
Looks like a standard benchmarking project that makes use of the jmh-gradle-plugin. But you don't want to add all those things to your project.
What YOU want to do is this:
I assume, the difference is clear. We replaced the originally used plugins with our own plugin, that adds our cherry on top (later).
The Gradle Docs quickly lead the way to an interface and some additional details which become handy when building your own plugins. This is how the CroLaBeFraJavaPlugin class looks like:
Tha CroLaBeFra Java part is really straight forward, because there is JMH and the already existing plugin for it in place. Nevertheless, it encapsulates the Microbenchmarking Framework and its configuration, so you could write your own custom strategy and create an additional Plugin next to it….
I wrote the very same thing for Hayai benchmarks which works slightly different, compared to JMH but the CroLaBeFra wrapper plugin should behave the same way as the Java one does.
The CroLaBeFra C++ Plugin does all those things for you, even if you are not using the cross-language part of it. So it is also a standalone integration of Hayai benchmarks into your tool chain with just one additional 'apply plugin':
Due to the more loose coupling and the naming flexibility with the native components, you have to define some properties according to your build output:
In order to get a deeper understanding, how those things fit together, please have a look at the C++ Plugin sources.
You come that far,let's get to an end. Both plugins produce their very own results with special settings, and different ways of execution. That makes comparison hard, but not impossible.
A JMH specialty which is worth a discussion at this stage: A well-known characteristic of the JVM is its cold-start behavior. Have you ever seen it? Do you know its impact? It can be made visible with JMH because there exists a special mode called SingleShotTime. Normally, JMH spends quite a lot of effort to circumvent it. But my goal is, to show a "worst", "average" and "best" in my report. Why am I doing this? Because the cold-start phase exists! It is there and that's fine! No one should bash the JVM for that, because in fact, it does a terrific job when optimizing the code, and that should be made visible! Consequently, the "worst" value value for Java benchmarks will be quite high, but still, I think it is interesting to see within the big picture. In contrast the compile time optimization of common C/C++ compilers will lead to numbers which are close together, but that's fine as well and is typical for static runtime code.
I told you that both JMH and Hayai are able run their benchmarks the same way, with the same identifiers, and the two previously presented plugins allow to create those results in a single build chain. This ultimately leads to the cherry on top, the funky mothership of plugins, the CroLaBeFra mothership.
The given numbers read as follows: Fastest, average, worst time spent per iteration. An iteration had a batch size of 1000000 calls of the according benchmark method. More is ok as well but probably increasing the number of iterations might make more sense. Less than 1000000 might not be enough reasonable amount of time. In general, the more iterations you can afford from a time consumption perspective, the more precise will be your results in general.
This is a set of results, that I have grabbed on my 6 years old MacBookPro under conditions which would make every performance expert cry - not due to its excellence of course. However, today we are not talking about the execution environment, but the possibility to have a one-shot command, which produces a combined report just like that one.
It all depends. The mathSqrt benchmark seems to be a good showcase to visualize the execution and measurement overhead of the two different micro benchmarking frameworks. Additionally, it is not yet proven, that the processor instructions are exactly the same in both worlds (although there should be the same sqrt intrinsic in the ASM code. But who knows whether there is additional 'noise'?). But these questions rectify a new blog post. For today, I would like to invite you to try out my showcase project yourself.
Further articles of this series: