The dmore/chrome-mink-driver README begins,
Mink driver for controlling Chrome without the overhead of Selenium.
It communicates directly with Google Chrome over HTTP and WebSockets, which allows it to work at least twice as fast as Chrome with Selenium. For Chrome 59+ it supports headless mode, eliminating the need to install a display server, and the overhead that comes with it. This driver is tested and benchmarked against a behat suite of 1800 scenarios and 19000 steps.
It can successfully run it in less than 18 minutes with Chrome 60 headless. The same suite running against Chrome 58 with xvfb and Selenium takes ~60 minutes.
The whole text needs an update (hopefully nobody is using Chrome 59 now, xvfb I'm uncertain) but it's the performance assertion that catches my eye. For a while I have wished for a cross-driver matrix to check what this looks like today.
While horsing around with Github Actions this weekend I accidentally struck the first cross-driver sample I've seen. It's a short test suite but interesting to finally see some numbers.
Here's what I see:
| Driver | Single "Run tests" (seconds) | 10x "Run tests" (seconds) | Notes |
|---|---|---|---|
| dmore/chrome-mink-driver + Chromium v124 | 34 | 334 | heck! |
| Selenium2 + Chrome v94 | 66 | 669 | |
| Selenium2 + Firefox v47 | 67 | 652 | skips 8 tests (@mouse-events) |
| Webdriver-Classic + Chrome v153 | 63 | 536 | |
| Webdriver-Classic + Firefox v156 | 75 | 580 |
Sources
- https://github.com/minkphp/driver-testsuite/pull/103 original PR with cross-browser tests
- https://github.com/xurizaemon/driver-testsuite/actions
- https://github.com/xurizaemon/driver-testsuite/actions/runs/38076873825 (official)
- https://github.com/xurizaemon/driver-testsuite/actions/runs/38075609043 (unofficial)
- https://github.com/xurizaemon/driver-testsuite/actions/runs/38082730795/job/114302813853 (unofficial, x10)
- https://github.com/xurizaemon/driver-testsuite/actions/runs/38082760974/job/114302898179 (official, x10)
This set of tests tells us only about a handful of samples of Mink's driver-testsuite, run in Github Actions, but it's interesting to see! I discovered a WIP cross-driver Github Actions PR (minkphp/driver-testsuite#103) which was a leap in the right direction. I used that to get some tests I wanted running in the upstream driver test suite, then realised a fuller cross-browser matrix was within reach. The Mink official tests (Webdriver, Selenium2) browser versions are based on the smoke tests PR I found; dmore/chrome-mink-driver is also tested with a v154 Chrome in its own project CI.
I maintain dmore/chrome-mink-driver with a handful of other users and I think it's a nice example of a healthy, slow-cooking project. It's a useful component for web testing, and has stayed healthy beyond the author's or my expectations. Folks drop in with occasional fixes, releases ship occasionally, it's largely stable even when Chrome makes big changes. The project CI validates against a couple of popular containerised Chrome's for testing with.
The Mink driver component was authored by Dorian More in 2017 (circa when the performance claim stems from) and remains useful and even popular (271K Packagist installs in last 30 days) today.
This component is a browser-controlling mink driver, meaning it's capable of testing JS. The performance seems to come from talking Chrome DevTools Protocol directly ... IDK, I just maintain here.
Comparing similar components, behat/mink-selenium2-driver has 362K installs in the same period, and mink/webdriver-classic-driver has 36K. I don't aim to compete with alternatives, I'm happy that there are several options and happy that this option is useful.
dmore/chrome-mink-driver is Chrome/Chromium-specific (meaning it supports a single dominant player, which I dislike), and that also means it is ultimately ephemeral - an Alphabet business decision could kill it at any point. For now, it thrives a little longer.
Reflecting on the test run times, and health / longeivity of the project, I do feel better about not chasing every issue in the queue. Ability to stabilise and maintain is an important part of a healthy system.
I'm really grateful to all the people who contribute to the project's health, especially Dorian More, for the original work and for releasing it to a community of caretakers; Andrew Coulton, for your deep knowledge and insight; and Chris Caldwell, for your dedication to solving challenging bugs.
Super shout out to each and every one of many folks who drop in with a small fix for someting they encountered - my thanks for your patience (and patch tools) while it gets around to being merged.
If you happen to have a JS/browser Behat test suite that takes a while to run (haha), and think perhaps dmore/chrome-mink-driver might speed things up, I would LOVE you to share your experiences. I have opened chrome-mink-driver#178: Real world performance experience update for this - let me know your story!
I will forewarn that the project has no resource allocated for a storm of new user bug reports - I will be grateful, but there is no guarantee of immediate response :)
Man, the Github Actions pages really lag out my Firefox? That's enough for today.