Testing at Gizra
We have covered the subject of automatic tests quite a lot here at Gizra and I’m sure that there are a lot of posts and articles covering the subject. However, I’m having a bizarre experience lately while writing new features that I had to share it with you folks. Keep in mind that I don’t write a lot of blog posts and I always feared following up on my previous smashing hit Bootstrap custom breakpoint :)
So the experience I want to talk about was that of insecurity, which was quite odd for me considering I’m always proud of my code until @amitaibu reviews it and smashes my ego to pieces (which I learned to love).
Chances are that you already use Travis or another cool CI to execute your tests, and everyone politely waits for the CI checks before even thinking about merging, right? More likely, waiting your turn becomes a pain and you click on the merge: it’s a trivial change and you need it now. If this happens often, then it’s the responsibility of those who worked on those scripts that Travis crunches to make some changes. There are some trivial and not so trivial options to make the team always be willing to wait for the completion.
This blog post is for you if you have a project with Travis integration, and you’d like to maintain and optimize it, or just curious what’s possible. Users of other CI tools, keep reading, many areas may apply in your case too.
Unlike other performance optimization areas, doing before-after benchmarks is not so crucial, as Travis mostly collects the data, you just have to make sure to do the math and present the numbers proudly.
To start, if your
.travis.yml lacks the
cache: directive, then you might start in the easiest place: caching dependencies. For a Drupal-based project, it’s a good idea to think about caching all the modules and libraries that must be downloaded to build the project (it uses a buildsystem, doesn’t it?). So even a variant of:
or for Drush
It’s explained well in the verbose documentation at Travis-ci.com. Before your script is executed, Travis populates the cache directories automatically from a successful previous build. If your project has only a few packages, it won’t help much, and actually it can make things even slower. What’s critical is that we need to cache slow-to-generate, easy-to-download materials. Caching a large ZIP file would not make sense for example, caching many small ones from multiple origin servers would be more beneficial.
From this point, you could just read the standard documentation instead of this blog post, but we also have icing on the cake for you. A Drupal installation can take several minutes, initializing all the modules, executing the logic of the install profile and so on. Travis is kind enough to provide a bird’s-eye view on what eats up build time:
Execution speed measurements built in the log
Mind the bottleneck when making a decision on what to cache and how.
For us, it means cache of the installed, initialized Drupal database and the full document root. Cache invalidation is hard, we can’t change that, but it turned out to be a good compromise between complexity and execution speed gain, check our examples:
Do your homework and cache what’s the most resource-consuming to generate, SQL database, built source code or compiled binary, Travis is here to assist with that.
There are two reasons to pay attention to software versions.
Use Pre-installed Versions
Travis uses containers of different distributions, let’s say you use
trusty, the default one these days, then if you choose PHP 7.0.7, it’s pre-installled, in case of 7.1, it’s needed to fetch separately and that takes time for every single build. When you have production constraints, that’s almost certainly more important to match, but in some cases, using the pre-installed version can speed things up.
And moreover, let’s say you prefer MariaDB over MySQL, then do not
sudo and start to install it with the package manager, as there is the add-on system to make it available. The same goes for Google Chrome, and so on.
Stick to what’s inside the image already if you can. Exploit that possibility of what Travis can fetch via the YML definition!
Use the Latest and (or) Greatest
If you ever read an article about the performance gain from migrating to PHP 7, you sense the importance of selecting the versions carefully. If your build is PHP-execution heavy, fetching PHP 7.2 (it’s another leap, but mind the backward incompatibilities) could totally make sense and it’s as easy as can be after making your code compatible:
Almost certainly, a similar thing could be written about Node.js, or relational databases, etc. If you know what’s the bottleneck in your build and find the best performing versions – newer or older – it will improve your speed. Does that conflict with the previous point about pre-installed versions? Not really, just measure which one helps your build the most!
Make it Parallel
When a Travis job is running, 2 cores and 4 GBytes of RAM is available – that’s something to rely on! Downloading packages should happen in parallel.
gulp and other tools like that might use it out of the box: check your parameters and configfiles. However, on the higher level, let’s say you’d like to execute a unit test and a browser-based test, as well. You can ask Travis to spin up two (or more) containers concurrently. In the first, you can install the unit testing dependencies and execute it; then the second one can take care of only the functional test. We have a fine-grained example of this approach in our Drupal-Elm Starter, where 7 containers are used for various testing and linting. In addition to the great execution speed reduction, the benefit is that the result is also more fine-grained, instead of having a single boolean value, just by checking the build, you have an overview what can be broken.
All in all, it’s a warm fuzzy feeling that Travis is happy to create so many containers for your humble project:
If it's independent, no need to serialize the execution
The available memory is currently between 4 and 7.5 GBytes , depending on the configuration, and it should be used as much as possible. One example could be to move the database main working directory to a memory-based filesystem. For many simpler projects, that’s absolutely doable and at least for Drupal, a solid speedup. Needless to say, we have an example and on client projects, we saw 15-30% improvement at SimpleTest execution. For traditional RMDBS, you can give it a try. If your DB cannot fit in memory, you can still ask InnoDB to fill memory.
Think about your use case – even moving the whole document root there could be legitimate. Also if you need to compile a source code, doing it there makes sense as well.
Build Your Own Docker Image
If your project is really exotic or a legacy one, it potentially makes sense to maintain your own Docker image and then download and execute it in Travis. We did it in the past and then converted. Maintaining your image means recurring effort, fighting with outdated versions, unavailable dependencies, that’s what to expect. Still, even it could be a type of performance optimization if you have lots of software dependencies that are hard to install on the current Travis container images.
+1 - Debug with Ease
To work on various improvements in the Travis integration for your projects, it’s a must to spot issues quickly. What worked on localhost, might or might not work on Travis – and you should know the root cause quickly.
In the past, we propagated video recording, now I’d recommend something else. You have a web application, for all the backend errors, there’s a tool to access the logs, at Drupal, you can use Drush. But what about the frontend? Headless Chrome is neat, it has built-in debugging capability, the best of which is that you can break out of the box using Ngrok. Without any X11 forwarding (which is not available) or a local hack to try to mimic Travis, you can play with your app running in the Travis environment. All you need to do is to execute a Debug build, execute the installation part (
travis_run_before_script), start Headless Chrome (
google-chrome --headless --remote-debugging-port=9222), download Ngrok, start a tunnel (
ngrok http 9222), visit the exposed URL from your local Chrome and have fun with inspection, debugger console, and more.
Working on such improvements has benefits of many kinds. The entire development team can enjoy the shorter queues and faster merges, and you can go ahead and apply part of the enhancements to your local environment, especially if you dig deep into database performance optimization and make the things parallel. And even more, clients love to hear that you are going to speed up their sites, as this mindset should be also used at production.
Elm’s type system is sufficiently sophisticated that you’ll often want to make
fine-grained distinctions between roughly similar types. In a
recent project, for instance, we ended up
with a separate type for a
Mother and a
Child.1 Now, a
is a little different than a
Child. Most obviously, mothers have children,
whereas (at least, in our data model) children do not. So, it was nice for them
to be separate types. In fact, there were certain operations which could be done
Mother but not a
Child (and vice versa). So it was nice to be able to
enforce that at the type level.
Mother and a
Child clearly have a lot in common as well. For this
reason, it sometimes felt natural to write functions that could accept either. So, in
those cases, it was a little awkward for them to be separate types. Something
was needed to express a relationship between the two types.
What alternatives are available to do this sort of thing in Elm? Which did we
end up choosing? For answers to these questions, and more, read on!
There were no fathers in our app’s data model. ↩
Once you start writing apps (and packages) in Elm, it’s
Yet when implementing features for paying clients, it doesn’t always make sense
them in pure Elm. In fact, sometimes it isn’t even possible!
necessary – ports!
Yet ports aren’t always the right answer, and there are several alternatives
which can be useful in certain situations.
For the purposes of this post, I’m going to assume that you’re familiar with
the many cases in which ports work well, and focus instead on a few cases where
you might want to try something else:
- When you want synchronous answers.
- When you need some context when you get the answer.
I tell my kids all the time that they can’t have both - whether it’s ice cream and cake or pizza and donuts - and they don’t like it. It’s because kids are uncorrupted, and their view of the world is pretty straightforward - usually characterized by a simple question: why not?
And so it goes with web projects:
Stakeholder: I want it to be like [insert billion dollar company]’s site where the options refresh as the user makes choices.
Me: [Thinks to self, “Do you know how many millions of dollars went into that?”] Hmm, well, it’s complicated…
Stakeholder: What do you mean? I’ve seen it in a few places [names other billion dollar companies].
Me: [Gosh, you know, you’re right] Well, I mean, that’s a pretty sophisticated application, and well, your current site is Drupal, and well, Drupal is in fact really great for decoupled solutions, but generally we’d want to redo the whole architecture… and that’s kind of a total rebuild…
Stakeholder: [eyes glazed over] Yeah, we don’t want to do that.
But there’s is a way.
If you happen to know Brice - my colleague and Gizra’s CEO - you probably have picked up that he doesn’t get rattled too easily. While I find myself developing extremely annoying ticks during stressful situations, Brice is a role model for stoicism.
Combine that with the fact that he knows I dislike speaking on the phone, let alone at 6:53pm, almost two hours after my work day is over, you’d probably understand why I was surprised to get a call from him. “Surprised” as in, immediately getting a stomach ache.
The day I got that call from him was a Sale day. You see, we have this product we’ve developed called ״Circuit Auction״, which allows auction houses to manage their catalog and run live, real-time, auction sales - the “Going once, Going twice” type.
- “Listen Bruce,” (that’s how I call him) “I’m on my way to working out. Did something crash?”
I don’t always think that the worst has happened, but you did just read the background.
I was expecting a long pause. In a way, I think he kind of enjoys those moments, where he knows I don’t know if it’s good or bad news. In a way, I think I actually do somehow enjoy them myself. But instead he said, “Are you next to a computer?”
- “No. I’m in the car. Should I turn back? What happened?”
I really hate to do this, but in order for his next sentence to make sense I have to go back exactly 95 years, to 1922 Tokyo, Japan.
Last week, we covered how to break down decoders by starting from the innermost, or topmost, part.
But what if you’re having trouble breaking things down from the top?
(Or you’re dealing with a really complex JSON schema?)
This week, let’s look at it from a different perspective: the outermost structure in (or the bottom up!)
A reader of the JSON Survival Kit wrote me with a question (lightly edited):
I really can’t figure out how to parse this–will your book help with nested JSON where the keys are different 2 or 3 levels deep?
If not, then I’ll just give up on Elm–as this is the first project that I’m trying to do, and something as basic as this, I’m finding impossible.
The biggest mindset shift you need to succeed with JSON Decoding is to think of your decoders like bricks.
(I’ve written about this before, and it’s chapter 1 of The JSON Survival Kit.)
You can combine bricks to build whatever you like; the same is true of decoders!
The one runtime exception nearly every Elm developer will encounter sooner or later is this one, dealing with recursive JSON decoders:
Uncaught TypeError: Cannot read property ‘tag’ of undefined at runHelp
ContextLet’s say you are writing a decoder for a recursive structure:https://medium.com/media/df8ff3ab37ef34dbc9448b2e4de39a75/hrefWith hypothetical JSON looking like this:https://medium.com/media/4cc288b6dbf33b922d6961340a645ad7/hrefA first attemptThe most straightforward approach to this type of decoder, is to create a branch-decoder, a leaf-decoder and a tree-decoder, which would look something like this:https://medium.com/media/6b631a0e4873d6eae488b3a6238ad0e8/hrefHowever, Elm is an eager language, and functions are evaluated as soon as all of their arguments are passed. In the above example, where decoders are simply values, we’re dealing with recursively defined values, and you can’t do that in an eager language. Elm will, of course, point this out in its usual, friendly manner.https://medium.com/media/93d6928f2a64e31dea3288ad76f4b4d5/hrefIntroducing lazinessAfter reading the linked document, you know you need to introduce laziness using — in this case — Json.Decode.lazy. You may be wondering where to put the call to lazy: should you lazily refer from decoder to branchDecoder, or should it be the other way around?The slightly surprising answer is this:
There is no way to know for sure.
I figured it would be fun to take a tiny function and explain how it works line by line.https://medium.com/media/a2a9d0a1397e23b4532f2fb98cfe76ca/hrefLet’s examine that, line by line, function by function. Noting down the link to the documentation, the signature of each function used and what the inferred types look like at that point should prove — if nothing else — interesting to some!https://medium.com/media/9c2c32674172cb3125b68dad349bcadd/hrefDelayed HTTP requests in Elm, line-by-line was originally published in Ilias Van Peer on Medium, where people are continuing the conversation by highlighting and responding to this story.
elm-reactor is an underrated tool. Not only does it do on-demand recompilation of Elm source code, but it can serve up other assets, too.But did you know you can serve your own HTML with live-compiled Elm code, too? This is useful if you need JS interop or want to start your program with flags.The trick here is that elm-reactor exposes a “magical” /_compile directory — any elm file prefixed with that path will be pulled in and compiled on page-load.For example, start with a folder-structure like this:myProject/|- elm-package.json|- index.html`- src/ `- Main.elmPlacing your index.html at the same level as your elm-package.json means that running elm-reactor from your myProject folder will allow you to point your browser to http://localhost:8000/index.htmlAs for the contents of your index.html, start with something like this:<html><head> <style> /* custom styles? Sure! */ </style></head><body> <!-- Relative to index.html, main.elm lives in `src/Main.elm`. --> <!-- Prefixing that with `/_compile/` gives us magic! --> <script src="/_compile/src/Main.elm"></script> <script> var app = Elm.Main.fullscreen() // You could also pass flags, or setup some ports, ... </script></body></html>There, all set!Note that elm-reactor has also learned how to serve quite a few other file types with the correct content type headers, so pulling in some CSS, images or JSON should work, too.Shout-out to @ohanhi for the tip!Elm reactor and custom HTML was originally published in Ilias Van Peer on Medium, where people are continuing the conversation by highlighting and responding to this story.
Chances are that you already using Travis or another Cool CI to execute your tests. Very often getting boolean or textual output from the execution is enough, because knowing which tests are failing is a good starting point to start to debug the problematic code. In our case, with WebdriverI/O (WDIO) and with an architecture where the frontend and backend are decoupled, it’s much more complicated.
It might be that the browser could not click on an element, or the frontend could not contact the backend, or the frontend has a runtime error (well, you might be faced with it, but at Gizra we use Elm, where it is practically impossible). Who knows, even the browser could crash due to lack of memory - the same applies to Travis too. One solution is to manually start reproducing what Travis does. It’s fun the first time, but doing it again and again is just a waste of time. But recently, our CTO, Amitai gave excellent pointers about dockerized Selenium and insisted that having video recordings is much better than simple static screenshots - and it was so true.
These days at Gizra - on client projects - we can benefit by knowing exactly how and why our browser-based tests failed. The fact that we already used Docker inside Travis helped a lot, but this additional video recording on the browser-based test makes the life of the developers much easier.
Let’s overview what’s bundled into Drupal Elm Starter, and who is responsible for what.
- Upon a push, GitHub invokes Travis to start a build, that’s just the standard for many projects on GitHub for a long time.
- Travis executes a set of shell scripts according to the build matrix. The only noteworthy thing is that using the build matrix with environment variables can be used to test the things in parallel - like one element of the matrix is the WDIO test, and another element could be any kind of Lint to scrutinize the code quality.
- From this point, we only focus on one element of the build matrix. Docker Compose launches two containers, one with the application and the test code, the other with a Selenium Grid. It also helps the containers talk to each other via expressive hostnames.
- The WDIO executes our test suites, but the Selenium host is not localhost, but rather the address of the other Docker container. This way Zalenium is able to record a video of the WDIO tests, it hosts the browser, the Selenium Grid and ffmpeg to encode the movie on-the-fly.
- Google Drive hosts the videos of the failed tests. To use Google Drive programmatically, several steps are needed, but the gdrive uploader tool has excellent documentation.
- In the very end, Gizra Robot posts a comment on the conversation thread of the pull request. Adding a robot user to GitHub is not different from adding a human - you can create a new GitHub user and dedicate it to this purpose. The exact process is documented in the repository.
You can see an example video of the test on a recent pull request. The icing on the cake is that if you receive the GitHub notification email to your GMail inbox, you can launch a video straight from there via a YouTube player!
WebdriverI/O in action
I joined Gizra three months ago, and the Gizra Way’s time-box/escalation system helped a lot to accomplish this task, where many layers of the CI stack were new to me. Needless to say, debugging Travis is hard. And then, you need to wait. And wait. A lot. Then your issue has a timebox on it, so hard things must be done quickly and by following best practices.
Seems impossible, right?
My experience is that this rigorous workflow helped me to find creative ways to solve the problems (not talking about ugly hacks here - just merely changing the way to find proper solutions), if the complexity is adequately calibrated to the developer, it triggers good stress that helps in problem solving too and contributes to the work satisfaction.
Let’s see how I was led to make it happen.
It seems to be obvious that you need to break the problem into smaller chunks, but when the testability is so problematic, you must follow this principle very strictly. In this case, the most helpful was to test the different units in the simplest environment as possible. For instance there’s a Bash script that’s responsible for the GitHub upload. Instead of launching the script via Travis or via a similar local environment, in the native local environment, just feeding the script with the proper environment variables, what Travis would do, helped to speed up the process to almost real time debuggability.
Even a small Bash construct can be extracted and tested separately. Same for a
curl invocation that posts a comment on GitHub. So in the end, I enjoyed the efficiency that came from the way of testing all the things with the minimally needed context - without all the hassle.
Invest in easy troubleshooting
It was a strong sign that we wanted to invest a significant amount to have this functionality at our project template, at Elm Starter, just to help future work. Similarly on the low level, it was mandatory at some point to be able to SSH into the Travis build. It’s enabled for private repositories, but in our case, it was mandatory to write to Travis support and this way, for our public repository, it was possible to use this functionality. It helped a lot to understand why the process behaves differently than at the local environment.
Contributing what you can
During the implementation, there were some issues with Zalenium, the side container, which provided Selenium Grid and the video recording (https://github.com/zalando/zalenium/pull/92). It got merged to upstream after 12 days, mostly the time the maintainer waited for my answer. It is just a little documentation fix, but it might save fellow developers frustration. On my side, I had the confirmation from the most capable person that I should not try to use
--abort-on-exit with that container. Such scenarios reinforces the best practice, give back what you have, either it is knowledge, a patch or a full-blown solution.
The solution that is publicly available at the repository is easy to re-use in any project that has a similar browser-based test, the only criteria is that it should support the execution on a Selenium Grid. You might capture videos of your pure Simpletest, Behat, WDIO or Nightwatch.js (and who knows what kind of other test frameworks are out there in the wild) test suite and port this code from Drupal Elm Starter to easily understand why your test fails, the only criteria is that you should be able to execute Zalenium Docker container aside. Pull requests are more than welcome to make the process more robust or even sleeker!