There are a number of posts about decoding recursive JSON in Elm, but I still found it was a slightly tricky task, particularly because the way the Elm compiler deals with recursion requires a workaround in the decoder code to avoid rather confusing runtime errors. In this post, I'd like to tell the story of implementing recursive decoders.
The path to a recursive decoder
The basic structure of the JSON we need to decode is like this (this happens to be the PostgreSQL EXPLAIN command output):...
I've written a static site generator to produce my site (yaks have to be shaved, after all). Unlike thousands of other static site generators out there, mine allows me to write pages in Elm, specifically because I wanted to be able to use style-elements and get away from CSS.
I'm using eeue56/elm-static-html to convert Elm to plain HTML, and I'm about to switch to eeue56/elm-static-html-lib (thanks to Noah for his many contributions to Elm!).
But how does it work? Normally, our views produce ...
I've seen a lot of discussion and concern around Elm development activity recently. This comment on Hacker News is a good example: "Most packages I use and Elm itself is weeks and months old, makes me worried that it's just stale". Someone else paraphrased their boss's opinion as "Looks like it's dead now, no releases in over a year".
The concern is that this sentiment either puts people off trying Elm, or it makes them consider not using Elm anymore, or it stops them from recommending Elm to ...
The main question I've seen people ask is: how do I extract a value from a union type?
Suppose you have a union type like this:
= Id Int
| Uuid String
A UserId value can contain either an Int or a String. The solution for getting to those values is simple because Elm provid...
A question I've seen quite often from people new to Elm is "How do I read types like Html Msg"? When the function signature is view : Model -> Html Msg, does view return some kind of function which takes a Msg argument? Or does it return a function that returns a Msg? Or is Html instead some kind of type decorator that extends Msg?
When you start out with Elm, this stuff can be confusing - I know I stumbled over it for a while.
To understand what's going on, a useful analogy to recall is the ...
Testing at Gizra
We have covered the subject of automatic tests quite a lot here at Gizra and I’m sure that there are a lot of posts and articles covering the subject. However, I’m having a bizarre experience lately while writing new features that I had to share it with you folks. Keep in mind that I don’t write a lot of blog posts and I always feared following up on my previous smashing hit Bootstrap custom breakpoint :)
So the experience I want to talk about was that of insecurity, which was quite odd for me considering I’m always proud of my code until @amitaibu reviews it and smashes my ego to pieces (which I learned to love).
Chances are that you already use Travis or another cool CI to execute your tests, and everyone politely waits for the CI checks before even thinking about merging, right? More likely, waiting your turn becomes a pain and you click on the merge: it’s a trivial change and you need it now. If this happens often, then it’s the responsibility of those who worked on those scripts that Travis crunches to make some changes. There are some trivial and not so trivial options to make the team always be willing to wait for the completion.
This blog post is for you if you have a project with Travis integration, and you’d like to maintain and optimize it, or just curious what’s possible. Users of other CI tools, keep reading, many areas may apply in your case too.
Unlike other performance optimization areas, doing before-after benchmarks is not so crucial, as Travis mostly collects the data, you just have to make sure to do the math and present the numbers proudly.
To start, if your
.travis.yml lacks the
cache: directive, then you might start in the easiest place: caching dependencies. For a Drupal-based project, it’s a good idea to think about caching all the modules and libraries that must be downloaded to build the project (it uses a buildsystem, doesn’t it?). So even a variant of:
or for Drush
It’s explained well in the verbose documentation at Travis-ci.com. Before your script is executed, Travis populates the cache directories automatically from a successful previous build. If your project has only a few packages, it won’t help much, and actually it can make things even slower. What’s critical is that we need to cache slow-to-generate, easy-to-download materials. Caching a large ZIP file would not make sense for example, caching many small ones from multiple origin servers would be more beneficial.
From this point, you could just read the standard documentation instead of this blog post, but we also have icing on the cake for you. A Drupal installation can take several minutes, initializing all the modules, executing the logic of the install profile and so on. Travis is kind enough to provide a bird’s-eye view on what eats up build time:
Execution speed measurements built in the log
Mind the bottleneck when making a decision on what to cache and how.
For us, it means cache of the installed, initialized Drupal database and the full document root. Cache invalidation is hard, we can’t change that, but it turned out to be a good compromise between complexity and execution speed gain, check our examples:
Do your homework and cache what’s the most resource-consuming to generate, SQL database, built source code or compiled binary, Travis is here to assist with that.
There are two reasons to pay attention to software versions.
Use Pre-installed Versions
Travis uses containers of different distributions, let’s say you use
trusty, the default one these days, then if you choose PHP 7.0.7, it’s pre-installled, in case of 7.1, it’s needed to fetch separately and that takes time for every single build. When you have production constraints, that’s almost certainly more important to match, but in some cases, using the pre-installed version can speed things up.
And moreover, let’s say you prefer MariaDB over MySQL, then do not
sudo and start to install it with the package manager, as there is the add-on system to make it available. The same goes for Google Chrome, and so on.
Stick to what’s inside the image already if you can. Exploit that possibility of what Travis can fetch via the YML definition!
Use the Latest and (or) Greatest
If you ever read an article about the performance gain from migrating to PHP 7, you sense the importance of selecting the versions carefully. If your build is PHP-execution heavy, fetching PHP 7.2 (it’s another leap, but mind the backward incompatibilities) could totally make sense and it’s as easy as can be after making your code compatible:
Almost certainly, a similar thing could be written about Node.js, or relational databases, etc. If you know what’s the bottleneck in your build and find the best performing versions – newer or older – it will improve your speed. Does that conflict with the previous point about pre-installed versions? Not really, just measure which one helps your build the most!
Make it Parallel
When a Travis job is running, 2 cores and 4 GBytes of RAM is available – that’s something to rely on! Downloading packages should happen in parallel.
gulp and other tools like that might use it out of the box: check your parameters and configfiles. However, on the higher level, let’s say you’d like to execute a unit test and a browser-based test, as well. You can ask Travis to spin up two (or more) containers concurrently. In the first, you can install the unit testing dependencies and execute it; then the second one can take care of only the functional test. We have a fine-grained example of this approach in our Drupal-Elm Starter, where 7 containers are used for various testing and linting. In addition to the great execution speed reduction, the benefit is that the result is also more fine-grained, instead of having a single boolean value, just by checking the build, you have an overview what can be broken.
All in all, it’s a warm fuzzy feeling that Travis is happy to create so many containers for your humble project:
If it's independent, no need to serialize the execution
The available memory is currently between 4 and 7.5 GBytes , depending on the configuration, and it should be used as much as possible. One example could be to move the database main working directory to a memory-based filesystem. For many simpler projects, that’s absolutely doable and at least for Drupal, a solid speedup. Needless to say, we have an example and on client projects, we saw 15-30% improvement at SimpleTest execution. For traditional RMDBS, you can give it a try. If your DB cannot fit in memory, you can still ask InnoDB to fill memory.
Think about your use case – even moving the whole document root there could be legitimate. Also if you need to compile a source code, doing it there makes sense as well.
Build Your Own Docker Image
If your project is really exotic or a legacy one, it potentially makes sense to maintain your own Docker image and then download and execute it in Travis. We did it in the past and then converted. Maintaining your image means recurring effort, fighting with outdated versions, unavailable dependencies, that’s what to expect. Still, even it could be a type of performance optimization if you have lots of software dependencies that are hard to install on the current Travis container images.
+1 - Debug with Ease
To work on various improvements in the Travis integration for your projects, it’s a must to spot issues quickly. What worked on localhost, might or might not work on Travis – and you should know the root cause quickly.
In the past, we propagated video recording, now I’d recommend something else. You have a web application, for all the backend errors, there’s a tool to access the logs, at Drupal, you can use Drush. But what about the frontend? Headless Chrome is neat, it has built-in debugging capability, the best of which is that you can break out of the box using Ngrok. Without any X11 forwarding (which is not available) or a local hack to try to mimic Travis, you can play with your app running in the Travis environment. All you need to do is to execute a Debug build, execute the installation part (
travis_run_before_script), start Headless Chrome (
google-chrome --headless --remote-debugging-port=9222), download Ngrok, start a tunnel (
ngrok http 9222), visit the exposed URL from your local Chrome and have fun with inspection, debugger console, and more.
Working on such improvements has benefits of many kinds. The entire development team can enjoy the shorter queues and faster merges, and you can go ahead and apply part of the enhancements to your local environment, especially if you dig deep into database performance optimization and make the things parallel. And even more, clients love to hear that you are going to speed up their sites, as this mindset should be also used at production.
Elm’s type system is sufficiently sophisticated that you’ll often want to make
fine-grained distinctions between roughly similar types. In a
recent project, for instance, we ended up
with a separate type for a
Mother and a
Child.1 Now, a
is a little different than a
Child. Most obviously, mothers have children,
whereas (at least, in our data model) children do not. So, it was nice for them
to be separate types. In fact, there were certain operations which could be done
Mother but not a
Child (and vice versa). So it was nice to be able to
enforce that at the type level.
Mother and a
Child clearly have a lot in common as well. For this
reason, it sometimes felt natural to write functions that could accept either. So, in
those cases, it was a little awkward for them to be separate types. Something
was needed to express a relationship between the two types.
What alternatives are available to do this sort of thing in Elm? Which did we
end up choosing? For answers to these questions, and more, read on!
There were no fathers in our app’s data model. ↩
I'm working on a tool that handles PostgreSQL EXPLAIN output in JSON format.
The data consists of a tree of nodes representing different parts of a query execution plan:
Sort on zone_idHash JoinSeq Scan on zonesHashSeq Scan on projects
Each node has a lot of attributes (more than 10), but with a significant portion of attributes common to all nodes.
The large number of attributes led me to use the Json.Decode.Pipeline package because it makes them easier to handle.
First attempt: universal...
Once you start writing apps (and packages) in Elm, it’s
Yet when implementing features for paying clients, it doesn’t always make sense
them in pure Elm. In fact, sometimes it isn’t even possible!
necessary – ports!
Yet ports aren’t always the right answer, and there are several alternatives
which can be useful in certain situations.
For the purposes of this post, I’m going to assume that you’re familiar with
the many cases in which ports work well, and focus instead on a few cases where
you might want to try something else:
- When you want synchronous answers.
- When you need some context when you get the answer.
I tell my kids all the time that they can’t have both - whether it’s ice cream and cake or pizza and donuts - and they don’t like it. It’s because kids are uncorrupted, and their view of the world is pretty straightforward - usually characterized by a simple question: why not?
And so it goes with web projects:
Stakeholder: I want it to be like [insert billion dollar company]’s site where the options refresh as the user makes choices.
Me: [Thinks to self, “Do you know how many millions of dollars went into that?”] Hmm, well, it’s complicated…
Stakeholder: What do you mean? I’ve seen it in a few places [names other billion dollar companies].
Me: [Gosh, you know, you’re right] Well, I mean, that’s a pretty sophisticated application, and well, your current site is Drupal, and well, Drupal is in fact really great for decoupled solutions, but generally we’d want to redo the whole architecture… and that’s kind of a total rebuild…
Stakeholder: [eyes glazed over] Yeah, we don’t want to do that.
But there’s is a way.
If you happen to know Brice - my colleague and Gizra’s CEO - you probably have picked up that he doesn’t get rattled too easily. While I find myself developing extremely annoying ticks during stressful situations, Brice is a role model for stoicism.
Combine that with the fact that he knows I dislike speaking on the phone, let alone at 6:53pm, almost two hours after my work day is over, you’d probably understand why I was surprised to get a call from him. “Surprised” as in, immediately getting a stomach ache.
The day I got that call from him was a Sale day. You see, we have this product we’ve developed called ״Circuit Auction״, which allows auction houses to manage their catalog and run live, real-time, auction sales - the “Going once, Going twice” type.
- “Listen Bruce,” (that’s how I call him) “I’m on my way to working out. Did something crash?”
I don’t always think that the worst has happened, but you did just read the background.
I was expecting a long pause. In a way, I think he kind of enjoys those moments, where he knows I don’t know if it’s good or bad news. In a way, I think I actually do somehow enjoy them myself. But instead he said, “Are you next to a computer?”
- “No. I’m in the car. Should I turn back? What happened?”
I really hate to do this, but in order for his next sentence to make sense I have to go back exactly 95 years, to 1922 Tokyo, Japan.
Last week, we covered how to break down decoders by starting from the innermost, or topmost, part.
But what if you’re having trouble breaking things down from the top?
(Or you’re dealing with a really complex JSON schema?)
This week, let’s look at it from a different perspective: the outermost structure in (or the bottom up!)
A reader of the JSON Survival Kit wrote me with a question (lightly edited):
I really can’t figure out how to parse this–will your book help with nested JSON where the keys are different 2 or 3 levels deep?
If not, then I’ll just give up on Elm–as this is the first project that I’m trying to do, and something as basic as this, I’m finding impossible.
The biggest mindset shift you need to succeed with JSON Decoding is to think of your decoders like bricks.
(I’ve written about this before, and it’s chapter 1 of The JSON Survival Kit.)
You can combine bricks to build whatever you like; the same is true of decoders!