Announcing mono-traversable, chunked-data, and conduit-combinators
February 16, 2014
I'm happy to announce the release of a number of packages today, in particular:
- mono-traversable 0.3
- chunked-data 0.1 (first release)
- conduit-combinators 0.1 (first release)
- classy-prelude, classy-prelude-conduit, and classy-prelude-yesod 0.8
I want to discuss the purpose of these packages and recent changes. I'm very excited about some of the opportunities these packages are presenting for future growth and use.
mono-traversable's core concept is abstracting over a common pattern: data
structures that look like containers, but which may be monomorphic. The primary
example of this is
Text, though this abstraction has proven
useful for other cases as well, such as unboxed and storable
some typeclass constraint limits the allowed value type.
In my experience so far, the most powerful abstraction has been
MonoTraversable, you don't end up losing any
flexibility in data types, since we do not need to retain the original data
structure. As such, I've switched over to using MonoFoldable variants of
functions in place of Foldable ones in all of my new code.
In addition to this core abstraction, mono-traversable provides abstractions
Maps, non-empty data types, and sequences. This last point can
serve as a replacement for a number of list-like abstractions which have been
implemented separately over the years.
Since the 0.2 release, Greg and I have worked on the following improvements:
- Additional instances for Either (thanks to João Cristóvão).
- An optimized
- Greatly improved test coverage.
- Remove a number of type classes. As an example, we used to have a separate typeclass for non-empty types which had elements which could be ordered, which allowed for a total and optimized implementation of
minimum. Instead, in version 0.3,
maximumExfunction, which can be optimized for an individual datatype, but may throw an exception, and then any non-empty data type may use
maximumas a total function. We've done similar things for
- Added unsafe functions like
unsafeHeadwhen you need more performance (thanks to John Lato for the idea).
- Greatly expanded the collection of
- A new
Data.MinLenmodule (still experimental), which extends the concept of non-null. Instead of simply encoding size as a boolean, we use type-level naturals to encode the minimum known length of a value. This allows, for example, such total functions as
head $ tail $ mlappend listLen1 listLen1. Greg and I have decided to release this initially as an experimental module and, if it works well, replace the current
The core functionality of
mono-traversable is pretty stable, and I highly
encourage people to give it a shot.
mono-traversable started as extracting some of the typeclass-based
functionality from classy-prelude in a principled, law-abiding manner. About
half of that functionality fell under the rubrik of monomorphic containers. The
other half was an abstraction of different kinds of chunked data: reading a
chunk of data from a
Handle, textual encoding/decoding, lazy sequences,
builders, and zipping.
chunked-data contains the remainder of this
This package should be considered somewhat experimental. However, most of the code is simply extracted from classy-prelude, where it's already had quite a bit of testing. So those of you who are somewhat adventurous should definitely jump in and play with it now. If you're more conservative, give in another month or so before experimenting.
This is the release I'm most excited about. There's quite a lot going on here, and I'll probably write a separate blog post going into the details of this package. But here's the basic idea.
Until now, the primary combinator collection for conduit was the
Data.Conduit.List module. This module contains many commonly used functions,
isolate. However, there are two things I'm unhappy with in
the current module:
The API only works on non-chunked data. For chunked data (like Text and ByteString), you need to use Data.Conduit.Text and Data.Conduit.Binary, respectively. John Lato and I have discussed this a bit in the past, and he made a very convincing argument to me that providing a unified chunked data API is superior.
The naming scheme of Data.Conduit.List does not always encourage best practices. For example,
takeconsumes all incoming values into memory. This was a decision inherited from enumerator, and made sense back when the library was first created. But common usage has changed.
This doesn't mean that there's anything broken in Data.Conduit.List, and therefore I have no intention of breaking backwards compatibility by changing the functions there. However, I think it's time to introduce a new, more modern module providing even more combinators, chunked variations, and a more consistent naming scheme.
That's where conduit-combinators comes into play. As a quick rundown:
- provides generalized versions of all (unless I missed something) functions from Data.Conduit.List, Data.Conduit.Binary, and Data.Conduit.Text
- The package makes heavy use of mono-traversable and chunked-data to let this happen, which is what originally instigated my work on those two packages in this development cycle.
- since it's higher on the dependency chain, it can depend on packages like vector, and provide specific functions for it (like
- adds a lot of missing functions (like
- exports a module intended for qualified import:
- exports a new module,
Conduit, which provides a one-stop-shop for all commonly needed conduit functionality. Combinator names are munged by appending a
I know many people out there are quite happy with qualified imports, but for
those of you like me who enjoy writing a short import list and then having all
of the functionality you need available, I think the new
Conduit module will
be a real pleasure.
Since I've been giving stability forecasts, I'll do the same thing here. For
the basic functions like
sinkList, it's highly unlikely that there
will ever be a change. However, as I get feedback on the library, I'm likely
to make some tweaks to behavior of other functions. One example: there are two
sets of behavior that the
drop function could reasonably have:
- Drop the given number of items, and let the next monadically composed component consume the rest. In this case,
dropwould be a
- Drop the given number of items, and then yield the rest of the items downstream. In this case,
dropwould be a
I've chosen the former, but I'm open to discussion on the topic.
So if you're using conduit in any significant way in your codebase, it's definitely worth checking out this new package. I wouldn't port large codebases over to use it yet unless you're okay needing to refactor again in the next month or so. But I anticipate this package reaching a stable point in the very near future (less than 3 months from now), as it will be a vital component of some work I'm doing at FP Complete.
One nice fact about this library: it is currently sitting at 100% HPC test coverage, and I intend to maintain that level going forward.
The most exciting thing about the 0.8 release of classy-prelude is what's not there: any significant code! After the work I've mentioned above, classy-prelude (and -conduit and -yesod) has now become a simple re-export module for functionality defined elsewhere. This is great: the core generalizations have been made available for wider usage, and classy-prelude is merely a convenience for getting at all of that functionality at once.
Since classy-prelude is based on the packages above, the same stability guidelines apply. In particular, classy-prelude-conduit may see quite a bit of turbulence as conduit-combinators changes. And if you used the chunked-data or Data.MinLen re-exports from ClassyPrelude, those may be updated in the future. But the vast majority of classy-prelude has remained stable for the past number of versions (accounting for well over a year), and will continue to do so in the future.
One question that I've raised and am looking for feedback on: should ClassyPrelude export mtl data types and functions? I opened a Github issue for discussion, and am interested if others have any thoughts on the matter.
Look forward to some blog posts demonstrating usage of these libraries over the next few weeks. And if you have any recommendations for improvement, please bring them up!