Giving up on Ruby packaging

I have finally reached a decision regarding my involvement in the Debian Ruby packaging efforts. I have decided to stop. This has been a very hard decision to make. I have invested huge amounts of time in that work over the years. I still love the language, and will continue to use it on a daily basis for my own developments. I still hope that it will succeed. I know that some people will be disappointed by that decision (and that others will think “your work was useless anyway, people should use RVM and rubygems”).

But I also know that I won’t be able to push for all the required changes alone. I just don’t have the time, nor the motivation. For the record, here are the changes I would have liked to see in the Ruby community.

The core Ruby development community should mature.
The core Ruby development community is still dominated by Japanese developers. While not a bad thing in itself, it is easily explained by the fact that the main development mailing list, where most of the important decisions are taken, is in japanese. ruby-dev@ should be closed, and all the technical discussions should happen on the english-speaking ruby-core@ list instead.

The release management process should also improve. Currently, it looks like a total mess. The following Ruby development branches are actively maintained:
ruby_1_8 (106 commits over the last six months)
ruby_1_8_6 (4 commits over the last six months)
ruby_1_8_7 (35 commits over the last six months)
ruby_1_9_1 (4 commits over the last six months)
ruby_1_9_2 (227 commits over the last six months)
trunk (1543 commits over the last six months)

While the state of the ruby_1_8_6 and ruby_1_9_1 branches is clear (very important bugfixes only), the state of all of the other branches is rather unclear.
What’s the stable Ruby branch? 1.8 or 1.9? If it’s 1.9, why are people still actively developing in the ruby_1_8 branch? How long will they continue to be maintained in parallel, dividing the manpower? Is a Ruby 1.8.8 release to be expected? Will it be ABI/API compatible with 1.8.7? Is the ruby_1_8_7 branch really bugfixes-only? How much testing of it has been done? If it’s bugfixes-only and regression-free, I should push it to Debian squeeze, due to be released in a few weeks. But would you recommend that? Due to past breakages in the ruby_1_8_7 branch, it’s unlikely that we will do it.
Is the ruby_1_9_2 a regression-free, bugfix-only branch? If yes, isn’t 227 commits over 6 months a lot? What will be the version of the next release of “trunk”? When is it expected? Will it be ABI-compatible with the current ruby_1_9_2 branch? API-compatible?
New releases in the 1.8.7 and 1.9.2 branches were done on december 25th. Why were they no betas or RCs allowing wider testing? How much testing has been done behind the scenes?

Most of those questions have no clear answer. The Ruby development community should build a common understanding of the status of the various branches, and of their release expectations. Releasing on december 25th of each year sounds fun, but is releasing when everybody is on vacation really a good idea?

It would be fantastic to have something similar to Python Enhancement Proposals in the Ruby community. But having open discussions in english about the major issues would already be great.

Ruby is not just the interpreter.
The Ruby development community should clearly define what the Ruby platform is. There are some big players, like Rails, and newer interpreter releases should not be done before ensuring that those big players still work.

Also, since we have alternative Ruby interpreters, like JRuby, Rubinius and MacRuby, we need a clear process on how they integrate with the rest of the ecosystem. For example, having each of them rely on their own outdated fork of the whole stdlib is ridiculous, since it’s not where they compete.

The Ruby community should acknowledge that RVM and Rubygems are not for everybody. People who say so should be laughed at. Of course, RVM and Rubygems are nice tools for some people. But it is completely wrong to believe that compiling from source using RVM should be the standard way of installing Ruby, or that all people interested in installing Redmine should know that Ruby has its own specific packaging system. The Ruby community should work with their target platforms to improve how Ruby is distributed instead of reinventing the wheel. That includes Debian, but also RedHat-based distros, for example. It is likely that it won’t be possible to reach a one-size-fits-all situation. But that’s real life.

Some people in the Ruby community should stop behaving like assholes. As one of the Debian Ruby maintainers, I have been routinely accused of creating crippled packages on purpose (FTR, I don’t think that the Debian packages are crippled, despite what the rumors says). Debian is not the only target of that. Just yesterday, someone called for abandonning YARV (the new Ruby VM in Ruby 1.9), calling it Yet Another Random Vailure. This kind of comments is really hurting the people who are investing their free time in Ruby, and is turning away people who consider getting involved. In Debian, we have had a lot of problems getting people to help with Ruby maintenance since they are getting shit from the community all the time.

So, what’s the future for Ruby in Debian?

  • For the interpreter, the two other maintainers, Akira Yamada and Daigo Moriwaki, are of course free to continue their work, and I wish them good luck.
  • For the pkg-ruby-extras team, which maintains most of the Ruby libraries and applications, the future is less clear. The team was already badly understaffed, and I feel that I should probably clean up its status by orphaning/removing the packages that are unmaintained otherwise. This won’t affect all the packages that the team maintains: some packages (like redmine) are actively maintained, and will stay.
  • For me, it just means I will have more time for other things (possibly not Free Software-related). If things improve dramatically, I might also come back to Ruby packaging at some point.

Update: there’s also a number of interesting comments about this post on this site.
Update 2: First, thanks a lot for all the interesting comments. I will make some follow-up posts trying to summarize what was said. It seems that this post also triggered some reactions on ruby-core@, with Charles Olivier explaining the JRuby stdlib fork, and Yui Naruse clarifying that all questions are welcomed on ruby-core@. This is great, really.

82 thoughts on “Giving up on Ruby packaging

  1. I think it is good, that the ruby-dev is still mainly in Japanese. Keeps Ruby clean. Not everybody has to learn English. Why not just learn Japanese?

  2. I am one of the original authors of Rubygems who committed to the cvs repository on Jim Weirich’s usb keychain at the back of the bar in the middle of rubyconf. We didn’t think of what we were doing as reinventing the wheel; one of the major problems with existing packaging systems at the time was distribution to multiple platforms. There existed no user-friendly way for one of the popular packaging systems to be deployed on a platform that used a competing system (e.g. deploying a slack tarball on redhat or vice versa). In particular, we were concerned with Windows users, who don’t even know what a deb or rpm is.

    We wanted developers to be able to publish their Ruby software without having to repackage the software for each target platform. We wanted the publishing process to be easy, to encourage developers to build new and interesting software. And we wanted to make the software accessible to users, without an intermediate packager or approver, since that would mean each piece of software must reach critical mass before it can enter the ecosystem. In these things I think we succeeded.

    There are a number of areas in which we failed. From the beginning, Matz wanted Rubygems to allow a system administrator to install and uninstall packages using the tools with which he is familiar. Thus a gem should be easily convertible to a deb or rpm with minimal effort. Given the widespread usage of tools such as cpan2rpm, we expected that this would be easily solvable; we did not anticipate that this would be as difficult a problem as it has proven to be.

    I’m also afraid that I am responsible for the design which caused heated discussion over whether rubygems conforms to the LFS. We wanted to build a working prototype in two hours. This meant we could spend time figuring out where to dump stuff, how to convince the interpreter to look for scripts/extensions there, how to save those install locations so that packages can be uninstalled, how to handle failure in the middle of the install, etc. Alternatively, we could dump everything in a single directory and the uninstall operation is simply removal of that directory (much like many admins treat /opt). I felt this was a reasonable way to get something up and running in a short amount of time, so I argued for it, and after some discussion, the group finally agreed. I think we made the right decision at the time, but without a packaging expert present, it was impossible to foresee the long-term consequences.

    I still feel that a gem2deb or gem2rpm tool is feasible, but as I am no longer an active ruby developer, I cannot say specifically what is involved here. At a minimum I suspect it would require 1) adding metadata to the gemspec required by each target packaging system, and 2) building a mapping of package names so that package requirements are handled properly (e.g. does the ruby extension depend on sdl-dev, libsdl-dev, sdl-devel, or libsdl-devel? different platforms unfortunately name their packages very differently).

    Lastly, please do not judge the behavior of the entire Ruby community solely on the actions of Ryan Davis (zenspider). Same goes for Austin Ziegler. These two people are well known for being harsh and blunt. The rest of us really are quite pleasant.

    I think you’ll find that any online community of sufficient size has a couple of people like this, and the only way to deal with it is to either live with it or be exclusive. (the problem becomes more difficult when the people involved aren’t just trolls, but instead do positively contribute to the community, as Ryan and Austin do).

    In particular, some of _your_ comments come across a little harsh as well. You also come across as somewhat uninformed (have you watched Matz’s keynote? did you know that he plans to submit Ruby to ISO?). Instead of judging the community based on online behavior, come to one of the regional or international conferences and see what the community is really about. I think you’ll be pleasantly surprised.

    (If you have been to one of the conferences, then I apologize, and I regret that you and I have not met. Perhaps in the future?)

  3. To several people who commented about submitting patches to Rubygems:

    it’s a bit naive to think that you can fix everything with patches. Yes, there
    are problems that can be fixed with patches. But fixing fundamental design
    decisions with patches is often not possible.
    Regarding the fact that gem update –system is disabled with Debian, I was not
    a big fan of this decision (which I did not originally make), and I actually
    reverted that to a compromise solution a few months ago: now, it’s still
    disabled by default, but you can override it if you set an environment
    variable.

    @Paul Brannan:
    I agree that I’m probably a bit misinformed. I’ve never been to a Ruby
    conference (the fact that I’m doing all my Ruby work as a volunteer, and that
    most of the conferences require travel and registration, makes this a bit
    difficult).

    Regarding the online community, I think that it’s important to stand up when
    facing offensive behaviour, because a minority will destroy the image of the
    whole community. We used to have a difficult community in Debian, and it has
    improved a lot over the recent years.

  4. I tried to comment this on the other site, but it wouldn’t let me use LP OpenID:

    I’m half a sysadmin and half an (inhouse) developer at my
    current workplace. We use FusionForge (aka GForge aka Source-
    forge) there, which is written in PHP. It has taken me quite
    a while to get us moved from some version put there and then
    updated / hacked on in-place to a _packaged_ version (from a
    local repository), but I don’t regret it and, while the
    cow-orkers hate the overhead and additional work, they see
    the benefit. You have the same version (not version*s*!) on
    all instances/systems, can easily upgrade stuff (to fix that
    was also quite some work), etc.

    Furthermore, if you use a packaged version and dist-upgrade
    regularily, you get security upgrades automatically. Espe-
    cially for “web development”, you DO WANT that. So, everyone
    who installs NOT from a distro packaging on a production sy-
    stem is bringing himself into risk and his own fault.

    I can only say, guess we’re lucky to *not* have switched to
    Redmine some time ago…

  5. Interesting read. An aside on the mercurial comment – for what it’s worth it’s starting to get decent traction in the .Net world due to it’s simplicity of set-up on windows

  6. @Np237: You’re quite right, it’s problematic to depend on different versions of the same library. The whole point of libraries is to share them, preserving memory (or at least they should) and reducing the maintenance burden (security updates, etc).

    However, the real world has a tendency of messing up the best laid plans. If you have gems or libraries that are used by a lot of projects, chances are some projects will be faster in moving on to the newest (major) releases than others. This happens not just in the ruby world. I seem to remember with the transition from libgtk1.2 to libgtk2.0 that there were some stragglers for years (GnuCash was it?). It looks like the same will happen with libgtk2.0 -> libgtk3.0. So Debian plays games (don’t mean this in a bad way) with package names, and enables parallel installs of libraries already.

    It seems that with dynamic languages like Python and Ruby, there’s less discipline with respect to APIs, in that things evolve more quickly. But I’m hoping (and I think, starting to see) that these developer communities will soon be using better version numbering (semver.org), so at least we know what to expect.

    That being said, I still think multiple versions will always be needed. Ruby or not, there’s always transitions going on between major versions of libraries.

    Try installing redmine on lenny with lenny-backports right now for instance (you can’t: redmine needs rails 2.2, lenny has 2.1 and lenny-backports has 2.3 ). And that’s with just one rails project in the pool. I’ve got 22 rails projects running on my servers right now (without rvm, just debian ruby+rubgems), and they need rails 2.1.x, 2.2.x, 2.3.x, and 3.0.x. Forward porting only happens within minor or patch versions (short term that means everything to 2.3.x and 3.0.x). Even a maintained project like Redmine won’t move to 3.0.x anytime soon.

    It’s a shame you seem to be arguing that multiple versions shouldn’t happen at all. Clearly it’s already happening in Debian (even with C or C++ libs), even more so in the Ruby community. If dpkg had better support for it, we would probably have easier Debian release management (there’s a lot of flag days now where everything needs to be re-uploaded at the same time), and be able to support Ruby better. I agree multiple versions should still be discouraged, but having dpkg support is better than pretending it doesn’t happen at all.

  7. @Lucas: you should make a proposal for a talk! Presenters get a reduced rate for registration. Regional conferences often waive the fee altogether (and I would imagine travel would be cheaper to a regional conference).

    @mirabilos: different production systems have different requirements. In our systems we are less concerned about security holes and bugs than we are about subtle changes in behavior. Even a minor revision change involves risk, which means testing before deployment, which costs time money. For that reason we don’t upgrade _anything_ except when necessary.

    Our use case isn’t common, and you make a good point that many users _do_ want the automatic updates that the system packaging gives them. Unfortunately no distribution supports every conceivable piece of software out there;
    what should a sysadmin do if he wants an update for software that his distribution doesn’t support?

    (it’s a difficult problem which IMO lies partially outside the scope of either debian or rubygems)

  8. @nona, the comparison to GTK+ is so irrelevant that it makes you look silly. You cannot compare a library which has not changed its API/ABI for 8 years to modules that change it at every single minor release. Especially when the transition to GTK+ 3.0 was well coordinated and the library was designed from the start for parallel installations, with everything in the build tools being able to pick the right one. If you want a comparison in the C/C++ world, pick up the Boost nightmare. Yet even with Boost, we manage to have only one version in the distribution at the same time.

    You are right that these issues could be fixed by proper API and ABI versioning, but the best way we know to do that is the build system: with pkg-config, you can pick the correct API version and pass the build flags relevant to the version being used in the source. With proper soname versioning, you can have the binary pick the correct library and know when it won’t be able to find it. The same goes for .Net with its GAC. Unfortunately, in Python and Ruby, there is no build process. And what is a strength when you want to do a script or prototype an application becomes a weakness when you build a large scale project. Whenever the interface of a dependency changes, you just don’t know it, and you have to wait for the crash (which happens at runtime, of course).

  9. @nona: You don’t seem to understand a difference between API and behavior change for major number change and minor one, do you?

  10. It’s sad to see this kind of post as #62, which is written by a person whose mother language is not one of Indo-European languages since he/she forgot to put “a” in front of the last word. I guess he/she did not distinguish Japanese language from Japanese people.

    Thanks Lucas, and other contributors who make this world happier, at least in my opinion.

    As for the language issue, yes, it’s pretty tough to switch my brain-working-mode and write in English while talking in Japanese and the same is true for writing in Japanese while working in English-speaking environment. When I had been in the US working in English-speaking environment, the first 10-20 minutes of my conversation with my wife in Japanese was really tough while the first 10-20 minutes of my conversation with my close friends at work was really tough. The switching of my brain is the core of the problem as for the language, not languages themselves. The problem is not that tough when I was in China where I needed to speak Chinese languages, whether Mandarin or Cantonese (many words have been imported to Japanese, though grammar is totally different). The same is true for my days (without my wife) in France and Germany: I had very little difficulty in speaking English at work while speaking French and German out of work.

    Recently, working in Japan, I receive and reply many e-mails, 30% in Japanese and 60% in English. But I make it a rule not to use both languages at the same time (without a separation of an hour), in order to avoid unfruitful hardwork of switching brain-working-mode. I read and reply all the e-mails in Japanese by 3pm and do so for all the e-mails in English after 3pm.

    Since I’m a scholar working in world-wide communities as well as an educator working in a university in Japan, this is inevitable in my case. No way. I’ve accepted the fact that English sentences look as if they were written in Devanagari when I happen to read e-mails in English.

    But, I can see no point doing like me for Ruby. Why core members does not separate issues into a domestic part and an international part? It would be true that reports from reporters in Japanese are valuable, and thus it should be in the domestic part while all the development should be carried out in the international part. Synchronizing both brings about not only strong headache like mine but also fruitless work (switching brain-mode).

    For that reason, I’ve been avoiding to write my own technical paper in Japanese since it’s tougher, though my Japanese is far better than my English.

    Other, more important issues for Ruby, let me just read others’ opinion. Again, thanks for your time and contributions, every bit, for happier world, at least in my opinion.

  11. @Np237: I used GTK+ as an example, because I wanted to show that even with a extremely well-maintained API/ABI, the archive still contained multiple (major) versions of GTK+. My point is that (even well-coordinated) transitions (between major versions) don’t usually happen in one flag day.

    @Anonymous is right that I glossed over the differences between what’s OK in minor and major releases, and certainly the stuff that’s been going on in even _patch_ releases of Rails is so not OK; but that’s why I mentioned semver.org, and my hope that the Ruby guys are starting to realize the importance of proper versioning, even if we end up with rails 5.0.0 in a very short time.

    If I understand correctly, you guys seem to be arguing that Debian would need separate packages for major versions (so f.e. a rails2 and rails3 package – not the case right now), and count on the developers to be disciplined enough to use proper versioning, ie. no API/ABI or behaviour changes in minor versions. Something that as @Np237 said is harder to verify with dynamic languages where everything happens at runtime. But if developers can’t stick to these rules, others shouldn’t be using their lib.

    I guess I’d personally want to be more accomodating to users and developers wanting to use something despite API/behaviour “accidents” – which are harder to avoid in Ruby and Python. And I’d also like to avoid namespace pollution of the archive -something Debian historically doesn’t seem to worry too much about for major versions.

  12. Also – you can cut the problem down in Debian by focussing on *applications* rather than *libraries*. Many Ruby developers are never intending to publish their apps, but want to sell access to them instead – Debian shouldn’t be jumping through more hoops just to provide a production environment for closed software. Especially when those hoops are pretty much impossible within its conservative packaging framework.

    So the next version of Debian would only have Rails 2.3, or whatever – not a problem. Rails at least *is* stabilising and after another release or two, the old libraries will be good enough for lots of applications.

    I just don’t think Debian should, or can, worry about the problems of applications that reference their dependencies through github version tags! Those applications are just not ready for out-of-the-box deployment on any long-term OS release. They will always need another, riskier layer of systems administration on top, without the kinds of guarantees that Debian’s security team can provide.

  13. Regarding the ability of ruby core developpers to speak english, dont forget that *all* commit messages to the ruby interpreter are in english

  14. I posted my description of JRuby’s stdlib maintenance policy before I saw your posts. I just wanted to preempt any questions about why JRuby now ships the Ruby 1.9 singleton.rb for Ruby 1.8 mode.

    I agree with most of your comments in principal. On the JRuby project, we have released dozens of releases over the past few years, always forcing major version numbers to their own branches and not making breaking changes (or even drastic feature additions) on those branches. We’ve been very careful about updating stdlib and other libraries, and we make great efforts to ensure we’re as backward-compatible as possible even across major releases.

    In my opinion, what’s missing most from ruby core is a feeling of responsibility to keep everything working on all platforms with *every single commit*. On the JRuby project, we run as many as 5 different test suites on every commit, and another couple every night across various configuration settings. We run tests on OS X, Linux, and Windows on every commit. And we consider it a point of pride that JRuby runs consistently across so many platforms.

    JRuby runs more tests than any other Ruby implementation. JRuby runs more CI builds on more platforms than any other implementation. I think we’re doing it right.

    Of course I’d also like to see “ruby-core” include other implementations by default, since decisions for “C Ruby” have drastic effects on JRuby and friends. C Ruby is now just one implementation among many, even if it is the “reference implementation”, and as a result there needs to be more consideration for how the future of Ruby will affect and be represented by other implementations.

  15. The whole world deals with legacy applications and static libraries in production. At the same time creates and develops new applications. It is necessary to mature the way of versioning in Ruby community of course, but to say “I’m tired” and go away does not help.

    I’m a brazilian programmer. I develop applications for two servers, one with Debian Stable and some other Red Hat much, much older version. Thanks to the RVM and Rubygems I can combine both jobs in one development environment.

    No matter who is responsible for decisions on the production environment. What really matters is that at some time are taken, the developer must take this into account but should not limit its development environment for it.

    Rubygems and RVM might not be for everyone, but all we have to adapt to circumstances. Transforming the gems on distribution-specific packages definitely is not a good idea; may be useful for gems compiled locally, but not for the entire ecosystem.

  16. I suspect that even if ruby developers only spoke English, it’d still be dominated by people from Japan. Many open source projects have network effects, where it’s more popular in one country than another.

    If you want people to be mature, please don’t use the word “asshole”.

    “FTR, I don’t think that the Debian packages are crippled, despite what the rumors says)”

    You don’t link to a comment by Matz saying they aren’t crippled, or a comment by James Earl, you link to a comment by a newbie. And newbie only says they’ve heard that there’s these problems.

    If people high up in the community are saying incorrect things, then you could claim a smear campaign. If the incorrect claims are coming from relative newbies, then it’s probably not a smear campaign, but genuine misunderstandings.

    (To be honest, I did get a little suspicious when I heard that Rubygems on Debian had problems, and that the Debian community tells you to use Debian packages rather than rubygems. It seemed to be a bit of a coincidence. I also get the impression the Debian community tends to say “Don’t use rubygems” when I’d rather them say “Sure, you can use Rubygems, but here’s some added benefits of using Debian packages”. But I’m not a expert on these things, and ruby works perfectly well on our Ubuntu box.)

    @Steven Obua:

    The difference between Indo-European languages is much less than the difference between English and Japanese.

    When I was learning a bit of Norwegian, I noticed that much of the time, it was just saying English words with a different accent. I doubt that learning Japanese will be that easy.

  17. I think you make a good point about release management, and it being really confusing to figure out what the status of various version numbers present past and future is.

    I think that’s pretty much the only good point up there, but it is a good one.

    I don’t see a problem with rubygems being an integral part of ruby, it works well.

  18. Sad to hear this lucas. But i do agree with your point, RVM & rubygems cannot be accepted as the only way of managing ruby as a platform. The release management process is also screwed i agree, after all we should have learned something from Perl/Python development, the pain of maintaining multiple branches. This also exponentially increases the effort required for maintaining stable ports ans libs.

  19. Ruby is Japan’s baby, after all. Let them grow it on their own (they haven’t really grown out of their isolationist policy in the last 500 years ;)

    As for the rest of the English-LOLing world, let them use Perl.

  20. Many of the technologies that ruby programmers are working with move very fast. It is a red herring for people to insult ruby programmers for their versioning. Even if every single gem followed perfect versioning practice, you will everyday as a ruby programmer encounter entirely valid reasons why you need three different versions of a particular gem installed at once.

    This is entirely valid and necessary for ruby programming AND deployment. Debian folks are threatened by the pace that ruby moves and feel it violates the prime directive of debian to be stable and secure. I understand that, but I still need my code to be able to specify which version gem I want to use. That is not negotiable.

    In my collective, we have three debian developers and as many ruby programmers. We love each other, but we still fight internally over how to handle ruby in debian. So, the debian/ruby blood feud has hit close to home.

    I think http://www.debgem.com/ (commercial and proprietary packaging of gems) is the sad and monstrous outcome from the unresolved feud.

    Maybe at the next debconf there can be a peace summit?

  21. Pingback: Azithromycin.
  22. I have been using and developing on the standard debian packages for many years across a boatload of machines. They are great. Sometimes I will compile from source and sometimes use rvm. As many have pointed out, these are different solutions for different problems. The great thing is that they can coexist (at least they coexist in my world). Thanks for years of hard work Lucas!

  23. On Nike Free design inspiration, Toby said: “We have found that the original grasslands in Africa and the Caribbean, the people running barefoot, their legs stronger, powerful and flexible, which makes us believe that the natural movement way make athletes stronger. “NIKE FREE family’s goal is to bring people’s feet from the traditional sports shoes fetters, use your feet to control shoes and enjoy the sport the most realistic feel. NIKE FREE series for the first time since last year designated in the world has been running supplies shop for sale, get a lot of sports fans.

Comments are closed.