Charlie, Jesse, Mitchell, Yehuda, and I recently engaged in a discussion on Twitter over whether JRuby should aim to support Ruby's C API. I think supporting the C API should be a priority because C extensions are still used (and useful) in many popular gems. Charlie made the case that the problem could be solve through FFI. Mitchell chimed in to say FFI wasn't quite ready for prime time, especially on Windows. Yehuda suggested having JRuby versions of every gem with C extensions.

So, Rubyists, what do you think? Is supporting gems with C extensions on JRuby important to you, and if so, what's the best approach do you favor?

Here are some of the salent points discussed on Twitter:

I'll make my case here...

Supporting the C API would be a reasonable idea...EXCEPT...

* The API exposes many details of MRI's internals that are either impossible to implement or implement efficiently. For example, macros that access the class/hash/array internals; string byte[] access; VM primitives...
* The C API does not support multiple runtimes in the same process, so many typical JRuby deploy scenarios completely break.
* The C API does not make any concurrency guarantees, so JRuby must prevent concurrent execution of any ext code.
* The C API does not make any object liveness guarantees, so C ext support in JRuby must burden the JVM's GC with a number of arbitrarily-lived objects, cleaned up via finalizers and other nasty tricks.

Here is a different case to be made, which we have given lip service to, but have not really went down this path....

Ultimately, MRI has a C extension API which are their internals and it also makes assumptions which suck for natively-threaded impls.

Nothing prevents us from making a new native C extension API for MRI which also works well on JRuby and Rubinius. This new API would not be mandatory but over time would get used because it would be documented, work well across implementations, and the scope would not be the entire MRI codebase. If it is a reasonable API, then it might just replace the old one by MRI.

This is not a short-term proposition, but I also think this has merit for MRI as an implementation itself. out of space

There are two ways to look at the C API. You can look at it as:
* a part of the Ruby ecosystem
* a way of interfacing with the Ruby runtime

Taking the first view: is the C API an important part of the Ruby ecosystem? Traditionally, it has been used for speed and for interfacing with large, preexisting C libraries. For the former case, we should focus on making Ruby faster. For the later case, we should improve FFI.

Taking the second view: the C API for Ruby is horrendous and has been badly abused over the years. If extending Ruby in C is important, then let's design an API with that purpose in mind. The current API was more of an accident than the result of willful intention. (mruby could be a starting point here...)

A new API could allow MRI the ability to relocate objects, which would open up an entire new world of performance. Lacking generational GC in an object mad language like Ruby is a bummer. Also, if MRI ever wants to consider native threads they will need to rewrite the C ext library anyways.

So looking at something like JNI and also working with MRI to come up with a memory model will basically solve world hunger...

My feeling is that JRuby should still make a best effort to support the C API. It seemed like JRuby was making good progress on this in 1.6 and then abandoned it with the release of 1.7.

If supporting the C API cripples JRuby performance, then there should be a mode to disable it. Basically, I believe JRuby should prioritize compatibility. Those who want better performance can opt-in by removing any gems with C extensions and disabling C API support.

More recommended reading for those unacquainted with the subject:

Tom, this is an interesting possibility. How would what you're proposing compare to FFI?

Erik, FFI/DL/Fiddle are runtime bindings to a DLL. Those have some place (and I wish they filled the entire problem space), but they have the issue of exploding when something like a struct has an extra field in it (which happens when a call is made). Native C extensions have a compile phase which might still blow up but it will happen when you compile the extension. Many people prefer this static binding nature over the dynamic binding of FFI (is this ironic?). [Note: Don't confuse this with cext just dumping core...they can do this regardless of how they are bound]

Native C extension APIs also give extra bidirectional access to the runtime itself. So you can alloc an object from C or lock something so that it won't get collected.

I think it's actually a good thing that JRuby decided to shift focus away from slavishly copying MRI 100% (even the bad parts).

With MacRuby we also started out thinking that 100% MRI compatibility (and eventually replacing MRI on OS X) was the goal. Now, it seems like 95% or 98% compatibility, with a strong focus on building OS X desktop apps, would be a better goal.

Likewise, I'd rather rather see JRuby focus on the 98% case and make server-based app performance amazing. Developer man-hours are a limited resource and we already have MRI if you need a 100% MRI-compatible implementation of Ruby.

Tom, would it be possible to define an alternative C API in a gem? If so, would the gem itself need to use C extensions? If not, have you discussed this option with the MRI team?

I would like to see JRuby support the current C API but not stop there. I agree that the current C API is a mess and needs reworking. I think JRuby is on the forefront of multi-threaded Ruby and is in an excellent position to propose an API. I would love to get involved in helping define that API as a C API user if someone from JRuby can provide the perspective for the Ruby/JRuby side.

The new C extension API could be a gem written in terms of the current native C extension API since that API is the entire Ruby code base :)

Eventually it could be merged an no longer be a gem.

If you look at mruby, Matz obviously learned a lot from Lua's C API (arguable the Gold Standard™ when it comes to extending dynamic languages with C). For example, mruby now has a runtime state struct that gets passed around. This points to a trivial way of isolating multiple threads, multiple runtimes, etc.

I actually think a mruby C API gem, implemented on top of the existing C API, wouldn't be impossible...

We would love someone to get some docs on (or have knowledge of) JNI and model something after it for Ruby. JNI for the JVM is a well-understood example of making a native extension API and I think it would be workable for a Ruby runtime. Our main issue is the time and desire to implement a lot of C code for MRI. I really think it should be done but I do not think I am that person...

Joshua, I agree with you about Matz learning how to do this better than before. I honestly don't think he saw this as a significant use-case for many years.

I also should say things like context objects would be stuff that would sort of be faked out in a new API since the MRI internals do not have structs like that, but the API itself must be written in terms of it in order to give them a path to internal refactoring.

Right. I've only had a cursory overview of the mruby API, but if it's modeled on the Lua API then the context objects are generated and consumed as opaque structs by the API calls. So, for a gem implementation they could just be empty, and all VM context could be gotten the "traditional" way.

Still, this would need a lot more thought and study...

I am going to think about this while drinking beer and playing video games...I will see where this thread goes when I get up tomorrow and read it :)

Joshua, regarding your point about compatibility, I'm okay with 98% compatibility as long as 100% compatibility is still a goal. When a major implementation gives up on trying to achieve that last 2%, it fractures the community. Library authors must decide between using those 2% features—which, in this case, can improve performance—or being incompatible. I think MacRuby/RubyMotion is different scenario, because libraries are not typically built to run on both MacRuby/RubyMotion and MRI, so compatibility is less important.

Tom, sounds good! Thanks for participating.

Erik, I agree in principle, but I think you have to consider that MRI probably contains 105% of what is "Ruby". That is, because there's only been one implementation for so long, it's hard to know what's really essential in MRI. For example, here's a confusing situation where the result of defining #to_str on an object to be compared with a string is unclear, because the only existing implementation to treat this case is MRI (and it's seemingly broken): bugs.ruby-lang.org

I think we can agree that *some form* of C API is part of this mythical "100% Ruby". One way JRuby can help is by focusing MRI on what's really important in Ruby, including the C API. Slavishly copying a bad API is not a help.

Re: support C API anyway

One reason we decided to deprecate the C API support is because we started to get lots of bugs about it being slow that were unfixable. Is it worth supporting if it's so slow it's unusable? At the end of the day it just creates more work for us and no value for users.

FWIW, Wayne Meissner (the JRuby native integration guy) has made some attempts at a new C API here: github.com

It's rather C++ish, but it would not be difficult to morph it from env->func style into xrb_func(env, ...) style.

Joshua's suggestion about mruby's API is a great one. I noticed that API recently myself, and it is definitely designed better (i.e. designed *at all*) than MRI's API.

And if we can do it in a gem...that's awesome.

Joshua, I understand how frustrating it can be to implement what seem to be bugs. That said, if many programs depend on these bugs, then this is necessary for compatibility. (On the other hand, if very few programs depends on these bugs or there exists an easy workaround, I believe this is less important.) However, what we're discussing is not re-implementing bugs. We're talking about implementing a major feature of Ruby, upon which many common libraries depend.

I also take your point that resources are not infinite and that the JRuby team must prioritize certain things over others. I'm very thankful of all the time they've put into building JRuby and generally agree with the priorities they've chosen…

I just want to live in a world where Ruby users can freely switch between competing implementations without having to modify their programs. I believe that achieving this reality will be a very good thing for JRuby.

Today, switching to JRuby is not that easy since many applications depend on at least one gem with C extensions.

Right, this is why I laid out at first that there are two angles to this argument. I think it's not in question that the existing C API in MRI is bad.

Then the question becomes how vitally important is it to the community. For my part, in the past 2 years I've ported two ~100K LOC Rails apps from MRI to JRuby with relatively little trouble (most of the trouble is in the actual app serving platform, and that has come a long way in the last 2 years). On the flip side, I've seen gems and other code break going from 1.9.2 to 1.9.3.

I think we need to know the full extent of difficulties that people are having, and then we could make a better assessment of what's less work: full C API compatibility or updated versions of the more popular gems.

However, "full C API compatibility" is even an open-ended question. For example, the rb_string_value_ptr method in the C API returns a "char *" pointed at the actual array of bytes for the string. The end result of this is that it is possible (and I've seen this in gems-in-the-wild) to call this method to get a handle on a Ruby String object's actual character storage, and then update the characters in the string in place.

This flat out **won't work** in almost any other implementation. It's also probably not intentionally a part of the C API. And yet, it is possible and it does happen.

The right thing would have been to make rb_string_value_ptr return a "const char *" at the least...

The real-world use-case that started this thread is my desire to use Peter's Oj gem as a dependency of my T gem. Oj delivers noticeably faster JSON parsing performance on MRI vs. any other library, including stdlib. At the same time, I don't want to break compatibility with JRuby. Peter has done his part to support JRuby: github.com

I've actually gone back and forth on which JSON library to use, based on JRuby compatibility at the time of release. Version 1.3 did not depend on Oj. Versions 1.4 and 1.5 did, because Oj worked on JRuby 1.6. Then JRuby 1.7 was released and now I'll probably remove the Oj dependency from the next release.

I'm being forced to choose between speed on MRI and compatibility with JRuby. This choice sucks.

Oj works on JRuby 1.7 with the exception of one of the JSON recursive walk functions which uncovers a bug in the current JRuby MRI. All other operations still work for now.

This has been a painful few releases, especially when you bring rvm into the mix.

On the JSON side of things, I think MultiJSON solves the cross-platform JSON compatibility and will be as fast as its users can support.

The problem that I have as a gem *user* is that if I want to use a gem which has a dependency on a gem that has a cext, (simple_xlsx_writer depends on fast_xs) I currently have to create a jruby version of the cext gem and "pretend" to be it in order to fool Bundler, etc.

While that is a giant PITA, it is more convenient than having a "backdoor" multithreaded throttle be introduced or a partially-complete works-sometimes 'support' -- both of which happen because the cext API has too few guarantees.