First: this branch isn't ready to be merged. It's still a bit sloppy and incomplete. I'm looking for feedback on the basic approach, API design, etc. If the basic approach for the API is okay (whether or not the actual implementation is good), I'd like to split it into a series of much smaller simpler PRs (re-implementing as necessary). I'm hopeful that someone can tell me there's a simpler, easier way to accomplish what I wrote in this draft PR. ;)

Issues

EventMachine is full of old SSL code, which isn't following best practices. It is much easier to write insecure code than to write secure code: there are no secure defaults. Debugging errors can be a pain. EventMachine's interface with OpenSSL feels very different from stdlib's, which can lead to confusion (and bugs and security holes).

Additionally, the internal API for setting the context: EventMachine.set_tls_params is a big mess, and it's only going to get messier if we keep using the same approach.

In particular, I wanted to add a default ca_store, the ability to configure ca_path and ca_file, and a verify_hostname option which uses the openssl gem's algorithm. I want to delegate all certificate and identity verification to the openssl library, and have the ability to easily diagnose specific errors from ruby.

Proposal:

My proposal is to move as close as possible to stdlib's API and implementation , while maintaining a reasonable backward compatibility with the original EM API.

API changes

1. The API for internal methods, such as EM.set_tls_params, EM.start_tls, and even e.g. EM.get_cipher_* do not to be protected from change. Applications and other gems should not rely on those. However it might be worth doing a quick survey of reverse dependencies on rubygems.org to see if anyone is using them.

2. Where the default EM.start_tls with no parameters defaults are insecure, we should break compatibility. The new defaults should mimic stdlib's OpenSSL::SSL::SSLContext::DEFAULT_PARAMS as much as it is feasible to do so. And we should run a RFC6125 hostname verification procedure by default just like stdlib (see OpenSSL::SSL::SSLSocket#post_connection_check and OpenSSL::SSL.verify_certificate_identity).

3. Instead of passing a dozen or more parameters from EM::Connection#start_tls to EM.set_tls_parms to t_set_tls_parms to ConnectionDescriptor::SetTlsParms to a bunch of instance variables on ConnectionDescriptor to instance variables on SslContext_t and SslBox_t to the actual underlying SSL_CTX and SSL objects, we should create a ruby class that wraps all of the SSL_CTX params. This current PR unloads that ruby object into into a C struct which is delivered into a SslContext_t object. But ideally we could make our ruby object just a TypedData_Wrap_Struct around the SSL_CTX and replace SslContext_t altogether. Both the OpenSSL project and the openssl gem encapsulated SSL_CTX into a data type. So should EM! We can entirely dispose of EM.set_tls_parms and simply change start_tls to take our SSL_CTX wrapper and whatever other options remain for the SSL object, e.g. SNI hostname.

4. Adding a ruby class that is analogous to OpenSSL::SSL::SSLSocket seems less necessary: the EventMachine approach has always been query methods and callback methods on the EM::Connection module/class. However, EM's C++ code has already found it useful to split off SslBox_t, and keeping a small separation allows us to change our API inside our SSL and SSL_CTX wrappers, while maintaining backwards compatibility via EM:::Connection. E.g. this could be an opportunity to have a single cipher method instead of three get_cipher_* methods. And peer_cert can return OpenSSL::X509::Certificate objects (from stdlib), instead of simple PEM strings. And EM::SSL::X509StoreContext can be sent to the EM::SSL::Context#verify_callback. Etc.

5. The direction I'm going is that the ruby EM::Connection class doesn't need to know about SSL at all. Instead of calling EM::Connection#start_tls(**params) and using the two connection callback methods, the new recommended API could mimic stdlib:

module Client
  # shares the underlying SSL_CTX
  CTX = EM::SSL::Context.new
  CTX.set_params ca_file: "my/internal/ca.pem"

  def post_init
    @ssl = EM::SSL::Connection.new(CTX, hostname: "foo.example")
    @ssl.connect
  end
end

backwards compatibility

1. We could retain EM::Connection#start_tls as just a simple implementation of the above. It can support both the old API and the new API. If a context is sent in, great! Use it. Otherwise, take the arguments Hash and use it to build a new context like above.
2. Where compatibility has been broken for more secure defaults, it should still be simple to opt into the original behavior. This could be done with something like:

start_tls **EventMachine::OLD_INSECURE_DEFAULTS, hostname: "foo.example"

😜

3. Where the API has been deprecated, we could create a EM::SSL::BackwardsCompatible mixin that could be used to bring back the existing behavior. It could be included in the next release but deprecated and available but not included in EM::Connection in the next release. That could allow the basic EM::Connection code to be a little bit simpler.

Implementation changes

So... this PR is sloppy. Sorry. Consider it a first-draft proof of concept?

Moving the implementation as closely as possible to stdlib isn't a nice-to-have. It's critically important: stdlib has far more active maintainers and resources devoted to it. In my branch, I copied and pasted large chunks of code (C and Ruby) from the openssl gem. This was just a quick exploration. Ideally, we should be able to "inherit" bugfixes and improvements from stdlib. This could be accomplished in a few ways:

1. Fork and Vendor openssl gem -- write a script that imports the latest upstream gem and applies a series of simple transformations and patches to make it work within EventMachine. This is the direction my PR has been leaning towards. I'm fairly confident I could get it there eventually, but it's time consuming to set up, its not fool-proof, and there's no guarantee that the script will remain functional through openssl updates.
2. Extract SSL_CTX, SSL, etc from VALUE -- Another alternative might be to hack create actual stdlib objects, hack the wrapped T_DATA objects to extract the underlying SSL_CTX, SSL, etc, and call SSL_dup to have something that's been configured by ruby but is completely under EM control. I'm not 100% certain that's feasible for SSL but I think it could be doable with `SSL_CTX. One very big concern of mine is that I don't know C linking and ABIs well enough to know if it's actually safe to do this.
3. Link against openssl gem -- A similar alternative, require openssl include some of its header files into project.h, and link directly against wherever possible. Again, I'm not really sure how "safe" this is wrt openssl ABIs. But it might enable us to use the C functions defined in the gem, without ABI incompatibility issues.
4. Use openssl ruby interface -- We could directly attach stdlib's OpenSSL::SSL::SSLSocket to the underlying IO, and then proxy reads and writes through that SSLSocket. Essentially, replace SslBox_t *SslBox with VALUE SSLSocket, and replace every if (SslBox) {...} in ConnectionDescripter with a similar if (SslSocket) { ... }. Although calling it from C/C++, we'd only use the ruby API. In this case, EventMachine wouldn't even need to include the openssl headers.

I did some research, and there do appear to be occasional minor ABI incompatibilities between different versions of openssl?
https://abi-laboratory.pro/?view=timeline&l=openssl So I don't know if that can be made safe. I'm only a C/C++ tourist. ;)

But pushing as much of our SSL code into stdlib's openssl gem as possible seems like a wise approach. Perhaps there's a set of patches we could submit to the openssl gem which might let us use it more directly without sacrificing performance?