mruby-regexp: correct regexp offset conversion for UTF-8 and binary strings by nattzn · Pull Request #6915 · mruby/mruby

nattzn · 2026-06-26T12:14:46Z

Summary

This PR follows up on the review feedback from #6913 and fixes regexp offset conversion and binary string handling in mruby-regexp.

The main fixes are:

Return character offsets from Regexp#=~ when MRB_UTF8_STRING is enabled
Preserve byte-indexed positions for ASCII-8BIT / BINARY strings
Make the regexp engine advance byte-by-byte for forced binary strings
Preserve the existing UTF-8 regexp character behavior for normal non-binary strings
Preserve the observable mruby behavior for non-MRB_UTF8_STRING builds

Details

Return character offsets from `Regexp#=~`

In MRB_UTF8_STRING builds, Regexp#=~ now converts the internal byte offset to a public character offset before returning it.

Without MRB_UTF8_STRING, the existing byte-offset behavior is preserved.

Keep `Regexp#match(str, pos)` semantics unchanged

Regexp#match(str, pos) and Regexp#match?(str, pos) keep the same public semantics. This PR fixes the internal position conversion so that ASCII-8BIT / BINARY strings keep byte-indexed pos values instead of being converted as UTF-8 character offsets.

Normal non-binary strings in MRB_UTF8_STRING builds continue to use character offsets, and non-MRB_UTF8_STRING builds continue to use byte offsets.

Pass binary string state into the regexp engine

mrb_re_exec now receives whether the subject string is binary. This allows the regexp engine to choose the correct advancement behavior:

Normal non-binary strings: advance by UTF-8 regexp character length
ASCII-8BIT / BINARY strings: advance by one byte

This applies to regexp execution paths including:

Pike VM execution
Backtracking execution
. matching
Character classes
lookahead / lookbehind recursion
zero-width match advancement in gsub

Small internal helpers were added to centralize the byte-vs-UTF-8 behavior:

mrb_re_charlen
mrb_re_decode_char
mrb_re_utf8_continuation_p

Affected behavior

The table below summarizes the effective units used by the affected APIs and regexp behavior.

Bold entries indicate behavior fixed by this PR.

Method / behavior	No `MRB_UTF8_STRING`	`MRB_UTF8_STRING` normal non-binary strings	`mruby-encoding` + `ASCII-8BIT` / `BINARY` strings
`Regexp#=~` return value	byte offset	character offset	byte offset
`Regexp#match(str, pos)` `pos`	byte offset	character offset	byte offset
`Regexp#match?(str, pos)` `pos`	byte offset	character offset	byte offset
`MatchData#begin(n)` / `#end(n)`	byte offset	character offset	byte offset
`.` / character class advancement	UTF-8 regexp character	UTF-8 regexp character	byte
`String#scan(/./)`	UTF-8 regexp character	UTF-8 regexp character	byte
`String#sub(/./)` / `String#gsub(/./)`	UTF-8 regexp character	UTF-8 regexp character	byte
zero-width match advancement	UTF-8 regexp character	UTF-8 regexp character	byte

Note that “UTF-8 regexp character” here means the unit used by the regexp engine while matching, such as how far . advances. It is different from public character offsets returned by APIs such as Regexp#=~ or MatchData#begin.

For normal non-binary strings, the existing regexp engine behavior is preserved. For example, \xC3\xA9 continues to be treated as one regexp character by ..

For forced ASCII-8BIT / BINARY strings, regexp matching now advances byte-by-byte.

CRuby compatibility

Compatibility was checked against CRuby 4.0.4.

The checked cases include:

Normal non-binary strings
Forced ASCII-8BIT / BINARY strings
Out-of-range and negative pos behavior

In the checked cases, the mruby-encoding build now matches CRuby behavior.

Representative examples:

s = "\xC3\xA9x"
bin = s.dup.force_encoding("ASCII-8BIT")

s.scan(/./).map(&:bytes)
# => [[195, 169], [120]]

s.sub(/./, "Z").bytes
# => [90, 120]

bin.scan(/./).map(&:bytes)
# => [[195], [169], [120]]

bin.sub(/./, "Z").bytes
# => [90, 169, 120]

/x/.match(bin, 2).begin(0)
# => 2

Tests

Added tests covering three behavior groups:

Normal non-binary strings and public character offsets
Non-MRB_UTF8_STRING builds and existing byte-offset behavior
Forced ASCII-8BIT / BINARY strings and byte-by-byte regexp advancement

Verification

Successfully across three configurations:

no MRB_UTF8_STRING, no mruby-encoding
MRB_UTF8_STRING enabled, no mruby-encoding
MRB_UTF8_STRING enabled with mruby-encoding

…tring

gemini-code-assist

Code Review

This pull request adds support for binary (ASCII-8BIT) strings in the mruby regexp engine by introducing a binary flag to the execution functions and bypassing UTF-8 decoding when processing binary strings. Feedback on the changes suggests capping byte_off to the string length in re_byte_to_char to prevent potential out-of-bounds memory reads, and adding a fast-path check in re_char_to_byte to avoid scanning the string when the character offset is clearly out of bounds.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-06-26T12:16:12Z

+  struct RString *s = RSTRING(str);
+  if (RSTR_SINGLE_BYTE_P(s) || RSTR_BINARY_P(s)) return byte_off;
+
+  const char *p = RSTR_PTR(s);


To prevent potential out-of-bounds memory reads, we should defensively cap byte_off to the actual string length len before iterating through the string bytes. If byte_off is larger than len, it could lead to reading past the allocated buffer of the string.

struct RString *s = RSTRING(str); if (RSTR_SINGLE_BYTE_P(s) || RSTR_BINARY_P(s)) return byte_off; mrb_int len = RSTR_LEN(s); if (byte_off > len) byte_off = len; const char *p = RSTR_PTR(s);

gemini-code-assist · 2026-06-26T12:16:12Z

  if (char_off < 0) {
+    mrb_int char_len = re_byte_to_char(mrb, str, len);
    char_off += char_len;
+    if (char_off < 0) return -1;
  }


For positive out-of-bounds offsets where char_off > len, we can immediately return -1 without scanning the entire string. Since a UTF-8 character is at least 1 byte, the character length of a string can never exceed its byte length. Adding this fast-path check avoids an O(N) scan of the string when the offset is clearly out of bounds.

if (char_off < 0) { mrb_int char_len = re_byte_to_char(mrb, str, len); char_off += char_len; if (char_off < 0) return -1; } else if (char_off > len) { return -1; }

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a7c5ab620c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-26T12:22:13Z

    if (captures[1] == captures[0]) {
      if (captures[1] < slen) {
-        int clen = mrb_re_utf8_charlen(s + captures[1], s + slen);
+        int clen = mrb_re_charlen(s + captures[1], s + slen, binary);


Keep block gsub zero-width advancement byte-based for binary

This binary-aware zero-width advance only fixes the replacement-string C path. When the subject is forced ASCII-8BIT and the pattern can match empty (// or /(?=.)/), the block implementation still advances in mrbgems/mruby-regexp/mrblib/string_regexp.rb with rest[0], which returns the whole UTF-8 character even for the forced-binary string; e.g. bin.gsub(//) { '-' } skips the 0xA9 byte while bin.gsub(//, '-') now advances byte-by-byte. Please mirror this byte-vs-UTF-8 step in the block path so the two String#gsub forms stay consistent for binary strings.

Useful? React with 👍 / 👎.

nattzn · 2026-06-26T13:47:23Z

Addressed three points:

Capped byte_off to the actual string byte length before scanning in re_byte_to_char, preventing
possible out-of-bounds reads for oversized byte offsets.
Added a fast path in re_char_to_byte for positive char_off > len, returning -1 immediately since a
character offset cannot exceed the byte length.
Updated block-form String#gsub to use the same binary-aware zero-width advance as the replacement-string path. Binary strings now advance by byte, while UTF-8 strings advance by character. Since I could not find a suitable Ruby-level API for checking the internal binary flag, I added the internal helper Regexp.__binary_string?(str). Added regression tests for gsub(//) and gsub(/(?=.)/) to keep both forms consistent.

mruby-regexp: correct regexp offset conversion for UTF-8 and binary s…

a7c5ab6

…tring

nattzn requested a review from matz as a code owner June 26, 2026 12:14

github-actions Bot added the mrbgems label Jun 26, 2026

gemini-code-assist Bot reviewed Jun 26, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 26, 2026

View reviewed changes

nattzn added 2 commits June 26, 2026 22:09

mruby-regexp: guard regexp offset conversions

28b48d8

mruby-regexp: align binary zero-width gsub block advance

9b8520f

matz merged commit a014f4d into mruby:master Jun 27, 2026
18 of 20 checks passed

nattzn deleted the fix/pr6913 branch June 27, 2026 08:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

mruby-regexp: correct regexp offset conversion for UTF-8 and binary strings#6915

mruby-regexp: correct regexp offset conversion for UTF-8 and binary strings#6915
matz merged 3 commits into
mruby:masterfrom
nattzn:fix/pr6913

nattzn commented Jun 26, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jun 26, 2026

Uh oh!

gemini-code-assist Bot Jun 26, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 26, 2026

Uh oh!

nattzn commented Jun 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nattzn commented Jun 26, 2026

Summary

Details

Return character offsets from Regexp#=~

Keep Regexp#match(str, pos) semantics unchanged

Pass binary string state into the regexp engine

Affected behavior

CRuby compatibility

Tests

Verification

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

nattzn commented Jun 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Return character offsets from `Regexp#=~`

Keep `Regexp#match(str, pos)` semantics unchanged