Skip to content

Memory leak while iterating over commits and their diffs? #766

Description

@jfremy

I have some code that iterates over a range of commits and tries to extract some information from each commit (which requires to look at the diff of each commit). For longer histories, this systematically ends up with a process out memory error or with a core dump.

I made a modified examples program that triggers it (just put it in nodegit examples directory and run it).
It will iterate over the commits and trigger the error.

Here's the code:

var nodegit = require("../");
var path = require("path");

var repo;

function process(commit) {
  "use strict";
  console.log("commit " + commit.sha());
  return commit.getDiff().then(function (diffList) {
    return Promise.all(diffList.map(function (diff) {
      return diff.patches().then(function (patches) {
        return Promise.all(patches.map(function (patch) {
          return patch.hunks().then(function (hunks) {
            return Promise.all(hunks.map(function (hunk) {
              return hunk.lines().then(function (lines) {
                return true;
              });
            }));
          });
        }));
      })
    })).then(function () {
      if(commit.parentcount() === 0) {
        return false;
      } else {
        return repo.getCommit(commit.parentId(0));
      }
    });
  });
}

function iterate(commit) {
  "use strict";

  return process(commit).then(function(childcommit) {
    if(!childcommit) {
      return true;
    } else {
      return iterate(childcommit);
    }
  });
}

// This code examines the diffs between a particular commit and all of its
// parents. Since this commit is not a merge, it only has one parent. This is
// similar to doing `git show`.

nodegit.Repository.open(path.resolve(__dirname, "../.git"))
  .then(function (re) {
    repo = re;
    return repo.getHeadCommit();
  })
  .then(function (commit) {
    return iterate(commit);
  })
  .done(function () {
    "use strict";
    console.log("done");
  })

I get an error similar to the following:

<--- Last few GCs --->

   29305 ms: Scavenge 1398.7 (1457.0) -> 1398.7 (1457.0) MB, 5.0 / 0 ms (+ 1.8 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
   30477 ms: Mark-sweep 1398.7 (1457.0) -> 1398.7 (1457.0) MB, 1172.2 / 5 ms (+ 2.7 ms in 2 steps since start of marking, biggest step 1.8 ms) [last resort gc].
   31407 ms: Mark-sweep 1398.7 (1457.0) -> 1398.7 (1457.0) MB, 930.4 / 5 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x72d20d36ae1 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [/home/jfremy/prog/nodejs/nodegit/node_modules/nodegit-promise/lib/node-extensions.js:24] [pc=0x28f3e3723a14] (this=0x72d20d04131 <undefined>,resolve=0x1460e70041d1 <JS Function done (SharedFunctionInfo 0x30acef4d48d1)>,reject=0x1460e7004189 <JS Function (SharedFunctionInfo 0x30acef4d4979)>)
    2: tryCallTwo(aka tryCallTwo) [/home/jfremy/prog/nodejs/n...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abandon (core dumped)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions