What is SCL​

Sequence Configuration Language (SCL) is a programming language for forensic and ediscovery technicians. The typical use case involves reading, writing, and manipulating large amounts of data so performance is critical. The language is implemented in C# so performance improvements are achieved by optimizing C# code.

Measuring Performance​

Before we improve performance we need to measure it. This means benchmarking. I created a new console project and added the excellent benchmark dot net package to it.

As we want to optimize real world performance I found some real world data to run the benchmarks on. This data is a 1MB concordance file (very similar to a CSV) with 1000 rows and 44 columns. The data comes from the ENRON dataset but nevertheless I was very careful to add it to .gitignore and not commit it to the open source repository.

I wrote several SCL scripts representing different workflows. For example this script removes a column from the data:

- <path> = PathCombine ['Data', 'TestData.dat']
- FileRead <path>
| FromConcordance
| EntityMap (EntityRemoveProperty  <> Property: "Custodian" )
| ToConcordance
| StringLength

Note that StringLength at the end. This is to ensure that the stream of concordance data is read to the end (and therefore all of the data is converted to concordance). In a real world scenario one would probably write the data to a file or send it across the internet but doing that would make the benchmarks a lot more fiddly and the code that writes to a file is outside my control anyway so there is not much point in benchmarking it.

Benchmarks​

Here is a summary of all the benchmarks and the initial results, running on a 12th gen Intel i7 CPU on Windows.

Note that all times are in microseconds, so the longest sequences are taking about one third of a second.

Performance Improvements​

Replacing Task with ValueTask​

SCL steps all have a Run method which returns the result wrapped in a Task. One of the performance optimizations I was eager to try was wrapping the result in a ValueTask instead. This should reduce the number of heap allocations per step by one and I was expecting a small performance improvement for sequences which run cheap steps inside a loop.

The tradeoff is that there are some things you can do with a Task but not with a ValueTask such as awaiting them multiple times. Fortunately the Run method was only being used in a few places and I was able to easily check that none of the missing functionality was being used.

The results of this update:

Finding a Bug​

A 2% performance improvement on the slowest benchmark was nice but I was actually hoping for a larger improvement. The StringReplace sequence was taking about three times as long as the ConvertDates sequence despite doing what seemed like a lot less work.

I thought that the reason might be that StringReplace converts the data at the end into JSON rather that concordance like most of the other benchmarks. I chose to do this initially because I wanted to make sure that performing an action on the data first didn't make the conversion to JSON slower. At this stage I wrote additional benchmarks to check if that was what was happening. It wasn't - converting to JSON was consistently ~6000μs faster than converting to concordance across the board.

Thinking again, I thought the difference might be down to the fixed cost of running a step. ConvertDates uses the Transform step which applies a JSON schema to every step in an entity stream whereas StringReplace is using the ArrayMap step to call the StringReplace step on every entity in the stream. This means that the StringReplace sequence is running about 1000 very cheap steps where ConvertDates is calling one very expensive step.

I decided to use the Visual Studio performance profiler to try and find out what was going on. I stared by using the CPU usage tool as I thought the CPU was the most likely performance bottleneck.

I discovered that a lot of time was being spent in the IRunnableStep<T>.Run method. This is not unexpected but what was unexpected was that it was spending almost three times as long in there as in the CompoundStep<T>.Run method despite it having been designed as a thin wrapper around that method which exists purely to enable logging.

Looking at the method code I immediately spotted the problem:

async Task<Result<T, IError>> IRunnableStep<T>.Run(IStateMonad stateMonad, CancellationToken cancellationToken)
{
    using (stateMonad.Logger.BeginScope(Name))
    {
        object[] GetEnterStepArgs()
        {
            var properties = AllProperties
                .ToDictionary(x => x.Name, x => x.Serialize(SerializeOptions.SanitizedName));

            return new object[] { Name, properties };
        }

        LogSituation.EnterStep.Log(stateMonad, this, GetEnterStepArgs());

        var result = await Run(stateMonad, cancellationToken);

        if (result.IsFailure)
        {
            LogSituation.ExitStepFailure.Log(stateMonad, this, Name, result.Error.AsString);
        }
        else
        {
            var resultValue = result.Value.Serialize(SerializeOptions.SanitizedName);

            LogSituation.ExitStepSuccess.Log(stateMonad, this, Name, resultValue);
        }

        return result;
    }
}

The EnterStep arguments are being created - using an expensive dictionary - every time the step is run, regardless of the logging level (they should only be logged if the log level is Trace and the benchmarks are using Info). Clearly what happened was that the LogSituation.EnterStep.Log method once took a Func<object[]> but was refactored to use an object[] at some point and I missed the performance implications when updating this invocation.

I fixed the code and also checked other invocations of the method to make sure I wasn't making the same mistake somewhere else (I wasn't). Then I ran the benchmarks again:

The three benchmarks which run steps in a loop all got a lot faster, and the BasicSCL sequence got almost twice as fast. This just goes to show that finding and fixing performance bugs is often massively more effective than micro-optimizing heap allocations.

Rearchitecting Entities​

The other way to get big performance wins is to use the right collections in the right places. SCL has the type entity which is used to represent data. It's a key-value or property-value container, similar to a javascript object. The most obvious way to represent this in C# was to use a Dictionary<String, Object> and that was essentially how the Entity type was implemented - though using the SCL-friendly types StringStream and ISCLObject instead. I wanted to try a different implementation using two ImmutableArrays instead. ImmutableArray is an extremely thin wrapper around an array and has exactly the same performance characteristics.

I refactored the entity type to use an ImmutableArray<StringStream> for the property names and ImmutableArray<ISCLObject> for the values. I made them private to ensure that they would always be kept in sync. I updated the implementations of all the entity methods but not the signatures. Keeping the interface the same meant that the refactor was very smooth and I hardly had to change any code outside the entity class.

Making this change also allowed me to sneak in a few other performance optimizations. When creating entities from a consistent data structure (such as a CSV file or a SQL table) I could let all the entities share the same headers collection. Note that this would have been a bad idea if I had used a regular rather than immutable array.

I ran the benchmarks again and compared to the previous results:

This change almost ❗ doubled ❗ performance across the board (note that the BasicSCL sequence does not use entities). I was very happy.

The table and charts below show the complete journey. Some of the steps were now several times as fast and this difference would likely be magnified in real world scenarios that perform multiple transformations since the costs of reading the file and converting between data formats are fixed.

Takeaways​

The top takeaways from this project are:

• Fixing bugs is often the best way to improve performance
• Dictionaries are slow - plain old arrays are often faster even when a Dictionary or Hashmap is the "theoretically" correct choice.
• Refactoring can be easier than you expect, especially if the code you are changing is well insulated and the API to that code remains consistent.

One final thing - even though the ValueTask update only showed a very small improvement initially, that improvement was orthogonal to the other improvements. It shaved 5ms off the initial StringReplace benchmark and once I had completed the other improvements I tried reverting just the ValueTask changes and running the benchmarks again. StringReplace was now about 5ms slower. Had I done this change last instead of first it would have looked like an 8% improvement instead of a 2% one. This shows the value of small incremental improvements. Sometimes they don't just add up - they multiply!

In this blog post I'll show how to use SCL to combine all the date fields into one and how to handle multiple formats

Sample Data​

The following is the CSV file I'll be using in this example.

"Date","Time","TimeZone","Location"
"January, 01 2000","01:23:45","+00:00","UK"
"December, 31 1999","15:23:45","-10:00","Hawaii"
"January, 01 2000","02:23:45","+01:00","France"
"January, 01 2000","06:53:45","+05:30","India"
"01/01/2000","01:23:45","+00:00","Iceland"
"31/12/1999","15:23:45","-10:00","Hawaii"
"01/01/2000","02:23:45","+01:00","Germany"
"01/01/2000","06:53:45","+05:30","Sri Lanka"

It has the same date and time written in eight different ways with two different time formats and four different timezones.

As an additional headache, the Date, Time, and Timezone data is spread across three different fields.

Download the example file here. Or just copy the example above.

Reading the Data​

To read the data, make sure you have the File System and Structured Data connectors configured, then run:

- <data> = FileRead 'dates-example.csv' | FromCSV

This reads the contents of the file and pipes the resulting stream into the FromCSV step which converts it into entities.

The rest of this process works with any data source, not just CSV.

Combining the Date Fields​

To parse the dates correctly, we need to combine the three fields. We can use ArrayMap to add another field which is the concatenation of all three.

- <newdata> = <data>
| ArrayMap (In <> Set: 'FullDate' To: $"{<>['Time']} {<>['Date']} {<>['TimeZone']}")

Here we use ArrayMap which maps every entity in our array (FromCSV produces an Array of Entities) to a new value.

Here the new value is the same entity but with an additional 'FullDate' field added which is the values of the three other fields concatenated together using an interpolated string.

The Schema​

In order to convert this date string to an actual date we need to create a schema to tell the Transform step what the data is supposed to look like.

- <schema> = (
    'type': 'object'
    'additionalProperties': false
    'properties': (
      'Location': ('type': 'string')
      'FullDate': ('type': 'string', 'format': 'date-time')
    )
    'required': ['FullDate', 'Location']
  )

To explain the schema:

• 'type': 'object' tells the Transform step to expect an entity.
• 'additionalProperties': False will remove all the properties which are not listed in the schema. This is how we get rid of the now redundant 'Date', 'Time', and 'TimeZone' fields.
• 'properties': is the list of properties to expect and their types.
• 'Location': ('type': 'string') the location field is just a plain string.
• 'FullDate': ('type': 'string', 'format':'date-time') The date field is also a string but the 'format':'date-time' tells the Transform step to convert it into a DateTime.
• 'required': ['Date', 'Location' ] tells the Transform step to give an error or warning if any of these fields are missing

The Date Formats​

- <dateFormats> = ['HH:mm:ss MMMM, dd yyyy zzz', 'HH:mm:ss dd/MM/yyyy zzz']

For the Transform step to correctly parse the dates, we need to tell it what date formats to expect. In this case we give it an array with the two different formats present in the data. It will try the first one first, then the second, and if that is unsucessful it will give you an error message.

If we had just one date format we could give just that string rather than an array with one element.

If we had several full date columns with a different format per column we could give an entity mapping column names to formats but that is not required in this case.

SCL is run inside C# and uses the C# Date parsing features and format specifiers. See this document which explains how those work and how to use them but briefly:

• HH:mm:ss means hours, minutes, and seconds separated by colons
• MMMM is the full word name for the month while 'MM' is the two digit representation
• dd and yyyy are day number and year respectively
• zzz is the timezone hours and minutes offset from UTC. The timezone specifiers can be quite fiddly and you may have to use multiple formats to get it right (e.g. if the timezone is omitted or just '0' for UTC but another format everywhere else)

The Transform Step​

Now that we have our data, our schema, and our date formats we can run the Transform step.

<result> = <newdata> | Transform Schema: <schema> DateInputFormats: <dateFormats>

The Transform step has a lot of optional arguments but apart from DateInputFormats the defaults are good enough for us in this case.

We could have set

• ErrorBehavior to control what happens if a transformation is not possible. By default you get an error but you could get a warning or ignore it entirely. This also lets you control whether or not to output the entity which caused the error.
• ArrayDelimiters lets you specify which characters can be array delimiters. This way you could transform strings like 'alpha|beta|gamma' into arrays.
• BooleanTrueFormats, BooleanFalseFormats, and NullFormats allow you to specify which strings can be converted to True, False, and Null respectively. As with the DateInputFormats you can do this on a column by column basis if needed.
• CaseSensitive allows you to specify whether or not the above formats are case sensitive. This is false by default.
• RoundingPrecision lets you control how close a floating point number needs to be to an integer to be rounded to it.

Exporting the Results​

We can output the data also as a CSV, but this time with tabs as the separator.

<result> | ToCSV DateTimeFormat: 'yyyy-MM-dd HH:mm:ss' Delimiter: "\t"

DateTimeFormat lets us specify how the DateTime is formatted in the output.

Final Output​

In the final output we see that all dates are now the same, which is correct:

Location    FullDate
UK    01:23:45 2000-01-01
Hawaii    01:23:45 2000-01-01
France    01:23:45 2000-01-01
India    01:23:45 2000-01-01
Iceland    01:23:45 2000-01-01
Hawaii    01:23:45 2000-01-01
Germany    01:23:45 2000-01-01
Sri Lanka    01:23:45 2000-01-01

Final SCL​

- <schema> = (
    'type': 'object'
    'additionalProperties': false
    'properties': (
      'Location': ('type': 'string')
      'FullDate': ('type': 'string', 'format': 'date-time')
    )
    'required': ['FullDate', 'Location']
  )

- <dateFormats> = ['HH:mm:ss MMMM, dd yyyy zzz', 'HH:mm:ss dd/MM/yyyy zzz']

- <data>
| FromCSV
| ArrayMap (In <> Set: 'FullDate' To: $"{<>['Time']} {<>['Date']} {<>['TimeZone']}" )
| Transform Schema: <schema> DateInputFormats: <dateFormats>
| ToCSV DateTimeFormat: 'yyyy-MM-dd HH:mm:ss' Delimiter: "\t"

Try on the Playground​

You can try this SCL on the playground, just copy, paste, and run.

- <data> = """
"Date","Time","TimeZone","Location"
"January, 01 2000","01:23:45","+00:00","UK"
"December, 31 1999","15:23:45","-10:00","Hawaii"
"January, 01 2000","02:23:45","+01:00","France"
"January, 01 2000","06:53:45","+05:30","India"
"01/01/2000","01:23:45","+00:00","Iceland"
"31/12/1999","15:23:45","-10:00","Hawaii"
"01/01/2000","02:23:45","+01:00","Germany"
"01/01/2000","06:53:45","+05:30","Sri Lanka"
"""

- <schema> = (
    'type': 'object'
    'additionalProperties': false
    'properties': (
      'Location': ('type': 'string')
      'FullDate': ('type': 'string', 'format': 'date-time')
    )
    'required': ['FullDate', 'Location']
  )

- <dateFormats> = ['HH:mm:ss MMMM, dd yyyy zzz', 'HH:mm:ss dd/MM/yyyy zzz']

- <data>
| FromCSV
| ArrayMap (In <> Set: 'FullDate' To: $"{<>['Time']} {<>['Date']} {<>['TimeZone']}" )
| Transform Schema: <schema> DateInputFormats: <dateFormats>
| ToCSV DateTimeFormat: 'yyyy-MM-dd HH:mm:ss' Delimiter: "\t"
| Print