using Programming;

A Blog about some of the intrinsics related to programming and how one can get the best out of various languages.

Getting started with programming and getting absolutely nowhere (Part 17)

How close are two words?

Lesson 16: Increasing the level of de-obsfucation

Today's lesson is going to continue on our "spam filter" adventure, but the next step it to start supporting similar-but-not-quite-identical words.

Anyone who's ever typed or written text has most assuredly spelled a word or two wrong, probably more than once. (In fact, I spell 'appropriate' wrong every day — for some reason I always end up with 'apporpriate', I think it's the speed at which I type.) These may be mistakes, or they may be intentional. Sometimes we spell something wrong for a reason.

Often times "spammers" will spell words wrong on-purpose — instead of just swapping letters out for other, similar letters, they'll actually mis-spell something knowing the filter is searching for the correctly spelled term. Of course, a good spam-filter (which is what we plan to build) can detect some of this and make decisions based on that fact itself.

As of right now we're testing several cases of our filters against the following terms: "nope"; "fail"; "leet", today we're going to add some more cases, such as "noope", and "failing". These are going to be two more words we match against, but we're also now going to return how effectively the filter matched the term. That is, "nope" would match against "nope" for a 100% match, but "noope" would not, so we'll devise a manner of indicating what the success-rate is.

We're also going to try to prevent some false-alarms. A word like "true" doesn't really match any of our filters, but it's fair to say that it shares letters with two of them: it shares one letter with "nope", and two letters with "leet". We know that it's not even close, but we'll have to tell the computer that before we continue.

This can also be the basis of a 'spell-checker'

At this moment we're devising a piece of code that could serve as a trivial spell-checker, we're going to have to design it that way because we have to support mis-spellings, and the easiest way to do so is to design it as a spell-checker.

We're also going to do some TDD here — for those unfamiliar, TDD is simply "test-driven development", or the idea that you build tests that have an input and expected output, and then start writing the code to get there. We would write a test such as:

let expected = 1.0
let inputTerm = "nope"
let inputFilter = ["nope"]
let actual = inputTerm |> matchFilters inputFilter |> Seq.tryHead |> Option.map snd |> (function | None -> 0.0 | Some v -> v)
printfn "%s" (if actual = expected then "Pass" else "Fail")

This uses the AAA pattern, or "Arrange, Act, Assert". That is:

  • Arrange: prepare the conditions for the test, this is the input and output. What are we looking for? What are we given?
  • Act: perform the operation for the test, this maps the input(s) to the output(s). What do we get?
  • Assert: determine whether the expected output and actual output match. What really happened?

When I do TDD I always write a test, then write the function being tested, and repeat. I want to start with the main "framework" that we'll use (though this is a basic testing framework, there are others that are far better than this, I recommend using one of them):

module Tests =
    module Assert =
        let private fn fn msg expected actual =
            if fn expected actual then None
            else (msg expected actual) |> Some
        let equal expected actual = fn (=) (sprintf "expected: %A; actual: %A") expected actual
        let notEqual expected actual = fn (<>) (sprintf "expected not: %A; actual: %A") expected actual
        let largerThan expected actual = fn (>) (sprintf "expected greater than: %A; actual %A") expected actual
        let largerEqual expected actual = fn (>=) (sprintf "expected greater than / equal to: %A; actual %A") expected actual
        let smallerThan expected actual = fn (<) (sprintf "expected smaller than: %A; actual %A") expected actual
        let smallerEqual expected actual = fn (<=) (sprintf "expected smaller than / equal to: %A; actual %A") expected actual
        let print assertion =
            match assertion with
            | None -> printfn "Pass"
            | Some msg -> printfn "Fail: %s" msg

    // Define tests

    let runTests () =
        let tests =
            [ // List tests
              ]
        tests |> List.iter ((|>) () >> Assert.print)

This is obviously not the easiest solution to use, but it works. Drop all your test methods in the tests list, and define them before runTests, and life should be good. As an example, our first test would look like:

let ``A term that exactly matches a word in the filter should return 1.0 as match`` () =
    let expected = 1.0
    let inputTerm = "nope"
    let inputFilter = ["nope"]
    let actual = inputTerm |> matchFilters inputFilter |> Seq.tryHead |> Option.map snd |> (function | None -> 0.0 | Some v -> v)
    Assert.equal expected actual

let runTests =
    let tests = [``A term that exactly matches a word in the filter should return 1.0 as match``]
    tests |> List.iter ((|>) () >> Assert.print)

Pretty easy, right?

Write the matchFilters function

Now we start writing the matchFilters function. We know what our inputs should be: a list of filters and a term to test, and we expect that it returns some sort of sequenced tuple. Specifically, we're going to return every term matched, and the "accuracy" of that match.

Initially, we can obviously define this pretty easily to pass our test:

let matchFilters filters term = filters |> List.choose (fun filter -> if term = filter then (filter, 1.0) |> Some else None)

If we run our tests now we should get Pass printed out. Pretty easy, right? Now we could have actually written let matchFilters filters term = ("", 1.0), and that would have been completely valid and satisfied our tests. It's not wrong. We still made all our tests pass. The version I wrote just happens to satisfy what the next test we'll write is as well.

Next we should probably test that a word like "abcd" doesn't accidentally return 1.0. If we write matchFilters as the version in the previous paragraph, it would do so, so we want a test for it:

let ``A term that does not match any words in the filter should return no matches`` () =
    let expected = 0.0
    let inputTerm = "abcd"
    let inputFilter = ["nope"]
    let actual = inputTerm |> matchFilters inputFilter |> Seq.tryHead |> Option.map snd |> (function | None -> 0.0 | Some v -> v)
    Assert.equal expected actual

Now we are forced to write a version that will at the very least return 1.0 for the first test, and 0.0 for this second one. You should now get the idea of TDD: write a test, build the method to make the test pass, repeat. Ideally, you don't trust a test until you've proven that the current version of the method made it change. That is: the test should fail first, then build a version of the method that passes. Think of it as a red light turning green: you can truly trust the traffic light once you've seen that happen, because you know the basic "is it just stuck on green?" test completed, and told you "nope, it's cycling like normal."

Now, interestingly, our second test isn't actually quite right, but I'm going to leave it as is. We said it should return no matches, but we tested it in a manner that loses that idea. It doesn't matter in the end, because no matches is synonymous with 0% match, but it's a good point to keep in mind.

Before we continue, let's make life a little easier on us. We should remember from previous lessons that our goal is always to write clear, concise, informative code. We can make matchFilters slightly better with that in mind:

let matchFilters filters term =
    let matchFilter filter =
        if term = filter then (filter, 1.0) |> Some
        else None
    filters
    |> Seq.ofList
    |> Seq.choose matchFilter

Now when we're ready to build better criteria, we just have to swap out matchFilter. Much easier than dealing with that silly lambda.

Build our spell-checking algorithm (sort-of)

Alright, on to the hard part. For the first iteration of the algorithm, we're simply going to test what letters are in both terms, then divide that by the total letters from both terms for the percentage. This will return 100% for "nope"/"nope", and 0% for "nope"/"abcd". This will also return a partial match for "nope"/"true" and "nope"/"leet".

Doing this is actually quite simple, we have a few options, the one I'll use is to create a concatenated array of both the strings, and then group them, and divide each group size by 2. As an example, consider we have "nope"/"true", when we concat we'll get [|'n'; 'o'; 'p'; 'e'; 't'; 'r'; 'u'; 'e'|], when we group the characters and get the counts we'll have [|('n', 1); ('o', 1); ('p', 1); ('e', 2); ('t', 1); ('r', 1); ('u', 1)|], if we sum floor(group.Count / 2) we get 1, which we then double and find that 2 of the total 8 characters were in each string, or 25%.

let matchFilters filters (term : string) =
    let matchFilter (filter : string) =
        let termChars = term.ToCharArray()
        let filterChars = filter.ToCharArray()
        let concated = termChars |> Array.append filterChars
        let grouped = concated |> Array.groupBy id |> Array.map (fun (c, a) -> (c, a |> Array.length))
        let sum = grouped |> Array.fold (fun acc (c, i) -> acc + i / 2) 0 |> float |> (*) 2.0
        if sum <= 0.0 then None
        else (filter, sum / (term.Length + filter.Length |> float)) |> Some
    filters
    |> Seq.ofList
    |> Seq.choose matchFilter
    |> Seq.sortByDescending snd

So that's what we end up with, for now. (We'll probably improve this in a later lesson.) We added the Seq.sortByDescending snd to order them by best match first. If we run our two existing tests we should get "Pass" and "Pass". We didn't create tests for this yet (well, I did, you probably haven't) so I recommend that before dumping this in your code you build a couple basic tests.

let testFn filters = matchFilters filters >> Seq.tryHead >> Option.map snd >> (function | None -> 0.0 | Some v -> v)

let ``A term that partially matches a word in the filter should return 0.* as match`` () =
    let expected = 0.25
    let inputTerm = "true"
    let inputFilter = ["nope"]
    let actual = (inputFilter, inputTerm) ||> testFn
    Assert.equal expected actual

let ``A term that mostly matches a word in the filter should return 0.* as match`` () =
    let expected = 0.75
    let inputTerm = "mope"
    let inputFilter = ["nope"]
    let actual = (inputFilter, inputTerm) ||> testFn
    Assert.equal expected actual

This method, of course, assumes that we only care about what characters are present and that the order is irrelevant. We'll (at some point) want to redefine it to support matching terms such that terms that have the same letters but not in the same position are not exactly a 1.0, etc. For now, this will do.

Finally, come up with a "threshold" for acceptance

When we do a "heuristic" match like this, we need to define a threshold of tolerance. We don't want "true" to match the "nope" filter because it only shares an e, we wnat to define a minimum-level-of-acceptability. For this, I usually start with a somewhat middle-ground value, knowing that 0.5 means 50% of our words were a match, I go with 0.75, or 75%. This gives us an initial feeling for where we stand, and we can tune that later.

Now I'm going to redefine things a little, don't fear, just to keep it a little cleaner:

let matchFilters filters (term : string) =
    let matchFilter (filter : string) =
        let termChars = term.ToCharArray()
        let filterChars = filter.ToCharArray()
        let concated = termChars |> Array.append filterChars
        let grouped = concated |> Array.groupBy id |> Array.map (fun (c, a) -> (c, a |> Array.length))
        let sum = grouped |> Array.fold (fun acc (c, i) -> acc + i / 2) 0 |> float |> (*) 2.0
        if sum <= 0.0 then None
        else (filter, sum / (term.Length + filter.Length |> float)) |> Some
    filters
    |> Seq.ofList
    |> Seq.choose matchFilter
let bestFilter filters =
    matchFilters filters >> Seq.sortByDescending snd >> Seq.tryHead >> Option.map snd

Pretty simple, we just moved our Seq.sortByDescending out, and now we'll go ahead and add a function for meetsThreshold which takes our "filter" and threshold:

let meetsThreshold threshold (filter, percent) = percent > threshold

Time to modify our previous matchedFilters

Finally, the fun part. We'll modify the matchedFilters to use our new algorithm. I'm only going to pick the best-matched filter, and we're only going to pick which transformation of the term it was that best-matched the filter, so we'll end up with the following:

let matchedFilters term =
    term
    |> transformToLowerTerm
    |> Array.map (bestFilter filters >> (function | None -> ([||], 0.0) | Some f -> f))
    |> Array.sortByDescending snd
    |> Array.filter (meetsThreshold 0.75)
    |> Array.tryHead
    |> (function | None -> [] | Some a -> [a])

I did make one modification to matchFIlters: the inner-function was defined to use strings, we want to use our int-arrays, so we'll change:

let matchFilters filters (term : string) =
    let matchFilter (filter : string) =
        let termChars = term.ToCharArray()
        let filterChars = filter.ToCharArray()
        let concated = termChars |> Array.append filterChars

To:

let matchFilters filters (term : int[]) =
    let matchFilter (filter : int[]) =
        let concated = term |> Array.append filter

And we should have the new algorithm. We should get some matched terms:

val matchedTerms : (string * string list) list =
  [("ℕope", ["nope"]); ("𝑵ope", ["nope"]); ("ռope", ["nope"]);
   ("nope", ["nope"]); ("𝕱ail", ["fail"]); ("𝓕ail", ["fail"]);
   ("l33t", ["leet"]); ("1337", ["leet"]); ("noope", ["nope"])]

We notice that "failing" is not in there - we can add it by tuning that 0.75 threshold from before. And, finally, if you want to include the percent that it matched the filter in your matchedTerms, replace it with the following:

let matchedTerms =
    terms
    |> List.map (fun t -> (t, t |> matchedFilters))
    |> List.filter (snd >> List.length >> any)
    |> List.map (fun (t, f) -> (t |> String.fromCodePoints, f |> List.map (fun (f, s) -> (f |> String.fromCodePoints, s))))

We just change the fst >> String.fromCodePoints to a lambda, which maps the first item to the string, and the second item as the value still.


Alright, so that sums up todays lesson. I know it got a bit rambly towards the end but that's what happens when you have absolutely no sleep and write this over a 5-day period. I actually have a plan for the next one, which involves more spreadsheet stuff, so tune-in next time for another fabulous adventure in learning how to program (but the hard way). Good luck and enjoy!

Loading