Building a Twitter Bot: Part 2 (Finish it all up)

Lesson 20: Building a Twitter Bot

Today I want to finish up our Twitter Bot. There's really not a lot left to do, we just have to tie a bunch of loose-ends up, parameterize some stuff and make it reusable. (Remember our previous discussion: write good, readable, usable code.)

Identify Dependencies

There are always dependencies in code, you'll never get around this. I want to identify the ones in our code today, and use them to push us to the next stage of development.

• Dependency 1: the Consumer Key and Secret (hell, include the Access Token and Secret);
• Dependency 2: the text file (our lyric file);
• Dependency 3: the directory for said text file;
• Dependency 4: what to split on (\r\n\r\n in our case, but it could be anything);
• Dependency 5: the account ID;

This actually sums our dependencies up quite nicely. Now I want to define them in some sort of "configuration" file, in our case I'm going to use JSON because it's just so damn convenient. Let's define the aforementioned JSON config file:

{
      "ConsumerKey": "abcd1234",
      "ConsumerSecret": "abcd1234",
      "AccessToken": "1234-abcd1234",
      "AccessTokenSecret": "abcd1234",
      "AccountId": 9223372036854775807,
      "TextFile": "File.txt",
      "Split": "\r\n",
      "BaseDir": "."
}

Too easy. We can define the entire thing as 8 lines, that describe the type of data we expect. These are all the dependencies we have. All of them. Wrapped up with a nice little bow.

The next step is to bring those dependencies into our code. This is bewilderingly easy with F#: we're going to Install-Package FSharp.Data if you're using NuGet (just type that in the Package Manager Console), if you use packet then you're smarter than me and know exactly how to install it. (I know nothing of packet, but I hear it's nice.)

Now I save the config JSON above in a file called Config.sample.json, we'll need it to pull into F# so that we can define a type for the configuration. This is literally one line (well two, with the open):

open FSharp.Data
type Parameters = JsonProvider<"Config.sample.json">

And done. The JsonProvider will read our Config.sample.json file, and define an entire type for us to use. We can read something in as follows:

let parameters = configFile |> File.ReadAllText |> Parameters.Parse

Easy enough. The configFile will be a filename, I default to Config.json but you can use whatever you like.

Fix our Percent Encoding

If you follow me on Twitter, you'll notice that I built a bot at the request of a good friend / colleague of mine, and I recently tweeted that I broke it. (Which I did, but I fixed it.)

This issue was entirely in our percentEncode, and as I mentioned way back in the last lesson, we didn't account for Unicode characters. This was bad, at least for this new bot (for the Don McLean bot it was a non-issue). This is fixed easily enough, and I want to give you the code to do so:

open System.Text
let encoding = Encoding.UTF8
let percentEncode : string -> string =
    encoding.GetBytes
    >> Array.collect (fun x -> 
        match x with
        | cint when cint = 0x2Duy
                 || cint = 0x2Euy
                 || cint = 0x5Fuy
                 || cint = 0x7Euy
                 || (cint >= 0x30uy && cint <= 0x39uy)
                 || (cint >= 0x41uy && cint <= 0x5Auy)
                 || (cint >= 0x61uy && cint <= 0x7Auy) -> [|cint|]
        | cint -> sprintf "%%%s" (cint.ToString("X")) |> Seq.toArray |> Array.map byte)
    >> Array.map char
    >> String

So our previous code relied on analyzing the char, which is completely inappropriate for Twitter when dealing with non-ASCII text. The char in .NET is a UTF-16 character, whereas Twitter works with UTF-8. So, we have to make an adjustment for that. Instead of using the char, we just use the System.Text.Encoding.UTF8.GetBytes(str) function to get a byte-array for the text we need to encode, then we iterate through it, test each char for the necessity of encoding (because UTF-8 is full ASCII-128 compatible, we don't need to do anything special, our same character codes will work), and return the same resultant String. So nothing comsuming percentEncode has to change (awesome!)._Activator

Analyze a Timeline

So now we get to the hard part. For the bot to be successful I wanted it to be capable of analyzing it's timeline and determining what position it was in to tweet next. This would mean no state has to be retained anywhere, you could in fact run the .exe from any computer and it would still continue the correct sequence.

To do this, we have to read a timeline. To do that, we have to abstract our OAuth / API handling code. UGH

Alright, so this isn't actually that hard. We want to build a composition chain that goes through the full API request.

If you lost our sign function, I have it right here:

let sign httpMethod endpoint oauthParams queryParams postParams =
    Array.concat [|oauthParams; queryParams; postParams|]
    |> Array.sortBy fst
    |> Array.map keyValueToStr
    |> stringJoin "&"
    |> Array.singleton
    |> Array.append [|httpMethod |> httpMethodToStr; endpoint|]
    |> Array.map percentEncode
    |> stringJoin "&"
    |> hasher

You'll see it's slightly different. I have this httpMethodToStr function which, by the looks of things, takes something called httpMethod and converts it to a string. Ah right, I should share that!

type HttpMethod = | Get | Post
let httpMethodToStr = function | Get -> "GET" | Post -> "POST"

Because I like having the help of the compiler, I built a quick HttpMethod type which is just a Get or Post, and the httpMethodToStr converts that to the capital-case value.

The other helper there:

let keyValueToStr (key, value) = sprintf "%s=%s" key (value |> percentEncode)

Again, pretty simple. So now we can see that our sign function is really just some chains. You'll also see I have a hasher, which we didn't have last time.

let hmacSha1Hash (encoding : Encoding) (key : string) (str : string) : string =
    use hmacSha1 = new HMACSHA1(key |> encoding.GetBytes)
    str |> encoding.GetBytes |> hmacSha1.ComputeHash |> Convert.ToBase64String
let baseOauthParams = [|("oauth_signature_method", "HMAC-SHA1"); ("oauth_version", "1.0"); ("oauth_consumer_key", parameters.ConsumerKey); ("oauth_token", parameters.AccessToken)|]
let hasher = hmacSha1Hash encoding ([|parameters.ConsumerSecret; parameters.AccessTokenSecret|] |> stringJoin "&")

Here we see our parameters from the config come into play. We pull the appropriate static OAuth parameters (signature method, version, etc.) into a standard array, then we build a hasher which is a partially-applied function which will only require us to add the final string to hash. The key and encoding are embedded.

For ease of use, I built a sendRequest, which looks as follows:

let sendRequest (submitMethod, oauthString, url : string, postParams) =
    try
        use wc = new WebClient()
        wc.Headers.Add(HttpRequestHeader.Authorization, oauthString)
        match submitMethod with
        | Post -> wc.UploadString(url, postParams)
        | Get -> wc.DownloadString(url)
        |> Some
    with
    | :? WebException as ex ->
        use sr = new StreamReader(ex.Response.GetResponseStream())
        printfn "Failure (%s): %A" ex.Message (sr.ReadToEnd())
        None
    | ex ->
        printfn "Failure: %A" ex
        None

This was really quite simple. Add headers, determine submit method, send the request, return. If it errors, print the error and return None.

Now we previously had a buildTweetRequest which would do a lot of work to create a signed OAuth request. Today we took a lot of that away and instead created a buildBaseRequest, which can be a more abstract version:

let baseBuildRequest submitMethod path postParams queryParams =
    let url = [baseUrl; path] |> stringJoin ""
    let timestamp = DateTime.Now |> timeToEpoch
    let nonce = Guid.NewGuid().ToString("N")
    let oauthParams =
        [|("oauth_timestamp", timestamp.ToString()); ("oauth_nonce", nonce)|]
        |> Array.append baseOauthParams
    let oauthString =
        [|("oauth_signature", sign submitMethod url oauthParams queryParams postParams)|]
        |> Array.append oauthParams
        |> formOAuthString
    let queryString = queryParams |> Array.map keyValueToStr |> stringJoin "&"
    let queryString = if queryString.Length > 0 then sprintf "?%s" queryString else queryString
    let postString = postParams |> Array.map keyValueToStr |> stringJoin "&"
    (submitMethod, oauthString, (sprintf "%s%s" url queryString), postString)

You'll notice that it does a lot of the boilerplate, but it requires quite a few parameters to function properly. This will create the base request, which can be directly piped to sendRequest. This means our new buildTweetRequest looks as follows:

let buildTweetRequest tweet =
    baseBuildRequest Post "statuses/update.json" [||] [|("status", tweet)|]

Again, nothing complex, we just build a base request with the appropriate parameters, and then return it.

Consumption of sending a tweet at this point is literally tweet |> buildTweetRequest |> sendRequest. You can even check if it succeeeded or failed.

Of course, we aren't worried about that yet. We want to get a timeline first:

let buildGetTimeline (userid : int64) =
    baseBuildRequest Get "statuses/user_timeline.json" [||] [|("user_id", userid.ToString()); ("count", "25"); ("tweet_mode", "extended")|]

Oh yeah, getting a timeline is super smooth. Adding new API support, in fact, is very simple and easy. We might be in the home stretch.

let getTimelineTweets = buildGetTimeline >> sendRequest

OK, maybe not. We have the timeline now, but we have to build out the ability to read it.

I have a Twitter Timeline.sample.json which I'll upload as an attachment to this post, which I use to create a type-provider type:

type Timeline = JsonProvider<"Twitter Timeline.sample.json">

I also created a Tweet type where I only care about the important things:

type Tweet = { TId : int64; Text : string; Truncated : bool; CreatedAt : DateTime }

Because I don't give a hoot about the rest of the data, I made it a slimmed down version of an actual tweet.

We're also going to need to consume statuses/show.json, which I'm providing below:

let buildShowRequest (id : int64) =
    baseBuildRequest Get "statuses/show.json" [||] [|("id", id.ToString()); ("tweet_mode", "extended")|]

This is another pretty basic function, it takes a tweet ID and gives us the request to send to get that tweet.

Finally, to get the timeline into memory, we need a timelineToTweets function:

let timelineToTweets (timeline : string) : Tweet array option =
    let mapFn (x : Timeline.Root) = { TId = x.Id; Text = x.FullText; Truncated = x.Truncated; CreatedAt = DateTime.ParseExact(x.CreatedAt.Substring(0, 19), "ddd MMM dd HH:mm:ss", null) }
    let idsEqual t1 t2 = t1.TId = t2.TId
    let tweets = timeline |> Timeline.Parse |> Array.map mapFn
    let idsLookup = tweets |> Array.filter (fun x -> x.Truncated) |> Array.map (fun x -> x.TId)
    idsLookup
    |> Array.choose (buildShowRequest >> sendRequest >> Option.map (Timeline.Parse >> Array.map mapFn))
    |> Array.concat
    |> Some
    |> Option.map (fun x ->
        tweets |> Array.map (fun t1 ->
            x |> Array.filter (idsEqual t1) |> function | [|t|] -> t | _ -> t1))

It should be easy to see that we're just loading a timeline JSON into memory, checking what tweets are "truncated", then looking those up, and mapping them to a new tweet array. (If we had to get a new tweet, we'll use it, otherwise we keep the original.)

Alright, so we have an array of tweets (timeline) in memory, what do we do next? Well, we need to calculate what index we were most recently at, so we can find the next one in the list.

Because our lyric program has several tweets that will be identical in the same sequence, we need to look at more than one tweet in a row. I pulled 25 tweets from the timeline (look at the count parameter in buildGetTimeline), then make sure that we get as many matches as possible.

let getIndex (tweets : Tweet array) (comparisons : string array) =
    let comparisons = comparisons |> Array.rev |> Array.map newLineReplace
    let tweets =
        tweets
        |> Array.sortByDescending (fun x -> x.CreatedAt)
        |> Array.map (fun x -> x.Text)
        |> Array.filter (fun x -> comparisons |> Array.contains x)
    let rec alg skip =
        if skip >= comparisons.Length then 0
        else
            let si = comparisons |> Array.skip skip |> Array.findIndex ((=) tweets.[0]) |> (+) skip
            let matches = tweets |> Array.fold (fun (r, i) t -> (r && (comparisons.[i % comparisons.Length] = t), i + 1)) (true, si) |> fst
            if matches then si else alg (si + 1)
    if tweets.Length = 0 then 0 else alg 0

This is a somewhat convoluted function, but it basically starts at a tweet, then determines if it matched from there on in the comparisons (reference) array. If it does, then that's our current index.

Consume it all

Consuming this whole thing is done by the following function:

member this.run send =
    let path = [rootDir; parameters.TextFile] |> stringJoin ""
    let file = File.ReadAllText(path)
    let parts = file.Split([|parameters.Split|], StringSplitOptions.RemoveEmptyEntries) |> Array.filter (Seq.length >> (>=) 280) |> Array.map (fun x -> x.Trim())

    let existingTweets =
        parameters.AccountId
        |> getTimelineTweets
        |> Option.map timelineToTweets
        |> Option.flatten
        |> function | None -> [||] | Some v -> v
    let newStatus = parts.[(parts.Length - getIndex existingTweets parts) % parts.Length] |> newLineReplace
    let req = buildTweetRequest newStatus

    if send then (req, newStatus, req |> sendRequest)
    else (req, newStatus, None)

Wait, member? That's new.

So F# allows us to define types with members, and in fact, I defined almost all the functions above as let definitions in the type, and then the only necessary member is the single function above. This can be run by a client to either send the next tweet, or just load and return it. I built that in for the case when we want to debug, and see what it's going to send next without actually doing so.

All in all, the entire TwitterBot is 150 lines of F#:

module TwitterBot
open System
open System.IO
open System.Net
open System.Security.Cryptography
open System.Text
open FSharp.Data

type HttpMethod = | Get | Post
type Tweet = { TId : int64; Text : string; Truncated : bool; CreatedAt : DateTime }
type Parameters = JsonProvider<"Config.sample.json">
type Timeline = JsonProvider<"Twitter Timeline.sample.json">

let encoding = Encoding.UTF8
let percentEncode : string -> string =
    encoding.GetBytes
    >> Array.collect (fun x -> 
        match x with
        | cint when cint = 0x2Duy
                 || cint = 0x2Euy
                 || cint = 0x5Fuy
                 || cint = 0x7Euy
                 || (cint >= 0x30uy && cint <= 0x39uy)
                 || (cint >= 0x41uy && cint <= 0x5Auy)
                 || (cint >= 0x61uy && cint <= 0x7Auy) -> [|cint|]
        | cint -> sprintf "%%%s" (cint.ToString("X")) |> Seq.toArray |> Array.map byte)
    >> Array.map char
    >> String

let httpMethodToStr = function | Get -> "GET" | Post -> "POST"
let newLineReplace (str : string) = str.Replace("\r\n", " / ")
let stringJoin sep (strs : string seq) = String.Join(sep, strs)
let keyValueToStr (key, value) = sprintf "%s=%s" key (value |> percentEncode)
let keyValueToQuotedStr (key, value) = sprintf "%s=\"%s\"" key (value |> percentEncode)
let timeToEpoch (time : DateTime) =
    let epoch = DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
    (time.ToUniversalTime() - epoch).TotalSeconds |> bigint
let hmacSha1Hash (encoding : Encoding) (key : string) (str : string) : string =
    use hmacSha1 = new HMACSHA1(key |> encoding.GetBytes)
    str |> encoding.GetBytes |> hmacSha1.ComputeHash |> Convert.ToBase64String

type Bot(configFile) =
    let parameters = configFile |> File.ReadAllText |> Parameters.Parse
    let baseUrl = "https://api.twitter.com/1.1/"
    let rootDir = sprintf "%s%s" parameters.BaseDir "\\"
    let baseOauthParams = [|("oauth_signature_method", "HMAC-SHA1"); ("oauth_version", "1.0"); ("oauth_consumer_key", parameters.ConsumerKey); ("oauth_token", parameters.AccessToken)|]
    let hasher = hmacSha1Hash encoding ([|parameters.ConsumerSecret; parameters.AccessTokenSecret|] |> stringJoin "&")

    let formOAuthString : (string * string) array -> string =
        Array.sortBy fst
        >> Array.map keyValueToQuotedStr
        >> stringJoin ", "
        >> sprintf "OAuth %s"

    let sign httpMethod endpoint oauthParams queryParams postParams =
        Array.concat [|oauthParams; queryParams; postParams|]
        |> Array.sortBy fst
        |> Array.map keyValueToStr
        |> stringJoin "&"
        |> Array.singleton
        |> Array.append [|httpMethod |> httpMethodToStr; endpoint|]
        |> Array.map percentEncode
        |> stringJoin "&"
        |> hasher

    let sendRequest (submitMethod, oauthString, url : string, postParams) =
        try
            use wc = new WebClient()
            wc.Headers.Add(HttpRequestHeader.Authorization, oauthString)
            match submitMethod with
            | Post -> wc.UploadString(url, postParams)
            | Get -> wc.DownloadString(url)
            |> Some
        with
        | :? WebException as ex ->
            use sr = new StreamReader(ex.Response.GetResponseStream())
            printfn "Failure (%s): %A" ex.Message (sr.ReadToEnd())
            None
        | ex ->
            printfn "Failure: %A" ex
            None

    let baseBuildRequest submitMethod path postParams queryParams =
        let url = [baseUrl; path] |> stringJoin ""
        let timestamp = DateTime.Now |> timeToEpoch
        let nonce = Guid.NewGuid().ToString("N")
        let oauthParams =
            [|("oauth_timestamp", timestamp.ToString()); ("oauth_nonce", nonce)|]
            |> Array.append baseOauthParams
        let oauthString =
            [|("oauth_signature", sign submitMethod url oauthParams queryParams postParams)|]
            |> Array.append oauthParams
            |> formOAuthString
        let queryString = queryParams |> Array.map keyValueToStr |> stringJoin "&"
        let queryString = if queryString.Length > 0 then sprintf "?%s" queryString else queryString
        let postString = postParams |> Array.map keyValueToStr |> stringJoin "&"
        (submitMethod, oauthString, (sprintf "%s%s" url queryString), postString)

    let buildShowRequest (id : int64) =
        baseBuildRequest Get "statuses/show.json" [||] [|("id", id.ToString()); ("tweet_mode", "extended")|]
    let buildGetTimeline (userid : int64) =
        baseBuildRequest Get "statuses/user_timeline.json" [||] [|("user_id", userid.ToString()); ("count", "25"); ("tweet_mode", "extended")|]
    let buildTweetRequest tweet =
        baseBuildRequest Post "statuses/update.json" [||] [|("status", tweet)|]
    let getTimelineTweets = buildGetTimeline >> sendRequest

    let timelineToTweets (timeline : string) : Tweet array option =
        let mapFn (x : Timeline.Root) = { TId = x.Id; Text = x.FullText; Truncated = x.Truncated; CreatedAt = DateTime.ParseExact(x.CreatedAt.Substring(0, 19), "ddd MMM dd HH:mm:ss", null) }
        let idsEqual t1 t2 = t1.TId = t2.TId
        let tweets = timeline |> Timeline.Parse |> Array.map mapFn
        let idsLookup = tweets |> Array.filter (fun x -> x.Truncated) |> Array.map (fun x -> x.TId)
        idsLookup
        |> Array.choose (buildShowRequest >> sendRequest >> Option.map (Timeline.Parse >> Array.map mapFn))
        |> Array.concat
        |> Some
        |> Option.map (fun x ->
            tweets |> Array.map (fun t1 ->
                x |> Array.filter (idsEqual t1) |> function | [|t|] -> t | _ -> t1))

    let getIndex (tweets : Tweet array) (comparisons : string array) =
        let comparisons = comparisons |> Array.rev |> Array.map newLineReplace
        let tweets =
            tweets
            |> Array.sortByDescending (fun x -> x.CreatedAt)
            |> Array.map (fun x -> x.Text)
            |> Array.filter (fun x -> comparisons |> Array.contains x)
        let rec alg skip =
            if skip >= comparisons.Length then 0
            else
                let si = comparisons |> Array.skip skip |> Array.findIndex ((=) tweets.[0]) |> (+) skip
                let matches = tweets |> Array.fold (fun (r, i) t -> (r && (comparisons.[i % comparisons.Length] = t), i + 1)) (true, si) |> fst
                if matches then si else alg (si + 1)
        if tweets.Length = 0 then 0 else alg 0

    member this.run send =
        let path = [rootDir; parameters.TextFile] |> stringJoin ""
        let file = File.ReadAllText(path)
        let parts = file.Split([|parameters.Split|], StringSplitOptions.RemoveEmptyEntries) |> Array.filter (Seq.length >> (>=) 280) |> Array.map (fun x -> x.Trim())

        let existingTweets =
            parameters.AccountId
            |> getTimelineTweets
            |> Option.map timelineToTweets
            |> Option.flatten
            |> function | None -> [||] | Some v -> v
        let newStatus = parts.[(parts.Length - getIndex existingTweets parts) % parts.Length] |> newLineReplace
        let req = buildTweetRequest newStatus

        if send then (req, newStatus, req |> sendRequest)
        else (req, newStatus, None)

This is almost everything necessary, the only thing left to do is run it.

Build a robust client

Because I want robust bot, and I want to be able to hot-swap config files, I set our bot up so that it takes two command line arguments (if you want):

• a boolean to indicate whether or not to send;
• a string to indicate the configuration file to load;

Basically, I want to specify the following:

type Parms = { Send : bool; Config : string }

Well that's easy. I've demonstrated this before, but my default pattern for this type of situation is as follows:

let stripQuotes (s : string) = if s.StartsWith "\"" && s.EndsWith "\"" then s.Substring(1, s.Length - 2) else s
let defParms = { Send = false; Config = "Config.json" }
let parms = 
    match argv with
    | [|s; c|] -> { defParms with Send = (s = "true"); Config = c |> stripQuotes }
    | [|s|] -> { defParms with Send = (s = "true") }
    | _ -> defParms
printfn "%A" parms

Essentially, I specify reasonable defaults, then when a command-line interface calls it, you can send arguments to override.

Then, the basic setup:

let bot = parms.Config |> TwitterBot.Bot
let request, tweet, response = bot.run parms.Send
let encodedTweet = tweet |> percentEncode
printfn "Request: %A" request
printfn "Tweet: (%i chars) %A" tweet.Length tweet
printfn "Encoded: (%i chars) %A" encodedTweet.Length encodedTweet
printfn "Response: %A" response

If the Parms.Send was true, the tweet sent, so in the case it's not then I do some human prompting:

if parms.Send |> not then
    let rec getResponse () =
        printfn "Send tweet? (Y for yes, N for no)"
        let key = Console.ReadKey().Key
        printfn ""
        match key with
        | ConsoleKey.Y -> true
        | ConsoleKey.N -> false
        | k ->
            printfn "Invalid key: %A" k
            () |> getResponse

    match () |> getResponse with
    | true -> 
        let request, tweet, response = bot.run true
        let encodedTweet = tweet |> percentEncode
        printfn "Request: %A" request
        printfn "Tweet: (%i chars) %A" tweet.Length tweet
        printfn "Encoded: (%i chars) %A" encodedTweet.Length encodedTweet
        printfn "Response: %A" response
        printfn "Press enter to exit..."
        Console.ReadLine() |> ignore
    | false -> ()

This means we get the best of both worlds: human-interface compatibility, and argument compatibility. Overall, the whole file is 47 lines:

open System
open TwitterBot

type Parms = { Send : bool; Config : string }

[<EntryPoint>]
let main argv =
    let stripQuotes (s : string) = if s.StartsWith "\"" && s.EndsWith "\"" then s.Substring(1, s.Length - 2) else s
    let defParms = { Send = false; Config = "Config.json" }
    let parms = 
        match argv with
        | [|s; c|] -> { defParms with Send = (s = "true"); Config = c |> stripQuotes }
        | [|s|] -> { defParms with Send = (s = "true") }
        | _ -> defParms
    printfn "%A" parms

    let bot = parms.Config |> TwitterBot.Bot
    let request, tweet, response = bot.run parms.Send
    let encodedTweet = tweet |> percentEncode
    printfn "Request: %A" request
    printfn "Tweet: (%i chars) %A" tweet.Length tweet
    printfn "Encoded: (%i chars) %A" encodedTweet.Length encodedTweet
    printfn "Response: %A" response

    if parms.Send |> not then
        let rec getResponse () =
            printfn "Send tweet? (Y for yes, N for no)"
            let key = Console.ReadKey().Key
            printfn ""
            match key with
            | ConsoleKey.Y -> true
            | ConsoleKey.N -> false
            | k ->
                printfn "Invalid key: %A" k
                () |> getResponse

        match () |> getResponse with
        | true -> 
            let request, tweet, response = bot.run true
            let encodedTweet = tweet |> percentEncode
            printfn "Request: %A" request
            printfn "Tweet: (%i chars) %A" tweet.Length tweet
            printfn "Encoded: (%i chars) %A" encodedTweet.Length encodedTweet
            printfn "Response: %A" response
            printfn "Press enter to exit..."
            Console.ReadLine() |> ignore
        | false -> ()
    0

I include the stripQuotes as when we send a path from argument -> program we may need to quote it, which will be included in the argv element. So, we just need to strip them when we expect them.

And that's it, we've built the whole bot, and we can now automate it entirely in one tight, self-contained program. There's only one external dependency, and even that is only for easy of development. We built all the functionality we wanted, and then some, and did it without any major pain points.

Shoutout to Janelle Shane, who's work inspired the second bot written with this script, and who's work also encouraged me to finish it up and tie this two-section blog post series up. I'll be continuing on with the next installment of "Getting Started with Programming and Getting Absolutely Nowhere" in the next post.

Files: Twitter Timeline.sample.json (70.11 kb); Config.sample.json (245.00 bytes); TwitterBot.fs (7.23 kb); Program.fs (1.76 kb)

Building a Twitter Bot

Lesson 19: Cleaning up our previous work

So today I want to do something fun. Periodically, when developing software, it's nice to do something that's not a regular-old-business-problem every once in a while. Recently, I learned there was a bot that only tweeted portions of "Africa" by "Toto". This is a grand idea, but there's just one slightly better song to use: American Pie by Don McLean.

Now another interesting point is that Twitter recently doubled the character limit for tweets: 140 -> 280 characters. So we could actually build our Don McLean's American Pie bot to tweet up to that length of characters. So we're going to go through the entire development lifecycle here, and we'll do it all in F# rather quickly.

Step 1: identify the problem (plus solution in this case)

The first step is to identify what our "problem" to be solved is. Well, isn't it obvious? No one has built a bot for American Pie by Don McLean yet! That's a real problem! Our solution will be to build said bot, and allow it to tweet the lyrics to American Pie on a regular basis.

One might think the first step is to browse the Twitter API, but not yet. We'll get to that in a moment, the first step is to analyze the song and determine how we want to split it.

I have heard this song so many times I happen to be able to type it from memory, which I did, then I evaluated the lyrics for accuracy.

Now usually we would take this time to design a solution, but there really isn't much to design, we basically need the following:

• Provide groupings of lyrics to the bot;
• Provide the timing / delay to the bot;
• Get it credentials;
• Periodically send a tweet of the next lyric group;

In larger software you would make this multiple steps, here we just make it one because of how simple it is.

Step 2: provide groupings of lyrics to the bot

This is easy, and literally three lines of code in F#:

let path = @"C:\Users\Elliott Brown\Desktop\American Pie Lyrics.txt"
let file = System.IO.File.ReadAllText(path)
let parts = file.Split([|"\r\n\r\n"|], System.StringSplitOptions.None)

We literally load the file with the lyrics (attached to this post), and then split it on double line breaks. That's it, we now have our lyric "parts". We can verify that each part will fit into a tweet by testing:

let partLengths = parts |> Array.map Seq.length

We should verify that the largest group we have is 212 characters. This will fit well within a tweet, so we should be good to continue.

Step 3: provide the timings / delays to the bot

For this, we want to check the Twitter API limiting information. We will see that it tells us that we're limited to 2,400 tweets per day, for those not keen on the math, that's 100 tweets an hour, which means we could send: 1.667 tweets per minute or 0.6 minutes between each tweet. We won't come near that limit, we'll go with 5 minutes between each tweet, for a total of 20 per hour. This should be well within the API limitations, and it should allow us to do exactly what we want.

Initially, we'll define this as follows:

let minutesBetweenTweets = 5.0

Step 4: get it credentials / API tokens

This is a little harder, we actually have to mess with Twitter for a moment now. What we're going to do is head over to http://apps.twitter.com and register an application. The rules have changed a while ago, and you now need a phone number attached to your account to register an application. For whatever reason twitter flagged the account I created and locked it, but it was easy to unlock. (Not sure why, plenty of examples of folks doing something like this.)

Alright, so once all that is done, we'll want to get a Consumer Key, Consumer Secret, and generate an Access Token and Access Token Secret. I'm going to use mine as the examples, but only for signature validation and I'm going to be regenerating them afterwards.

let consumerKey = "AMbmXRe0nKymYOv23rzpBkggN";
let consumerSecret = "LIzIQNW79G5p8EyNbGjgxgkzvnjp7OImc6AdNKvbDPIPfReK5B";
let accessToken = "934423086732075008-qWUnoqnWByTTYJzKNrK8GT50MMYXE5B";
let accessTokenSecret = "fT1bo6TMVgLjLf74b16OIkdUAeyPamhk62si8QR1Xb2KJ";

Step 5: sign a request

The first thing we need to know when accessing the Twitter API is that all requests are REST with special HTTP authorization headers. We also need to "sign" requests, as documented on the Twitter Developers website.

To sign the request we need to know a few things:

• The HTTP Method (POST in our case);
• The raw endpoint URL (https://api.twitter.com/1.1/statuses/update.json in our case);
• The query-string and OAuth parameters and values:
  • Query string: status
  • OAuth Parameters:
  • oauth_consumer_key (consumerKey);
  • oauth_nonce (we'll generate this);
  • oauth_signature_method (HMAC-SHA1);
  • oauth_timestamp (Unix Epoch current time);
  • oauth_token (accessToken);
  • oauth_version (1.0);

Once we have all that, we can begin the signing process. The next thing we have to do is "percent encode" the invalid characters. Twitter has a handy guide on doing this, which I have written code for below:

let percentEncode (str : string) =
str
    |> Seq.collect (fun x -> 
        let cint = x |> int
        match cint with
        | 0x2D | 0x2E | 0x5F | 0x7E -> [|x|]
        | cint when (cint >= 0x30 && cint <= 0x39)
                 || (cint >= 0x41 && cint <= 0x5A)
                 || (cint >= 0x61 && cint <= 0x7A) -> [|x|]
        | cint -> (sprintf "%%%s" (cint.ToString("X"))).ToCharArray())
    |> Seq.toArray
    |> System.String

Now I didn't account for any Unicode symbols, but the top three examples on that page should be properly encoded.

After we build this percent encoding, we need to sign our request. Signing is not nearly as hard as it could be here, we actually have some examples to fall back on. Basically, the process is as follows:

• Append the POST, Query String, and OAUTH parameters together;
• Sort alphabetically by key, then value (Twitter does not allow duplicate keys across the three, so sorting by key will be the only requirement here);
• Percent encode all values (not keys, since keys have nothing requiring encoding);
• Join the key and value into key=value;
• Join all the key=value strings into key=value&key2=value2...;
• Build the base string: `method&URL (percent encoded)&parameters (percent encoded again);
• Build the signing key: consumerSecret&accessTokenSecret, if accessTokenSecret is a blank string, leave the & in the result;
• Using HMAC-SHA1 with the singing key, hash the base string;
• Base-64 encode the result;

This is actually surprisingly easy with F#, we can do each step on it's own, or do a couple at a time. The key/value array sorting and such is pretty simple, so I do them in a quick chain.

let sign method endpoint oauthParams queryParams postParams =
    let keyValues =
        oauthParams
        |> Array.append queryParams
        |> Array.append postParams
        |> Array.sortBy fst
        |> Array.map (fun (k, v) -> (k, v |> percentEncode))
    let concatedStr =
        ("&", keyValues |> Array.map (fun (key, value) -> sprintf "%s=%s" key value))
        |> System.String.Join
    let baseStr = sprintf "%s&%s&%s" method (endpoint |> percentEncode) (concatedStr |> percentEncode)
    let signKey =
        sprintf "%s&%s" consumerSecret accessTokenSecret
        |> System.Text.Encoding.ASCII.GetBytes
    use hmacSha1 = new System.Security.Cryptography.HMACSHA1(signKey)
    baseStr
    |> System.Text.Encoding.ASCII.GetBytes
    |> hmacSha1.ComputeHash
    |> System.Convert.ToBase64String

As you can see, we do everything as best we can to maintain ease of readability and follow what best-practices we must. This makes the entire process quite painless, we pass in our parameters and get the signature result.

Step 6: form a request

If we look at the status update API documentation, we'll see that the status update in general is actually pretty simple. We just POST to https://api.twitter.com/1.1/statuses/update.json with a query-string parameter of status={{StatusText}}, this is trivial with F#.

The first step is forming our OAuth string:

let formOAuthString p =
    let paramsStr =
        (", ", p
        |> Array.sortBy fst
        |> Array.map (fun (k, v) -> sprintf "%s=\"%s\"" (k |> percentEncode) (v |> percentEncode)))
        |> System.String.Join
    sprintf "OAuth %s" paramsStr

We know the values in this section won't have double-quotes, so there's no need to guard here.

When dealing with F# (.NET) dates and times, we need to convert them to the 'unix epoch' times (which Twitter expects):

let timeToEpoch (time : System.DateTime) =
    (time.ToUniversalTime() - System.DateTime(1970, 1, 1, 0, 0, 0)).TotalSeconds

Next, we form up our parameters and such for signing:

let url = "https://api.twitter.com/1.1/statuses/update.json"

let newStatus = parts.[0]
let newStatus = newStatus.Replace("\r\n", " / ")
printfn "%s" (newStatus |> percentEncode)

let timestamp = System.DateTime.Now |> timeToEpoch |> bigint
let nonce = System.Guid.NewGuid().ToString("N")
let oauthParams =
    [|("oauth_signature_method", "HMAC-SHA1")
      ("oauth_version", "1.0")
      ("oauth_consumer_key", consumerKey)
      ("oauth_timestamp", timestamp.ToString())
      ("oauth_token", accessToken)
      ("oauth_nonce", nonce)|]
let postParams = [||]
let queryParams = [|("status", newStatus)|]
let signature = sign "POST" url oauthParams queryParams postParams

let oauthParams = [|("oauth_signature", signature)|] |> Array.append oauthParams
let oauthString = oauthParams |> formOAuthString

And that's it, we have the request formed. The final part is to do the POST itself.

Step 7: POST the request

To POST the request to Twitter we basically do three things: create a WebClient, add the Authorization header; send the request. F# with .NET makes this trivial, and we can even handle failure cases really easily.

try
    use wc = new System.Net.WebClient()
    wc.Headers.Add("Authorization", oauthString)
    let response = wc.UploadData(sprintf "%s?status=%s" url (newStatus |> percentEncode), [||]) |> System.Text.Encoding.UTF8.GetString
    printfn "Success: %s" response
with
| :? System.Net.WebException as ex ->
    use sr = new System.IO.StreamReader(ex.Response.GetResponseStream())
    printfn "Failure (HTTP %A): %A" ex.Status (sr.ReadToEnd())
| ex -> printfn "Failure: %A" ex

And that's it. Our Twitter bot now works. You'll also notice that I included a .Replace("\r\n", " / "), Twitter is funky in how it treats line-breaks, and if you actually include the real line breaks it doesn't respect them properly. It also fails OAuth verification, so we have to do something about that. My solution: replace them with a slash indicating line breaks, which is somewhat frequently used in lyric sharing. I made this a separate call for a reason: we're going to see in the next post how to turn that initial parts.[0] into a calculation as to which portion of the lyrics we are currently in. (Such that we don't need the application to run constantly, we can just run it once, and it will decide what the next lyric to post is.)

Step 8: clean things up

Now this entire application is about 90 lines for me, and works great, but there's a lot of ugly there, because we don't import (see: open) anything from .NET, and we have to use the String.Join which is just ugly. Let's fix that up a bit:

module TwitterBot =
    open System
    open System.IO
    open System.Net
    open System.Security.Cryptography
    open System.Text

    let private consumerKey = "AMbmXRe0nKymYOv23rzpBkggN"
    let private consumerSecret = "LIzIQNW79G5p8EyNbGjgxgkzvnjp7OImc6AdNKvbDPIPfReK5B"
    let private accessToken = "934423086732075008-qWUnoqnWByTTYJzKNrK8GT50MMYXE5B"
    let private accessTokenSecret = "fT1bo6TMVgLjLf74b16OIkdUAeyPamhk62si8QR1Xb2KJ"

    let private url = "https://api.twitter.com/1.1/statuses/update.json"
    let private encoding = Encoding.ASCII

    let stringJoin sep (strs : string seq) = String.Join(sep, strs)

    let timeToEpoch (time : DateTime) =
        let epoch = DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
        (time.ToUniversalTime() - epoch).TotalSeconds |> bigint

    let percentEncode : string -> string =
        Seq.collect (fun x -> 
            match x |> int with
            | 0x2D | 0x2E | 0x5F | 0x7E -> [|x|]
            | cint when (cint >= 0x30 && cint <= 0x39)
                     || (cint >= 0x41 && cint <= 0x5A)
                     || (cint >= 0x61 && cint <= 0x7A) -> [|x|]
            | cint -> (sprintf "%%%s" (cint.ToString("X"))).ToCharArray())
        >> Seq.toArray
        >> String

    let hmacSha1Hash (key : string) (str : string) : string =
        use hmacSha1 = new HMACSHA1(key |> encoding.GetBytes)
        str |> encoding.GetBytes |> hmacSha1.ComputeHash |> Convert.ToBase64String

    let sign method endpoint oauthParams queryParams postParams =
        let concatedStr =
            Array.concat [|oauthParams; queryParams; postParams|]
            |> Array.sortBy fst
            |> Array.map (fun (key, value) -> sprintf "%s=%s" key (value |> percentEncode))
            |> stringJoin "&"
        [|method; endpoint; concatedStr|]
        |> Array.map percentEncode
        |> stringJoin "&"
        |> hmacSha1Hash ([|consumerSecret; accessTokenSecret|] |> stringJoin "&")

    let formOAuthString : (string * string) array -> string =
        Array.sortBy fst
        >> Array.map (fun (k, v) -> sprintf "%s=\"%s\"" (k |> percentEncode) (v |> percentEncode))
        >> stringJoin ", "
        >> sprintf "OAuth %s"

    let buildTweetRequest tweet =
        let timestamp = DateTime.Now |> timeToEpoch
        let nonce = Guid.NewGuid().ToString("N")
        let oauthParams =
            [|("oauth_signature_method", "HMAC-SHA1")
              ("oauth_version", "1.0")
              ("oauth_consumer_key", consumerKey)
              ("oauth_timestamp", timestamp.ToString())
              ("oauth_token", accessToken)
              ("oauth_nonce", nonce)|]
        let postParams = [||]
        let queryParams = [|("status", tweet)|]
        let oauthString =
            [|("oauth_signature", sign "POST" url oauthParams queryParams postParams)|]
            |> Array.append oauthParams
            |> formOAuthString
        (oauthString, sprintf "%s?status=%s" url (tweet |> percentEncode), String.Empty)

    let run () =
        let path = @"C:\Users\Elliott Brown\Desktop\American Pie Lyrics.txt"
        let file = File.ReadAllText(path)
        let parts = file.Split([|"\r\n\r\n"|], StringSplitOptions.RemoveEmptyEntries)

        let newStatus = parts.[0]
        let newStatus = newStatus.Replace("\r\n", " / ")
        let oauthString, url, postParams = buildTweetRequest newStatus

    try
            use wc = new WebClient()
            wc.Headers.Add("Authorization", oauthString)
            let response = wc.UploadString(url, postParams)
            printfn "Success: %s" response
        with
        | :? WebException as ex ->
            use sr = new StreamReader(ex.Response.GetResponseStream())
            printfn "Failure (%s): %A" ex.Message (sr.ReadToEnd())
        | ex -> printfn "Failure: %A" ex

After all this rewriting, and making everything much clearer, my total line count did not change. It literally didn't change at all (even with new open statements), I did change total line-count during this change, that's huge. I guess my point here is that you should never worry about the line count, always write code readable and understandable first. Then deal with trimming lines down if absolutely necessary. (I usually don't worry about it until it's become unreasonable — several thousand lines, that is.)

Step 9: admire our handiwork

You can see the result of our handiwork on Twitter, you'll start seeing more and more tweets pop in there after we do the next lesson, when I demonstrate how we can build a "smart algorithm" to decide what to Tweet next.

The takeaway from this lesson should be two things: 1. We can actually do some fun / entertaining things in programming; 2. We always have another opportunity to practice;

I really want you to try to find something to do with programming that you enjoy, then try to learn how to do it. You can pick anything, easy, hard, whatever you want, just pick something that you like. The easiest way to convince yourself to keep learning is to find something you enjoy, and work towards it.

Also, worry not about the "security" issue of me sharing keys and secrets, I regenerated all four before this post.