using Programming;

A Blog about some of the intrinsics related to programming and how one can get the best out of various languages.

.NET Gotcha: System.Net.WebException: The remote server returned an error

F# / .NET: Gotcha's

The dreaded System.Net.WebException: The remote server returned an error mess...

If you've ever used the WebClient to target RESTful API's in .NET, you probably hit this error at least once:

System.Net.WebException: The remote server returned an error: {Some Error}.
   at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
   at System.Net.WebClient.DownloadString(Uri address)
   at System.Net.WebClient.DownloadString(String address)

Scenario: you're just chugging along, working on some stuff, then an exception comes in and BAM, messes up your whole day.

I had this happen to a coworker yesterday. He was working on something and kept wondering why the endpoint was giving him a 404: Not Found in a WebException. He'd never really hit this before, because this was the first time he needed to consume a RESTful API in .NET.

What's happening here?

When the .NET WebClient sends a request, it's backed by a WebRequest, which happens to have someone's "good" idea: when we hit a non-200 status code, throw an exception. That makes sense, right? If we hit a 404, it makes sense to throw an exception, right?

Sure, in 1990. It's 2019 though, we use the 404 to mean something. Some genius out there said "hey, when building RESTful API's, just send an HTTP status code with your 'thing', we already have a list of them." The problem is that up, down, left and right people were treating 300, 400, and 500 status codes as actual errors, and things would break. Well, .NET is no exception.

So how do we fix it?

Yes, this is the easy part. We have to use a try / catch or try / with, depending on the language (OH NO!).

Effectively, the following code will throw a System.Net.WebException: The remote server returned an error: (404) Not Found. on line 7, when called on line 11. We'll fix that.

open System.Net
let urlBase = "https://httpstat.us/"

let getCode code =
    use wc = new WebClient()
    wc.Headers.Add(HttpRequestHeader.Accept, "application/json")
    code |> sprintf "%s%i" urlBase |> wc.DownloadString

200 |> getCode |> printfn "%s"
202 |> getCode |> printfn "%s"
404 |> getCode |> printfn "%s"

So, what do we do?

Easy, catch the WebException and read the response:

open System.IO
open System.Net
let urlBase = "https://httpstat.us/"

let getCode code =
    try
        use wc = new WebClient()
        wc.Headers.Add(HttpRequestHeader.Accept, "application/json")
        code |> sprintf "%s%i" urlBase |> wc.DownloadString
    with :? WebException as ex ->
        use sr = new StreamReader(ex.Response.GetResponseStream())
        sr.ReadToEnd()

200 |> getCode |> printfn "%s"
202 |> getCode |> printfn "%s"
404 |> getCode |> printfn "%s"

This isn't perfect, we'll catch all WebException issues, so we want one more "guard" clause:

open System.IO
open System.Net
let urlBase = "https://httpstat.us/"

let getCode code =
    try
        use wc = new WebClient()
        wc.Headers.Add(HttpRequestHeader.Accept, "application/json")
        code |> sprintf "%s%i" urlBase |> wc.DownloadString
    with :? WebException as ex when ex.Status = WebExceptionStatus.ProtocolError ->
        use sr = new StreamReader(ex.Response.GetResponseStream())
        sr.ReadToEnd()

200 |> getCode |> printfn "%s"
202 |> getCode |> printfn "%s"
404 |> getCode |> printfn "%s"

Bam. Now we only catch ProtocolError exceptions, like 400, 500, etc.

.NET Gotcha: TimeSpan: Invalid Format String

F# / .NET: Gotcha's

This is another annoying "feature" in .NET. For some reasons, TimeSpan.ToString with a custom format string does all sorts of wonky stuff, particularly it will give you the "Invalid Format String" error on any unescaped character that isn't a valid format specifier.

Basically, take this list: Custom TimeSpan Format Strings and treat it literally.

Want to add a space between days and hours/minutes/seconds? Escape it: d\ hh\:mm\:ss. Yes, even spaces.

F# Gotcha: Giraffe Task "FS0708"

F# / .NET: Gotcha's

If you are a user of Giraffe with ASP.NET Core and F#, you might have come across the following error:

FS0708: This control construct may only be used if the computation expression builder defines a 'Bind' method

This happens if you try to use a let! expression inside a task computation expression:

task {
    let! x = ...Async()

The problem here is that task as a computation expression doesn't define a "Bind" method, from what I can tell. There is, however, a quick workaround:

open FSharp.Control.Tasks.V2.ContextInsensitive

For some reason, this open fixes it. I assume that it allows the async computation-expression version of the Bind to be used, so that the code above works.

There's been discussion around this, from what I can tell, but it hasn't really resolved this issue, not sure if it's a version thing, or entirely unrelated: Q: remove async await bind from task {}?.

The Giraffe folks do a great job, so I want to make sure you can work around this particular issue if and when you encounter it.