Can't sort column in report - "Baseline Due Date Variance"

I created a report that looks a milestones and has a column for "Baseline Due Date Variance". After I run the report, I try to sort by "Baseline Due Date Variance", but it will not sort. Even when I remove blanks or empty entries, it does not work. 

For example, when I sorted by "Baseline Due Date Variance" descending, it ordered them as following:

-70 d
-15 d
-70 d
-25 d
-45 d, etc.

I can only sort the report by "Due Date"