Duration to Numeric not working

I am trying to calculate the difference between an actual and forecast date using the following formula:

If(not IsNull([Master Data - PA Date]), ToNumber(ToText([Master Data - Scheduled Finish Date - CCD] - [CCD at T - 7] )))

The only results that are populating are the ones where both dates match, Then I get a 0 as expected.  but I don't get positive or negative numbers displaying if the dates do not match.

------------------------------
Paul Peterson
------------------------------