The field type of 'Date/Time' must have a time, thus even if you leave it blank the time is defaulted.
It is usually best to have those fields separated on the data entry portion, and then have them combined later if needed.
If its possible for you, I'd recommend making a 'date' field and a 'time of day' field. Then you can easily create the formula field to default to what you need.
Generally speaking its always easier to combine, then to separate.
i.e. First Name & Last Name as two separate fields, can always be combined to one, but its harder to separate full name into its components later.