Using an IF statement to return an unwelcome value to "" caused the chart to behave as you described, and I was experiencing the same problem.
However, when I substituted #N/A for "," The graphic ignored the erroneous data (it's vital to notice that there are no quotation marks, thus it should read #N/A and not "#N/A"). Even with an erroneous FALSE statement, it still functioned correctly; the only change was that #NAME? was returned as the error in the cell rather than #N/A. To illustrate what I mean, let me use a fictitious IF statement:
=IF(A1>A2,A3,"")
---> Returned "" into cell when statement is FALSE and plotted on chart
(this is unwanted as you described)
=IF(A1>A2,A3,"#N/A")
---> Returned #N/A as text when statement is FALSE and plotted on chart
(this is also unwanted as you described)
=IF(A1>A2,A3,#N/A)
---> Returned #N/A as Error when statement is FALSE and does not plot on chart (Ideal)
=IF(A1>A2,A3,a)
---> Returned #NAME? as Error when statement is FALSE and does not plot on chart
(Ideal, and this is because any letter without quotations is not a valid statement)