ValueError could not convert string to float id

Question

I'm running this python script:

#!/usr/bin/python

import os,sys
from scipy import stats
import numpy as np

f=open('data2.txt', 'r').readlines()
N=len(f)-1
for i in range(0,N):
    w=f[i].split()
    l1=w[1:8]
    l2=w[8:15]
    list1=[float(x) for x in l1]
    list2=[float(x) for x in l2]
    result=stats.ttest_ind(list1,list2)
    print result[1]

I am getting this error:

ValueError: could not convert string to float: id

When I do it for only one line in the interactive section, instead of for loop using a script, it works.

>>> from scipy import stats
>>> import numpy as np
>>> f=open('data2.txt','r').readlines()
>>> w=f[1].split()
>>> l1=w[1:8]
>>> l2=w[8:15]
>>> list1=[float(x) for x in l1]
>>> list1
[5.3209183842, 4.6422726719, 4.3788135547, 5.9299061614, 5.9331108706, 5.0287087832, 4.57...]

Can someone explain why this is happening and how can I resolve this?

Answer 1

The problem is that in your code some lines don't have valid float data. In some lines, there is a text id that can't be converted to float. When you do it in an interactive prompt you are trying only the first line, so it's better to print the line where you are getting this error and you will know the wrong line.

 For example:

#!/usr/bin/python

import os,sys
from scipy import stats
import numpy as np

f=open('data2.txt', 'r').readlines()
N=len(f)-1
for i in range(0,N):
    w=f[i].split()
    l1=w[1:8]
    l2=w[8:15]
    try:
        list1=[float(x) for x in l1]
        list2=[float(x) for x in l2]
    except ValueError,e:
        print "error",e,"on line",i
    result=stats.ttest_ind(list1,list2)
    print result[1]

I hope this helps.