Spyder not counting lines in datafile correctly

[rkiko]

I conducted a coding class and use code to open a txt file with pandas. I need to skip parts of the header, using skiprows = 57. Some of the students reported that the code works with skiprows = 114 and when one checks the datafile in the spyder window, one sees that the file is shown to have 114 rows. Other students reported that even 114 did not work. So, spyder in some cases did corrupt the line number of the file on read in...

Data file shown on computer of a student:

Code working with skiprows set to 114:

The data file that needs to be read in:

Station_1_Schwentine.txt

File as shown on my Mac, with correct line number (also e.g. as shown in BBedit).

[ccordoba12]

Hey @rkiko, thanks for reporting. You said:

I conducted a coding class and use code to open a txt file with pandas. I need to skip parts of the header, using skiprows = 57. Some of the students reported that the code works with skiprows = 114 and when one checks the datafile in the spyder window, one sees that the file is shown to have 114 rows. Other students reported that even 114 did not work. So, spyder in some cases did corrupt the line number of the file on read in...

I think the problem could be that the file is saved by Spyder with a different line ending than the one with which it was originally created (each operating system has different characters to represent the end of a line). And that's what's causing the problem your students are seeing.

What Spyder version are you and your students using?

[ccordoba12]

@dalthviz, could you check what you get on Windows when opening the file posted by @rkiko? Thanks!

[dalthviz]

When I open the file on Windows with Spyder 6.0.2 from the Windows installer I see the file line numbers as shown in the macOS screenshot (so the data is after the line 57):

Checking the screenshot with the file that works with 114 as skiprows, seems to me like \n got somehow duplicated. I was unable to do something from Spyder to make the file to be in such state but checking with notepad++ (which has an extend find and replace that allows to search/replace special characters like \n) I was able to make the data be after the line 114 when replacing \n with \n\n:

That operation I guess could be though as mixing CRLF with LF line endings? Is there any Spyder Editor option/preference that could do such a change/replace @ccordoba12 ?

[ccordoba12]

No, there is not way to do that from Spyder.

@rkiko, did your students only open the file you posted and see it as in the first screenshot you posted? Or did they make some changes to it?

[rkiko]

@ccordoba12, according to my knowledge, the students did not change the file. I however asked them again and also asked for info about operating system and spyder version. Will get back to you asap.

[ccordoba12]

Thanks @rkiko! I think we should be able to get to the bottom of this with that info.

[rkiko]

Here is some info from the students:

Student 1:

"""
My spyder version is 5.5.1, OS is macOS 12.0.1 (Apple M1 chip), Python 3.12.7 64-bit | Qt 5.15.2 | PyQt5 5.15.10

When opening a data file, Spyder displays this warning:

"""

Student 2: works on Mac, with spyder version 5.5.1, has the same problem

Student 3: works on a Microsoft Windows (on a Surface Pro 7), now updated to spyder version 6.0.1, no longer has the issue.

So, the problem might be fixed between 5.5.1 and 6.0.1.

I have not yet asked the students with the problems to update their spyder, so that one could ask for more diagnostics.

Let me know if I should ask them to update.

Cheers,
Rainer.