Python 3.4 text file write failure

[kclulow]

Please use the following bug reporting template to help produce actionable and reproducible issues. Please try to ensure that the reproduction is minimal so that the team can go through more bugs!

• A brief description
  When writing a large CSV file (greater than 50000 lines) by reading from a pymssql cursor and using the standard csv writer, the application hangs within the csvwriter.writelines method and fails to return after writing all lines.

Same code runs successfully on Centos7, Ubuntu, Red hat distros

• Expected results
  Method returns successfully
• Actual results (with terminal output if applicable)
  Method hangs
• Required packages and commands to install
  python3
  pip3
  pymssql

[aseering]

@kclulow -- interesting; could you share some example Python code that shows how you are calling writelines()? Preferably a minimal snippet of code that reproduces this issue.

I don't know offhand of any known issues with writing to files. But there are some known issues with socket I/O ( #575 #610 #616 ). If you're passing writelines() a generator s.t. it's streaming data over the network socket from pymssql as it's writing data to the file, then one of those issues might be the root cause.

[kclulow]

@aseering

`

# serialize a SQL Server cursor result to a csv.
def write_file(filename, cursor, do_headers = False, do_compress = False):

print_time()
if(do_compress==True):
    filename += ".gz"    
    csv_file = gzip.open(filename, "wt", encoding='utf-8')
else:
    csv_file = open(filename, "wt", encoding='utf-8')

csv_writer = csv.writer(csv_file)
if do_headers == True: csv_writer.writerow([i[0] for i in cursor.description]) # write headers to CSV
csv_writer.writerows(cursor) # hangs here if cursor has more than 50k rows
csv_file.close()

print_time()
return filename`

[kclulow]

Yeah 616 sounds very like the behaviour here. Also note, this call is executing over a high bandwidth high latency link between Sydney Australia and Las Vegas USA which may exacerbate #616

[sunilmut]

Marking this duplicate of #616. The fix for #616 is in the dev branch and in-bound for Insider build. Will appreciate if you can validate this once the fix hits the Insider build. Will ping this thread once the fix is available.