Saturday 18 May 2019

io redirection - Why is writing to a file from my python script getting overwritten

This might be a simple OS question rather than programming question so we might need to migrate this to another community.



I made a mistake and came across some weird behavior that I'd like to understand what's happening.



In my python script, I call a java program someone wrote. They use Logger to log some msgs and I'm writing the java program output to outfile. I also want to redirect the output of my script to another file b/c I don't want to search thru the sea of msgs from the java program.




out = sys.argv[2]
...
...
with open(out, 'w') as outfile:
# call some java programs via Popen but log
# some msgs after all the java programs have run
cmd = ["java", "-cp", "blah.jar", "com.main.blah", "param1", "param2"]
proc = subprocess.Popen(cmd, stdout=outfile)
proc.wait()

....
....
print "got here to test where this msg gets written"


When running my script, I redirected stdout and stderr to the same file name by accident.



python script.py input output > output 2>&1



Due to this accident, I came across this weird behavior. It seems like the redirected stdout and stderr gets written at the beginning of the file "output" rather than the end and thus overwrites any existing msgs from the java program



I'm just curious why this is happening. I'm guessing that the file pointer for the python script kept moving forward because the java program kept writing log msgs but the file pointer for the OS redirect didn't move b/c it didn't know anything was being written by the script, and thus, when the redirect's turn came, it wrote where it thinks is the end of the file. Is this correct? Hopefully someone can enlighten me on this situation. Thanks.



P.S. I wrote the command "correctly" and the java program output did go to one file and the redirect did go to another file



UPDATE:
Thanks to all the views and comments/answers. I think I didn't word my question correctly so I might delete this question. I don't think this is a duplicate question. This is not a question on appending or writing both stdout and stderr to the same file. I know how to do that. This is a question on having two things trying to write to the same file but the output gets all mixed up. I had a guess as to what was happening and put that in my original post but it seems like what I wrote isn't clear.

No comments:

Post a Comment

php - file_get_contents shows unexpected output while reading a file

I want to output an inline jpg image as a base64 encoded string, however when I do this : $contents = file_get_contents($filename); print &q...