Large File Management

We will begin with a discussion about managing large file sizes, or huge stacks of small to medium size files. Fundimental constraints such as internet connection speed, hard drive size, RAM, and processing speed can make big differences when managing huge files. A few tips to get you started:

Turn off Updates

Most computers have automatic updates enables, especially if you are using an enterprise system (school computers or work computers). You will want to disable updates whenever you are working on something that will take a long while to complete (batch processing or a long download). Automatic updates have the ability to override your process and re-start the computer, leading to many tears. Avoid these tears, talk to your administrator about disabling automatic updates.

Internet speed

Even with a fairly fast connection it may take hours to days to download your dataset. There is little that you can do about this. When this is an issue I like to begin downloads before I go to sleep, or before I leave in the morning. On most personal internet connections demand surges in the AM when people are getting ready for work, and surges even more in the evening when people are streaming Netflix and whatnot. Avoid these times for peak performance.

Storage

External hard drives are getting really cheap. As of writing this (May 2015) a terrabite worth of external storage is about $50. Buy two and keep a backup of everything you do! You could consider backing your stuff up on Dropbox, but for these enormous file sizes that takes a lot of bandwidth.

Processing Speed

Again, there isn’t much that you can do about processing speed other than buying a faster machine. You should turn off all other processes (music, internet browser, etc) and let your computer do its thing. Running anything through an emulator will certainly cause lag, so select the appropriate software when possible.

Continue to FTP Process

Back to Index