Squirting big data

You may recall that a couple of weeks ago, I mentioned that that Squirt, a regex compliant command line FTP client, was falling over when handling very large files. I have finally found the time to take a look into this and finding a solution proved to be a lot easier than I expected.

In short, what the get and put functions now do is generate a list of files to squirt and then processes each file in its own FTP instance. This approach does make the app a little slower when handling multiple small files, but this is a cost worth paying for the massively increased reliability.

The code is a bit of a mess at the moment, and in dire need of some refactoring, but it can be found at GitHub if you are interested in how the workaround works.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s