Google
 
Web pyadmin.blogspot.com

Friday, September 08, 2006

A tricky secure bulk file upload

Last week i came across a situation that, I hav to upload a bulk file of 52 MB through a very unsteady internet connection.This python program is for that,which splits a files into pieces and do an secure copy and reconstruct the file there,at the destination.This script uses the python implementation of expect command in unix,namely pexpect.This is a very useful class,which can be used to execute almost all commands and we can spawn a new process and we can control the session of the process through this class.
here's the script


#!/usr/bin/python
import pexpect
import os
import sys
import getpass

failedlist=[]
def splitter (filename,size,prefix):
os.popen('split -a3 -b %s %s %s'%(size,filename,prefix))
fin,fout=os.popen2('ls %s*'%(prefix))
list=fout.readlines()
return list
def doSsh (user, host, password, command):
"""This runs a command on the remote host. This returns a
pexpect.spawn object. This handles the case when you try
to connect to a new host and ssh asks you if you want to
accept the public key fingerprint and continue connecting.
"""
ssh_newkey = 'Are you sure you want to continue connecting'
child = pexpect.spawn('ssh -l %s %s %s'%(user, host, command))
i = child.expect([pexpect.TIMEOUT, ssh_newkey, 'password: '])
if i == 0: # Timeout
print 'ERROR!'
print 'SSH could not login. Here is what SSH said:'
print child.before, child.after
return None
if i == 1: # SSH does not have the public key. Just accept it.
child.sendline ('yes')
child.expect ('password: ')
i = child.expect([pexpect.TIMEOUT, 'password: '])
if i == 0: # Timeout
print 'ERROR!'
print 'SSH could not login :'
print child.before, child.after
return None
child.sendline(password)
return child
def doScp(filename,user,host,passwd,dest_dir):
ssh_newkey = 'Are you sure you want to continue connecting'
print 'doing scp'
string='scp '+filename+' '+user+'@'+host+':'+dest_dir
print string
child = pexpect.spawn(string)
print 'after spawning'
i = child.expect([pexpect.TIMEOUT, ssh_newkey, 'password: '])
if i == 0: # Timeout
print 'ERROR!'
print 'Copying failed: '
print child.before, child.after
failedlist.append(filename)
return None
if i == 1: # SSH does not have the public key. Just accept it.
child.sendline ('yes')
child.expect ('password: ')
i = child.expect([pexpect.TIMEOUT, 'password: '])
if i == 0: # Timeout
print 'ERROR!'
print 'Copying Failed :'
print child.before, child.after
return None
child.sendline(passwd)
child.expect([pexpect.TIMEOUT, '100%'])
print 'uploading done'
return child


def doUpload(list,user,passwd,host,dest_dir):
for file in list:
doScp(file,user,host,passwd,dest_dir)

list=splitter('install','1m','.splitted.tmp.')
doUpload(list,'root','password','192.168.1.2','/root/')
doSsh('root','192.168.1.2','password','cat .splitted.tmp.* > install')
doSsh('root','192.168.1.2','password','rm -f .splitted.tmp.*')



19 Comments:

At 9:10 AM, Anonymous Anonymous said...

Unlimited Earnings Potential - http://1greatfuture.com

Our company is rapidly growing and offers you an extraordinary income helping others succeed. The primary requirement is to follow up on client inquiries and point them in the right direction. It is stress free, rewarding and straightforward work.

For complete details: http://1greatfuture.com


(Please feel free to delete this post if you don't want it on your blog. Thanks for the informative blog and opportunity to post.)

 
At 11:29 AM, Blogger Unknown said...

This comment has been removed by a blog administrator.

 
At 2:10 AM, Anonymous Anonymous said...

Sorry, but I have to tell you...

You wasted your time with this one. Try this sometime:

man rsync

Then start with this as a first test:

rsync -e ssh -avz myLargeFile.dat \
user@host:/dir/to/upload/to/

You get, for free, a secure compressed resumable file transfer over any (even unreliable) network. If it fails in the middle, simply run the same command again and it will continue where it left off.

 
At 9:15 AM, Blogger Sagar said...

My friend, if you know indentation is everything in Python.

your code is no more useful thus.

 
At 6:25 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 12:32 PM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 4:12 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 4:12 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 12:46 PM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 11:01 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 8:30 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 8:30 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 8:43 AM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 11:43 AM, Blogger 凱許倫 said...

This comment has been removed by a blog administrator.

 
At 9:38 AM, Blogger 建邱勳 said...

This comment has been removed by a blog administrator.

 
At 2:40 AM, Blogger 雅王任 said...

This comment has been removed by a blog administrator.

 
At 12:20 PM, Anonymous Anonymous said...

This comment has been removed by a blog administrator.

 
At 10:01 PM, Blogger 孫邦柔 said...

This comment has been removed by a blog administrator.

 
At 8:49 PM, Blogger 孫邦柔 said...

This comment has been removed by a blog administrator.

 

Post a Comment

<< Home