Discussion List Archives

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Possible addition for EPC-L agenda


For the EPC-L committee meeting at Glasgow if this is time.

For the pan-crystallographic mirrors:
Would a possible addition be to discuss replacing the
WGET FTP based mirroring  with the RSYNC protocol and program.
(both WGET and RSYNC are open source freely available software)

The main visible advantage (besides being a possible
order of magnitude more efficient) is that it can
delete old files and directories if the disappear from
the server.
(now much of the material presently on the IUCr mirrors
is redundant?).  Also, if there is concern of inappropriate
material seeping into future mirrors - Rsync would delete
this material on the regional mirrors once it was deleted
on the primary one.  WGET/FTP does not and the inappropriate
material would have to be manually deleted on each 
regional mirror.

------

Other advantages include: can work via Secure Shell - thus
passwords are encrypted and uses SSH compression.  For ASCII files, 
it only transfers the differences between the files.
(thus if one line in a 100 Meg file has changed - in theory it
only transfers the one line rather than the entire 100 Meg File)
RSYNC was made for this type of mirroring - WGET/FTP was not.
Is easy to set permissions and only allow specified computers
to log in via rsync - as well as have password validation 
via Secure Shell session so passwords cannot be sniffed over
the network.

------

While there can be some nuances with RSYNC with greater
than 1 Gig mirrors - overall I have had very good exeriences
with RSYNC for not only mirroring material over to the CCP14
site (www.netlib.org, www.gnu.org)  but also doing a daily
maintenance of the regional CCP14 mirrors in Australia and 
Canada (which are 2.3 Gig in size).

If needbe, I can use the CCP14 web-server as guinea-pig to
trial this.

Lachlan.

-- 
Lachlan M. D. Cranswick

Collaborative Computational Project No 14 (CCP14)
    for Single Crystal and Powder Diffraction
Daresbury Laboratory, Warrington, WA4 4AD U.K
Tel: +44-1925-603703  Fax: +44-1925-603124
E-mail: l.cranswick@dl.ac.uk  Ext: 3703  Room C14
                           http://www.ccp14.ac.uk

Reply to: [list | sender only]
International Union of Crystallography

Scientific Union Member of the International Science Council (admitted 1947). Member of CODATA, the ISC Committee on Data. Partner with UNESCO, the United Nations Educational, Scientific and Cultural Organization in the International Year of Crystallography 2014.

International Science Council Scientific Freedom Policy

The IUCr observes the basic policy of non-discrimination and affirms the right and freedom of scientists to associate in international scientific activity without regard to such factors as ethnic origin, religion, citizenship, language, political stance, gender, sex or age, in accordance with the Statutes of the International Council for Science.

Useful links