rSync instead of (S)FTP upload


#1

Hi,

It would be great to have an rSync actions instead of (s)FTP. Saves a lot of time, and this could also help with keeping the file-system in sync (it is a pain that an upload from the file-system does not remove files).

Love buddy so far !


#2

This would be great!.

We we’ve been testing rsync upload from a PHP docker action. And syncing/uploading ~100MB of data takes only 5 seconds instead of 20 minutes with sftp.

One other option would be to have the buddy public/private key available in docker actions. But that’s more of a trick.

A rsync action would “need”:

  • access to buddy ssh pub/private key
  • ssh params: user/host/port/(pass)
  • source dir
  • destination dir
  • rsync exclude options
  • option to set rsync params.

Hope this makes any sense!

Love buddy too!


#3

Hi there,

We’re using ‘beam’ for this which is essentially a wrapper for rsync. Beam is an open source deployment tool we developed for deploying websites to servers: https://github.com/heyday/beam - it’s great as a standalone deployment tool, but can also be used within CI/CD tools like Buddy. It works with a JSON formatted config file stored in the project where you can store things like file excludes and targets [servers]. A few of the Beam safeguard features don’t really apply when running beam within the Buddy environment - like locking a target to a git branch, defaulting to deployments from a temp directory, but these can be overridden with command line params.

To use within a pipeline just add a PHPUnit Docker action which gives you a PHP environment. Then configure as follows:

Add this to the Packages & Setup Commands (you can also delete the phpunit install):
# Install Beam
curl https://getbeam.io/installer -O; php installer
mv beam /usr/local/bin/beam

In the Run Commands section add:
echo “Host *\n\tStrictHostKeyChecking no” >> /root/.ssh/config
beam self-update
beam up [target] --no-prompt --working-copy

You’ll also need to generate an SSH key called id_rsa as a Buddy environment variable (either for the pipeline, project or workspace) - this’ll allow the SSH access that rsync requires once you’ve added the associated public key to your server’s authorised_keys file.

You can also pass in rsync parameters.

We have it working great for us and file deployments are significantly faster given it’s using rsync.


#4

Thanks for the info Shane… We will definitely have a look !

ps (Always good to meet a fellow Silverstriper in weird places… :wink: )


#5

I second a rsync option. However, I got beam after a little time struggling to see why it wouldn’t run beam in a subdirectory. Turns out I had to have the beam.json in the directory I wanted to run it in otherwise it would traverse up directories until it found a beam.json file and use the directory it was in for the rsync.


#6

Hey guys! Thanks for your contribution, awesome work! FYI, we’re going to add a dedicated rsync action to the file transfer list if you’d like to use it instead of (S)FTP to save you the hassle.

PS. @lars-lemon8 “weird places?” What do you mean by “weird places?” :wink:


#7

@Kivlov , great news, Thanks!

ps(Weird , but interesting ! :wink:


#8

Sorry for reviving this old thread, but I was wondering why the rsync option requires me to upload a new ssh key instead of offering to use buddy’s key (in the dropdown) in the same way the sftp option does?


#9

@ken, the thing is that buddy key contains the passphrase and it’s not possible to pass it in the non-interactive mode.
However, we’ll add such option soon.