Studio:Transferring Files

Most modern devices have some sort of filesystem and it is often necessary to move files between the host platform and the target device during testing. This article briefly reviews some of the options for transferring files to and from your device.

FTP or SCP

If your device is unix based, it is possible that you have one or both of these services available on the device. There are standard, well-documented perl packages for dealing with these protocols: Net::FTP and Net::SSH2 (see scp_get and scp_put). If you have one of these services available on your device, they should be seriously considered since they provide a standard, well-tested protocol for transferring files.

netcat

While FTP and SCP are often omitted from embedded linux builds, it's not uncommon to have netcat available (particularly if you are using BusyBox for your tools implementation). Netcat provides simple sockets based data transfer capability. It is possible to craft perl code that can transfer files using basic sockets communication and netcat on the target side. STRIDE provides one such implementation in the Linux SDK utilities (%STRIDE_DIR%\SDK\Linux\lib\perl\TargetUtilities.pm). This module provides RemoteGet and RemotePut methods that use telnet to establish a remote session with the device and netcat to transfer files. If you have these services available, you can likely use this module without modification. If you require some other form of login or remote process invocation, you might find it helpful to refer to this module when implementing your own netcat-based solution.

custom device tools

If your device runs a proprietary OS and has a filesystem, it is likely that someone has developed custom tools for transferring data and/or files to the device. You should consult with your system team to determine what options are available for these non-standard systems. If it turns out that communication channels are available on the device for file transfer, it is likely that you could craft perl code to automate the communcation. For instance, perl has several libraries available for doing serial communication. Other data interfaces are also probably available - we recommend searching for available packages at CPAN to see if something suits your needs.

STRIDE Runtime Payloads

If you prefer, you can build file transfer capability into your application using the STRIDE messaging subsystem. We propose some source code here that can be incorporated into your target application code. Once incorporated, the interface we define can be invoked from host-based script code to do file transfers to/from the device. The code presented here is sample code - you should modify it accordingly for your platform and needs

In this example code, we define a single target function that controls the sending or reading of file data. The data and final bytes tally are delivered by STRIDE broadcast messages.

Device Implementation

header (definitions)

#pragma once
#include <sr.h>

/* this is the number of bytes we attempt to read
 * and send in one message.  You may want to tune 
 * this parameter for efficiency with your stdio
 * implementation and STRIDE runtime. In general, 
 * larger chunks will result in fewer broadcast 
 * messages which should improve transfer time
 */
#define FILETRANSFER_DATA_CHUNK 1024*8
#define MAX_FILE_PATH 255
#define FILETRANSFER_BASE_SMID 10

#define BRD_FILEREADY (FILETRANSFER_BASE_SMID+1 | srMT_BRD | srST_RSP_VAL)
#define BRD_FILEDATA  (FILETRANSFER_BASE_SMID+2 | srMT_BRD | srST_RSP_VAL)
#define BRD_FILEDONE  (FILETRANSFER_BASE_SMID+3 | srMT_BRD | srST_RSP_VAL)

typedef struct
{
    unsigned char* pData;
    int bytes;
} FileData_t;

typedef struct
{
    unsigned int totalBytes;
} FileDone_t;

typedef enum 
{
    FILETRANSFER_SEND = 0,
    FILETRANSFER_RECEIVE
} TransferType_e;

#ifdef __cplusplus
extern "C" {
#endif

unsigned int transfer_file(TransferType_e type, char* file_path);

#ifdef _SCL
#pragma scl_msg(BRD_FILEREADY)
#pragma scl_msg(BRD_FILEDONE, FileDone_t)
#pragma scl_ptr_sized(FileData_t, pData, "RETURN", "PRIVATE", FILETRANSFER_DATA_CHUNK, bytes)
#pragma scl_msg(BRD_FILEDATA, FileData_t)

#pragma scl_function(transfer_file)
#pragma scl_string(transfer_file, file_path, MAX_FILE_PATH)
#endif

#ifdef __cplusplus
}
#endif

Source (implementation)

This implementation either subscribes-to or broadcasts the STRIDE messages related to file transfer. This implementation uses the stdio interfaces for file I/O - you should change this as appropriate for your target device.

Note: the first #include statement below includes the definitions above - change the statement to use whichever file you place the definitions in.

#include "FileTransfer.h"
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "srconn.h"
#pragma warning(disable: 4996)

unsigned int transfer_file(TransferType_e type, char* file_path)
{
    unsigned int retval = 0;    
    srWORD wSTID        = 0;
    srDWORD dwNID       = 0;
    FILE* fStream       = NULL;
    char openMode[8];

    /* initialization, STID lookup */
    if (srQueryName("strideIM", &wSTID) != srOK) {
        srPrintError("srQueryName for strideIM failed");
        goto transfer_quit;
    }

    if (srQuerySTID(&dwNID, wSTID) != srOK) {
        srPrintError("srQuerySTID failed");
        goto transfer_quit;
    }

    srSubscribe(wSTID, srBOX_5, srCONNECT_STATUS_B_SMID, srTRUE);
    srSubscribe(wSTID, srBOX_5, BRD_FILEREADY, srTRUE);

    if ((file_path == NULL) || (strlen(file_path) == 0)) {
        srPrintError("No file provided");
        goto transfer_error;
    }

    if (type == FILETRANSFER_SEND) 
        strcpy(openMode, "rb");
    else if (type == FILETRANSFER_RECEIVE)
        strcpy(openMode, "wb");
    else {
        srPrintError("invalid transfer type");
        goto transfer_error;
    }

    if ((fStream = fopen(file_path, openMode)) == NULL) {
        srPrintError("file open failed");
        goto transfer_error;
    }

    if (type == FILETRANSFER_SEND) {
        /* file being sent to the host from the target */
        srDWORD dwMsgInst = 0;
        palDWORD dwEvents = palALL_EVENTS_MASK;
        srDWORD dwSMIDRead = 0;
        srWORD  wReadRet = srOK;
        srBoxInfo_t tBoxInfo;
        srWORD wDummySize = 0;
        unsigned char readBuf[FILETRANSFER_DATA_CHUNK];
        
        /* wait for the READY message to be broadcast before sending data */
        palWait(dwNID, &dwEvents);
        if (dwEvents & palSTOP_EVENT) {
            /* repost the stop event */
            palNotify(dwNID, palSTOP_EVENT);
            goto transfer_done;
        }

        if (srQueryBox(wSTID, srBOX_5, &tBoxInfo) != srOK) {
            srPrintError("unable to read box for ready message");
            goto transfer_done;
        }

        wReadRet = srRead(wSTID,
                          srBOX_5,
                          0,
                          &dwSMIDRead,
                          NULL,
                          &wDummySize,
                          &dwMsgInst);

        if ((wReadRet != srOK) || (dwSMIDRead != BRD_FILEREADY)) {
            srPrintError("did not get ready message");
            goto transfer_done;
        }

        while (1) {
            size_t bytesRead = fread(readBuf, sizeof(unsigned char), FILETRANSFER_DATA_CHUNK, fStream);
            if (bytesRead > 0) {
                FileData_t data;
                srWORD wEptr0 = srPTR_EMPTY;
                data.pData = readBuf;
                data.bytes = bytesRead;
                srPtrSetup(wSTID, BRD_FILEDATA, (srWORD)(0), srNULL, bytesRead*sizeof(unsigned char),
                            srMSGDIR_RESPONSE, srPTRDIR_RSP_RET, srPTR_USAGE_PRIVATE, &wEptr0);
                srBroadcast(wSTID, BRD_FILEDATA, (srBYTE*)&data, sizeof(FileData_t));
                retval += bytesRead;
            }

            if (feof(fStream)) {
                break;
            }

            if (ferror(fStream)) {
                srPrintError("file read error");
                break;
            }
        }
    }
    else {
        /* file sent from host to target */
        srBOOL  bReceiving = srTRUE;
        srSubscribe(wSTID, srBOX_5, BRD_FILEDATA, srTRUE);
        srSubscribe(wSTID, srBOX_5, BRD_FILEDONE, srTRUE);
        srBroadcast(wSTID, BRD_FILEREADY, NULL, 0);

        while(bReceiving == srTRUE) {
            srDWORD dwMsgInst = 0;
            palDWORD dwEvents = palALL_EVENTS_MASK;
            srBoxInfo_t tBoxInfo;
            palWait(dwNID, &dwEvents);
            if (dwEvents & palSTOP_EVENT) {
                /* repost the stop event */
                palNotify(dwNID, palSTOP_EVENT);
                break;
            }

            if (srQueryBox(wSTID, srBOX_5, &tBoxInfo) == srOK) {
                srDWORD dwSMIDRead;
                srBYTE  pyMsgBuff[sizeof(FileData_t)];
                srWORD  wMsgSize;
                srWORD  wReadRet = srRead(wSTID,
                                          srBOX_5,
                                          sizeof(pyMsgBuff),
                                          &dwSMIDRead,
                                          pyMsgBuff,
                                          &wMsgSize,
                                          &dwMsgInst);
                switch (dwSMIDRead)
                {
                    case BRD_FILEDATA: {
                        FileData_t* pData = (FileData_t*)pyMsgBuff;
                        size_t bytesWritten = fwrite(pData->pData, sizeof(unsigned char), pData->bytes, fStream);
                        retval += bytesWritten;
                        if (bytesWritten != pData->bytes) {
                            srPrintError("file write error - byte mismatch");
                            bReceiving = srFALSE;
                        }

                        if (ferror(fStream)) {
                            srPrintError("file write error");
                            bReceiving = srFALSE;
                        }

                        break;
                    }
                    case BRD_FILEDONE: {
                        FileDone_t* pData = (FileDone_t*)pyMsgBuff;
                        if (pData->totalBytes != retval) {
                            srPrintError("did not receive expected bytes");
                        }
                        bReceiving = srFALSE;
                        break;
                    }
                    case srCONNECT_STATUS_B_SMID: {
                        srConnectStatusRsp_t* status = (srConnectStatusRsp_t*)pyMsgBuff;
                        if(status->eConnection==srCONNECTION_CLOSED) {
                            srPrintError("srCONNECTION_CLOSED");
                            bReceiving = srFALSE;
                        }
                        break;
                    }
                }
            }
            srReadComplete(wSTID, dwMsgInst);
        }

        srSubscribe(wSTID, srBOX_5, BRD_FILEDATA, srFALSE);
        srSubscribe(wSTID, srBOX_5, BRD_FILEDONE, srFALSE);
    }

transfer_done:
    srSubscribe(wSTID, srBOX_5, BRD_FILEREADY, srFALSE);
    srSubscribe(wSTID, srBOX_5, srCONNECT_STATUS_B_SMID, srFALSE);
    fclose(fStream);
transfer_error:
    if (type == FILETRANSFER_SEND) {
        FileDone_t   doneData;
        doneData.totalBytes = retval;
        srBroadcast(wSTID, BRD_FILEDONE, (srBYTE*)&doneData, sizeof(FileDone_t));
    }
transfer_quit:
    return retval;
}

Host Implementation

Here is example perl code that contains a single subroutine to drive the transfer of files to/from the host using the transfer_file function on the target. You should modify the error reporting and default file logic as you see fit for your testing environment.

use strict;
use File::Spec;
use File::Basename;
use MIME::Base64;
use IO::File;
use Time::HiRes qw(gettimeofday tv_interval sleep);

use vars qw($ascript $messages $studio);
Win32::OLE->Option(Warn=>3);

$messages = $ascript->Messages;

sub TransferFile
{
    my %args = @_;
    $ascript->{WaitTimeoutPeriod} = 2000;
    my $remote = $args{TargetFilePath} || die "target file path required";
    my $local = $args{HostFilePath};
    unless ($local) {
        if ($args{Send}) {
            die "local file path required when sending";
        }
        else {
            my $base = fileparse($remote);
            $local = File::Spec->catfile(File::Spec->tmpdir(), $base);        
        }
    }

    my $file = new IO::File(($args{Send} ? "<" : ">") . "$local") || die "can't open $local: $!";
    $file->binmode();

    $messages->Item("BRD_FILEREADY")->User->Subscribe();
    
    my $transfer = $ascript->Functions->Item("transfer_file")->User;
    $transfer->ParameterList->{type} = $args{Send} ? "FILETRANSFER_RECEIVE" : "FILETRANSFER_SEND";
    $transfer->ParameterList->{file_path} = $remote;
    $transfer->CallNonBlocking();
    sleep(0.5); #allow the transfer_file function to startup and subscribe

    my $totalBytes = 0;
    my $startTime = [gettimeofday];
    if ($args{Send}) {
        # send file to target
        my $event = $ascript->WaitForEvent();
        if ($event->Type eq "BroadcastMessageUser" && $event->Name eq "BRD_FILEREADY") {
            my $fileData = $messages->Item("BRD_FILEDATA")->Owner;
            my $fileDone = $messages->Item("BRD_FILEDONE")->Owner;
            my $buffer;
            my $readChunk = $ascript->Constants->Item("FILETRANSFER_DATA_CHUNK")->Value || 8*1024;
            while (my $bytes = $file->read($buffer, $readChunk)) {
                $totalBytes += $bytes;
                $fileData->Response->{bytes} = $bytes;
                $fileData->Response->pData->{B64String} = MIME::Base64::encode_base64($buffer);
                $fileData->Broadcast();
            }
            $fileDone->Response->{totalBytes} = $totalBytes;
            $fileDone->Broadcast();
        }
        else {
            die "Never received READY message";
        }
    }
    else {
        # receive file from target
        $messages->Item("BRD_FILEDATA")->User->Subscribe();
        $messages->Item("BRD_FILEDONE")->User->Subscribe();
        $messages->Item("BRD_FILEREADY")->Owner->Broadcast();
        while (1) {
            my $event = $ascript->WaitForEvent();
            if ($event->Type eq "BroadcastMessageUser" && $event->Name eq "BRD_FILEDATA") {
                my $bytes = $event->Response->bytes;
                my $data = MIME::Base64::decode_base64($event->Response->pData->B64String);
                my $realBytes = length($data);
                $totalBytes += $realBytes;
                unless ($realBytes == $bytes) {
                    $studio->Output->PrintMessage("WARN: got $realBytes, expected $bytes");
                }
                print $file $data;
            }
            elsif ($event->Type eq "BroadcastMessageUser" && $event->Name eq "BRD_FILEDONE") {
                my $total = $event->Response->totalBytes;
                unless ($total == $totalBytes) {
                    $studio->Output->PrintMessage("WARN: TOTALS - got $totalBytes, expected $total");
                }
                
                last;
            } 
            elsif ($event->Type eq "Error") {
                $studio->Output->PrintMessage("error while waiting: " . $event->Description);
                last;
            }               
        }
        $messages->Item("BRD_FILEDATA")->User->Unsubscribe();
        $messages->Item("BRD_FILEDONE")->User->UnSubscribe();
    }

    my $elapsed = tv_interval($startTime);
    $studio->Output->PrintMessage(($args{Send} ? "Sent" : "Received") . " $totalBytes bytes of $local in $elapsed seconds");
    $messages->Item("BRD_FILEREADY")->User->Unsubscribe();
    # fetch the return status of the target function
    # (this is not strictly necessary - really only for verification)
    my $event = $ascript->WaitForEvent();
    if (($event->Type eq "FunctionUser") && ($event->Name eq "transfer_file")) {
        my $retval = $event->ReturnValue;
        unless ($retval == $totalBytes) {
            $studio->Output->PrintMessage("WARN: TOTALS - counted $totalBytes but return value is $retval");
        }
    }
    else {
        $studio->Output->PrintMessage("WARN: never got return value from transfer_file");
    }

    $file->close();
    $ascript->{WaitTimeoutPeriod} = 0;
}

In order to execute a transfer, you could simply call this function like this:

# to fetch from the target device
TransferFile(
    Send => 0,
    TargetFilePath => '/tmp/target_file', 
    HostFilePath  => 'c:/file_from_target');

# to send to the target device
TransferFile(
    Send => 1,
    TargetFilePath => '/tmp/file_from_host', 
    HostFilePath  => 'c:/myfile');