..::LoDN Content Publishing - Web Interface::..

Logging In

From the LoDN welcome page (http://promise.cs.utk.edu/lodn) you can sign up for a new LoDN account, log in to an existing account, or browse LoDN's public directory as a guest. Logging in to your own account takes you to your home area, where you can upload or download files and manage your stored content. Logging in as a guest allows you to access stored content that other LoDN users have made available to the public.

Browse as Guest

From the LoDN welcome page, click the "browse files as guest" link to access LoDN's public directory. You do not need a LoDN account to browse as a guest. When browsing the public directory you can access published content made available by registered LoDN users. With LoDN, registered users have the option to "publish" an exNode (a pointer to stored content). Publishing an exNode makes a read-only version of the exNode publicly accessible. Guests can use the read-only exNode to download file content from the network without having access to any data management options—i.e. guests cannot alter or delete published data.

The public directory interface, like the interface for registered users, is based on an IMAP email interface, so the look and feel of the page should be familiar to most users. On the public directory page, you will find the current directory path displayed near the top of the left-hand frame; below this is a directory list that allows you to navigate the publicly accessible LoDN subdirectories.

• To open a subdirectory, click the directory name or folder icon. When you open a subdirectory, the exNode list (in the main frame of the page) will display the published exNodes in that folder.
  
  
• To move up one directory level, click "..".

From the exNode list in the main frame of the page, you may download file content, view an exNode file, or get a read-only exNode file.

• To download file content, click the exNode name for the file you wish to download; this will start the LoDN Java Download Client. Use the download client window to select a location and filename for the output file and indicate your connection speed, then click Download. (For more information about using the Download Client, see Download Client Help, also accessible from the download client window under Help–>Contents.)
  
  
• To download multiple files, use the check boxes to select one or more exNodes from the list, then choose "Download" from the Operations drop-down menu and click Go. The LoDN download client window will appear. Use the download client window to select your connection speed, then click Download. The selected files will be downloaded in succession and placed in the local directory specified in the "Output File" field.
  
  
• To view an exNode file (display a list of the storage allocations referenced by that exNode), click the " " icon to the right of the exNode name. Information found in the exNode file includes the allocation type, IP address and port number of storage depot, and expiration date and time for each data block.
  
  
• To get a read-only exNode file, click the " " icon. For more information about read-only exNodes, see the "Read-Only vs. Full Capability ExNodes" section below.

Note: While browsing as a guest you have access to read-only versions of published exNodes. However, you cannot perform data management operations (augment, delete, publish/unpublish) on stored content using a read-only exNode. If you wish to maintain your own copy of a published file, download the file content from the public directory, then upload the content using the LoDN upload client to produce an exNode with full (not just read-only) capabilities. Do not import a read-only exNode into your LoDN directory, as the exNode warmer (see "ExNodes and Warming") will not be able to maintain the stored content using a read-only exNode.

Your Home Area

The directory interface to your LoDN home area is based on the interface used for accessing IMAP email accounts, so the look and feel of your home area will most likely be familiar. Log in to your home area from LoDN's welcome page at http://promise.cs.utk.edu/lodn/. From your home area, you can upload and download content to and from network storage, and manage your stored content.

ExNodes and "Warming"

The logistical networking storage that LoDN uses is designed to be globally scalable and sharable. To offer maximum scalability, the storage allocations are lightweight and best effort; and for maximum sharability, they are also time-limited. Using best effort, time-limited storage means that reliability, fault tolerance, and extended storage duration are implemented by aggregating multiple weak storage allocations to achieve strong service characteristics not offered by any single allocation. The exNode is a metadata filetype that accommodates the aggregation of individual storage allocations into a "network file" by holding the metadata necessary to manage distributed content. For more info about logistical networking and LoDN, click here.

When a file is uploaded with LoDN, the file content is broken into small data blocks. Each data block is then replicated and the replicas are distributed across multiple remote storage locations (called storage "depots") accessed via the network, with each data block placed in a separate allocation. Scattering fragmented replicas of file content over multiple storage locations helps to ensure that stored content will remain accessible in case of a machine or network outage. An exNode is used to keep track of each block of stored data: where the block is stored, when its storage "lease" expires, etc. In essence, the exNode is a pointer to stored file content. The exNode acts like a pointer to file content stored over the network, in the same way that the UNIX inode is a pointer to storage allocations on a hard drive.

LoDN uses time-limited storage, so, in order to maintain file content in storage for longer than the length of an allocation's time limit, each allocation's lease must be periodically renewed. An exNode "warmer" is used to take the burden of exNode maintenance off the user. When you place an exNode in your LoDN directory (by uploading a file or importing an exNode), LoDN's warmer automatically renews the storage leases for you, and makes sure that enough copies of each data block are available to provide acceptable fault-tolerance. The warmer operates on all exNodes in your LoDN directory, whether they are "published" (made available for download by other LoDN users or guests) or "unpublished" (accessible only by you). ExNodes that are not warmed will deteriorate over time, as allocations holding blocks of the stored content expire and the data is erased from those expired allocations. If you wish to keep stored content available for an extended period (i.e., longer than a day or two), it is recommended that you place the exNode in your LoDN directory.

LoDN Directories

• To open a directory, click the directory name to the right of the folder icon. When you open a directory, the exNode list (in the main frame of the page) will display the exNodes in that folder.
  
  
• To move up one directory level, click "..".
  
  
• To create a new subdirectory within the current directory, enter a folder name in the "Create Directory" box, and then click Create.
  

Note: Placing an exNode in a LoDN directory means that it is maintained or "warmed" automatically by the LoDN server. ExNodes that are not warmed will deteriorate over time.

Uploading Data and ExNodes

From your home area there are two options for placing an exNode in your LoDN directory: uploading the contents of a file or importing an already existing exNode into the LoDN directory.

• To import an exNode created using the LoRS Tools, received via e-mail, etc., from your local system to your current LoDN directory: Enter a filepath in the box labeled "Upload ExNode," or click Browse and select a .xnd file to upload, then click Upload. Note: The LoDN exNode warmer will maintain exNodes in your LoDN directory (see "ExNodes and Warming" above), but the warmer cannot replace lost data from allocations that have already expired. Before importing, you may wish to use the LoRS Tools software (http://loci.cs.utk.edu/lors/) to "List" the exNode and confirm that at least one complete replica of the content is available, especially if the exNode has been out of a warmer for a day or more.
  
  
• To upload the contents of a file from your local system into storage and automatically create an exNode in your current LoDN directory:
  Click the "Launch Java Upload Client" button to start the LoDN Publisher Java Upload client. In the publisher window (Fig. 1), enter the name of the file to be uploaded in the box labeled "Input File" or click Browse to choose a file to upload. Adjust upload parameters (see below) and then click Upload. The "Transfer Progress" window will appear, showing the status of the upload. When the upload is complete, click OK to close the Transfer Progress window. If the new exNode does not immediately appear in your exNode directory list, use your browser's refresh command to reload the page.

Note: Placing an exNode in a LoDN directory does not automatically give others access to that exNode. An exNode in your directory is private until you Publish it (see below).

Information in this section can also be accessed from the LoDN Publisher window, under Help–>Contents.

Figure 1: The LoDN Upload Client

Enter the name of the file you wish to upload in the box provided, or click Browse to choose a file to upload.

Specifies the LoDN directory location where your exNode will be placed. When you upload a file, the upload client will create an exNode (a pointer to the stored content). The exNode is named by appending the .xnd extension to the name of the uploaded file. A copy of the exNode will be stored in the current LoDN directory, where it will be automatically maintained by LoDN's exNode "warmer".

The "Location Type" and "Location Code" boxes allows you to request that your data be stored near a specific geographical location. LoDN will place your data in available storage as near as possible to your specified location. If the data is to be downloaded from a remote location, consider storing the data near the download site. For content that will be broadly or globally distributed, you may upload content and then create copies of the content at multiple geographical locations using the "Augment" function (accessible in your LoDN home area).

To specify a geographical location, use the drop down menu to select a location type: "zip", "state", "country", or "airport". Then use the box labeled "Location Code" to enter the appropriate location code: a five digit zip code (US only), two letter state abbreviation (US only), three letter airport code, or two letter ISO country code. LoDN will store your content as near as possible to your specified location (dependent upon storage availability in the specified area).

Use the radio buttons to choose the type of connection you are using to access the Internet. Default values of "Number of Connections" and "Block Size" will be adjusted according to your connection type.

"Number of connections" refers to the maximum number of data transfer operations to be performed concurrently during upload. Each connection, or "thread," is an individual transfer (i.e. TCP) stream, delivering a separate block of data. Maximizing upload performance is a matter of balancing the number of connection threads with available resources (local memory, bandwidth, etc.). Using more connection threads usually means faster performance, provided that sufficient bandwidth and memory (on your local system) is available to the upload client. For transfers over public wide area networks, use one to three threads for TCP friendliness. For transfers over high-speed links (e.g., Abilene, ESNet, etc.), or dedicated links between research facilities, try using ten or more threads for maximum performance. See Figure 2 for further suggestions. To specify the number of connections, you may choose integer.

Figure 2: Recommended Connection Settings

Since the IBP storage depots used by LoDN offer a best effort service, LoDN relies on the combined use of replication, fragmentation, and striping of file fragments across multiple depots to strengthen assurance of sustained data accessibility. As on the hard drive of a computer, a file stored with IBP may be fragmented into many blocks of data, with each data block stored in a separate location. The fragments may be spread across several IBP depots, or stored in separate allocations within the same depot. A small file may be left whole (one fragment), while larger files can be split into many smaller data blocks. With LoDN, fragmenting the file into data blocks is controlled by specifying a block size. "Block Size" indicates the size of the data blocks into which your data will be divided. The number of fragments into which a file is partitioned will equal the total file size divided by the specified block size. If you choose a block size larger than the file size, the file will not be fragmented.

Enter a positive integer in the box labeled "Block Size," then use the radio buttons to choose units of kilobytes (KB) or megabytes (MB). Choose a block size small enough to cause sufficient fragmentation for fault tolerance considerations, but large enough to allow good upload performance. Larger block size allows faster uploads of large data sets by decreasing the number of allocation operations necessary to store the file content, but the effect is asymptotic. A block size between 512 KB and 2 MB is optimal for most routine applications.

"Copies" specifies the number of replicas of your content that should be stored. Storing multiple copies guards against inaccessibility due to a network outage or data loss if one storage allocation fails. However, the value of each additional replica is balanced by the storage capacity needed to house it. Please remember that the storage resources employed by LoDN are shared resources. Two to three copies provides sufficient backup in most instances. Use the drop-down menu labeled "Copies" to select a value.

As mentioned above, since the IBP storage depots used by LoDN offer a best effort service, LoDN relies on the combined use of replication, fragmentation, and striping of file fragments across multiple depots to strengthen assurance of sustained data accessibility. When a file is uploaded, the file content is divided into blocks of data. Those data blocks are replicated and then distributed over several IBP depots for storage. Using multiple storage depots to house data replicas provides greater fault tolerance in case one depot fails. Using multiple depots will also improve download performance, as the download client will maximize speed by retrieving data from the fastest source depot. Choose the number of depots over which the data will be distributed to ensure a reasonable level of fault tolerance and download performance for the number of data blocks to be stored. A value of five depots is optimal in most instances.

For more fine-grained control over uploading file content or adding replicas of stored content, consider using the Logistical Runtime System Tools (LoRS Tools) software. If you wish, you may use LoRS to upload file content to storage and generate an exNode, and then use LoDN's web interface to import the exNode into your LoDN directory. The LoRS software is engineered for high-performance, multi-threaded downloads and is available for all major OS platforms. The LoRS Tools software is available for free download from http://loci.cs.utk.edu/lors/downloads/ . Downloading and installing the LoRS Tools takes only a few minutes. For more information about installing and using the LoRS Tools software, please refer to the LoRS user manual, available at http://loci.cs.utk.edu/lors/distributions/LoRS_Manual_v0_82.pdf/.

Note: If you use LoRS to upload file content it is recommended that you import the exNode into LoDN as soon as possible to avoid data loss and take advantage of LoDN's exNode warmer (see "ExNodes and Warming" section above).

Managing ExNodes

In your home area, a list of the exNodes within the current directory is found in the main frame of the page. This exNode list behaves similarly to the familiar IMAP email interface—you can select or deselect items in the list using the check boxes and then perform operations on the selected items using commands from the drop-down menus. Some commands are also available by clicking the icons to the right of each exNode name.

Use the "Select" drop down menu to select all items in exNode list, deselect all items, or reverse the currently selected and unselected items.

Use the "Operation" drop down menu to perform a download, augment, delete, or publish/unpublish operation on one or more selected exNodes from the exNode list. If more than one exNode is selected, the chosen operation will act on each selected exNode sequentially. To select an exNode, check the box to the left of the exNode name. See Figure 3 for a description of the exNode operations.

Operations to be performed on only one exNode may be executed by clicking the quick command icons to the right of each exNode name. Clicking one of these icons is equivalent to selecting a single exNode and then choosing the corresponding operation from the operation drop-down menu. In addition to the operations available via the operation menu (download, augment, delete, and publish/unpublish), there is also a quick command icon for the view command. Figure 3 gives a description of each command.

Figure 3: ExNode Operations

Notes on the Augment Command:

Storage Type: The storage depots used to store file content offer two types of storage allocations, soft or hard. Soft storage allocations are made using free space in the local file system—space that may be reclaimed at any time according to the needs of that local system. Hard storage allocations are made using dedicated storage space that will not be reclaimed by the local system. Hard storage is more stable, but more scarce than soft storage. And, accessibility of both soft and hard storage allocations are vulnerable to machine and network outages. Since soft storage is more abundant, using soft storage increases the chances that storage space meeting your duration and location criteria will be available.

Duration: As mentioned earlier, LoDN uses time-limited storage. Each storage depot has its own local policy specifying the maximum duration of storage allocations (i.e., length of storage "lease") allowed on that depot. When using the Augment command, you specify the duration of storage that you desire. LoDN attempts to replicate your content on depots which offer time-limits greater than or equal to your specified duration. As most depots offer only short-term storage, available storage capacity is inversely related to requested duration. The shorter the requested duration, the more likely you will be to find available depots that meet your specifications.

Location: As with the Upload command, the "Location" box allows you to request that your data be stored near a specific geographical location. The Augment command is useful for creating replicas in a remote location, near expected download sites. To specify a geographical location, type either "zip", "state", "airport", or "country", followed by an equals sign, a space, and the appropriate location code: a five digit zip code (US only), two letter state abbreviation (US only), three letter airport code, or two letter ISO country code. You must include a space after the equals sign. For example, to store data in the vicinity of the University of Tennessee, Knoxville, enter "zip= 37996"; to store data near Jacksonville, Florida, enter "airport= JAX".

Read Only vs. Full Capability ExNodes

ExNodes are pointers to stored content, and are used to access and manage that content. Full exNodes include read, write, and manage capabilities that allow the user to perform the range of operations described in the "ExNode Management" section above. Read-only exNodes include only the read capability and allow the user to download the file content. Read-only exNodes are analogous to read-only files; users may access, but not alter content using a read-only exNode.

When you use LoDN to publish file content, a read-only version of the exNode is posted in the public directory. This allows other users and guests to use the read-only exNode to download file content, without granting them access to manage or delete the stored content. Users logged on as guests have access to read-only versions of published exNodes, but do not have access to the full capability exNodes.

You can access read-only and full capability versions of your exNodes from your home area. In the exNode list, clicking the " " icon downloads a read-only version of the exNode, while clicking the " " icon downloads the full capability exNode (these commands download the .xnd file, not the stored content). If you or your colleagues are LoRS Tools users, you might wish to send an exNode file via email, ftp, etc., to be downloaded using the LoRS Tools. (For more information about the LoRS Tools software, visit http://loci.cs.utk.edu/lors).

Remember that you cannot perform data management operations (augment, delete, publish/unpublish) on stored content using a read-only exNode. If you use a read-only exNode to access stored content and you wish to maintain your own copy of the content, use the read-only exNode to download the content, then use LoDN's upload client to store your own copy of the content and produce an exNode with full capabilities.

Note: Do not import a read-only exNode into your LoDN directory, as the exNode warmer (see "ExNodes and Warming") will not be able to maintain the stored content using a read-only exNode.

Content Distribution

You can make LoDN data files available by placing an active "LoDN link" on a webpage, sending a LoDN link by email, or sharing published content via the LoDN directory. To get a LoDN link url for sharing your file content, go to your LoDN home area and right-click the appropriate exNode name, choose Copy Shortcut, and paste the link where needed. You can copy and paste this link directly into a webpage, email, document, spreadsheet, etc., just as with any other url. Users access stored content by clicking a LoDN link, which starts the LoDN Download Client, and then use the download client to retrieve the content directly from network storage.

Keep in mind that if you are publishing content using a LoDN link, it will be helpful to your users if you include some descriptive information about the content (file size, file type, etc.) and some information about the LoDN Download client (that it requires Java Web Start, etc.). In order to use the LoDN Download client, a user must have Java Web Start installed on their computer. For Windows users, Java Web Start will be automatically detected and installed if necessary. Other users should manually install Java Web Start, which is included with the free download of the Java 2 Runtime Environment, Standard Edition (JRE) v1.4.2 (information and downloads are available at java.sun.com). When starting the LoDN client for the first time, it may take a few seconds for the client files to be configured. A dialog box will appear, verifying that you wish to install and run the LoDN Download client. LoDN's security certificate is digitally signed by the University of Tennessee and the user must accept the certificate prior to running the application for the first time.

! Note: LoDN is a tool designed to assist users in the managing and publication of their own content. Use of all content must conform to applicable copyright restrictions. The accounts of users who use copyrighted content inappropriately will be canceled. Thank you for your cooperation.

If you have any questions or suggestions for LoDN,please subscribe to the LoCI interest mailing list or send us an email at