File Connector

Version 21.0.7964


File Connector


The File Connector pulls files into the ArcESB flow from external directories, or pushes files from the ArcESB flow into to an external directory.

Overview

Each File Connector is configured with a path on disk that determines what folder it will read files from or write files to. This can be a local file path or a UNC path to access other file locations on the network.

When a File Connector receives a file from the configured path, it passes the received file off to the next connector in the Arc flow without modifying the file. In this way, File Connectors can be used to poll an external folder location for files to pull into the Arc flow.

When a file in the Arc flow is processed by a File Connector, that file is written to the folder at the configured path. In this way, File Connectors can be used to drop off processed files at a location external to the application.

File Connectors can be configured with username/password credentials to provide access to external filepaths that otherwise would not be accessible to the application. This allows Arc to pull files from protected folder locations without needing to grant the entire application access to the protected location.

The Path configured in the File Connector can be dynamically populated using macros; for more information, please see the Macros section.

Connector Configuration

This section contains all of the configurable connector properties.

Settings Tab

Connector Settings

Settings related to the core operation of the connector.

  • Connector Id The static name of the connector. All connector-specific files are held in a folder by the same name within the Data Directory.
  • Connector Description An optional field to provide free-form description of the connector and its role in the flow.
  • Path The external filepath where the connector will pull files from or push files to. The Path can include macros for dynamic evaluation as described in the Macros section.

Receive

Settings related to pulling files from the external filepath.

  • File Mask A glob pattern for filtering files that should be pulled from the configured Path. Only files matching the filemask will be pulled. This setting can be combined with Receive Filter in the Advanced tab to specify multiple filters.
  • Enable Timestamp Comparison Whether to cache the file names of files pulled from the external Path. If enabled, the connector will only pull files that have not been pulled before or have been modified since the last pull. The last modified timestamp is used to determine whether files have been recently modified.
  • Delete files (after received) Whether to remove files from the external path after they are pulled into the Arc flow.

Authentication

Settings for specifying user credentials that have access to the configured path. Only necessary if Arc does not already have permissions to access the external folder path. NOTE: Impersonation via the Authentication settings are only available in the Windows/.NET edition.

  • Username The user with permissions to access the external folder path.
  • Password The password for the specified user.
  • Domain The domain of the specified user, if applicable.

Automation Tab

Automation Settings

Settings related to the automatic processing of files by the connector.

  • Send Whether files arriving at the connector will automatically be sent to the external path.
  • Retry Interval The amount of time before a failed Send is retried.
  • Retry Maximum Attempts The maximum number of times a failed Send will be retried.
  • Receive Whether the connector should automatically poll the remote folder path for files to pull into the flow.
  • Receive Interval The interval between automatic Receive attempts.
  • Minutes The number of minutes to wait before Receiving. Only applicable when Receive Interval is set to Minute.
  • Minutes Past the Hour The minutes offset for an hourly schedule. Only applicable when Receive Interval is set to Hourly. For example, if this value is set to 5, the automation service will Receive at 1:05, 2:05, 3:05, etc.
  • Time The time within a given day that the Receive should occur. Only applicable when Receive Interval is set to Daily, or Weekly, or Monthly.
  • Day The day on which the Receive should occur. Only applicable when Receive Interval is set to Weekly or Monthly.
  • Cron Expression An arbitrary string representing a cron expression that determines when the Receive should occur. Only applicable when Receive Interval is set to Advanced.

Advanced Tab

Local Folders

Settings that determine the folder on disk that files will be sent/uploaded from, and the folder that they will be received/downloaded to.

  • Input Folder (Send) The connector can send/upload files placed in this folder. If Send Automation is enabled, the connector will automatically poll this location for files to process.
  • Output Folder (Receive) The connector will place received/downloaded files in this folder. If the connector is connected to another connector in the flow, files will not remain here and will instead be passed along to the Input/Send folder for the connected connector.
  • Processed Folder (Sent) After processing a file, the connector will place a copy of sent/uploaded files in this folder if Save to Sent Folder is enabled.

Performance

Settings related to the allocation of resources to the connector.

  • Max Workers The maximum number of worker threads that will be consumed from the threadpool to process files on this connector. If set, overrides the default setting from the Profile tab.
  • Max Files The maximum number of files that will be processed by the connector each time worker threads are assigned to the connector. If set, overrides the default setting from the Profile tab.

Other Settings

Settings not included in the previous categories.

  • Max Receive Files The maximum number of files that will be pulled in a single receive interval. If this value is not a positive integer, no limit will be applied.
  • Temp Send Path A directory to use as a staging folder before moving the file to its configured destination path.
  • Temp Send Prefix If specified, the connector will upload the file with the temporary prefix, then rename the file to its original filename after the operation is complete.
  • Temp Send Extension If specified, the connector will upload the file with the temporary extension, then rename the file to its original filename and extension after the operation is complete.
  • Overwrite Option Specifies how the connector should handle the case when it attempts to write a file to the external path and that file already exists. The connector can rename the current file to a unique filename, overwrite the existing file, append the current file to the existing file, skip the operation without error, or stop the operation and throw an error.
  • Download Excluded Prefixes A comma-delimited list of prefixes that instructs the connector not to download filenames that begin with a specified prefix.
  • Enable File Size Comparison Whether to cache downloaded file names and sizes; if True then files will only be downloaded if they have not been downloaded before or have changed in size.
  • Recurse Subdirectories Whether to download files in subfolders of the target remote path.
  • Receive Filter A glob pattern filter to determine which files should be downloaded from the remote storage (e.g. *.txt). Negative patterns may be used to indicate files that should not be downloaded (e.g. -*.tmp). This setting should be used when multiple File Mask patterns are desired. Multiple patterns may be separated by commas, with later filters taking priority except when an exact match is found.
  • Send Filter A glob pattern filter to determine which files in the Send folder will be uploaded by the connector (e.g. *.txt). Negative patterns may be used to indicate files that should not be uploaded (e.g. -*.tmp). Multiple patterns may be separated by commas, with later filters taking priority except when an exact match is found.
  • Enable File Size Comparison Whether to cache the filenames of files pulled by the connector. If enabled, the connector will only pull files that have not been pulled before, or have been modified since the last pull. The connector uses the file size to determine if the file has been modified recently.
  • Log Subfolder Scheme By default, logs for transactions processed by the connector will be stored in the Logs subfolder for the connector. For connectors that process many transactions, it may be desirable to further divide the logs based on the datetime they were generated. When this setting is set to Daily, logs generated on the same day will be grouped in a subfolder;; when this setting is set to Weekly, logs generated in the same week will be grouped in a subfolder; and so on.
  • Log Messages Whether the log entry for a processed file will include a copy of the file itself.
  • Save to Sent Folder Whether files processed by the connector should be copied to the Sent folder for the connector.

Miscellaneous

Settings for specific use cases.

  • Other Settings Allows configuration of hidden connector settings in a semicolon-separated list, like setting1=value1;setting2=value2. Normal connector use cases and functionality should not require use of these settings.

Establishing a Connection

File Connectors must have the appropriate permissions to read/write from the configured Path. Permissions issues are primarily a concern when the configured path is a UNC path to another server on the network, but may also arise when pushing or pulling files from a protected folder on the local disk.

If the user running Arc does not already have permission to access the Path, the Username, Password, and Domain fields can be set to a specific user that does have the appropriate permissions.

Sending and Receiving Files

Sending Files

The File Connector sends files from the Input/Send Folder to the external folder specified in Path. If Send Filter is specified in the Advanced tab, only files that match the filter will be sent. Files are automatically sent if Send Automation is enabled in the connector.

Receiving Files

The File Connector receives files from the external folder specified in Path and places them in the Output/Receive Folder of the connector. If the File Connector is connected to another connector in the flow, the file does not remain in this folder and is instead passed along to the next connected connector.

The connector will automatically poll the external folder for files to receive if Receive Automation is enabled in the Automation tab.

The connector will only pull files that match the specified File Mask. If Enable Timestamp Comparison or Enable File Size Comparison are enabled, the connector will cache file names pulled from the remote path and only receive files that have not been received before or have been modified since they were last received.

Macros

The Path configured in the File Connector can be dynamically populated using Macros with the following syntax:

%MacroName%

The following macros are supported:

  • Header:myHeaderName
  • Filename
  • FilenameNoExt
  • Ext
  • ShortDate
  • LongDate
  • ConnectorID

For these macros, “ext” refers the file extension for the file currently being processed by the connector.

Example

Some files need to be placed in different folders based on data included within the file, such as a header value promoted on the ArcESB message. If a file has a header called ‘Customer’ that contains the folder into which the file should be placed, the Path property of the File Connector could be set to the following:

/some/path/%Header:Customer%