AWS S3 Listener

AWS S3 Listener

Easily configure an Amazon S3 – AWS Simple Cloud Storage (S3) Listener or Adapter with the eiConsole.

The AWS S3 Listener is used to poll files from the Amazon Simple Cloud Storage Service (Amazon S3). To configure the AWS S3 Listener, select AWS S3 from the Listener Type drop-down menu.

AWS S3 Listener or Adapter Configuration in Pilotfish Engine

Listener (Adapter) Configuration Drop-Down List

 

Basic AWS S3 Listener Configuration Options

On the Basic tab, you should specify the Polling Interval for how often to scan the Amazon S3 Bucket. This is a container, where Amazon S3 stores data.

  • Polling Interval – used to specify the interval for how often to scan the Amazon S3 bucket
  • Bucket Name – used to configure the unique name of the bucket you need to access
  • Polling Directory – directory in the bucket to be polled
  • File/Folder Name Restriction – name of a file or folder to be picked up
  • File Extension Restriction – allows you to specify the file extensions that you will process
  • Specify Full File Path – if enabled, a full file path may be specified to pick up a particular file
  • Path to File/Folder – the path to the file to be picked up
  • Tokenizers – this allows you to specify a character after which to split the file into multiple transactions. By default, the eiConsole will treat the entire file as the data stream to be handled as 1 message. However, if each line of the file represents a different record you would enter a value, such as the new line delimiter in this space.

AWS S3 Listner or Adapter Configuration Options

AWS S3 Listener Basic Configuration Options

 

Advanced AWS S3 Listener Configuration Options

On the Advanced tab, you can set the maximum number of transactions to process at a time; whether or not sub-folders should be searched for files as well; whether or not reads in all matching files as a single stream; a number of lines to treat as header lines (the header of the first file is included and skipped in all other files); a regex expression which must be contained by the full folder path of the file to be picked up (typically will only be used if you are using the ‘Include subfolder’ option).

  • Initialize on Trigger Only – if enabled, the Listener doesn’t start up until a trigger initializes it
  • Allow Command-line Invocation – if enabled, the listener can be invoked using the CLI client application
  • Restart on Listening Error – if enabled, the listener will be restarted after an error occurs
  • FIFO Queue Name – the FIFO field enables a “First In, First Out” queuing mechanism between Listeners and Transports. If a “FIFO Queue Name” is provided, that name will be used as a key for a queue Transactions and will be pushed into before reaching a Transport. They’ll be ordered in this queue according to when the Listener created them.

AWS S3 Advanced Listener Configuration Options in PilotFish Software

AWS S3 Advanced Listener Configuration Options (top half of screen)

 

  • FIFO Queue Delay – this is the interval between updates or checks against that queue. Providing a queue name guarantees that a given Transport sends transactions in the same order the Listener created them in
  • Max Transaction Per Run – specifies the maximum number of transactions to process at a time
  • Include Subfolders – specifies whether or not sub-folders should be searched for files as well
  • Full Folder Path Restriction – a regex expression that must be contained by the full folder path of the file to be picked up (typically it will only be used if you are using the ‘Include subfolders’ option).
  • Combine File Into One Stream – this reads in all matching files as a single stream
  • Number of Header Lines – this is the number of lines to treat as header lines. The header of the first file is included and skipped in all other files

AWS S3 Advanced Listener Configuration Options in PilotFish Middleware

AWS S3 Advanced Listener Configuration Options (bottom half of screen)

 

AWS S3 Listener Transaction Logging Configuration Options

On the Transaction Logging tab, you can enable transaction events logging. That data can be logged by a Transaction Event Listener (TransactionEventListener).

  • Transaction Logging Enable – this checkbox allows transaction events originating from this listener to be logged by a Transaction Event Listener (TransactionEventListener)
  • Log Transaction Data – if enabled, logs transaction data body
  • Log Transaction Data Base64 – if enabled, logs transaction data body as Base64
  • Log Transaction Attribute – if enabled, logs transaction attributes
  • Log All Attributes – if enabled, no attributes will be filtered
  • Allowed Attributes – attributes that are allowed to be logged

AWS S3 Listener Transaction Logging Options in PilotFish Integration Engine

AWS S3 Listener Transaction Logging Configuration Options

 

AWS S3 Listener Inactivity Configuration Options

On the Inactivity tab you can specify:

  • Enable Inactivity Monitor – check this box to enable inactivity monitoring – this will throw a non-transaction exception if the specified number of transactions haven’t been processed in the specified time interval
  • Minimum Transactions to Expect – the number of transactions to expect to be completed per monitoring interval
  • Monitoring Interval – how often to check the specified number of transactions have been processed
  • Times to Monitor – if set, monitoring will be done during the defined times of day – to ignore, set start and end time equally
  • Days to Exclude from Monitoring – inactivity monitoring will not occur on the days specified
  • Include Errors in Transaction Count – if checked, transactions that attempted to start but failed at the Listener stage will also be counted

AWS Listener Inactivity Configuration Options in PilotFish

AWS S3 Listener Inactivity Configuration Options

 

AWS S3 Listener Throttling Configuration Options

On the Throttling tab, you can specify:

  • Throttling Mode – the throttling mode to use for limiting the number of transactions or messages emitted by this Listener. “Timed” will limit transactions based on time intervals, while “Concurrent” will limit based on a concurrent number of transactions. “Concurrent” mode requires a Throttling Response Processor step later in your Interface workflow to acknowledge completion.

AWS S3 Listener Throttling Mode Selections

AWS S3 Listener Throttling Mode

 

  • Throttling Mechanism – the mechanism to use for throttling message – “Blocking” prevents the Listener from continuing to process and emit messages altogether, while “Queued” pushes received messages into the Interface queue (or a default, in-memory queue).
  • Max Concurrent Messages – how many messages can be concurrently processed, either by time-based limits (“Allow X per Second”) or synchronous (“Allow X at any Time”)
  • Timed Emission Interval – the interval for time-based limits (“Allow X per X Timed Emission Interval”)
  • Synchronous Timeout Interval – the interval to wait for a synchronous response before failing

Throttling Configuration Options for AWS S3 Adapter or Listener in PilotFish Integration Engine

AWS S3 Listener Throttling Configuration Options

 

AWS S3 Listener Post-Process Configuration Options

On the Post-Process tab, you are allowed to specify the behavior of the system after a file has been picked up. You can Keep, Delete or Move the file. If you choose to Move a file, the Target directory becomes required.

AWS Listener Post-Process Configuration Options in PilotFish

AWS S3 Listener Post-Process Configuration Options

 

AWS S3 Listener Scheduling Configuration Options

On the Scheduling tab, you are allowed to specify the start times and end times for scheduled execution. Also, you can specify days of the week or particular dates to exclude from scheduling. The Time Zone drop-down menu allows you to specify the Time Zone that should be used for scheduling; by default, it is set to the Time Zone of the eiConsole during initial configuration.

AWS S3 Listener Scheduling Options in PilotFish Integration Software

AWS S3 Listener Scheduling Configuration Options

 

AWS S3 Listener Criteria Configuration Options

On the Criteria tab, you are allowed to specify additional criteria, such as:

  • Minimum Seconds Since File Modified – minimum number of seconds elapsed since file modified
  • Minimum Days Since File Modified – minimum number of days elapsed since file modified
  • Maximum Days Since File Modified – maximum number of days elapsed since file modified

AWS S3 Listener Criteria Options in PilotFish Integration Engine

AWS S3 Listener Criteria Configuration Options

 

AWS S3 Listener Credential Configuration Options

On the Credentials tab, you can set the necessary credentials for the connection.

  • Region – specify the AWS Region to connect to
  • Use Credentials From – specifies the method of getting the credentials. Note: choosing the ‘No Credentials’ option will only allow access to public buckets
  • Access Key ID – access Key ID for making calls to Amazon services
  • Secret Key – secret access key for making calls to Amazon services
  • Session Token – the STS session token for making calls to Amazon services. If not using STS, leave blank
  • AWS File Path – specifies the file path where the AWS profile configuration file is located
  • Profile Name – specifies the name of a local configuration profile

AWS S3 Listener Credential Configuration Options in PilotFish Interface Engine

AWS S3 Listener Credential Configuration Options

For more information please call us at 860 632 9900 or click the link below.

This is a unique website which will require a more modern browser to work! Please upgrade today!