File Connectors Guide

Connect to remote file systems via FTP, SFTP, and FTPS. Browse files, automate file processing with trigger rules, and integrate with the Script Engine.

Table of Contents

1. Overview
2. Connector Types
3. Creating an FTP Connector
4. Creating a Filesystem Connector
5. File Browser
6. File Operations
7. File Trigger Rules
8. Polling Configuration
9. Audit Logging
10. Best Practices

1. Overview

File Connectors enable DataMagik to integrate with external file systems for automated file processing, transfer, and monitoring. Two connector types are available:

• FTP Connectors — Connect to remote servers via FTP, SFTP, or FTPS protocols
• Filesystem Connectors — Access local or network file systems through on-premise connector agents

Key Features:

• Multiple Protocols — FTP, SFTP (SSH), and FTPS (TLS) support
• File Browser — Visual file browser for navigating and managing files
• Trigger Rules — Automatically process files matching patterns with scripts
• Polling — Interval-based or cron-scheduled file monitoring
• Full Audit Trail — Every file operation is logged with user, timing, and status
• Duplicate Prevention — Tracks processed files to avoid reprocessing

2. Connector Types

FTP Connectors

Connect to remote file servers using standard file transfer protocols:

Authentication Methods:

• Username and password
• SSH private key with optional passphrase (SFTP only)
• SFTP host key verification (Trust-On-First-Use)

Filesystem Connectors

Access local or network-mounted file systems through a DataMagik connector agent:

• Linked to a specific connector site and agent
• Operates within a configured base path
• Can be set to read-only mode to prevent writes and deletes
• Optional printing enabled flag for print job file output

3. Creating an FTP Connector

Step 1: Navigate to File Connectors

Go to Connectors → File Connectors in the main menu.

Step 2: Click "Add FTP Connector"

Step 3: Configure Connection Settings

For SFTP Connections

File Restrictions

Step 4: Test Connection

Click Test Connection to verify the connector can reach the server. Fix any issues before saving.

4. Creating a Filesystem Connector

Filesystem connectors use your on-premise connector agent to access local file systems.

Configuration

5. File Browser

The File Browser provides a visual interface for navigating and managing files across all connector types.

Accessing the File Browser

Navigate to Connectors → File Browser in the main menu.

Features

• Browse — Navigate directories across all connected file systems
• Upload — Upload files via multipart upload
• Download — Stream file downloads
• Preview — Read and preview file content
• Create Folders — Create new directories
• Move/Rename — Move or rename files and folders
• Delete — Remove files (with confirmation)

6. File Operations

All file operations are available via API for both FTP and filesystem connectors:

7. File Trigger Rules

Trigger rules automatically process files that match a pattern by executing a DataMagik script.

Creating a Trigger Rule

Post-Process Actions

How Triggers Work

1. Polling detects a file matching the trigger pattern
2. File metadata is captured (path, size, hash)
3. The associated script executes with the file context
4. The post-process action is applied (leave, delete, or move)
5. The file is recorded as processed to prevent reprocessing

Execution History

View trigger execution history to monitor which files were processed, when, and whether the script succeeded or failed.

8. Polling Configuration

Configure how frequently connectors check for new files.

Polling Settings

Cron Schedule Examples

Note: If both an interval and cron schedule are set, the cron schedule takes precedence.

Failure Handling

• Consecutive failures are tracked per polling job
• Retry backoff is applied after repeated failures
• Error details are recorded for troubleshooting

9. Audit Logging

Every file operation is recorded in the audit log with comprehensive details:

Access audit logs from Connectors → File Connectors to review file operation history.

10. Best Practices

Connection Setup

• Use SFTP when possible — Provides encrypted file transfer
• Use key-based auth for SFTP — More secure than passwords
• Enable passive mode for FTP — Works better through firewalls
• Test connections after setup — Verify connectivity before creating trigger rules

File Processing

• Use specific file patterns — invoice_*.csv is better than *.*
• Use Move post-process action — Archive processed files rather than deleting
• Set allowed extensions — Restrict file types for security
• Set max file size — Prevent oversized files from consuming resources

Polling

• Use cron for business-hours processing — Avoid unnecessary polling overnight
• Don't poll too frequently — Match interval to how often new files appear
• Monitor execution history — Check for failed triggers regularly

Security

• Credentials are encrypted at rest — Passwords and private keys are stored securely
• Use read-only mode — For filesystem connectors that should only read, never write
• Review audit logs — Monitor for unexpected file operations