Connectors
Connector Configuration
Configure sink and source connector instances with schemas, transforms, and plugin settings
After activating a connector, configure each instance through the Configuration system. Each instance has its own versioned configuration — you can create versions, activate a specific one, and roll back.
Sink Configuration
| Setting | Description |
|---|---|
| enabled | Whether the connector instance is active |
| streams | Which Iggy streams and topics to consume from |
| schema | Message format — JSON (default), Raw, Text, Protocol Buffers, FlatBuffers |
| batch_length | Number of messages to batch before sending |
| poll_interval | How often to poll for new messages |
| consumer_group | Optional consumer group for coordinated consumption |
| plugin_config | Plugin-specific settings (connection strings, credentials, table names, etc.) |
| transforms | Optional data transformations before sending |
Source Configuration
| Setting | Description |
|---|---|
| enabled | Whether the connector instance is active |
| streams | Which Iggy stream and topic to produce into |
| schema | Message format — JSON (default), Raw, Text, Protocol Buffers, FlatBuffers |
| batch_length | Number of messages to batch before producing |
| linger_time | Maximum time to wait before flushing a batch |
| plugin_config | Plugin-specific settings (source connection, polling interval, etc.) |
| transforms | Optional data transformations before producing |
Data Transforms
Apply field-level transformations to messages as they flow through connectors:
| Transform | Description |
|---|---|
| AddFields | Add new fields to messages |
| DeleteFields | Remove fields from messages |
| FilterFields | Keep only specified fields |
| UpdateFields | Modify existing field values |
| ProtoConvert | Convert to/from Protocol Buffers |
| FlatBufferConvert | Convert to/from FlatBuffers |
Custom transforms can be built by implementing the Transform trait in Rust.
Examples
PostgreSQL Sink
{
"name": "orders-to-postgres",
"values": {
"enabled": true,
"streams": [
{
"stream": "orders",
"topics": ["completed", "refunded"],
"schema": "json",
"batch_length": 100,
"poll_interval": "1s",
"consumer_group": "pg-sink-orders"
}
],
"plugin_config_format": "json",
"plugin_config": {
"connection_string": "postgres://user:pass@host:5432/orders",
"table": "order_events"
}
},
"activate": true
}Random Source (Development)
{
"name": "test-data-generator",
"values": {
"enabled": true,
"streams": [
{
"stream": "test_stream",
"topic": "test_topic",
"schema": "json",
"batch_length": 1000,
"linger_time": "5ms"
}
],
"plugin_config": {
"interval": "3000ms",
"max_count": 1000000,
"message_range": [1, 5],
"payload_size": 200
}
},
"activate": true
}