SFTP Source Connector: Missing identity file

[geofflhenderson]

Hi @jcustenborder,

I hope I am posting this in the correct place. I am using the following SFTP Source connector class: com.github.jcustenborder.kafka.connect.spooldir.SftpCsvSourceConnector

The connector is configured as follows (only the sftp setting, I omitted the rest):

{
"name": "CCS_DataCDR-Source-Sftp",
"config": {
"connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SftpCsvSourceConnector",
"tasks.max": "1",
"sftp.host": "mysftphost",
"sftp.port": 30022,
"sftp.username": "myuser",
"sftp.identity": "/home/myuser/.ssh/myprivatekeyfile",
}
}

The public key that matches /home/myuser/.ssh/myprivatekeyfile is in the authorized_keys file on my SFTP server.

I can successfully SFTP from my VM (Redhat) as follows, without being prompted for a password:

sftp -P 30022 -i /home/myuser/.ssh/myprivatekeyfile myuser@mysftphost

However, when trying from the connector (running on the same Redhat VM) I observe the following logs in /var/log/kafka/connect-distributed.log:

org.apache.kafka.connect.errors.ConnectException: Can not get connection to SFTP:
at com.github.jcustenborder.kafka.connect.spooldir.SftpConnection.init(SftpConnection.java:146)
at com.github.jcustenborder.kafka.connect.spooldir.SftpConnection.(SftpConnection.java:75)
at com.github.jcustenborder.kafka.connect.spooldir.SftpCsvSourceConnectorConfig.getSftpConnection(SftpCsvSourceConnectorConfig.java:180)
at com.github.jcustenborder.kafka.connect.spooldir.SftpCsvSourceTask.fileActions(SftpCsvSourceTask.java:43)
at com.github.jcustenborder.kafka.connect.spooldir.AbstractSourceTask.checkDirectory(AbstractSourceTask.java:95)
at com.github.jcustenborder.kafka.connect.spooldir.AbstractSourceTask.start(AbstractSourceTask.java:127)
at com.github.jcustenborder.kafka.connect.spooldir.SpoolDirSourceTask.start(SpoolDirSourceTask.java:45)
at com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceTask.start(SpoolDirCsvSourceTask.java:127)
at com.github.jcustenborder.kafka.connect.spooldir.SftpCsvSourceTask.start(SftpCsvSourceTask.java:29)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:219)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: com.jcraft.jsch.JSchException: Auth fail
at com.jcraft.jsch.Session.connect(Session.java:519)
at com.jcraft.jsch.Session.connect(Session.java:183)
at com.github.jcustenborder.kafka.connect.spooldir.SftpConnection.init(SftpConnection.java:140)
... 16 more

Any idea what the issue may be ?

The kafka-connect process is running as user cp-kafka-connect group confluent. I made sure the .ssh directory with my private key is owned by the same user/group with 700 permissions, and the private key also has the same ownership with 600 permissions.