Kafka Security
•Download as PPTX, PDF•
3 likes•2,652 views
This document discusses security features in Apache Kafka including SSL for encryption, SASL/Kerberos for authentication, authorization controls using an authorizer, and securing Zookeeper. It provides details on how these security components work, such as how SSL establishes an encrypted channel and SASL performs authentication. The authorizer implementation stores ACLs in Zookeeper and caches them for performance. Securing Zookeeper involves setting ACLs on Zookeeper nodes and migrating security configurations. Future plans include moving more functionality to the broker side and adding new authorization features.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Kafka Security
Similar to Kafka Security (20)
More from DataWorks Summit/Hadoop Summit
More from DataWorks Summit/Hadoop Summit (20)
Recently uploaded
Recently uploaded (20)
Kafka Security
- 1. Page1 © Hortonworks Inc. 2014 Kafka Security SSL, Kerberos & Authorization
- 2. Page2 © Hortonworks Inc. 2014 Who Are We? Sriharsha Chintalapani Apache Kafka Committer Apache Storm Committer & PMC Parth Brahmbhatt Apache Kafka Contributor Apache Storm Committer & PMC
- 3. Page3 © Hortonworks Inc. 2014 Why Kafka Security? • Kafka is becoming centralized data bus connecting external data sources to Hadoop eco system. • There are lot of requests/discussions in Kafka mailing lists to add security
- 4. Page4 © Hortonworks Inc. 2014 Why Kafka Security? • How can we prevent rogue agents to publishing/consuming data from Kafka • How can we encrypt the data that’s flowing through the network • How can we give permissions to a topic to specific group or users
- 5. Page5 © Hortonworks Inc. 2014 Kafka Security • We recognized the necessity of security in Kafka • Added wire encryption via SSL • Role Based authentication via SASL ( Kerberos) • Authorizer to add fine-grain access controls to Kafka topics per User, per Host.
- 6. Page6 © Hortonworks Inc. 2014 Kafka Networking
- 7. Page7 © Hortonworks Inc. 2014 Kafka Networking http://www.slideshare.net/jjkoshy/troubleshooting-kafkas-socket-server-from-incident-to-resolution
- 8. Page8 © Hortonworks Inc. 2014 Kafka Networking
- 9. Page9 © Hortonworks Inc. 2014 SSL
- 10. Page10 © Hortonworks Inc. 2014 Kafka Security – SSL • Kafka SSL / SASL requirements • No User-level API changes to clients • Retain length-encoded Kafka protocols • Client must authenticate before sending/receiving requests • Kafka Channel • Instead of using socket channel, we added KafkaChannel which consists a TransportLayer, Authenticator.
- 11. Page11 © Hortonworks Inc. 2014 Kafka Security – SSL • SSLTransportLayer • Before sending any application data, both client and server needs to go though SSL handshake • SSLTransportLayer uses SSLEngine to establish a non- blocking handshake. • SSLEngine provides a state machine to go through several steps of SSLhandshake
- 12. Page12 © Hortonworks Inc. 2014 Kafka Networking KafkaChannel TransportLayer Authenticator Kafka Server handshake authenticate
- 13. Page13 © Hortonworks Inc. 2014 Kafka Security – SSL
- 14. Page14 © Hortonworks Inc. 2014 Kafka Security – SSL • SSLTransportLayer • SocketChannel read • Returns encrypted data • Decrypts the data and returns the length of the data from Kafka protocols • SocketChannel Write • Writes encrypted data onto channel • Regular socketChannel returns length of the data written to socket. • Incase of SSL since we encrypt the data we can’t return exact length written to socket which will be more than actual data • Its important to keep track length of data written to network. This signifies if we successfully written data to the network or not and move on to next request.
- 15. Page15 © Hortonworks Inc. 2014 Kafka Security – SSL • Principal Builder • SSLTransportLayer gives hostname as authenticated user • X509Certificate has lot more information about a client identity. • PrincipalBuilder provides interface to plug in a custom PrincipalBuilder that has access to X509Certificate and can construct a user identity out of it. • Authenticator can use this custom principal to add ACLs
- 16. Page16 © Hortonworks Inc. 2014 Kafka Security – SSL
- 17. Page17 © Hortonworks Inc. 2014 Kafka Security – SSL • listeners=SSL://host.name:port • ssl.keystore.location • ssl.keystore.password • ssl.key.password • ssl.truststore.location • ssl.truststore.password • security.inter.broker.protocol (optional)
- 18. Page18 © Hortonworks Inc. 2014 SASL/Kerberos
- 19. Page19 © Hortonworks Inc. 2014 Kafka Security – SASL • Simple Authentication and Security Layer, or SASL • Provides flexibility in using Login Mechanisms • One can use Kerberos , LDAP or simple passwords to authenticate. • JAAS Login • Before client & server can handshake , they need to authenticate with Kerberos or other Identity Provider. • JAAS provides a pluggable way of providing user credentials. One can easily add LDAP or other mechanism just by changing a config file.
- 20. Page20 © Hortonworks Inc. 2014 Kafka Security – SASL • Pass JAAS config file as jvm parameter. - Djava.security.auth.login.config • JAAS Config file KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/vagrant/keytabs/kafka1.keytab" principal="kafka/host@EXAMPLE.COM"; }; KafkaClient { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka" keyTab="/vagrant/keytabs/client1.keytab" principal=”client/host@EXAMPLE.COM"; };
- 21. Page21 © Hortonworks Inc. 2014 Kafka Security – SASL Client Broker Connection Mechanism list Selected Mechanism & sasl data Evaluate and Response Sasl data Client Authenticated
- 22. Page22 © Hortonworks Inc. 2014 Kafka Security – Resources • SSL • https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka • SASL • https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61326390 • Vagrant Setup • SASL • https://github.com/harshach/kafka-vagrant/tree/master/ • SSL • https://github.com/harshach/kafka-vagrant/tree/ssl/
- 23. Page23 © Hortonworks Inc. 2014 Authorization
- 24. Page24 © Hortonworks Inc. 2014 Authorizer • Controls who can do what • Pluggable • Acl based approach
- 25. Page25 © Hortonworks Inc. 2014 Acl • Alice is Allowed to Read from Orders-topic from Host-1 Principal Permission Operation Resource Host Alice Allow Read Orders Host-1
- 26. Page26 © Hortonworks Inc. 2014 Principal • PrincipalType:Name • Supported types: User • Extensible so users can add their own types • Wild Card User:*
- 27. Page27 © Hortonworks Inc. 2014 Operation • Read, Write, Create, Delete, Alter, Describe, ClusterAction, All • Each API as an Operation VS Classification that maps to APIs.
- 28. Page28 © Hortonworks Inc. 2014 Resource • ResourceType:ResourceName • Topic, Cluster and ConsumerGroup • Wild card resource ResourceType:*
- 29. Page29 © Hortonworks Inc. 2014 Permissions • Allow and Deny • Anyone without an explicit Allow ACL is denied • Then why do we have Deny? • Deny works as negation • Deny takes precedence over Allow Acls
- 30. Page30 © Hortonworks Inc. 2014 Hosts • Why provide this granularity? • Allows authorizer to provide firewall type security even in non secure environment. • * as Wild card.
- 31. Page31 © Hortonworks Inc. 2014 Configuration • Authorizer class • Super users • Authorizer properties • Default behavior for resources with no ACLs
- 32. Page32 © Hortonworks Inc. 2014 SimpleAclAuthorizer • Out of box authorizer implementation. • Stores all of its ACLs in zookeeper. • In built ACL cache to avoid performance penalty. • Provides authorizer audit log.
- 33. Page33 © Hortonworks Inc. 2014 Client Broker Authorizer Zookeeper configure Read ACLs Load Cache Request authorize ACL match Or Super User? Allowed/Den ied
- 34. Page35 © Hortonworks Inc. 2014 CLI • Add, Remove and List acls • Convenience options: --producer and --consumer.
- 35. Page36 © Hortonworks Inc. 2014 Ranger Policy
- 36. Page37 © Hortonworks Inc. 2014 Ranger Auditing
- 37. Page38 © Hortonworks Inc. 2014 Ranger ACL management Audit
- 38. Page39 © Hortonworks Inc. 2014 Unsecure zookeeper
- 39. Page40 © Hortonworks Inc. 2014 Zookeeper • Kafka’s metadata store • Has its own security mechanism that supports SASL and MD5-DIGEST for establishing identity and ACL based authorization • Create , Delete directly interacts with zookeeper
- 40. Page41 © Hortonworks Inc. 2014 Securing zookeeper • Acl on zk nodes: user:cdrwa • Zookeeper.set.acl • ZkSecurityMigrator script • Credit where its due: Flavio Junqueira
- 41. Page42 © Hortonworks Inc. 2014 Client JAAS Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="zookeeper" keyTab="/vagrant/keytabs/kafka.keytab" principal="kafka/kafka@WITZEND.COM"; };
- 42. Page43 © Hortonworks Inc. 2014 Future • KIP-4: Move everything to server side, no direct interactions with zookeeper • Group Support • Pluggable Auditor • Delegation Tokens • Impersonation
- 43. Page44 © Hortonworks Inc. 2014 Summary • SSL for wire encryption • Sasl for authentication • Authorization • Secure Zookeeper Thanks to the community for participation.