More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to Oozie meetup - HA
Similar to Oozie meetup - HA (20)
Recently uploaded
Recently uploaded (20)
Oozie meetup - HA
- 1. 1 Oozie High Availability (HA) Robert Kanter
- 2. 2 High Availability • A system without non-planned downtime when partial failures occur • Typically achieved by having redundancies and removing single-points of failure • Our Goals • Don’t change the API or usage patterns • User doesn’t even have to know its HA
- 3. 3 The HA Solution Architectural Overview
- 4. 4 The HA Solution: Database • Oozie stores all state in a database • (submitted jobs, workflow definitions, etc) • Instead of a failover model, we want to run many Oozie servers against the same database • Active-Active HA • Also provides horizontal scalability • ZooKeeper for coordination
- 5. 5 The HA Solution: Database
- 6. 6 The HA Solution: Access • Users and client programs need a single address to connect (Web UI, REST/Java API, JobTracker callbacks, etc) • Load Balancer, Virtual IP, or DNS round-robin can be used to provide a single entry point to the Oozie servers • Technically also needs to be HA
- 7. 7 The HA Solution: Access
- 8. 8 The HA Solution: Log Streaming • Oozie’s log files are not in the database • Each Oozie Server only has access to its own logs • Jobs are not assigned to a specific Oozie server • What if Oozie Server A wants to get logs for a job processed by Oozie Server B? • Oozie Server A can ask Oozie Server B for its logs • Caveat: If an Oozie Server goes down, any logs from it will be unavailable until it is brought back up
- 9. 9 The HA Solution: Log Streaming
- 10. 10 How to Enable HA Configuration and Security
- 11. 11 How to Enable HA • Setup Load balancer, ZooKeeper ensemble, HA database, and multiple identically configured Oozie servers • Enable Oozie HA services: <property> <name>oozie.services.ext</name> <value> org.apache.oozie.service.ZKLocksService, org.apache.oozie.service.ZKXLogStreamingService, org.apache.oozie.service.ZKJobsConcurrencyService </value> </property>
- 12. 12 How to Enable HA • Point Oozie to ZooKeeper Ensemble: <property> <name>oozie.zookeeper.connection.string</name> <value>ZK_HOST1:2181,ZK_HOST2:2181</value> </property> • Point environment variable for callbacks to load balancer: export OOZIE_BASE_URL="http://loadbalancer:11000/oozie"
- 13. 13 How to Enable HA: Security • Extra step to configure Kerberos with Load Balancer: <property> <name> oozie.authentication.kerberos.principal </name> <value>HTTP/loadbalancer@REALM</value> </property> • Note: this currently prevents clients from talking directly to any Oozie server
- 14. 14 How to Enable HA: Security • Enable Kerberos connection to ZooKeeper and ACLs: <property> <name>oozie.zookeeper.secure</name> <value>true</value> </property> • ACLs prevent malicious users or programs from interfering with Oozie’s znodes
- 15. 15 Using Oozie with HA
- 16. 16 Using Oozie with HA • New Oozie CLI/REST API command to list all servers $ oozie admin -oozie http://loadbalancer:11000/oozie -servers hostA : http://hostA:11000/oozie hostB : http://hostB:11000/oozie hostC : http://hostC:11000/oozie • Log messages now include which server wrote them 2013-09-29 16:46:20,182 WARN org.apache.oozie.command.wf.ActionStartXCommand: SERVER[hostA] USER[root] GROUP[-] TOKEN[] APP[demo-wf] JOB[0000000-130925230553293-oozie-oozi-W] ACTION[0000000- 130925230553293-oozie-oozi-W@streaming-node] [***0000000- 130925230553293-oozie-oozi-W@streaming-node***]Action status=RUNNING
- 18. 18 To Do • HA support for SLAs and HCatalog integration • Sharelib Purging with HA • Log Streaming HA • With Kerberos, Oozie servers can’t talk to each other • Breaks log streaming, sharelibupdate • Other misc improvements
- 19. 19