I’ve been working on the Coursera Cloud Computing Specialization Capstone for several weeks and can make some observations. The capstone lasts 6 weeks but the first deliverable is due at the 3 week point so I have a good idea of what’s left with Hadoop subtask.
First, it looks like the capstone has good coverage of the concepts and techniques needed, at least according to my exposure to Hadoop development in the Udemy courses and books that I’ve been reading over the past few months.
Second, there is absolutely no way a non-programmer can do this. That’s disappointing since the rest of the specialization was largely code-free. (One class did require coding but the answers were pretty much in the study material.)
Third, the actual work (at least for the first two groups of questions) is straightforward if you’ve watched a Udemy course or read an introductory Hadoop book and paid attention. Call it 80% recognizing that there’s a solution and 20% finding the details. I didn’t have any problems writing both unit tests (MRUnit) and standalone (or pseudo-cluster?) functional tests using sample data.
BUT actually pushing this onto a multi-system cloud is another matter. To be honest I think I made a mistake – one of the instructor’s comments referred to us learning how to work in a real multi-system cloud environment and that we should not just run our applications in pseudo-server mode. I studied Spring Cloud. Learned a lot. Learned solutions to problems I didn’t know I had. But I didn’t learn how to set up a Hadoop cluster that can being in additional AWS EC2 instances. (But once they’re up I now know how to configure and monitor them!)
Now I’m focusing on Amazon EMR. I know it scales and I’m sure it has some way to handle the minimal configuration I need. (E.g., setting up a connection to a Cassandra instance.) (Oh, yeah, I also needed to learn about the care and feeding of Cassandra databases.) I’m sure it won’t be hard to pick up…
… but I can definitely feel the clock ticking as the deadline approaches. I really wish we had gotten a heads-up a month or more earlier to we could prepare better. Nothing has been particularly difficult to learn but there’s a lot of it.