Hello all, welcome back!
I’m sure each one of you has heard your fair share about COVID-19, so I will keep this part brief. There are clearly wildly different effects for everyone because of the lost ability to go onsite. In my situation, no Unanet employees can commute to the office, so I have been working from home. This has actually made my internship a bit more manageable because instead of having to work two 7.5 hour days per week, I can split up the 15 hours between more days. Plus, more at-home time means more time to work… right?
This past week, I figured out that I would have to tweak my trajectory. After finishing the deliverable described in ‘Week 1: Working with workflow,’ I moved onto figuring out how to integrate XML into the program. To sum up this goal, XML is a very structured way of storing information, much like notes taken in a classroom setting. My goal for this was to get Luigi and Python to take XML files as input and make tasks out of what they outline. However, after doing some research, I discovered that Luigi does not take XML files. Apparently this aspect makes the workflow framework more ‘lightweight.’ I told my onsite advisor, and thankfully, he said it’s OK to keep using Luigi. I am glad both because I don’t have to start from scratch with a new library, and because I think this will give me more hands-on experience with python (I will be hard-coding the tasks rather than just plugging them in through XML files).
With this newfound knowledge, my next goal was to make tasks that are closer to what they will be in the real application. For example, one task was to run a batch file that made two new files, one log.out and one log.err (these are the two outputs on standard programs). The task declared success if it found both files and if the log.err file had nothing written inside, while it crashed if the error file had anything written inside. Another set of tasks that I worked on is one that would gather all .log and .error files from a particular directory and pack them all into a .zip file.
I also started my search for a Graphical User Interface (GUI) that will display workflow to the user. After looking at various downloadable libraries compatible with Python and Luigi, I decided that the best option was to use ‘Luigid’, or ‘Luigi Daemon’, which is a service provided along with Luigi. Essentially, when running Luigi scripts, you can either use the local scheduler or the central scheduler. A scheduler allocates workers to the different tasks that you tell the scheduler to complete. The local scheduler is more lightweight and offline – usually used for testing purposes. The central scheduler (based in Luigid) is online and has its workers talk to the scheduler over HTTP. At the same time, Luigid displays everything that’s happening on a GUI found at http:\localhost:8082. This is a screenshot from Luigi’s website of what the ‘visualizer page’ looks like.
As you can see, it conveniently shows everything that you would want to know about your process, particularly pending tasks, running tasks, done tasks, and failed tasks.
Plans + reflection
Sadly, the picture that I included was a screenshot because I am having trouble getting Luigid up and running. My leading theory is that I can’t connect because of the security settings on my work computer. So my biggest task for next week is to get all of that sorted out. Along with this, another goal is to get closer to tasks that I want the final product to be running. As I go along, I expect the tasks to be more difficult.
As I said before, the quarantine does not impede my ability to work whatsoever, so I expect to continue making lots of progress. Stay safe (and tuned), everyone!