Mark Carroll's Journal

I have had a busy year. Applying for jobs since I returned from Asia, interviewing, sometimes intensively, then weighing offers and preparing to relocate: paperwork, making ready here for my departure, researching for my arrival. A couple of aspects of this relocation have had me interact with staff at various levels as my new employer has engaged companies who then subcontract to local partners.

I get a couple of quieter weeks this month. I am reasonably on top of ( preparing to relocate. ) My previous anxiety is thus somewhat calmed.

After that, the real effort follows after I move next month: ( setting up a new domicile, onboarding at work. )

My current employment is in a good state so that now feels calmer too. It feels fitting to be leaving having gotten a new round of work done on ( a complex software component. )

Approximately annually, at work we run the occasional large meeting, typically over three days, so that our users, developers and other interested parties can come together, talk to us, learn about how to use our software, tell everyone about how they are using it, we can plan future work, etc. With the pandemic we shifted this to being online via Zoom.

The meeting worked rather well, admittedly better than I had expected. I did miss one part of the previous meetings that I like, the ad-hoc small-group conversations in between sessions. Still, I believe that the effort was well worth it. Further, some mentioned that it is far more possible for them to attend online things, even ordinarily. We got to see and hear from each other and it is good to be able to put faces to the names we see in the support fora subsequently.

There were probably around eighty attendees in the main sessions and maybe around twenty in the workshops I was helping to run. My main jobs there were to check people against the registration list in shuffling them from the waiting room into the meeting and to keep track of questions from the chat to make sure nothing important was missed by the speaker; on lucky occasions I could field them myself.

Participants were worldwide, ranging from Australia through Asia and Europe to the Americas, so sessions were timed accordingly: on Thursday, workshops I helped with included a 11h to 13h one then, later, a repeat of it from 21h to 23h which attracted people from North America. I have taken this coming Monday and Friday off work as part of recovery but I know that some of my colleagues put much more than I into the meeting. I hope that they are sleeping well this weekend.

Unexpected developments left me with more vacation time for this calendar year than I had initially expected. I figured that I should finally perform a filial duty and take my parents' ashes on perhaps a last visit to Cornwall for me and scatter them there. I am conscious of how much of my thinking is, on its face, irrational. For example, whom does this trip benefit? It is not as if I believe in an afterlife. So much of what I believe turns out to arise simply from received wisdom about how one ought to behave. Rational or not, it seems the right thing to do so I have now made the related bookings for the middle of fall after the summer vacation folks will be long gone.

I also hope to move further toward sorting through my parents' photographs. I took backups of their hard drives and flash drives before I wiped them but had left things there. My work is on software for organizing, annotating and sharing one's images so I may as well start clearing the obvious clutter from my parents' data then arrange the remainder using the software I work on. I can't help but be reminded of Roy Batty's, All those moments will be lost in time, like tears in rain. I won't even know where some of their photographs were taken or of whom.

As usual, I follow what my instinct tells me: I shall preserve and eventually organize my parents' photographs. Regardless of if it is a good thing to do, it might be quite interesting. I will try not to trouble myself over what meaning those photographs had for them or that others should think similarly.

As a teenager I was beset by existential angst that had me reading authors from Dostoyevsky to Sartre for clues to how to give my life any meaning. In my thirties I was struck by a deep depression that took me years to climb out of. That healing process changed me and, among other things, somewhat inoculated me against despair. It's partly a matter of perception. The things that matter to me will someday be forgotten by all. I must accept that, whatever their value, the old things pass and make way for the new.

I have been happy to leave these inherited items alone, to procrastinate considerably. As well as the photographs from the computers I still have some physical items that I inherited, such as my father's coat which is presently in the attic. Likewise, I preserve the physical items but am in no rush to engage with them. It will get easier with time.

Both the hot bath and the embrocation were agreeable. Acetaminophen and codeine banished my headache and it has not returned. Also, the sun came out a little, even enough for some people to mow their yards. I still skipped hair-trimming but I ran some laundry and mostly finished the agenda for tomorrow morning's (not) stand-up meeting.

I guessed a fix for the failing test which tomorrow can be tried on the server whose configuration and data appear to trigger it well. This afternoon I repeatedly tried to trigger the problem on my work laptop but since applying the fix have been unable to reproduce it so I am cautiously optimistic.

Update: I got to run the test this evening thanks to others working today. Unfortunately my optimism was misplaced. We'll see what inspiration the new week brings.

I have been free from headaches today but my previous pain has developed into the right side of my ribcage being generally ornery. It feels like some kind of soft tissue strain and I had hoped that it would heal more quickly. It isn't greatly disabling; I can still sleep fine, I don't even need painkillers. Still, it doesn't much like my getting out of bed or inhaling deeply and I really don't relish having to sneeze or to change my footwear. This evening I tried applying an embrocation which at least feels nice, from the menthol or camphor or whatever; I don't know if it helps. The pain's mercurial, moving quickly between absent and, ha, you forgot and tried to move, now for your punishment.

My Ansible work got put aside because shortly before next week's software release I thought of a test we could do on that product and yesterday afternoon the test failed. I have my work laptop at home this weekend for gathering a bit more data and in case I think of why the problem might occur. It seems intermittent; I hope to at least grab some debug logs and other state for a good and a bad (but otherwise similar) run then compare though I fear that the smoking gun won't be in that difference. The intermittency makes me try to think of where there could be a relevant race condition.

I feel as if I have been pushing half-done work tasks onto the stack for the past half-year or so and I like to imagine that the coming half-year might let me resurrect and complete that series of interrupted tasks before I resurface and ask what comes next. It may not be very critical but I do prefer to leave things tidy.

For this coming week I am also to organize our daily morning meetings which are mostly about all what needs doing today and who is to do it. This ranges across errors in server logs, questions from users, code that needs review, whatever comes up. It's something that I've been helping with for a few years now.

The weather for the past couple of days has been what the Scots might term dreich though it looks to be clearing up tonight.

As expected, my shoulder and side are a little improved today; I still expect them to be fine by the weekend. After work yesterday I did still manage to get useful tasks done, mostly ones I would normally do at the weekend except for that we didn't get back from England until late Sunday afternoon, with more Lancashire cheese in hand. At work I have been making slow but steady progress with learning some basic Ansible; it helped to be pointed to our repository of roles which are amenable to git grep --recurse-submodules …. Today I got a simple additional role to seem to actually work despite my having had a headache for what must now be over twenty-four hours. Painkillers have at least reduced it somewhat. Unfortunately I am no closer to identifying triggers.

A few weeks ago I mentioned a slow, tricky period at work battling popular enterprise software libraries and, later, that things had started going better again. Last week I caused some amusement at a meeting in responding to a comment about writing parsers in PL/pgSQL that such would be one of the better tasks I have been given. Sure, my favorite parser writing so far was over a decade ago in Haskell, using Frisby and Parsec, but, while PL/pgSQL is as limited and basic as, say, PostScript (if we ignore the font stuff), that it is basic really helps. It is well-documented and the abstraction that it does present simply works.

In contrast, the frustrating technologies are those where they purport to offer a simple helpful view but the illusion is repeatedly shattered as unexplained dragons keep breaking out from inside. The best software for building things, of which I also offer Basser Lout as an example, is that where you ask even more than it seemed was promised and it still works seamlessly: one cannot help but suspect that some good, clean design underpins them rather than an unholy nest of worms that are not all entirely friends with each other.

Perhaps in contrast to that tricky period, recent weeks at work have felt rather productive. Admittedly, some of the to-do's that I checked off the list were trivial: a few are but minor changes to our codebase, even to comments, but still worth doing in my opinion. Others do involve more code, largely bugfixes, but again they were clearly bugs in our code, rather than unclearly arising from interactions with third-party code, so they were far more amenable to inquiry and correction than my challenges this time last month.

Yesterday morning I took a break from testing the next version of our Image Data Resource to join the School of Life Sciences' annual carols. Although there were many attendees, at first I was the only one from our immediate group who joined. I sing poorly but I have a churchgoing background so I was familiar enough with the words and melodies to try to encourage singing among the congregants in my vicinity. Given that we have many students and staff from overseas I suspect that it may have been some impediment to try to negotiate some of the archaic English, especially at the lively tempo: we rarely had more than a beat in between verses. The music was provided by piano, violin and saxophone, plus what I would guess was a tambourine that I heard but did not see. I was pleased that a couple of others still working on the testing looked down from the internal windows to see me there caroling and decided to come down and join in. I like to maintain what few traditions are still easy for me to come by and I rather enjoy the melodies of some carols such as We Three Kings and O Little Town of Bethlehem (to Forest Green).

Sometimes I change a bunch of code, get my changes working, then tidy up my mess, even working from the resulting large diff to recreate a series of commits that capture simpler logical steps. In thinking about some code reorganization yesterday I noticed that it includes activities that could be amenable to automated analysis:

( some vague ideas for that come to mind. )

Broadly I have the sense that things are going well if my code size is reducing and the average amount of in-scope state is also reducing: then I am mostly left with having to name the subparts. Those measures are among my proxies for if each part of the code is more easily understood and easier to write tests for. I can imagine that there has probably been some active research into the question of automated code tidying to make it clearer though analyzing data dependency patterns, detecting similar code and characterizing its differences, etc. I wonder if it has got anywhere.

This was especially in my mind over the past couple of weeks as I have had to ( do plenty of code tidying. )

Despite my tiredness I seem to remain effective at my day job. Last week mostly I focused on improving the process by which microscope images are transferred from client software into the OMERO server. This is a complex process with many substeps and variables depending on configuration and the kind of image data being imported.

I was largely productive but did run into a couple of interesting issues. One is that I had found my way to Java's ExecutorService class but would prefer to be notified when any of the submitted threads complete without having to poll them. I now find that the easiest solution may be to use ExecutorCompletionService so that will be a task for Monday.

Another issue is that for writing the imported image files on the server, together with parent directories, the import is running in the context of a single database transaction and our file storage has the actual filesystem objects but also corresponding ownership metadata in a database table that allows OMERO's permissions system to determine quickly who may access the file. The server complains when there is a database row or a filesystem object but not both.

There is a race condition when multiple imports need to create the same parent directory. One thread may create the filesystem object and write the ownership information. Before that thread's transaction commits, a different thread may detect that the filesystem object already exists and, thanks to isolation, not see the corresponding ownership information anytime soon. So far an easy but robust solution to this has not come to my mind. I do provide the workaround of allowing users to configure the server so that image data is written within a parent directory named for the server thread that is managing the import.