Skip to main content

commandline data analytics

Here are some patterns that were important for most  infrastructure functions - infrastructure operations,  software development, or even large scale infrastructure changes, migrations and predicting future states.

The slice:   "| grep | project"
The  map-reduce pipeline:   "| transform | project | sort |uniq -c|sort -nr"
The pivot:   "|project | pivot-and-aggregate"

The output of these patterns are typically fed into the execution pipeline.

These patterns are essential because they direct, focus and scale the primary output beyond what is possible by just being an expert.

There is nothing magical about any of those. But as a considered habit, an involuntary muscle memory, they can bring  enormous leverage and time line compression to any complex piece of work.

There is one pattern that is missing from this: classification/machine learning filter.

We need an  "ml-filter" to  make the future architect/webops engineer's end result to become a fully closed loop control system.



 

Popular posts from this blog

Why PI is not 4, math is great, and other mysteries.

The other day, I found myself with an interesting problem of approximating a circle with the enclosing square which seems to prove pi = 4.

The paradox was forwarded by a most interesting puzzle collector, Surajit Basu, a friend and life long inspiration. See Sonata for Unaccompanied Tortoise for why!



Here is the offending paradox:

























This is an example of how counterintuitive questions can be answered with a little calculus.

The key is to realize that no matter how closely we approximate the circle, the orthogonal lines of the approximation formed by inverting the square corners will never actually be tangential to the circle.

Note carefully that as you get closer to 90 degrees, the horizontal line is much longer than the vertical. Same goes with the approximation at 0 and 180 - the vertical line is much larger than the horizontal component.

If we take a quadrant of the circle - let's say the top left quadrant, moving counter clockwise from top to left -  we can imagine that each inf…

Ambition vs. Fear.

Most important things in life don't come to us. Nor do we get them by seeking/wanting them. It comes from letting go of the unimportant stuff.

The hardest part is letting go of the tendency to take the world as is. This is a habit of our past successes.

But success is not a destination, it is a STOP sign. You stop, wait, and move on. Too often, we are paralyzed by success into the fear of the new. We stall on the road to a new life. We need to break our inertia and move.

Our thoughts and thought habits are hard to break. But that is where we have to spend the most energy. Thoughts are always competing strands  - of worries of the past and anxieties for the future. For some of us, they are cleanly separated into rivers that nurture every place they travel. For most, they are like the torrents and trickles -- competing, rushing somewhere, stopping completely elsewhere, always mixing, morphing, competing, winning, losing.

Our thoughts are the potential difference between the two pole…

How transformation/change could be hard - an analogy from Life

The illusion of steady change
Most changes are gradual. Or it appears to be, as you read stories and narratives about successful people. Be it successful CEOs, scientists, artists, startup founders, you name it -- almost every one who has reached a state of success, it seems, has had a steady run of successes.

We want to believe that these fine folks transformed their lives because they worked hard,  had talent, or had help, or many other things -- and it was simply effort and reward.


But we all know this is not true. Otherwise, we'd be doing more of what we do best, every day.

What is it that can explain great changes? It is a transformation. Yes, it has gradual bits and habits thrown in. But in the end, they were ready to transform themselves. They were ready to re-work their brains, bodies and life into a new combination.

Example from the chemistry of Life
There is an example in Life that may shed some light. Treat this as an analogy, but it has some lessons.

To illustrate, …